Oct 31 19:18:48.662598 ip-10-0-131-94 systemd[1]: Starting Kubernetes Kubelet...
Oct 31 19:18:49.113135 ip-10-0-131-94 kubenswrapper[2584]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 19:18:49.113135 ip-10-0-131-94 kubenswrapper[2584]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 31 19:18:49.113135 ip-10-0-131-94 kubenswrapper[2584]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 19:18:49.113135 ip-10-0-131-94 kubenswrapper[2584]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 19:18:49.118469 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118371    2584 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 19:18:49.118469 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118469    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 19:18:49.118469 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118473    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 19:18:49.118570 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118477    2584 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 19:18:49.118570 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118480    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 19:18:49.118570 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118482    2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 19:18:49.118570 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118485    2584 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 19:18:49.118570 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118488    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 19:18:49.118570 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118490    2584 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 19:18:49.118570 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118493    2584 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 19:18:49.118570 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118496    2584 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 19:18:49.118570 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118498    2584 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 19:18:49.118570 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118502    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 19:18:49.118570 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118505    2584 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 19:18:49.118570 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118507    2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 19:18:49.118570 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118510    2584 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 19:18:49.118570 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118513    2584 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 19:18:49.118570 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118515    2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 19:18:49.118570 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118518    2584 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 19:18:49.118570 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118526    2584 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 19:18:49.118570 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118529    2584 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 19:18:49.118570 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118532    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 19:18:49.118570 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118535    2584 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 19:18:49.119057 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118538    2584 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 19:18:49.119057 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118540    2584 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 19:18:49.119057 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118543    2584 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 19:18:49.119057 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118545    2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 19:18:49.119057 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118548    2584 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 19:18:49.119057 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118551    2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 19:18:49.119057 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118553    2584 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 19:18:49.119057 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118556    2584 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 19:18:49.119057 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118558    2584 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 19:18:49.119057 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118561    2584 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 19:18:49.119057 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118563    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 19:18:49.119057 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118566    2584 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 19:18:49.119057 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118569    2584 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 19:18:49.119057 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118573    2584 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 19:18:49.119057 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118576    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 19:18:49.119057 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118578    2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 19:18:49.119057 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118581    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 19:18:49.119057 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118584    2584 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 19:18:49.119057 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118587    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 19:18:49.119057 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118590    2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 19:18:49.119577 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118592    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 19:18:49.119577 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118595    2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 19:18:49.119577 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118598    2584 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 19:18:49.119577 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118600    2584 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 19:18:49.119577 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118602    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 19:18:49.119577 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118605    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 19:18:49.119577 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118607    2584 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 19:18:49.119577 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118610    2584 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 19:18:49.119577 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118612    2584 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 19:18:49.119577 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118615    2584 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 19:18:49.119577 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118618    2584 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 19:18:49.119577 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118621    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 19:18:49.119577 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118623    2584 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 19:18:49.119577 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118625    2584 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 19:18:49.119577 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118630    2584 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 19:18:49.119577 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118634    2584 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 19:18:49.119577 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118637    2584 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 19:18:49.119577 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118639    2584 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 19:18:49.119577 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118642    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 19:18:49.120020 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118645    2584 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 19:18:49.120020 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118647    2584 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 19:18:49.120020 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118650    2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 19:18:49.120020 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118653    2584 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 19:18:49.120020 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118656    2584 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 19:18:49.120020 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118658    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 19:18:49.120020 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118661    2584 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 19:18:49.120020 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118664    2584 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 19:18:49.120020 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118667    2584 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 19:18:49.120020 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118672    2584 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 19:18:49.120020 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118675    2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 19:18:49.120020 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118679    2584 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 19:18:49.120020 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118681    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 19:18:49.120020 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118685    2584 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 19:18:49.120020 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118687    2584 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 19:18:49.120020 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118690    2584 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 19:18:49.120020 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118692    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 19:18:49.120020 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118695    2584 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 19:18:49.120020 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118698    2584 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 19:18:49.120480 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118700    2584 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 19:18:49.120480 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118703    2584 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 19:18:49.120480 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118705    2584 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 19:18:49.120480 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118708    2584 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 19:18:49.120480 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118711    2584 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 19:18:49.120480 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118714    2584 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 19:18:49.120480 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118716    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 19:18:49.120480 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118719    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 19:18:49.120480 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118721    2584 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 19:18:49.120480 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118724    2584 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 19:18:49.120480 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118726    2584 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 19:18:49.120480 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118729    2584 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 19:18:49.120480 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118731    2584 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 19:18:49.120480 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118734    2584 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 19:18:49.120480 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118737    2584 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 19:18:49.120480 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118739    2584 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 19:18:49.120480 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118742    2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 19:18:49.120480 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118745    2584 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 19:18:49.120480 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118747    2584 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 19:18:49.121104 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118751    2584 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 19:18:49.121104 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118754    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 19:18:49.121104 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118756    2584 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 19:18:49.121104 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118759    2584 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 19:18:49.121104 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118762    2584 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 19:18:49.121104 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.118765    2584 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 19:18:49.121104 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120424    2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 19:18:49.121104 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120432    2584 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 19:18:49.121104 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120435    2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 19:18:49.121104 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120438    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 19:18:49.121104 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120441    2584 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 19:18:49.121104 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120444    2584 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 19:18:49.121104 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120446    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 19:18:49.121104 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120449    2584 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 19:18:49.121104 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120452    2584 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 19:18:49.121104 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120456    2584 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 19:18:49.121104 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120459    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 19:18:49.121104 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120462    2584 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 19:18:49.121104 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120465    2584 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 19:18:49.121821 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120468    2584 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 19:18:49.121821 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120471    2584 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 19:18:49.121821 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120473    2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 19:18:49.121821 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120476    2584 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 19:18:49.121821 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120478    2584 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 19:18:49.121821 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120481    2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 19:18:49.121821 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120484    2584 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 19:18:49.121821 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120487    2584 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 19:18:49.121821 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120490    2584 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 19:18:49.121821 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120492    2584 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 19:18:49.121821 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120495    2584 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 19:18:49.121821 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120498    2584 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 19:18:49.121821 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120500    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 19:18:49.121821 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120503    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 19:18:49.121821 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120506    2584 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 19:18:49.121821 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120509    2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 19:18:49.121821 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120511    2584 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 19:18:49.121821 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120514    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 19:18:49.121821 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120517    2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 19:18:49.121821 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120520    2584 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 19:18:49.122368 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120522    2584 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 19:18:49.122368 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120525    2584 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 19:18:49.122368 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120527    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 19:18:49.122368 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120530    2584 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 19:18:49.122368 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120533    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 19:18:49.122368 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120535    2584 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 19:18:49.122368 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120538    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 19:18:49.122368 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120541    2584 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 19:18:49.122368 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120543    2584 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 19:18:49.122368 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120546    2584 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 19:18:49.122368 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120548    2584 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 19:18:49.122368 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120551    2584 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 19:18:49.122368 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120554    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 19:18:49.122368 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120557    2584 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 19:18:49.122368 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120559    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 19:18:49.122368 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120562    2584 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 19:18:49.122368 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120564    2584 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 19:18:49.122368 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120567    2584 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 19:18:49.122368 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120570    2584 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 19:18:49.122368 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120572    2584 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 19:18:49.122842 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120575    2584 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 19:18:49.122842 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120577    2584 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 19:18:49.122842 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120580    2584 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 19:18:49.122842 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120582    2584 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 19:18:49.122842 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120585    2584 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 19:18:49.122842 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120587    2584 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 19:18:49.122842 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120590    2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 19:18:49.122842 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120592    2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 19:18:49.122842 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120596    2584 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 19:18:49.122842 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120600    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 19:18:49.122842 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120603    2584 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 19:18:49.122842 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120606    2584 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 19:18:49.122842 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120608    2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 19:18:49.122842 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120611    2584 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 19:18:49.122842 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120614    2584 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 19:18:49.122842 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120616    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 19:18:49.122842 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120619    2584 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 19:18:49.122842 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120622    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 19:18:49.122842 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120625    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 19:18:49.123295 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120628    2584 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 19:18:49.123295 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120631    2584 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 19:18:49.123295 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120633    2584 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 19:18:49.123295 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120636    2584 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 19:18:49.123295 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120639    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 19:18:49.123295 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120641    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 19:18:49.123295 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120644    2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 19:18:49.123295 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120647    2584 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 19:18:49.123295 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120650    2584 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 19:18:49.123295 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120652    2584 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 19:18:49.123295 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120655    2584 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 19:18:49.123295 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120657    2584 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 19:18:49.123295 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120660    2584 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 19:18:49.123295 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120662    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 19:18:49.123295 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120665    2584 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 19:18:49.123295 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120667    2584 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 19:18:49.123295 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120670    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 19:18:49.123295 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120672    2584 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 19:18:49.123295 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120932    2584 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 19:18:49.123770 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120944    2584 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 19:18:49.123770 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120951    2584 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 19:18:49.123770 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120956    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 19:18:49.123770 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120966    2584 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 19:18:49.123770 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120973    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 19:18:49.123770 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120980    2584 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 19:18:49.123770 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120985    2584 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 19:18:49.123770 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120991    2584 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 19:18:49.123770 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.120996    2584 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 19:18:49.123770 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.121001    2584 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 19:18:49.123770 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.121006    2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 19:18:49.123770 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.121017    2584 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 19:18:49.123770 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.121022    2584 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 19:18:49.123770 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.121026    2584 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 19:18:49.123770 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.121030    2584 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 19:18:49.123770 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122211    2584 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 19:18:49.123770 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122221    2584 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 19:18:49.123770 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122225    2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 19:18:49.123770 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122229    2584 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 19:18:49.123770 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122232    2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 19:18:49.124262 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122235    2584 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 19:18:49.124262 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122238    2584 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 19:18:49.124262 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122241    2584 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 19:18:49.124262 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122243    2584 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 19:18:49.124262 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122246    2584 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 19:18:49.124262 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122249    2584 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 19:18:49.124262 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122252    2584 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 19:18:49.124262 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122255    2584 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 19:18:49.124262 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122257    2584 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 19:18:49.124262 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122260    2584 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 19:18:49.124262 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122264    2584 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 19:18:49.124262 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122271    2584 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 19:18:49.124262 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122274    2584 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 19:18:49.124262 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122277    2584 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 19:18:49.124262 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122280    2584 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 19:18:49.124262 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122283    2584 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 19:18:49.124262 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122285    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 19:18:49.124262 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122288    2584 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 19:18:49.124262 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122290    2584 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 19:18:49.124717 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122293    2584 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 19:18:49.124717 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122296    2584 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 19:18:49.124717 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122298    2584 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 19:18:49.124717 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122301    2584 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 19:18:49.124717 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122303    2584 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 19:18:49.124717 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122306    2584 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 19:18:49.124717 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122309    2584 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 19:18:49.124717 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122312    2584 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 19:18:49.124717 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122314    2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 19:18:49.124717 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122317    2584 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 19:18:49.124717 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122320    2584 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 19:18:49.124717 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122324    2584 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 19:18:49.124717 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122342    2584 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 19:18:49.124717 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122345    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 19:18:49.124717 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122348    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 19:18:49.124717 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122351    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 19:18:49.124717 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122353    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 19:18:49.124717 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122356    2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 19:18:49.124717 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122359    2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 19:18:49.124717 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122362    2584 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 19:18:49.125198 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122364    2584 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 19:18:49.125198 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122367    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 19:18:49.125198 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122370    2584 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 19:18:49.125198 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122372    2584 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 19:18:49.125198 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122375    2584 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 19:18:49.125198 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122378    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 19:18:49.125198 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122381    2584 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 19:18:49.125198 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122384    2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 19:18:49.125198 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122386    2584 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 19:18:49.125198 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122389    2584 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 19:18:49.125198 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122392    2584 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 19:18:49.125198 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122394    2584 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 19:18:49.125198 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122398    2584 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 19:18:49.125198 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122400    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 19:18:49.125198 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122403    2584 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 19:18:49.125198 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122406    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 19:18:49.125198 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122409    2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 19:18:49.125198 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122411    2584 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 19:18:49.125198 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122414    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 19:18:49.125650 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122417    2584 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 19:18:49.125650 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122419    2584 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 19:18:49.125650 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122422    2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 19:18:49.125650 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122426    2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 19:18:49.125650 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122429    2584 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 19:18:49.125650 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122432    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 19:18:49.125650 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122434    2584 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 19:18:49.125650 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122437    2584 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 19:18:49.125650 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122439    2584 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 19:18:49.125650 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122442    2584 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 19:18:49.125650 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122444    2584 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 19:18:49.125650 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122447    2584 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 19:18:49.125650 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122450    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 19:18:49.125650 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122452    2584 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 19:18:49.125650 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122455    2584 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 19:18:49.125650 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122458    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 19:18:49.125650 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122460    2584 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 19:18:49.125650 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122463    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 19:18:49.125650 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122465    2584 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 19:18:49.125650 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122468    2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 19:18:49.126110 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122473    2584 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 19:18:49.126110 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122477    2584 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 19:18:49.126110 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122480    2584 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 19:18:49.126110 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122483    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 19:18:49.126110 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122486    2584 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 19:18:49.126110 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122489    2584 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 19:18:49.126110 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122492    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 19:18:49.126110 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122495    2584 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 19:18:49.126110 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122497    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 19:18:49.126110 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122500    2584 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 19:18:49.126110 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122502    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 19:18:49.126110 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122505    2584 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 19:18:49.126110 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122508    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 19:18:49.126110 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122510    2584 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 19:18:49.126110 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122513    2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 19:18:49.126110 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122516    2584 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 19:18:49.126110 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122519    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 19:18:49.126110 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122522    2584 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 19:18:49.126110 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122525    2584 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 19:18:49.126571 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122527    2584 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 19:18:49.126571 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122530    2584 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 19:18:49.126571 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122532    2584 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 19:18:49.126571 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.122535    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 19:18:49.126571 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123242    2584 flags.go:64] FLAG: --address="0.0.0.0"
Oct 31 19:18:49.126571 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123251    2584 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 31 19:18:49.126571 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123258    2584 flags.go:64] FLAG: --anonymous-auth="true"
Oct 31 19:18:49.126571 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123263    2584 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 31 19:18:49.126571 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123268    2584 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 31 19:18:49.126571 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123271    2584 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 31 19:18:49.126571 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123276    2584 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 31 19:18:49.126571 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123280    2584 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 31 19:18:49.126571 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123283    2584 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 31 19:18:49.126571 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123286    2584 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 31 19:18:49.126571 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123290    2584 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 31 19:18:49.126571 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123293    2584 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 31 19:18:49.126571 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123296    2584 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 31 19:18:49.126571 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123299    2584 flags.go:64] FLAG: --cgroup-root=""
Oct 31 19:18:49.126571 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123302    2584 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 31 19:18:49.126571 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123305    2584 flags.go:64] FLAG: --client-ca-file=""
Oct 31 19:18:49.126571 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123308    2584 flags.go:64] FLAG: --cloud-provider="external"
Oct 31 19:18:49.126571 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123311    2584 flags.go:64] FLAG: --cluster-dns="[]"
Oct 31 19:18:49.126571 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123316    2584 flags.go:64] FLAG: --cluster-domain=""
Oct 31 19:18:49.126571 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123319    2584 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 31 19:18:49.127119 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123322    2584 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 31 19:18:49.127119 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123336    2584 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 31 19:18:49.127119 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123340    2584 flags.go:64] FLAG: --container-log-max-files="5"
Oct 31 19:18:49.127119 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123345    2584 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 31 19:18:49.127119 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123348    2584 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 31 19:18:49.127119 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123351    2584 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 31 19:18:49.127119 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123355    2584 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 31 19:18:49.127119 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123359    2584 flags.go:64] FLAG: --contention-profiling="false"
Oct 31 19:18:49.127119 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123362    2584 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 31 19:18:49.127119 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123365    2584 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 31 19:18:49.127119 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123368    2584 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 31 19:18:49.127119 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123371    2584 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 31 19:18:49.127119 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123375    2584 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 31 19:18:49.127119 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123378    2584 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 31 19:18:49.127119 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123381    2584 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 31 19:18:49.127119 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123384    2584 flags.go:64] FLAG: --enable-load-reader="false"
Oct 31 19:18:49.127119 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123387    2584 flags.go:64] FLAG: --enable-server="true"
Oct 31 19:18:49.127119 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123390    2584 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 31 19:18:49.127119 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123394    2584 flags.go:64] FLAG: --event-burst="100"
Oct 31 19:18:49.127119 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123397    2584 flags.go:64] FLAG: --event-qps="50"
Oct 31 19:18:49.127119 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123400    2584 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 31 19:18:49.127119 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123403    2584 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 31 19:18:49.127119 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123406    2584 flags.go:64] FLAG: --eviction-hard=""
Oct 31 19:18:49.127119 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123411    2584 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 31 19:18:49.127748 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123414    2584 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 31 19:18:49.127748 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123417    2584 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 31 19:18:49.127748 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123420    2584 flags.go:64] FLAG: --eviction-soft=""
Oct 31 19:18:49.127748 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123423    2584 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 31 19:18:49.127748 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123426    2584 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 31 19:18:49.127748 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123429    2584 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 31 19:18:49.127748 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123431    2584 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 31 19:18:49.127748 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123435    2584 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 31 19:18:49.127748 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123437    2584 flags.go:64] FLAG: --fail-swap-on="true"
Oct 31 19:18:49.127748 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123440    2584 flags.go:64] FLAG: --feature-gates=""
Oct 31 19:18:49.127748 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123445    2584 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 31 19:18:49.127748 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123448    2584 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 31 19:18:49.127748 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123451    2584 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 31 19:18:49.127748 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123455    2584 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 31 19:18:49.127748 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123458    2584 flags.go:64] FLAG: --healthz-port="10248"
Oct 31 19:18:49.127748 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123461    2584 flags.go:64] FLAG: --help="false"
Oct 31 19:18:49.127748 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123464    2584 flags.go:64] FLAG: --hostname-override="ip-10-0-131-94.us-east-2.compute.internal"
Oct 31 19:18:49.127748 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123467    2584 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 31 19:18:49.127748 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123470    2584 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 31 19:18:49.127748 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123473    2584 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 31 19:18:49.127748 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123477    2584 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 31 19:18:49.127748 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123480    2584 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 31 19:18:49.127748 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123483    2584 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 31 19:18:49.128299 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123489    2584 flags.go:64] FLAG: --image-service-endpoint=""
Oct 31 19:18:49.128299 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123497    2584 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 31 19:18:49.128299 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123500    2584 flags.go:64] FLAG: --kube-api-burst="100"
Oct 31 19:18:49.128299 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123503    2584 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 31 19:18:49.128299 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123506    2584 flags.go:64] FLAG: --kube-api-qps="50"
Oct 31 19:18:49.128299 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123509    2584 flags.go:64] FLAG: --kube-reserved=""
Oct 31 19:18:49.128299 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123512    2584 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 31 19:18:49.128299 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123515    2584 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 31 19:18:49.128299 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123518    2584 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 31 19:18:49.128299 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123521    2584 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 31 19:18:49.128299 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123524    2584 flags.go:64] FLAG: --lock-file=""
Oct 31 19:18:49.128299 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123527    2584 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 31 19:18:49.128299 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123530    2584 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 31 19:18:49.128299 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123533    2584 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 31 19:18:49.128299 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123543    2584 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 31 19:18:49.128299 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123546    2584 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 31 19:18:49.128299 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123549    2584 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 31 19:18:49.128299 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123552    2584 flags.go:64] FLAG: --logging-format="text"
Oct 31 19:18:49.128299 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123555    2584 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 31 19:18:49.128299 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123558    2584 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 31 19:18:49.128299 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123562    2584 flags.go:64] FLAG: --manifest-url=""
Oct 31 19:18:49.128299 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123565    2584 flags.go:64] FLAG: --manifest-url-header=""
Oct 31 19:18:49.128299 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123570    2584 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 31 19:18:49.128299 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123573    2584 flags.go:64] FLAG: --max-open-files="1000000"
Oct 31 19:18:49.128299 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123577    2584 flags.go:64] FLAG: --max-pods="110"
Oct 31 19:18:49.128901 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123581    2584 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 31 19:18:49.128901 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123584    2584 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 31 19:18:49.128901 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123588    2584 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 31 19:18:49.128901 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123591    2584 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 31 19:18:49.128901 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123594    2584 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 31 19:18:49.128901 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123597    2584 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 31 19:18:49.128901 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123600    2584 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 31 19:18:49.128901 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123608    2584 flags.go:64] FLAG: --node-status-max-images="50"
Oct 31 19:18:49.128901 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123612    2584 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 31 19:18:49.128901 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123615    2584 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 31 19:18:49.128901 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123618    2584 flags.go:64] FLAG: --pod-cidr=""
Oct 31 19:18:49.128901 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123621    2584 flags.go:64] FLAG: --pod-manifest-path=""
Oct 31 19:18:49.128901 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123624    2584 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 31 19:18:49.128901 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123627    2584 flags.go:64] FLAG: --pods-per-core="0"
Oct 31 19:18:49.128901 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123630    2584 flags.go:64] FLAG: --port="10250"
Oct 31 19:18:49.128901 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123633    2584 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 31 19:18:49.128901 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123636    2584 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-09d12504134ccfb62"
Oct 31 19:18:49.128901 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123639    2584 flags.go:64] FLAG: --qos-reserved=""
Oct 31 19:18:49.128901 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123642    2584 flags.go:64] FLAG: --read-only-port="10255"
Oct 31 19:18:49.128901 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123645    2584 flags.go:64] FLAG: --register-node="true"
Oct 31 19:18:49.128901 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123648    2584 flags.go:64] FLAG: --register-with-taints=""
Oct 31 19:18:49.128901 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123652    2584 flags.go:64] FLAG: --registry-burst="10"
Oct 31 19:18:49.128901 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123655    2584 flags.go:64] FLAG: --registry-qps="5"
Oct 31 19:18:49.128901 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123658    2584 flags.go:64] FLAG: --reserved-cpus=""
Oct 31 19:18:49.128901 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123661    2584 flags.go:64] FLAG: --reserved-memory=""
Oct 31 19:18:49.128901 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123665    2584 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 31 19:18:49.129543 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123668    2584 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 31 19:18:49.129543 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123671    2584 flags.go:64] FLAG: --rotate-certificates="false"
Oct 31 19:18:49.129543 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123674    2584 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 31 19:18:49.129543 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123677    2584 flags.go:64] FLAG: --runonce="false"
Oct 31 19:18:49.129543 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123680    2584 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 31 19:18:49.129543 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123683    2584 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 31 19:18:49.129543 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123687    2584 flags.go:64] FLAG: --seccomp-default="false"
Oct 31 19:18:49.129543 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123691    2584 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 31 19:18:49.129543 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123694    2584 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 31 19:18:49.129543 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123697    2584 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 31 19:18:49.129543 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123700    2584 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 31 19:18:49.129543 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123703    2584 flags.go:64] FLAG: --storage-driver-password="root"
Oct 31 19:18:49.129543 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123706    2584 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 31 19:18:49.129543 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123710    2584 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 31 19:18:49.129543 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123713    2584 flags.go:64] FLAG: --storage-driver-user="root"
Oct 31 19:18:49.129543 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123717    2584 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 31 19:18:49.129543 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123720    2584 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 31 19:18:49.129543 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123723    2584 flags.go:64] FLAG: --system-cgroups=""
Oct 31 19:18:49.129543 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123726    2584 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 31 19:18:49.129543 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123732    2584 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 31 19:18:49.129543 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123735    2584 flags.go:64] FLAG: --tls-cert-file=""
Oct 31 19:18:49.129543 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123737    2584 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 31 19:18:49.129543 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123742    2584 flags.go:64] FLAG: --tls-min-version=""
Oct 31 19:18:49.129543 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123745    2584 flags.go:64] FLAG: --tls-private-key-file=""
Oct 31 19:18:49.129543 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123748    2584 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 31 19:18:49.130135 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123751    2584 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 31 19:18:49.130135 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123754    2584 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 31 19:18:49.130135 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123757    2584 flags.go:64] FLAG: --v="2"
Oct 31 19:18:49.130135 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123762    2584 flags.go:64] FLAG: --version="false"
Oct 31 19:18:49.130135 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123766    2584 flags.go:64] FLAG: --vmodule=""
Oct 31 19:18:49.130135 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123770    2584 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 31 19:18:49.130135 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.123774    2584 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 31 19:18:49.130135 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.123904    2584 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 19:18:49.130135 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.123910    2584 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 19:18:49.130135 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.123913    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 19:18:49.130135 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.123916    2584 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 19:18:49.130135 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.123918    2584 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 19:18:49.130135 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.123921    2584 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 19:18:49.130135 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.123923    2584 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 19:18:49.130135 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.123926    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 19:18:49.130135 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.123929    2584 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 19:18:49.130135 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.123932    2584 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 19:18:49.130135 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.123935    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 19:18:49.130135 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.123937    2584 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 19:18:49.130135 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.123940    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 19:18:49.130135 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.123944    2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 19:18:49.130651 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.123947    2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 19:18:49.130651 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.123950    2584 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 19:18:49.130651 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.123953    2584 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 19:18:49.130651 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.123955    2584 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 19:18:49.130651 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.123958    2584 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 19:18:49.130651 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.123960    2584 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 19:18:49.130651 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.123963    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 19:18:49.130651 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.123966    2584 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 19:18:49.130651 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.123968    2584 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 19:18:49.130651 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.123971    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 19:18:49.130651 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.123973    2584 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 19:18:49.130651 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.123976    2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 19:18:49.130651 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.123978    2584 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 19:18:49.130651 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.123981    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 19:18:49.130651 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.123983    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 19:18:49.130651 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.123986    2584 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 19:18:49.130651 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.123989    2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 19:18:49.130651 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.123991    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 19:18:49.130651 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.123994    2584 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 19:18:49.131130 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.123998    2584 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 19:18:49.131130 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124001    2584 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 19:18:49.131130 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124003    2584 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 19:18:49.131130 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124006    2584 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 19:18:49.131130 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124008    2584 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 19:18:49.131130 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124011    2584 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 19:18:49.131130 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124014    2584 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 19:18:49.131130 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124016    2584 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 19:18:49.131130 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124019    2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 19:18:49.131130 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124022    2584 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 19:18:49.131130 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124024    2584 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 19:18:49.131130 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124027    2584 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 19:18:49.131130 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124032    2584 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 19:18:49.131130 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124034    2584 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 19:18:49.131130 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124037    2584 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 19:18:49.131130 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124039    2584 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 19:18:49.131130 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124042    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 19:18:49.131130 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124045    2584 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 19:18:49.131130 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124049    2584 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 19:18:49.131581 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124053    2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 19:18:49.131581 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124055    2584 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 19:18:49.131581 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124058    2584 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 19:18:49.131581 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124061    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 19:18:49.131581 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124063    2584 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 19:18:49.131581 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124066    2584 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 19:18:49.131581 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124068    2584 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 19:18:49.131581 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124071    2584 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 19:18:49.131581 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124074    2584 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 19:18:49.131581 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124076    2584 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 19:18:49.131581 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124079    2584 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 19:18:49.131581 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124081    2584 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 19:18:49.131581 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124084    2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 19:18:49.131581 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124088    2584 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 19:18:49.131581 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124090    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 19:18:49.131581 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124093    2584 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 19:18:49.131581 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124095    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 19:18:49.131581 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124098    2584 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 19:18:49.131581 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124101    2584 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 19:18:49.131581 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124103    2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 19:18:49.132130 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124107    2584 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 19:18:49.132130 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124111    2584 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 19:18:49.132130 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124114    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 19:18:49.132130 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124117    2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 19:18:49.132130 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124120    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 19:18:49.132130 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124124    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 19:18:49.132130 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124126    2584 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 19:18:49.132130 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124129    2584 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 19:18:49.132130 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124131    2584 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 19:18:49.132130 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124134    2584 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 19:18:49.132130 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124137    2584 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 19:18:49.132130 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124139    2584 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 19:18:49.132130 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124142    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 19:18:49.132130 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124145    2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 19:18:49.132130 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124147    2584 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 19:18:49.132130 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124150    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 19:18:49.132130 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124152    2584 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 19:18:49.132130 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124155    2584 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 19:18:49.132130 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124157    2584 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 19:18:49.132593 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124160    2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 19:18:49.132593 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124163    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 19:18:49.132593 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124165    2584 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 19:18:49.132593 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124168    2584 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 19:18:49.132593 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124170    2584 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 19:18:49.132593 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124173    2584 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 19:18:49.132593 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124177    2584 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 19:18:49.132593 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124179    2584 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 19:18:49.132593 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124182    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 19:18:49.132593 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124184    2584 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 19:18:49.132593 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124187    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 19:18:49.132593 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124189    2584 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 19:18:49.132593 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124192    2584 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 19:18:49.132593 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124195    2584 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 19:18:49.132593 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124197    2584 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 19:18:49.132593 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.124222    2584 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 19:18:49.132975 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124266    2584 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 19:18:49.132975 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124271    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 19:18:49.132975 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124274    2584 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 19:18:49.132975 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124280    2584 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 19:18:49.132975 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124283    2584 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 19:18:49.132975 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124286    2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 19:18:49.132975 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124289    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 19:18:49.132975 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124292    2584 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 19:18:49.132975 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124295    2584 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 19:18:49.132975 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124297    2584 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 19:18:49.132975 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124300    2584 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 19:18:49.132975 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124302    2584 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 19:18:49.132975 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124305    2584 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 19:18:49.132975 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124308    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 19:18:49.132975 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124311    2584 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 19:18:49.132975 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124314    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 19:18:49.132975 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124316    2584 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 19:18:49.132975 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124319    2584 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 19:18:49.132975 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124322    2584 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 19:18:49.132975 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124336    2584 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 19:18:49.133475 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124339    2584 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 19:18:49.133475 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124342    2584 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 19:18:49.133475 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124347    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 19:18:49.133475 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124350    2584 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 19:18:49.133475 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124352    2584 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 19:18:49.133475 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124355    2584 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 19:18:49.133475 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124357    2584 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 19:18:49.133475 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124360    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 19:18:49.133475 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124362    2584 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 19:18:49.133475 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124365    2584 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 19:18:49.133475 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124368    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 19:18:49.133475 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124370    2584 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 19:18:49.133475 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124373    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 19:18:49.133475 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124376    2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 19:18:49.133475 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124379    2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 19:18:49.133475 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124382    2584 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 19:18:49.133475 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124385    2584 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 19:18:49.133475 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124388    2584 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 19:18:49.133475 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124390    2584 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 19:18:49.133475 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124393    2584 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 19:18:49.133947 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124396    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 19:18:49.133947 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124398    2584 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 19:18:49.133947 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124401    2584 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 19:18:49.133947 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124404    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 19:18:49.133947 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124406    2584 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 19:18:49.133947 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124409    2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 19:18:49.133947 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124411    2584 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 19:18:49.133947 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124414    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 19:18:49.133947 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124417    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 19:18:49.133947 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124419    2584 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 19:18:49.133947 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124422    2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 19:18:49.133947 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124425    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 19:18:49.133947 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124427    2584 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 19:18:49.133947 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124430    2584 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 19:18:49.133947 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124434    2584 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 19:18:49.133947 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124436    2584 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 19:18:49.133947 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124439    2584 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 19:18:49.133947 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124442    2584 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 19:18:49.133947 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124444    2584 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 19:18:49.134415 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124447    2584 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 19:18:49.134415 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124449    2584 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 19:18:49.134415 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124452    2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 19:18:49.134415 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124454    2584 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 19:18:49.134415 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124457    2584 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 19:18:49.134415 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124459    2584 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 19:18:49.134415 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124463    2584 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 19:18:49.134415 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124466    2584 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 19:18:49.134415 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124470    2584 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 19:18:49.134415 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124473    2584 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 19:18:49.134415 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124476    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 19:18:49.134415 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124478    2584 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 19:18:49.134415 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124482    2584 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 19:18:49.134415 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124487    2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 19:18:49.134415 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124490    2584 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 19:18:49.134415 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124493    2584 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 19:18:49.134415 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124496    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 19:18:49.134415 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124498    2584 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 19:18:49.134415 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124501    2584 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 19:18:49.134415 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124504    2584 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 19:18:49.134881 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124506    2584 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 19:18:49.134881 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124509    2584 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 19:18:49.134881 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124511    2584 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 19:18:49.134881 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124514    2584 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 19:18:49.134881 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124517    2584 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 19:18:49.134881 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124519    2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 19:18:49.134881 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124522    2584 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 19:18:49.134881 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124525    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 19:18:49.134881 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124528    2584 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 19:18:49.134881 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124531    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 19:18:49.134881 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124533    2584 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 19:18:49.134881 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124536    2584 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 19:18:49.134881 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124538    2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 19:18:49.134881 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124542    2584 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 19:18:49.134881 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124546    2584 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 19:18:49.134881 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124549    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 19:18:49.134881 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124552    2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 19:18:49.134881 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124555    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 19:18:49.134881 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124558    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 19:18:49.135317 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124560    2584 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 19:18:49.135317 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124565    2584 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 19:18:49.135317 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124568    2584 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 19:18:49.135317 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124570    2584 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 19:18:49.135317 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124574    2584 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 19:18:49.135317 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124576    2584 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 19:18:49.135317 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124579    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 19:18:49.135317 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.124581    2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 19:18:49.135317 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.132092    2584 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 31 19:18:49.135317 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.132105    2584 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 31 19:18:49.135317 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.132118    2584 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 31 19:18:49.135317 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.132122    2584 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 31 19:18:49.135317 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132184    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 19:18:49.135317 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132189    2584 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 19:18:49.135317 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132193    2584 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 19:18:49.135317 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132197    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 19:18:49.135317 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132200    2584 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 19:18:49.135317 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132203    2584 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 19:18:49.135317 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132206    2584 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 19:18:49.135771 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132208    2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 19:18:49.135771 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132211    2584 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 19:18:49.135771 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132214    2584 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 19:18:49.135771 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132216    2584 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 19:18:49.135771 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132219    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 19:18:49.135771 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132221    2584 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 19:18:49.135771 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132224    2584 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 19:18:49.135771 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132228    2584 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 19:18:49.135771 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132233    2584 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 19:18:49.135771 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132235    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 19:18:49.135771 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132238    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 19:18:49.135771 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132241    2584 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 19:18:49.135771 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132243    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 19:18:49.135771 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132246    2584 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 19:18:49.135771 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132250    2584 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 19:18:49.135771 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132253    2584 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 19:18:49.135771 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132256    2584 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 19:18:49.135771 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132259    2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 19:18:49.135771 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132262    2584 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 19:18:49.136235 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132265    2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 19:18:49.136235 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132267    2584 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 19:18:49.136235 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132271    2584 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 19:18:49.136235 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132274    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 19:18:49.136235 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132277    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 19:18:49.136235 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132280    2584 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 19:18:49.136235 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132282    2584 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 19:18:49.136235 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132285    2584 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 19:18:49.136235 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132288    2584 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 19:18:49.136235 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132290    2584 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 19:18:49.136235 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132293    2584 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 19:18:49.136235 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132296    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 19:18:49.136235 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132299    2584 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 19:18:49.136235 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132301    2584 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 19:18:49.136235 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132304    2584 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 19:18:49.136235 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132307    2584 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 19:18:49.136235 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132309    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 19:18:49.136235 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132312    2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 19:18:49.136235 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132314    2584 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 19:18:49.136235 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132317    2584 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 19:18:49.136748 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132320    2584 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 19:18:49.136748 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132323    2584 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 19:18:49.136748 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132339    2584 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 19:18:49.136748 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132343    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 19:18:49.136748 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132345    2584 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 19:18:49.136748 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132348    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 19:18:49.136748 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132351    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 19:18:49.136748 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132353    2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 19:18:49.136748 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132357    2584 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 19:18:49.136748 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132360    2584 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 19:18:49.136748 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132363    2584 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 19:18:49.136748 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132365    2584 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 19:18:49.136748 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132368    2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 19:18:49.136748 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132371    2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 19:18:49.136748 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132374    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 19:18:49.136748 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132377    2584 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 19:18:49.136748 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132380    2584 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 19:18:49.136748 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132383    2584 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 19:18:49.136748 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132385    2584 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 19:18:49.137200 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132388    2584 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 19:18:49.137200 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132390    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 19:18:49.137200 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132393    2584 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 19:18:49.137200 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132396    2584 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 19:18:49.137200 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132398    2584 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 19:18:49.137200 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132401    2584 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 19:18:49.137200 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132404    2584 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 19:18:49.137200 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132406    2584 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 19:18:49.137200 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132409    2584 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 19:18:49.137200 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132411    2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 19:18:49.137200 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132414    2584 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 19:18:49.137200 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132417    2584 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 19:18:49.137200 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132419    2584 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 19:18:49.137200 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132422    2584 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 19:18:49.137200 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132424    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 19:18:49.137200 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132426    2584 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 19:18:49.137200 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132429    2584 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 19:18:49.137200 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132432    2584 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 19:18:49.137200 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132434    2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 19:18:49.137200 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132436    2584 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 19:18:49.137761 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132439    2584 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 19:18:49.137761 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132442    2584 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 19:18:49.137761 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132445    2584 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 19:18:49.137761 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132449    2584 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 19:18:49.137761 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132451    2584 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 19:18:49.137761 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132454    2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 19:18:49.137761 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132457    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 19:18:49.137761 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132459    2584 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 19:18:49.137761 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132462    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 19:18:49.137761 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132465    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 19:18:49.137761 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132467    2584 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 19:18:49.137761 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132470    2584 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 19:18:49.137761 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132473    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 19:18:49.137761 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132475    2584 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 19:18:49.137761 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132478    2584 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 19:18:49.137761 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132480    2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 19:18:49.137761 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132482    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 19:18:49.137761 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132485    2584 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 19:18:49.137761 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132487    2584 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 19:18:49.137761 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132490    2584 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 19:18:49.138221 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132494    2584 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 19:18:49.138221 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.132521    2584 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 19:18:49.138221 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132688    2584 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 19:18:49.138221 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132694    2584 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 19:18:49.138221 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132697    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 19:18:49.138221 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132700    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 19:18:49.138221 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132702    2584 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 19:18:49.138221 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132705    2584 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 19:18:49.138221 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132709    2584 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 19:18:49.138221 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132713    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 19:18:49.138221 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132716    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 19:18:49.138221 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132719    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 19:18:49.138221 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132722    2584 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 19:18:49.138221 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132724    2584 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 19:18:49.138221 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132727    2584 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 19:18:49.138586 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132731    2584 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 19:18:49.138586 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132734    2584 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 19:18:49.138586 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132736    2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 19:18:49.138586 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132739    2584 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 19:18:49.138586 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132742    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 19:18:49.138586 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132744    2584 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 19:18:49.138586 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132747    2584 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 19:18:49.138586 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132750    2584 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 19:18:49.138586 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132752    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 19:18:49.138586 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132755    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 19:18:49.138586 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132757    2584 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 19:18:49.138586 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132760    2584 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 19:18:49.138586 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132763    2584 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 19:18:49.138586 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132765    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 19:18:49.138586 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132768    2584 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 19:18:49.138586 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132770    2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 19:18:49.138586 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132773    2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 19:18:49.138586 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132775    2584 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 19:18:49.138586 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132778    2584 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 19:18:49.138586 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132780    2584 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 19:18:49.139066 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132782    2584 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 19:18:49.139066 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132785    2584 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 19:18:49.139066 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132787    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 19:18:49.139066 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132790    2584 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 19:18:49.139066 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132793    2584 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 19:18:49.139066 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132795    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 19:18:49.139066 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132797    2584 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 19:18:49.139066 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132800    2584 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 19:18:49.139066 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132803    2584 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 19:18:49.139066 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132805    2584 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 19:18:49.139066 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132808    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 19:18:49.139066 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132811    2584 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 19:18:49.139066 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132813    2584 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 19:18:49.139066 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132817    2584 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 19:18:49.139066 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132819    2584 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 19:18:49.139066 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132822    2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 19:18:49.139066 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132824    2584 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 19:18:49.139066 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132827    2584 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 19:18:49.139066 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132830    2584 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 19:18:49.139066 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132833    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 19:18:49.139531 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132835    2584 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 19:18:49.139531 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132838    2584 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 19:18:49.139531 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132840    2584 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 19:18:49.139531 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132843    2584 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 19:18:49.139531 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132845    2584 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 19:18:49.139531 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132848    2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 19:18:49.139531 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132850    2584 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 19:18:49.139531 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132853    2584 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 19:18:49.139531 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132855    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 19:18:49.139531 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132858    2584 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 19:18:49.139531 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132860    2584 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 19:18:49.139531 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132863    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 19:18:49.139531 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132865    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 19:18:49.139531 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132868    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 19:18:49.139531 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132871    2584 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 19:18:49.139531 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132873    2584 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 19:18:49.139531 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132875    2584 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 19:18:49.139531 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132878    2584 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 19:18:49.139531 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132881    2584 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 19:18:49.139531 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132883    2584 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 19:18:49.140031 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132886    2584 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 19:18:49.140031 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132889    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 19:18:49.140031 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132891    2584 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 19:18:49.140031 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132893    2584 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 19:18:49.140031 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132897    2584 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 19:18:49.140031 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132901    2584 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 19:18:49.140031 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132904    2584 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 19:18:49.140031 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132907    2584 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 19:18:49.140031 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132910    2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 19:18:49.140031 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132913    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 19:18:49.140031 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132915    2584 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 19:18:49.140031 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132918    2584 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 19:18:49.140031 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132920    2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 19:18:49.140031 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132923    2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 19:18:49.140031 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132925    2584 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 19:18:49.140031 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132928    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 19:18:49.140031 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132931    2584 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 19:18:49.140031 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132933    2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 19:18:49.140031 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132936    2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 19:18:49.140485 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132938    2584 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 19:18:49.140485 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132941    2584 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 19:18:49.140485 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132944    2584 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 19:18:49.140485 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132946    2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 19:18:49.140485 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132949    2584 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 19:18:49.140485 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132951    2584 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 19:18:49.140485 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132954    2584 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 19:18:49.140485 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132956    2584 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 19:18:49.140485 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132959    2584 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 19:18:49.140485 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132961    2584 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 19:18:49.140485 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132964    2584 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 19:18:49.140485 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132966    2584 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 19:18:49.140485 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132969    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 19:18:49.140485 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.132971    2584 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 19:18:49.140485 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.132994    2584 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 19:18:49.140485 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133031    2584 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 19:18:49.140889 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133037    2584 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 19:18:49.140889 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133048    2584 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 19:18:49.140889 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133052    2584 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 19:18:49.140889 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133054    2584 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 19:18:49.140889 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133058    2584 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 19:18:49.140889 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133061    2584 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 19:18:49.140889 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133066    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 19:18:49.140889 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133069    2584 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 19:18:49.140889 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133071    2584 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 19:18:49.140889 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133074    2584 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 19:18:49.140889 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133077    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 19:18:49.140889 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133080    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 19:18:49.140889 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133083    2584 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 19:18:49.140889 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133085    2584 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 19:18:49.140889 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133089    2584 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 19:18:49.140889 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133093    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 19:18:49.140889 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133096    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 19:18:49.140889 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133100    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 19:18:49.140889 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133104    2584 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 19:18:49.141438 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133106    2584 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 19:18:49.141438 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133109    2584 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 19:18:49.141438 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133112    2584 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 19:18:49.141438 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133114    2584 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 19:18:49.141438 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133117    2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 19:18:49.141438 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133119    2584 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 19:18:49.141438 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133122    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 19:18:49.141438 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133124    2584 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 19:18:49.141438 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133127    2584 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 19:18:49.141438 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133129    2584 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 19:18:49.141438 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133132    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 19:18:49.141438 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133134    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 19:18:49.141438 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133137    2584 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 19:18:49.141438 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133140    2584 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 19:18:49.141438 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133142    2584 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 19:18:49.141438 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133144    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 19:18:49.141438 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133147    2584 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 19:18:49.141438 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133150    2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 19:18:49.141438 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133153    2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 19:18:49.141438 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133158    2584 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 19:18:49.141977 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133161    2584 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 19:18:49.141977 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133163    2584 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 19:18:49.141977 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133166    2584 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 19:18:49.141977 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133171    2584 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 19:18:49.141977 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133173    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 19:18:49.141977 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133176    2584 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 19:18:49.141977 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133178    2584 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 19:18:49.141977 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133181    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 19:18:49.141977 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133184    2584 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 19:18:49.141977 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133186    2584 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 19:18:49.141977 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133189    2584 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 19:18:49.141977 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133191    2584 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 19:18:49.141977 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133194    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 19:18:49.141977 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133197    2584 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 19:18:49.141977 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133200    2584 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 19:18:49.141977 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133202    2584 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 19:18:49.141977 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133205    2584 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 19:18:49.141977 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133207    2584 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 19:18:49.141977 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133210    2584 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 19:18:49.141977 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133213    2584 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 19:18:49.142494 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133215    2584 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 19:18:49.142494 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133218    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 19:18:49.142494 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133220    2584 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 19:18:49.142494 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133223    2584 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 19:18:49.142494 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133225    2584 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 19:18:49.142494 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133228    2584 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 19:18:49.142494 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133230    2584 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 19:18:49.142494 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133233    2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 19:18:49.142494 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133235    2584 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 19:18:49.142494 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133238    2584 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 19:18:49.142494 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133241    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 19:18:49.142494 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133245    2584 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 19:18:49.142494 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133248    2584 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 19:18:49.142494 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133250    2584 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 19:18:49.142494 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133253    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 19:18:49.142494 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133258    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 19:18:49.142494 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133260    2584 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 19:18:49.142494 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133263    2584 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 19:18:49.142494 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133265    2584 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 19:18:49.142930 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133268    2584 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 19:18:49.142930 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133271    2584 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 19:18:49.142930 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133273    2584 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 19:18:49.142930 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133276    2584 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 19:18:49.142930 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133279    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 19:18:49.142930 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133281    2584 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 19:18:49.142930 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133284    2584 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 19:18:49.142930 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133288    2584 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 19:18:49.142930 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133291    2584 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 19:18:49.142930 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133294    2584 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 19:18:49.142930 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133297    2584 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 19:18:49.142930 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133300    2584 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 19:18:49.142930 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133302    2584 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 19:18:49.142930 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133305    2584 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 19:18:49.142930 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133307    2584 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 19:18:49.142930 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133310    2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 19:18:49.142930 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133313    2584 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 19:18:49.142930 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133315    2584 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 19:18:49.142930 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133318    2584 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 19:18:49.142930 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133320    2584 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 19:18:49.143461 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133323    2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 19:18:49.143461 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133338    2584 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 19:18:49.143461 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133341    2584 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 19:18:49.143461 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133344    2584 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 19:18:49.143461 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133349    2584 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 19:18:49.143461 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133352    2584 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 19:18:49.143461 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:49.133355    2584 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 19:18:49.143461 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.133405    2584 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 31 19:18:49.143461 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.136025    2584 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 31 19:18:49.143461 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.137053    2584 server.go:1014] "Starting client certificate rotation"
Oct 31 19:18:49.143461 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.137147    2584 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 19:18:49.143461 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.137176    2584 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 19:18:49.143461 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.139759    2584 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 19:18:49.143461 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.141315    2584 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 19:18:49.149910 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.149890    2584 log.go:25] "Validated CRI v1 runtime API"
Oct 31 19:18:49.153844 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.153828    2584 log.go:25] "Validated CRI v1 image API"
Oct 31 19:18:49.155070 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.155037    2584 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 31 19:18:49.159380 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.159355    2584 fs.go:135] Filesystem UUIDs: map[1a7924e8-5660-49f5-9f5f-4ca38580f698:/dev/nvme0n1p4 33970d72-81df-4c0e-bd75-9ac6392bcde7:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Oct 31 19:18:49.159456 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.159380    2584 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 31 19:18:49.164464 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.164344    2584 manager.go:217] Machine: {Timestamp:2026-10-31 19:18:49.163012102 +0000 UTC m=+0.373857279 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100004 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29c520b7507a46ec0dbcadf7ef2614 SystemUUID:ec29c520-b750-7a46-ec0d-bcadf7ef2614 BootID:ae04324e-8c8a-4491-92de-58aa3c832361 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e3:e7:bf:69 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e3:e7:bf:69 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ba:6a:7b:69:ee:3e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 31 19:18:49.164464 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.164457    2584 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 31 19:18:49.164623 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.164565    2584 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 31 19:18:49.165860 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.165833    2584 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 31 19:18:49.165992 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.165862    2584 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-94.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 31 19:18:49.166041 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.165995    2584 topology_manager.go:143] "Creating topology manager with none policy"
Oct 31 19:18:49.166041 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.166003    2584 container_manager_linux.go:311] "Creating device plugin manager"
Oct 31 19:18:49.166041 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.166026    2584 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 19:18:49.167926 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.167912    2584 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 19:18:49.169152 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.169143    2584 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 31 19:18:49.169556 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.169541    2584 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 31 19:18:49.173819 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.173806    2584 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 31 19:18:49.173941 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.173932    2584 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 31 19:18:49.173972 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.173961    2584 kubelet.go:493] "Attempting to sync node with API server"
Oct 31 19:18:49.173972 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.173971    2584 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 31 19:18:49.174028 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.173982    2584 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 31 19:18:49.174028 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.173990    2584 kubelet.go:405] "Adding apiserver pod source"
Oct 31 19:18:49.174028 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.173999    2584 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 31 19:18:49.174443 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.174430    2584 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 19:18:49.176943 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.176922    2584 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 31 19:18:49.182498 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.182483    2584 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 19:18:49.183485 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.183469    2584 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 31 19:18:49.183546 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.183500    2584 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 31 19:18:49.184246 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.184235    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 31 19:18:49.184283 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.184252    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 31 19:18:49.184283 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.184259    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 31 19:18:49.184283 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.184265    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 31 19:18:49.184283 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.184270    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 31 19:18:49.184283 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.184277    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 31 19:18:49.185046 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.185032    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 31 19:18:49.185080 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.185048    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 31 19:18:49.185080 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.185056    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 31 19:18:49.185080 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.185062    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 31 19:18:49.185080 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.185070    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 31 19:18:49.185080 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.185079    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 31 19:18:49.186589 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.186579    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 31 19:18:49.186633 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.186592    2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 31 19:18:49.191469 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.191455    2584 server.go:1263] "Started kubelet"
Oct 31 19:18:49.191557 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.191530    2584 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 31 19:18:49.191595 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.191545    2584 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 31 19:18:49.191628 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.191604    2584 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 31 19:18:49.192183 ip-10-0-131-94 systemd[1]: Started Kubernetes Kubelet.
Oct 31 19:18:49.195812 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.195790    2584 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 31 19:18:49.199447 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.198217    2584 server.go:317] "Adding debug handlers to kubelet server"
Oct 31 19:18:49.199447 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:18:49.199428    2584 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 31 19:18:49.199896 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.199716    2584 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-94.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 31 19:18:49.202625 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.202608    2584 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 31 19:18:49.202711 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.202623    2584 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 31 19:18:49.202711 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:18:49.202691    2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-94.us-east-2.compute.internal\" not found"
Oct 31 19:18:49.203443 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.203395    2584 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 31 19:18:49.203443 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.203432    2584 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 31 19:18:49.203620 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.203449    2584 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 31 19:18:49.203620 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:18:49.203466    2584 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 31 19:18:49.203620 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.203568    2584 reconstruct.go:97] "Volume reconstruction finished"
Oct 31 19:18:49.203620 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.203574    2584 reconciler.go:29] "Reconciler: start to sync state"
Oct 31 19:18:49.203801 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:18:49.203623    2584 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-94.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 31 19:18:49.203860 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.203847    2584 factory.go:55] Registering systemd factory
Oct 31 19:18:49.203913 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.203861    2584 factory.go:223] Registration of the systemd container factory successfully
Oct 31 19:18:49.204059 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.204044    2584 factory.go:153] Registering CRI-O factory
Oct 31 19:18:49.204059 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.204059    2584 factory.go:223] Registration of the crio container factory successfully
Oct 31 19:18:49.204149 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.204132    2584 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 31 19:18:49.204197 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.204150    2584 factory.go:103] Registering Raw factory
Oct 31 19:18:49.204197 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.204165    2584 manager.go:1199] Started watching for new ooms in manager
Oct 31 19:18:49.205015 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:18:49.203219    2584 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-94.us-east-2.compute.internal.18e3b366713cacc2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-94.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-94.us-east-2.compute.internal,},FirstTimestamp:2026-10-31 19:18:49.191427266 +0000 UTC m=+0.402272444,LastTimestamp:2026-10-31 19:18:49.191427266 +0000 UTC m=+0.402272444,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-94.us-east-2.compute.internal,}"
Oct 31 19:18:49.205207 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.205192    2584 manager.go:319] Starting recovery of all containers
Oct 31 19:18:49.214871 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.214856    2584 manager.go:324] Recovery completed
Oct 31 19:18:49.216563 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.216545    2584 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5ptdx"
Oct 31 19:18:49.216644 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:18:49.216608    2584 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-94.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 31 19:18:49.220189 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.220174    2584 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 19:18:49.222084 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.222062    2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-94.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 19:18:49.222166 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.222101    2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-94.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 19:18:49.222166 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.222118    2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-94.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 19:18:49.222888 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.222872    2584 cpu_manager.go:226] "Starting" policy="none"
Oct 31 19:18:49.222888 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.222887    2584 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 31 19:18:49.223016 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.222877    2584 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 19:18:49.223016 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.222905    2584 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 31 19:18:49.223627 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.223611    2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-94.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 19:18:49.223690 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.223638    2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-94.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 19:18:49.223690 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.223648    2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-94.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 19:18:49.225089 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.225076    2584 policy_none.go:50] "Start"
Oct 31 19:18:49.225135 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.225095    2584 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 31 19:18:49.225135 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.225106    2584 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 31 19:18:49.228218 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.228203    2584 policy_none.go:44] "Start"
Oct 31 19:18:49.229677 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.229660    2584 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5ptdx"
Oct 31 19:18:49.232631 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:18:49.232606    2584 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 31 19:18:49.232923 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:18:49.232833    2584 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-94.us-east-2.compute.internal.18e3b36673106da0  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-94.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-94.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-94.us-east-2.compute.internal,},FirstTimestamp:2026-10-31 19:18:49.222081952 +0000 UTC m=+0.432927135,LastTimestamp:2026-10-31 19:18:49.222081952 +0000 UTC m=+0.432927135,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-94.us-east-2.compute.internal,}"
Oct 31 19:18:49.239727 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:18:49.239643    2584 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-94.us-east-2.compute.internal.18e3b3667310d6ca  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-94.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-131-94.us-east-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-131-94.us-east-2.compute.internal,},FirstTimestamp:2026-10-31 19:18:49.222108874 +0000 UTC m=+0.432954050,LastTimestamp:2026-10-31 19:18:49.222108874 +0000 UTC m=+0.432954050,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-94.us-east-2.compute.internal,}"
Oct 31 19:18:49.264873 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.264850    2584 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 31 19:18:49.267227 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.267212    2584 manager.go:341] "Starting Device Plugin manager"
Oct 31 19:18:49.267303 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:18:49.267244    2584 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 31 19:18:49.267303 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.267267    2584 server.go:87] "Starting device plugin registration server"
Oct 31 19:18:49.267534 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.267520    2584 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 31 19:18:49.267590 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.267534    2584 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 31 19:18:49.267643 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.267626    2584 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 31 19:18:49.267754 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.267706    2584 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 31 19:18:49.267754 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.267719    2584 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 31 19:18:49.268258 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:18:49.268239    2584 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 31 19:18:49.268313 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:18:49.268292    2584 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-94.us-east-2.compute.internal\" not found"
Oct 31 19:18:49.352638 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.352610    2584 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 31 19:18:49.354389 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.353830    2584 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 31 19:18:49.354389 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.353853    2584 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 31 19:18:49.354389 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.353871    2584 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 31 19:18:49.354389 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:18:49.353902    2584 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 31 19:18:49.357110 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.357095    2584 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 19:18:49.368657 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.368593    2584 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 19:18:49.369738 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.369718    2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-94.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 19:18:49.369806 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.369755    2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-94.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 19:18:49.369806 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.369770    2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-94.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 19:18:49.369806 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.369797    2584 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-94.us-east-2.compute.internal"
Oct 31 19:18:49.377900 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.377886    2584 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-131-94.us-east-2.compute.internal"
Oct 31 19:18:49.377976 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:18:49.377906    2584 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-94.us-east-2.compute.internal\": node \"ip-10-0-131-94.us-east-2.compute.internal\" not found"
Oct 31 19:18:49.390398 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:18:49.390383    2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-94.us-east-2.compute.internal\" not found"
Oct 31 19:18:49.454633 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.454473    2584 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-94.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-94.us-east-2.compute.internal"]
Oct 31 19:18:49.454747 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.454675    2584 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 19:18:49.455562 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.455547    2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-94.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 19:18:49.455687 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.455578    2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-94.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 19:18:49.455687 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.455593    2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-94.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 19:18:49.457520 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.457505    2584 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 19:18:49.457662 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.457648    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-94.us-east-2.compute.internal"
Oct 31 19:18:49.457713 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.457676    2584 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 19:18:49.458204 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.458188    2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-94.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 19:18:49.458289 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.458212    2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-94.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 19:18:49.458289 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.458223    2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-94.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 19:18:49.458289 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.458195    2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-94.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 19:18:49.458289 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.458286    2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-94.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 19:18:49.458499 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.458306    2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-94.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 19:18:49.460046 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.460033    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-94.us-east-2.compute.internal"
Oct 31 19:18:49.460103 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.460056    2584 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 19:18:49.460734 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.460720    2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-94.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 19:18:49.460838 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.460747    2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-94.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 19:18:49.460838 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.460773    2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-94.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 19:18:49.490455 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:18:49.490433    2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-94.us-east-2.compute.internal\" not found"
Oct 31 19:18:49.499314 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:18:49.499294    2584 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-94.us-east-2.compute.internal\" not found" node="ip-10-0-131-94.us-east-2.compute.internal"
Oct 31 19:18:49.502053 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:18:49.502034    2584 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-94.us-east-2.compute.internal\" not found" node="ip-10-0-131-94.us-east-2.compute.internal"
Oct 31 19:18:49.505425 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.505411    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b8f50d0470923e3e6371360e86f8acbb-config\") pod \"kube-apiserver-proxy-ip-10-0-131-94.us-east-2.compute.internal\" (UID: \"b8f50d0470923e3e6371360e86f8acbb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-94.us-east-2.compute.internal"
Oct 31 19:18:49.505477 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.505432    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/96d3baf5b18698617651d9af1ffbc2b2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-94.us-east-2.compute.internal\" (UID: \"96d3baf5b18698617651d9af1ffbc2b2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-94.us-east-2.compute.internal"
Oct 31 19:18:49.505477 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.505451    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/96d3baf5b18698617651d9af1ffbc2b2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-94.us-east-2.compute.internal\" (UID: \"96d3baf5b18698617651d9af1ffbc2b2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-94.us-east-2.compute.internal"
Oct 31 19:18:49.591158 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:18:49.591127    2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-94.us-east-2.compute.internal\" not found"
Oct 31 19:18:49.605931 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.605895    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/96d3baf5b18698617651d9af1ffbc2b2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-94.us-east-2.compute.internal\" (UID: \"96d3baf5b18698617651d9af1ffbc2b2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-94.us-east-2.compute.internal"
Oct 31 19:18:49.606081 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.605937    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/96d3baf5b18698617651d9af1ffbc2b2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-94.us-east-2.compute.internal\" (UID: \"96d3baf5b18698617651d9af1ffbc2b2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-94.us-east-2.compute.internal"
Oct 31 19:18:49.606081 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.605966    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b8f50d0470923e3e6371360e86f8acbb-config\") pod \"kube-apiserver-proxy-ip-10-0-131-94.us-east-2.compute.internal\" (UID: \"b8f50d0470923e3e6371360e86f8acbb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-94.us-east-2.compute.internal"
Oct 31 19:18:49.606081 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.606018    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b8f50d0470923e3e6371360e86f8acbb-config\") pod \"kube-apiserver-proxy-ip-10-0-131-94.us-east-2.compute.internal\" (UID: \"b8f50d0470923e3e6371360e86f8acbb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-94.us-east-2.compute.internal"
Oct 31 19:18:49.606081 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.606022    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/96d3baf5b18698617651d9af1ffbc2b2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-94.us-east-2.compute.internal\" (UID: \"96d3baf5b18698617651d9af1ffbc2b2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-94.us-east-2.compute.internal"
Oct 31 19:18:49.606081 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.606022    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/96d3baf5b18698617651d9af1ffbc2b2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-94.us-east-2.compute.internal\" (UID: \"96d3baf5b18698617651d9af1ffbc2b2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-94.us-east-2.compute.internal"
Oct 31 19:18:49.691800 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:18:49.691716    2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-94.us-east-2.compute.internal\" not found"
Oct 31 19:18:49.792183 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:18:49.792161    2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-94.us-east-2.compute.internal\" not found"
Oct 31 19:18:49.802385 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.802365    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-94.us-east-2.compute.internal"
Oct 31 19:18:49.804508 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:49.804492    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-94.us-east-2.compute.internal"
Oct 31 19:18:49.892498 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:18:49.892467    2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-94.us-east-2.compute.internal\" not found"
Oct 31 19:18:49.993002 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:18:49.992943    2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-94.us-east-2.compute.internal\" not found"
Oct 31 19:18:50.093489 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:18:50.093464    2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-94.us-east-2.compute.internal\" not found"
Oct 31 19:18:50.136958 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:50.136942    2584 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 31 19:18:50.137365 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:50.137068    2584 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 31 19:18:50.194358 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:18:50.194321    2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-94.us-east-2.compute.internal\" not found"
Oct 31 19:18:50.237188 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:50.237148    2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 19:13:50 +0000 UTC" deadline="2028-07-01 15:26:48.880664342 +0000 UTC"
Oct 31 19:18:50.237188 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:50.237184    2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14612h7m58.643485616s"
Oct 31 19:18:50.266689 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:50.266621    2584 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 19:18:50.295092 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:18:50.295065    2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-94.us-east-2.compute.internal\" not found"
Oct 31 19:18:50.305525 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:50.305508    2584 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 19:18:50.306577 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:50.306560    2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-94.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 19:18:50.306643 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:50.306596    2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-94.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 19:18:50.306643 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:50.306612    2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-94.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 19:18:50.395849 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:18:50.395813    2584 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-94.us-east-2.compute.internal\" not found"
Oct 31 19:18:50.460484 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:50.460457    2584 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 19:18:50.481017 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:50.480987    2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96d3baf5b18698617651d9af1ffbc2b2.slice/crio-876a4b2c55217a62653fbcffac89b3ff4f237029ca500213a384afde189b8b85 WatchSource:0}: Error finding container 876a4b2c55217a62653fbcffac89b3ff4f237029ca500213a384afde189b8b85: Status 404 returned error can't find the container with id 876a4b2c55217a62653fbcffac89b3ff4f237029ca500213a384afde189b8b85
Oct 31 19:18:50.485385 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:50.485359    2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8f50d0470923e3e6371360e86f8acbb.slice/crio-87d4f3ec9faded6c0502d00c22586498bc6b8b44e0e3db741d6c1b45e313e9c6 WatchSource:0}: Error finding container 87d4f3ec9faded6c0502d00c22586498bc6b8b44e0e3db741d6c1b45e313e9c6: Status 404 returned error can't find the container with id 87d4f3ec9faded6c0502d00c22586498bc6b8b44e0e3db741d6c1b45e313e9c6
Oct 31 19:18:50.486707 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:50.486688    2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 19:18:50.503113 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:50.503092    2584 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-94.us-east-2.compute.internal"
Oct 31 19:18:50.518687 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:50.518640    2584 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 19:18:50.519428 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:50.519415    2584 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-94.us-east-2.compute.internal"
Oct 31 19:18:50.533057 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:50.533040    2584 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 19:18:50.831754 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:50.831682    2584 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 19:18:51.096386 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.096306    2584 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 19:18:51.176068 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.176034    2584 apiserver.go:52] "Watching apiserver"
Oct 31 19:18:51.193720 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.193676    2584 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 31 19:18:51.194496 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.194094    2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tmtsr","openshift-dns/node-resolver-58f52","openshift-image-registry/node-ca-5mwk4","openshift-multus/multus-hhnhg","kube-system/konnectivity-agent-kc9mq","kube-system/kube-apiserver-proxy-ip-10-0-131-94.us-east-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-94.us-east-2.compute.internal","openshift-multus/multus-additional-cni-plugins-6s8ft","openshift-ovn-kubernetes/ovnkube-node-mr9n2","kube-system/global-pull-secret-syncer-4pntt","openshift-cluster-node-tuning-operator/tuned-kzv9d","openshift-network-diagnostics/network-check-target-4j5p9","openshift-network-operator/iptables-alerter-c5fnq","openshift-multus/network-metrics-daemon-j6fmb"]
Oct 31 19:18:51.197218 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.196825    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4pntt"
Oct 31 19:18:51.197218 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:18:51.196900    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4pntt" podUID="4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761"
Oct 31 19:18:51.199159 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.199137    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-c5fnq"
Oct 31 19:18:51.201481 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.201453    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5mwk4"
Oct 31 19:18:51.203930 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.203907    2584 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 31 19:18:51.205205 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.205186    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hhnhg"
Oct 31 19:18:51.207922 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.207810    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kc9mq"
Oct 31 19:18:51.209660 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.209002    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 31 19:18:51.209660 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.209078    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9gfbm\""
Oct 31 19:18:51.209660 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.209170    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-dcmmt\""
Oct 31 19:18:51.209660 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.209231    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 31 19:18:51.209898 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.209802    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 31 19:18:51.209898 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.209891    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 31 19:18:51.210180 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.210164    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j6fmb"
Oct 31 19:18:51.210268 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:18:51.210239    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j6fmb" podUID="3382fc12-f376-4b15-95be-6cb371ae87bd"
Oct 31 19:18:51.210995 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.210972    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 31 19:18:51.210995 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.210976    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 31 19:18:51.211141 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.211043    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 31 19:18:51.211141 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.211054    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 31 19:18:51.213744 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.213720    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:18:51.214573 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.214546    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761-kubelet-config\") pod \"global-pull-secret-syncer-4pntt\" (UID: \"4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761\") " pod="kube-system/global-pull-secret-syncer-4pntt"
Oct 31 19:18:51.214655 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.214587    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bb36e52c-ab3f-4c51-84e0-d943187ef234-multus-cni-dir\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:18:51.214655 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.214613    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bb36e52c-ab3f-4c51-84e0-d943187ef234-host-var-lib-cni-bin\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:18:51.214655 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.214648    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bb36e52c-ab3f-4c51-84e0-d943187ef234-multus-daemon-config\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:18:51.214770 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.214685    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bb36e52c-ab3f-4c51-84e0-d943187ef234-host-run-multus-certs\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:18:51.214770 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.214688    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-w5n5s\""
Oct 31 19:18:51.214770 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.214717    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb36e52c-ab3f-4c51-84e0-d943187ef234-etc-kubernetes\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:18:51.214770 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.214755    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfwph\" (UniqueName: \"kubernetes.io/projected/bb36e52c-ab3f-4c51-84e0-d943187ef234-kube-api-access-tfwph\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:18:51.214914 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.214782    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761-dbus\") pod \"global-pull-secret-syncer-4pntt\" (UID: \"4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761\") " pod="kube-system/global-pull-secret-syncer-4pntt"
Oct 31 19:18:51.214914 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.214815    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bb36e52c-ab3f-4c51-84e0-d943187ef234-cnibin\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:18:51.214914 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.214838    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bb36e52c-ab3f-4c51-84e0-d943187ef234-multus-socket-dir-parent\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:18:51.214914 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.214881    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb36e52c-ab3f-4c51-84e0-d943187ef234-host-var-lib-kubelet\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:18:51.215112 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.214948    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/739472ea-5c22-44a3-9a81-acbf0755b3b4-host-slash\") pod \"iptables-alerter-c5fnq\" (UID: \"739472ea-5c22-44a3-9a81-acbf0755b3b4\") " pod="openshift-network-operator/iptables-alerter-c5fnq"
Oct 31 19:18:51.215112 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.214972    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/944a350f-aa7e-4921-8ddc-b0001fa05086-serviceca\") pod \"node-ca-5mwk4\" (UID: \"944a350f-aa7e-4921-8ddc-b0001fa05086\") " pod="openshift-image-registry/node-ca-5mwk4"
Oct 31 19:18:51.215112 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.214988    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bb36e52c-ab3f-4c51-84e0-d943187ef234-system-cni-dir\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:18:51.215112 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.215019    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bb36e52c-ab3f-4c51-84e0-d943187ef234-cni-binary-copy\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:18:51.215112 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.215041    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bb36e52c-ab3f-4c51-84e0-d943187ef234-host-var-lib-cni-multus\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:18:51.215112 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.215055    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bb36e52c-ab3f-4c51-84e0-d943187ef234-hostroot\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:18:51.215112 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.215070    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761-original-pull-secret\") pod \"global-pull-secret-syncer-4pntt\" (UID: \"4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761\") " pod="kube-system/global-pull-secret-syncer-4pntt"
Oct 31 19:18:51.215112 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.215107    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/739472ea-5c22-44a3-9a81-acbf0755b3b4-iptables-alerter-script\") pod \"iptables-alerter-c5fnq\" (UID: \"739472ea-5c22-44a3-9a81-acbf0755b3b4\") " pod="openshift-network-operator/iptables-alerter-c5fnq"
Oct 31 19:18:51.215542 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.215123    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6xws\" (UniqueName: \"kubernetes.io/projected/739472ea-5c22-44a3-9a81-acbf0755b3b4-kube-api-access-l6xws\") pod \"iptables-alerter-c5fnq\" (UID: \"739472ea-5c22-44a3-9a81-acbf0755b3b4\") " pod="openshift-network-operator/iptables-alerter-c5fnq"
Oct 31 19:18:51.215542 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.215150    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/944a350f-aa7e-4921-8ddc-b0001fa05086-host\") pod \"node-ca-5mwk4\" (UID: \"944a350f-aa7e-4921-8ddc-b0001fa05086\") " pod="openshift-image-registry/node-ca-5mwk4"
Oct 31 19:18:51.215542 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.215169    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lq8r\" (UniqueName: \"kubernetes.io/projected/944a350f-aa7e-4921-8ddc-b0001fa05086-kube-api-access-5lq8r\") pod \"node-ca-5mwk4\" (UID: \"944a350f-aa7e-4921-8ddc-b0001fa05086\") " pod="openshift-image-registry/node-ca-5mwk4"
Oct 31 19:18:51.215542 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.215194    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bb36e52c-ab3f-4c51-84e0-d943187ef234-host-run-k8s-cni-cncf-io\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:18:51.215542 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.215210    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bb36e52c-ab3f-4c51-84e0-d943187ef234-host-run-netns\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:18:51.215542 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.215233    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bb36e52c-ab3f-4c51-84e0-d943187ef234-multus-conf-dir\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:18:51.217842 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.217822    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6s8ft"
Oct 31 19:18:51.220435 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.220280    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-58f52"
Oct 31 19:18:51.223299 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.223283    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tmtsr"
Oct 31 19:18:51.226749 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.226725    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:18:51.230367 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.230096    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4j5p9"
Oct 31 19:18:51.230367 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:18:51.230173    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4j5p9" podUID="534f9972-b54f-4713-8326-4808fdf08386"
Oct 31 19:18:51.260983 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.260963    2584 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 31 19:18:51.261130 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.261020    2584 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-4xjvw"
Oct 31 19:18:51.292426 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.292352    2584 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-4xjvw"
Oct 31 19:18:51.305440 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.305416    2584 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 31 19:18:51.316254 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.316228    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761-dbus\") pod \"global-pull-secret-syncer-4pntt\" (UID: \"4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761\") " pod="kube-system/global-pull-secret-syncer-4pntt"
Oct 31 19:18:51.316395 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.316272    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4220431d-fa8a-4537-ad2e-e8ba6164f63d-cni-binary-copy\") pod \"multus-additional-cni-plugins-6s8ft\" (UID: \"4220431d-fa8a-4537-ad2e-e8ba6164f63d\") " pod="openshift-multus/multus-additional-cni-plugins-6s8ft"
Oct 31 19:18:51.316395 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.316303    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k8mg\" (UniqueName: \"kubernetes.io/projected/21dc10e8-ee61-46ae-83f7-67d1772f083c-kube-api-access-7k8mg\") pod \"node-resolver-58f52\" (UID: \"21dc10e8-ee61-46ae-83f7-67d1772f083c\") " pod="openshift-dns/node-resolver-58f52"
Oct 31 19:18:51.316395 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.316328    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk5qd\" (UniqueName: \"kubernetes.io/projected/638d688d-d5fc-480d-8c53-08ab2f1864b1-kube-api-access-gk5qd\") pod \"aws-ebs-csi-driver-node-tmtsr\" (UID: \"638d688d-d5fc-480d-8c53-08ab2f1864b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tmtsr"
Oct 31 19:18:51.316395 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.316370    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/65c98317-00d2-4ba8-8f66-9b34d720c553-etc-sysctl-conf\") pod \"tuned-kzv9d\" (UID: \"65c98317-00d2-4ba8-8f66-9b34d720c553\") " pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:18:51.316608 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.316398    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65c98317-00d2-4ba8-8f66-9b34d720c553-tmp\") pod \"tuned-kzv9d\" (UID: \"65c98317-00d2-4ba8-8f66-9b34d720c553\") " pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:18:51.316608 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.316425    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/638d688d-d5fc-480d-8c53-08ab2f1864b1-socket-dir\") pod \"aws-ebs-csi-driver-node-tmtsr\" (UID: \"638d688d-d5fc-480d-8c53-08ab2f1864b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tmtsr"
Oct 31 19:18:51.316608 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.316449    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/65c98317-00d2-4ba8-8f66-9b34d720c553-etc-sysctl-d\") pod \"tuned-kzv9d\" (UID: \"65c98317-00d2-4ba8-8f66-9b34d720c553\") " pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:18:51.316608 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.316472    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/65c98317-00d2-4ba8-8f66-9b34d720c553-run\") pod \"tuned-kzv9d\" (UID: \"65c98317-00d2-4ba8-8f66-9b34d720c553\") " pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:18:51.316608 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.316497    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bb36e52c-ab3f-4c51-84e0-d943187ef234-host-run-multus-certs\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:18:51.316608 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.316523    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761-kubelet-config\") pod \"global-pull-secret-syncer-4pntt\" (UID: \"4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761\") " pod="kube-system/global-pull-secret-syncer-4pntt"
Oct 31 19:18:51.316608 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.316550    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bb36e52c-ab3f-4c51-84e0-d943187ef234-multus-daemon-config\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:18:51.316608 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.316575    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-env-overrides\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:18:51.316608 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.316601    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/21dc10e8-ee61-46ae-83f7-67d1772f083c-hosts-file\") pod \"node-resolver-58f52\" (UID: \"21dc10e8-ee61-46ae-83f7-67d1772f083c\") " pod="openshift-dns/node-resolver-58f52"
Oct 31 19:18:51.316945 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.316626    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npsmn\" (UniqueName: \"kubernetes.io/projected/534f9972-b54f-4713-8326-4808fdf08386-kube-api-access-npsmn\") pod \"network-check-target-4j5p9\" (UID: \"534f9972-b54f-4713-8326-4808fdf08386\") " pod="openshift-network-diagnostics/network-check-target-4j5p9"
Oct 31 19:18:51.316945 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.316651    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-run-systemd\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:18:51.316945 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.316698    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-host-cni-netd\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:18:51.316945 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.316722    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq8nb\" (UniqueName: \"kubernetes.io/projected/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-kube-api-access-nq8nb\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:18:51.316945 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.316745    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/65c98317-00d2-4ba8-8f66-9b34d720c553-host\") pod \"tuned-kzv9d\" (UID: \"65c98317-00d2-4ba8-8f66-9b34d720c553\") " pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:18:51.316945 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.316770    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bb36e52c-ab3f-4c51-84e0-d943187ef234-multus-conf-dir\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:18:51.316945 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.316797    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bb36e52c-ab3f-4c51-84e0-d943187ef234-host-var-lib-cni-multus\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:18:51.316945 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.316824    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/944a350f-aa7e-4921-8ddc-b0001fa05086-host\") pod \"node-ca-5mwk4\" (UID: \"944a350f-aa7e-4921-8ddc-b0001fa05086\") " pod="openshift-image-registry/node-ca-5mwk4"
Oct 31 19:18:51.316945 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.316849    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bb36e52c-ab3f-4c51-84e0-d943187ef234-host-run-k8s-cni-cncf-io\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:18:51.316945 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.316878    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bb36e52c-ab3f-4c51-84e0-d943187ef234-host-run-netns\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:18:51.316945 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.316903    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-log-socket\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:18:51.317333 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.316977    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3382fc12-f376-4b15-95be-6cb371ae87bd-metrics-certs\") pod \"network-metrics-daemon-j6fmb\" (UID: \"3382fc12-f376-4b15-95be-6cb371ae87bd\") " pod="openshift-multus/network-metrics-daemon-j6fmb"
Oct 31 19:18:51.317333 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.317005    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfwph\" (UniqueName: \"kubernetes.io/projected/bb36e52c-ab3f-4c51-84e0-d943187ef234-kube-api-access-tfwph\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:18:51.317333 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.317031    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-run-openvswitch\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:18:51.317333 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.317059    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:18:51.317333 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.317088    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4220431d-fa8a-4537-ad2e-e8ba6164f63d-system-cni-dir\") pod \"multus-additional-cni-plugins-6s8ft\" (UID: \"4220431d-fa8a-4537-ad2e-e8ba6164f63d\") " pod="openshift-multus/multus-additional-cni-plugins-6s8ft"
Oct 31 19:18:51.317333 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.317113    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21dc10e8-ee61-46ae-83f7-67d1772f083c-tmp-dir\") pod \"node-resolver-58f52\" (UID: \"21dc10e8-ee61-46ae-83f7-67d1772f083c\") " pod="openshift-dns/node-resolver-58f52"
Oct 31 19:18:51.317333 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.317166    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/65c98317-00d2-4ba8-8f66-9b34d720c553-etc-kubernetes\") pod \"tuned-kzv9d\" (UID: \"65c98317-00d2-4ba8-8f66-9b34d720c553\") " pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:18:51.317333 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.317191    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/65c98317-00d2-4ba8-8f66-9b34d720c553-lib-modules\") pod \"tuned-kzv9d\" (UID: \"65c98317-00d2-4ba8-8f66-9b34d720c553\") " pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:18:51.317333 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.317215    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/65c98317-00d2-4ba8-8f66-9b34d720c553-etc-tuned\") pod \"tuned-kzv9d\" (UID: \"65c98317-00d2-4ba8-8f66-9b34d720c553\") " pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:18:51.317333 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.317245    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bb36e52c-ab3f-4c51-84e0-d943187ef234-multus-socket-dir-parent\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:18:51.317333 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.317271    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-ovn-node-metrics-cert\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:18:51.317333 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.317300    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/638d688d-d5fc-480d-8c53-08ab2f1864b1-device-dir\") pod \"aws-ebs-csi-driver-node-tmtsr\" (UID: \"638d688d-d5fc-480d-8c53-08ab2f1864b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tmtsr"
Oct 31 19:18:51.317755 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.317337    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78hn4\" (UniqueName: \"kubernetes.io/projected/65c98317-00d2-4ba8-8f66-9b34d720c553-kube-api-access-78hn4\") pod \"tuned-kzv9d\" (UID: \"65c98317-00d2-4ba8-8f66-9b34d720c553\") " pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:18:51.317755 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.317364    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/944a350f-aa7e-4921-8ddc-b0001fa05086-serviceca\") pod \"node-ca-5mwk4\" (UID: \"944a350f-aa7e-4921-8ddc-b0001fa05086\") " pod="openshift-image-registry/node-ca-5mwk4"
Oct 31 19:18:51.317755 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.317388    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bb36e52c-ab3f-4c51-84e0-d943187ef234-system-cni-dir\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:18:51.317755 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.317429    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-host-kubelet\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:18:51.317755 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.317452    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-host-run-netns\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:18:51.317755 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.317469    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-etc-openvswitch\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:18:51.317755 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.317492    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-host-cni-bin\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:18:51.317755 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.317510    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-ovnkube-script-lib\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:18:51.317755 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.317528    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4lbw\" (UniqueName: \"kubernetes.io/projected/4220431d-fa8a-4537-ad2e-e8ba6164f63d-kube-api-access-v4lbw\") pod \"multus-additional-cni-plugins-6s8ft\" (UID: \"4220431d-fa8a-4537-ad2e-e8ba6164f63d\") " pod="openshift-multus/multus-additional-cni-plugins-6s8ft"
Oct 31 19:18:51.317755 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.317543    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-run-ovn\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:18:51.317755 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.317560    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-host-run-ovn-kubernetes\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:18:51.317755 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.317577    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/638d688d-d5fc-480d-8c53-08ab2f1864b1-etc-selinux\") pod \"aws-ebs-csi-driver-node-tmtsr\" (UID: \"638d688d-d5fc-480d-8c53-08ab2f1864b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tmtsr"
Oct 31 19:18:51.317755 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.317597    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/638d688d-d5fc-480d-8c53-08ab2f1864b1-sys-fs\") pod \"aws-ebs-csi-driver-node-tmtsr\" (UID: \"638d688d-d5fc-480d-8c53-08ab2f1864b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tmtsr"
Oct 31 19:18:51.317755 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.317618    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/65c98317-00d2-4ba8-8f66-9b34d720c553-etc-systemd\") pod \"tuned-kzv9d\" (UID: \"65c98317-00d2-4ba8-8f66-9b34d720c553\") " pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:18:51.317755 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.317632    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/65c98317-00d2-4ba8-8f66-9b34d720c553-sys\") pod \"tuned-kzv9d\" (UID: \"65c98317-00d2-4ba8-8f66-9b34d720c553\") " pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:18:51.317755 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.317649    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bb36e52c-ab3f-4c51-84e0-d943187ef234-multus-cni-dir\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:18:51.317755 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.317676    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb36e52c-ab3f-4c51-84e0-d943187ef234-etc-kubernetes\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:18:51.318242 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.317692    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-host-slash\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:18:51.318242 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.317709    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4220431d-fa8a-4537-ad2e-e8ba6164f63d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6s8ft\" (UID: \"4220431d-fa8a-4537-ad2e-e8ba6164f63d\") " pod="openshift-multus/multus-additional-cni-plugins-6s8ft"
Oct 31 19:18:51.318242 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.317725    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t62j5\" (UniqueName: \"kubernetes.io/projected/3382fc12-f376-4b15-95be-6cb371ae87bd-kube-api-access-t62j5\") pod \"network-metrics-daemon-j6fmb\" (UID: \"3382fc12-f376-4b15-95be-6cb371ae87bd\") " pod="openshift-multus/network-metrics-daemon-j6fmb"
Oct 31 19:18:51.318242 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.317764    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bb36e52c-ab3f-4c51-84e0-d943187ef234-cnibin\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:18:51.318242 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.317786    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb36e52c-ab3f-4c51-84e0-d943187ef234-host-var-lib-kubelet\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:18:51.318242 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.317803    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/05ca774c-33f8-4368-a677-7c1a98b61bf7-konnectivity-ca\") pod \"konnectivity-agent-kc9mq\" (UID: \"05ca774c-33f8-4368-a677-7c1a98b61bf7\") " pod="kube-system/konnectivity-agent-kc9mq"
Oct 31 19:18:51.318242 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.317820    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-systemd-units\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:18:51.318242 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.317838    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-var-lib-openvswitch\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:18:51.318242 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.317874    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4220431d-fa8a-4537-ad2e-e8ba6164f63d-cnibin\") pod \"multus-additional-cni-plugins-6s8ft\" (UID: \"4220431d-fa8a-4537-ad2e-e8ba6164f63d\") " pod="openshift-multus/multus-additional-cni-plugins-6s8ft"
Oct 31 19:18:51.318242 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.317897    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4220431d-fa8a-4537-ad2e-e8ba6164f63d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6s8ft\" (UID: \"4220431d-fa8a-4537-ad2e-e8ba6164f63d\") " pod="openshift-multus/multus-additional-cni-plugins-6s8ft"
Oct 31 19:18:51.318242 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.317915    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/638d688d-d5fc-480d-8c53-08ab2f1864b1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tmtsr\" (UID: \"638d688d-d5fc-480d-8c53-08ab2f1864b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tmtsr"
Oct 31 19:18:51.318242 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.317933    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/739472ea-5c22-44a3-9a81-acbf0755b3b4-host-slash\") pod \"iptables-alerter-c5fnq\" (UID: \"739472ea-5c22-44a3-9a81-acbf0755b3b4\") " pod="openshift-network-operator/iptables-alerter-c5fnq"
Oct 31 19:18:51.318242 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.317950    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bb36e52c-ab3f-4c51-84e0-d943187ef234-cni-binary-copy\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:18:51.318242 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.317972    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bb36e52c-ab3f-4c51-84e0-d943187ef234-hostroot\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:18:51.318242 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.317994    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/65c98317-00d2-4ba8-8f66-9b34d720c553-etc-modprobe-d\") pod \"tuned-kzv9d\" (UID: \"65c98317-00d2-4ba8-8f66-9b34d720c553\") " pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:18:51.318242 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.318044    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/65c98317-00d2-4ba8-8f66-9b34d720c553-etc-sysconfig\") pod \"tuned-kzv9d\" (UID: \"65c98317-00d2-4ba8-8f66-9b34d720c553\") " pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:18:51.318242 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.318064    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/65c98317-00d2-4ba8-8f66-9b34d720c553-var-lib-kubelet\") pod \"tuned-kzv9d\" (UID: \"65c98317-00d2-4ba8-8f66-9b34d720c553\") " pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:18:51.318810 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.318082    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761-original-pull-secret\") pod \"global-pull-secret-syncer-4pntt\" (UID: \"4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761\") " pod="kube-system/global-pull-secret-syncer-4pntt"
Oct 31 19:18:51.318810 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.318099    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/739472ea-5c22-44a3-9a81-acbf0755b3b4-iptables-alerter-script\") pod \"iptables-alerter-c5fnq\" (UID: \"739472ea-5c22-44a3-9a81-acbf0755b3b4\") " pod="openshift-network-operator/iptables-alerter-c5fnq"
Oct 31 19:18:51.318810 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.318117    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6xws\" (UniqueName: \"kubernetes.io/projected/739472ea-5c22-44a3-9a81-acbf0755b3b4-kube-api-access-l6xws\") pod \"iptables-alerter-c5fnq\" (UID: \"739472ea-5c22-44a3-9a81-acbf0755b3b4\") " pod="openshift-network-operator/iptables-alerter-c5fnq"
Oct 31 19:18:51.318810 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.318134    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lq8r\" (UniqueName: \"kubernetes.io/projected/944a350f-aa7e-4921-8ddc-b0001fa05086-kube-api-access-5lq8r\") pod \"node-ca-5mwk4\" (UID: \"944a350f-aa7e-4921-8ddc-b0001fa05086\") " pod="openshift-image-registry/node-ca-5mwk4"
Oct 31 19:18:51.318810 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.318152    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/638d688d-d5fc-480d-8c53-08ab2f1864b1-registration-dir\") pod \"aws-ebs-csi-driver-node-tmtsr\" (UID: \"638d688d-d5fc-480d-8c53-08ab2f1864b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tmtsr"
Oct 31 19:18:51.318810 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.318176    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bb36e52c-ab3f-4c51-84e0-d943187ef234-host-var-lib-cni-bin\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:18:51.318810 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.318195    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/05ca774c-33f8-4368-a677-7c1a98b61bf7-agent-certs\") pod \"konnectivity-agent-kc9mq\" (UID: \"05ca774c-33f8-4368-a677-7c1a98b61bf7\") " pod="kube-system/konnectivity-agent-kc9mq"
Oct 31 19:18:51.318810 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.318211    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-node-log\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:18:51.318810 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.318231    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-ovnkube-config\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:18:51.318810 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.318256    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4220431d-fa8a-4537-ad2e-e8ba6164f63d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6s8ft\" (UID: \"4220431d-fa8a-4537-ad2e-e8ba6164f63d\") " pod="openshift-multus/multus-additional-cni-plugins-6s8ft"
Oct 31 19:18:51.318810 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.318576    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761-dbus\") pod \"global-pull-secret-syncer-4pntt\" (UID: \"4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761\") " pod="kube-system/global-pull-secret-syncer-4pntt"
Oct 31 19:18:51.318810 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.318685    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bb36e52c-ab3f-4c51-84e0-d943187ef234-host-run-multus-certs\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:18:51.318810 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.318737    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761-kubelet-config\") pod \"global-pull-secret-syncer-4pntt\" (UID: \"4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761\") " pod="kube-system/global-pull-secret-syncer-4pntt"
Oct 31 19:18:51.319223 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.318876    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bb36e52c-ab3f-4c51-84e0-d943187ef234-multus-conf-dir\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:18:51.319223 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.318906    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bb36e52c-ab3f-4c51-84e0-d943187ef234-host-var-lib-cni-multus\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:18:51.319223 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.318932    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/944a350f-aa7e-4921-8ddc-b0001fa05086-host\") pod \"node-ca-5mwk4\" (UID: \"944a350f-aa7e-4921-8ddc-b0001fa05086\") " pod="openshift-image-registry/node-ca-5mwk4"
Oct 31 19:18:51.319223 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.318957    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bb36e52c-ab3f-4c51-84e0-d943187ef234-host-run-k8s-cni-cncf-io\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:18:51.319223 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.318985    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bb36e52c-ab3f-4c51-84e0-d943187ef234-host-run-netns\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:18:51.319223 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.319056    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 31 19:18:51.319438 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.319266    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bb36e52c-ab3f-4c51-84e0-d943187ef234-cnibin\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:18:51.319438 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.319278    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bb36e52c-ab3f-4c51-84e0-d943187ef234-multus-socket-dir-parent\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:18:51.319438 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.319296    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb36e52c-ab3f-4c51-84e0-d943187ef234-host-var-lib-kubelet\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:18:51.319624 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.319462    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bb36e52c-ab3f-4c51-84e0-d943187ef234-system-cni-dir\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:18:51.319624 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.319550    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bb36e52c-ab3f-4c51-84e0-d943187ef234-multus-cni-dir\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:18:51.319624 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.319561    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/739472ea-5c22-44a3-9a81-acbf0755b3b4-host-slash\") pod \"iptables-alerter-c5fnq\" (UID: \"739472ea-5c22-44a3-9a81-acbf0755b3b4\") " pod="openshift-network-operator/iptables-alerter-c5fnq"
Oct 31 19:18:51.319624 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.319614    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bb36e52c-ab3f-4c51-84e0-d943187ef234-host-var-lib-cni-bin\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:18:51.319824 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:18:51.319651    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 19:18:51.319824 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:18:51.319715    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761-original-pull-secret podName:4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761 nodeName:}" failed. No retries permitted until 2026-10-31 19:18:51.819693134 +0000 UTC m=+3.030538339 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761-original-pull-secret") pod "global-pull-secret-syncer-4pntt" (UID: "4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761") : object "kube-system"/"original-pull-secret" not registered
Oct 31 19:18:51.319824 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.319712    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb36e52c-ab3f-4c51-84e0-d943187ef234-etc-kubernetes\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:18:51.319824 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.319712    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 31 19:18:51.320035 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.320000    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bb36e52c-ab3f-4c51-84e0-d943187ef234-cni-binary-copy\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:18:51.320087 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.320042    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/944a350f-aa7e-4921-8ddc-b0001fa05086-serviceca\") pod \"node-ca-5mwk4\" (UID: \"944a350f-aa7e-4921-8ddc-b0001fa05086\") " pod="openshift-image-registry/node-ca-5mwk4"
Oct 31 19:18:51.320268 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.320247    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bb36e52c-ab3f-4c51-84e0-d943187ef234-hostroot\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:18:51.320374 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.320316    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/739472ea-5c22-44a3-9a81-acbf0755b3b4-iptables-alerter-script\") pod \"iptables-alerter-c5fnq\" (UID: \"739472ea-5c22-44a3-9a81-acbf0755b3b4\") " pod="openshift-network-operator/iptables-alerter-c5fnq"
Oct 31 19:18:51.320486 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.320458    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 31 19:18:51.320486 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.320476    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 31 19:18:51.320599 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.320497    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 31 19:18:51.321737 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.321714    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 31 19:18:51.321832 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.321748    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 31 19:18:51.321922 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.321902    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 31 19:18:51.323435 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.323204    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 31 19:18:51.323435 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.323238    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 31 19:18:51.323435 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.323263    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-j7nlh\""
Oct 31 19:18:51.323435 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.323271    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-cs8fb\""
Oct 31 19:18:51.323435 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.323210    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-bbst6\""
Oct 31 19:18:51.323435 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.323294    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 31 19:18:51.323435 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.323311    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-kbgxr\""
Oct 31 19:18:51.323435 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.323346    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 31 19:18:51.323435 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.323359    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 31 19:18:51.323435 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.323390    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-zwg2q\""
Oct 31 19:18:51.323435 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.323400    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 31 19:18:51.323435 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.323423    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 31 19:18:51.324730 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.324702    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 31 19:18:51.326181 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.326004    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 31 19:18:51.326181 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.326043    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 31 19:18:51.326181 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.326138    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-rs77w\""
Oct 31 19:18:51.326181 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.326156    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 31 19:18:51.330582 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.330559    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bb36e52c-ab3f-4c51-84e0-d943187ef234-multus-daemon-config\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:18:51.350138 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.349741    2584 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 31 19:18:51.356320 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.354502    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfwph\" (UniqueName: \"kubernetes.io/projected/bb36e52c-ab3f-4c51-84e0-d943187ef234-kube-api-access-tfwph\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:18:51.356320 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.355401    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6xws\" (UniqueName: \"kubernetes.io/projected/739472ea-5c22-44a3-9a81-acbf0755b3b4-kube-api-access-l6xws\") pod \"iptables-alerter-c5fnq\" (UID: \"739472ea-5c22-44a3-9a81-acbf0755b3b4\") " pod="openshift-network-operator/iptables-alerter-c5fnq"
Oct 31 19:18:51.357056 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.357027    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lq8r\" (UniqueName: \"kubernetes.io/projected/944a350f-aa7e-4921-8ddc-b0001fa05086-kube-api-access-5lq8r\") pod \"node-ca-5mwk4\" (UID: \"944a350f-aa7e-4921-8ddc-b0001fa05086\") " pod="openshift-image-registry/node-ca-5mwk4"
Oct 31 19:18:51.361499 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.361449    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-94.us-east-2.compute.internal" event={"ID":"b8f50d0470923e3e6371360e86f8acbb","Type":"ContainerStarted","Data":"87d4f3ec9faded6c0502d00c22586498bc6b8b44e0e3db741d6c1b45e313e9c6"}
Oct 31 19:18:51.362796 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.362772    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-94.us-east-2.compute.internal" event={"ID":"96d3baf5b18698617651d9af1ffbc2b2","Type":"ContainerStarted","Data":"876a4b2c55217a62653fbcffac89b3ff4f237029ca500213a384afde189b8b85"}
Oct 31 19:18:51.418686 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.418658    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-env-overrides\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:18:51.418686 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.418691    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/21dc10e8-ee61-46ae-83f7-67d1772f083c-hosts-file\") pod \"node-resolver-58f52\" (UID: \"21dc10e8-ee61-46ae-83f7-67d1772f083c\") " pod="openshift-dns/node-resolver-58f52"
Oct 31 19:18:51.418889 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.418710    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npsmn\" (UniqueName: \"kubernetes.io/projected/534f9972-b54f-4713-8326-4808fdf08386-kube-api-access-npsmn\") pod \"network-check-target-4j5p9\" (UID: \"534f9972-b54f-4713-8326-4808fdf08386\") " pod="openshift-network-diagnostics/network-check-target-4j5p9"
Oct 31 19:18:51.418889 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.418731    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-run-systemd\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:18:51.418889 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.418753    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-host-cni-netd\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:18:51.418889 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.418811    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-host-cni-netd\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:18:51.418889 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.418815    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/21dc10e8-ee61-46ae-83f7-67d1772f083c-hosts-file\") pod \"node-resolver-58f52\" (UID: \"21dc10e8-ee61-46ae-83f7-67d1772f083c\") " pod="openshift-dns/node-resolver-58f52"
Oct 31 19:18:51.418889 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.418813    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-run-systemd\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:18:51.419192 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.418939    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq8nb\" (UniqueName: \"kubernetes.io/projected/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-kube-api-access-nq8nb\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:18:51.419192 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.418973    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/65c98317-00d2-4ba8-8f66-9b34d720c553-host\") pod \"tuned-kzv9d\" (UID: \"65c98317-00d2-4ba8-8f66-9b34d720c553\") " pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:18:51.419192 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.419004    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-log-socket\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:18:51.419192 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.419030    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3382fc12-f376-4b15-95be-6cb371ae87bd-metrics-certs\") pod \"network-metrics-daemon-j6fmb\" (UID: \"3382fc12-f376-4b15-95be-6cb371ae87bd\") " pod="openshift-multus/network-metrics-daemon-j6fmb"
Oct 31 19:18:51.419192 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.419054    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-run-openvswitch\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:18:51.419192 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.419079    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/65c98317-00d2-4ba8-8f66-9b34d720c553-host\") pod \"tuned-kzv9d\" (UID: \"65c98317-00d2-4ba8-8f66-9b34d720c553\") " pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:18:51.419192 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.419082    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:18:51.419192 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.419130    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:18:51.419192 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.419127    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4220431d-fa8a-4537-ad2e-e8ba6164f63d-system-cni-dir\") pod \"multus-additional-cni-plugins-6s8ft\" (UID: \"4220431d-fa8a-4537-ad2e-e8ba6164f63d\") " pod="openshift-multus/multus-additional-cni-plugins-6s8ft"
Oct 31 19:18:51.419192 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.419164    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21dc10e8-ee61-46ae-83f7-67d1772f083c-tmp-dir\") pod \"node-resolver-58f52\" (UID: \"21dc10e8-ee61-46ae-83f7-67d1772f083c\") " pod="openshift-dns/node-resolver-58f52"
Oct 31 19:18:51.419719 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.419229    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/65c98317-00d2-4ba8-8f66-9b34d720c553-etc-kubernetes\") pod \"tuned-kzv9d\" (UID: \"65c98317-00d2-4ba8-8f66-9b34d720c553\") " pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:18:51.419719 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.419238    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-env-overrides\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:18:51.419719 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.419250    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-log-socket\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:18:51.419719 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:18:51.419258    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 19:18:51.419719 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.419165    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4220431d-fa8a-4537-ad2e-e8ba6164f63d-system-cni-dir\") pod \"multus-additional-cni-plugins-6s8ft\" (UID: \"4220431d-fa8a-4537-ad2e-e8ba6164f63d\") " pod="openshift-multus/multus-additional-cni-plugins-6s8ft"
Oct 31 19:18:51.419719 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.419310    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-run-openvswitch\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:18:51.419719 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.419319    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/65c98317-00d2-4ba8-8f66-9b34d720c553-etc-kubernetes\") pod \"tuned-kzv9d\" (UID: \"65c98317-00d2-4ba8-8f66-9b34d720c553\") " pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:18:51.419719 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.419257    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/65c98317-00d2-4ba8-8f66-9b34d720c553-lib-modules\") pod \"tuned-kzv9d\" (UID: \"65c98317-00d2-4ba8-8f66-9b34d720c553\") " pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:18:51.419719 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:18:51.419325    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3382fc12-f376-4b15-95be-6cb371ae87bd-metrics-certs podName:3382fc12-f376-4b15-95be-6cb371ae87bd nodeName:}" failed. No retries permitted until 2026-10-31 19:18:51.919306939 +0000 UTC m=+3.130152123 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3382fc12-f376-4b15-95be-6cb371ae87bd-metrics-certs") pod "network-metrics-daemon-j6fmb" (UID: "3382fc12-f376-4b15-95be-6cb371ae87bd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 19:18:51.419719 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.419383    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/65c98317-00d2-4ba8-8f66-9b34d720c553-etc-tuned\") pod \"tuned-kzv9d\" (UID: \"65c98317-00d2-4ba8-8f66-9b34d720c553\") " pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:18:51.419719 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.419407    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/65c98317-00d2-4ba8-8f66-9b34d720c553-lib-modules\") pod \"tuned-kzv9d\" (UID: \"65c98317-00d2-4ba8-8f66-9b34d720c553\") " pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:18:51.419719 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.419415    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-ovn-node-metrics-cert\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:18:51.419719 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.419441    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/638d688d-d5fc-480d-8c53-08ab2f1864b1-device-dir\") pod \"aws-ebs-csi-driver-node-tmtsr\" (UID: \"638d688d-d5fc-480d-8c53-08ab2f1864b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tmtsr"
Oct 31 19:18:51.419719 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.419468    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78hn4\" (UniqueName: \"kubernetes.io/projected/65c98317-00d2-4ba8-8f66-9b34d720c553-kube-api-access-78hn4\") pod \"tuned-kzv9d\" (UID: \"65c98317-00d2-4ba8-8f66-9b34d720c553\") " pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:18:51.419719 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.419494    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-host-kubelet\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:18:51.419719 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.419521    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-host-run-netns\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:18:51.419719 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.419535    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/638d688d-d5fc-480d-8c53-08ab2f1864b1-device-dir\") pod \"aws-ebs-csi-driver-node-tmtsr\" (UID: \"638d688d-d5fc-480d-8c53-08ab2f1864b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tmtsr"
Oct 31 19:18:51.420614 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.419546    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-etc-openvswitch\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:18:51.420614 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.419575    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-host-cni-bin\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:18:51.420614 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.419603    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-ovnkube-script-lib\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:18:51.420614 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.419611    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21dc10e8-ee61-46ae-83f7-67d1772f083c-tmp-dir\") pod \"node-resolver-58f52\" (UID: \"21dc10e8-ee61-46ae-83f7-67d1772f083c\") " pod="openshift-dns/node-resolver-58f52"
Oct 31 19:18:51.420614 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.419629    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4lbw\" (UniqueName: \"kubernetes.io/projected/4220431d-fa8a-4537-ad2e-e8ba6164f63d-kube-api-access-v4lbw\") pod \"multus-additional-cni-plugins-6s8ft\" (UID: \"4220431d-fa8a-4537-ad2e-e8ba6164f63d\") " pod="openshift-multus/multus-additional-cni-plugins-6s8ft"
Oct 31 19:18:51.420614 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.419657    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-run-ovn\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:18:51.420614 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.419684    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-host-run-ovn-kubernetes\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:18:51.420614 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.419711    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-etc-openvswitch\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:18:51.420614 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.419714    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/638d688d-d5fc-480d-8c53-08ab2f1864b1-etc-selinux\") pod \"aws-ebs-csi-driver-node-tmtsr\" (UID: \"638d688d-d5fc-480d-8c53-08ab2f1864b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tmtsr"
Oct 31 19:18:51.420614 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.419754    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/638d688d-d5fc-480d-8c53-08ab2f1864b1-sys-fs\") pod \"aws-ebs-csi-driver-node-tmtsr\" (UID: \"638d688d-d5fc-480d-8c53-08ab2f1864b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tmtsr"
Oct 31 19:18:51.420614 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.419768    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/638d688d-d5fc-480d-8c53-08ab2f1864b1-etc-selinux\") pod \"aws-ebs-csi-driver-node-tmtsr\" (UID: \"638d688d-d5fc-480d-8c53-08ab2f1864b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tmtsr"
Oct 31 19:18:51.420614 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.419780    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/65c98317-00d2-4ba8-8f66-9b34d720c553-etc-systemd\") pod \"tuned-kzv9d\" (UID: \"65c98317-00d2-4ba8-8f66-9b34d720c553\") " pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:18:51.420614 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.419812    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-host-run-netns\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:18:51.420614 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.419820    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/65c98317-00d2-4ba8-8f66-9b34d720c553-sys\") pod \"tuned-kzv9d\" (UID: \"65c98317-00d2-4ba8-8f66-9b34d720c553\") " pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:18:51.420614 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.419854    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-host-slash\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:18:51.420614 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.419879    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4220431d-fa8a-4537-ad2e-e8ba6164f63d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6s8ft\" (UID: \"4220431d-fa8a-4537-ad2e-e8ba6164f63d\") " pod="openshift-multus/multus-additional-cni-plugins-6s8ft"
Oct 31 19:18:51.420614 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.419835    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-host-kubelet\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:18:51.421487 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.419906    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t62j5\" (UniqueName: \"kubernetes.io/projected/3382fc12-f376-4b15-95be-6cb371ae87bd-kube-api-access-t62j5\") pod \"network-metrics-daemon-j6fmb\" (UID: \"3382fc12-f376-4b15-95be-6cb371ae87bd\") " pod="openshift-multus/network-metrics-daemon-j6fmb"
Oct 31 19:18:51.421487 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.419934    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-host-cni-bin\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:18:51.421487 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.419947    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/05ca774c-33f8-4368-a677-7c1a98b61bf7-konnectivity-ca\") pod \"konnectivity-agent-kc9mq\" (UID: \"05ca774c-33f8-4368-a677-7c1a98b61bf7\") " pod="kube-system/konnectivity-agent-kc9mq"
Oct 31 19:18:51.421487 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.419968    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-systemd-units\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:18:51.421487 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.419975    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/65c98317-00d2-4ba8-8f66-9b34d720c553-sys\") pod \"tuned-kzv9d\" (UID: \"65c98317-00d2-4ba8-8f66-9b34d720c553\") " pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:18:51.421487 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.419985    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-var-lib-openvswitch\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:18:51.421487 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.420003    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4220431d-fa8a-4537-ad2e-e8ba6164f63d-cnibin\") pod \"multus-additional-cni-plugins-6s8ft\" (UID: \"4220431d-fa8a-4537-ad2e-e8ba6164f63d\") " pod="openshift-multus/multus-additional-cni-plugins-6s8ft"
Oct 31 19:18:51.421487 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.420008    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/638d688d-d5fc-480d-8c53-08ab2f1864b1-sys-fs\") pod \"aws-ebs-csi-driver-node-tmtsr\" (UID: \"638d688d-d5fc-480d-8c53-08ab2f1864b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tmtsr"
Oct 31 19:18:51.421487 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.420044    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-host-slash\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:18:51.421487 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.420021    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4220431d-fa8a-4537-ad2e-e8ba6164f63d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6s8ft\" (UID: \"4220431d-fa8a-4537-ad2e-e8ba6164f63d\") " pod="openshift-multus/multus-additional-cni-plugins-6s8ft"
Oct 31 19:18:51.421487 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.420085    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/638d688d-d5fc-480d-8c53-08ab2f1864b1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tmtsr\" (UID: \"638d688d-d5fc-480d-8c53-08ab2f1864b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tmtsr"
Oct 31 19:18:51.421487 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.420112    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/65c98317-00d2-4ba8-8f66-9b34d720c553-etc-modprobe-d\") pod \"tuned-kzv9d\" (UID: \"65c98317-00d2-4ba8-8f66-9b34d720c553\") " pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:18:51.421487 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.420140    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/65c98317-00d2-4ba8-8f66-9b34d720c553-etc-sysconfig\") pod \"tuned-kzv9d\" (UID: \"65c98317-00d2-4ba8-8f66-9b34d720c553\") " pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:18:51.421487 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.420200    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/65c98317-00d2-4ba8-8f66-9b34d720c553-var-lib-kubelet\") pod \"tuned-kzv9d\" (UID: \"65c98317-00d2-4ba8-8f66-9b34d720c553\") " pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:18:51.421487 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.420211    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4220431d-fa8a-4537-ad2e-e8ba6164f63d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6s8ft\" (UID: \"4220431d-fa8a-4537-ad2e-e8ba6164f63d\") " pod="openshift-multus/multus-additional-cni-plugins-6s8ft"
Oct 31 19:18:51.421487 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.420269    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/65c98317-00d2-4ba8-8f66-9b34d720c553-var-lib-kubelet\") pod \"tuned-kzv9d\" (UID: \"65c98317-00d2-4ba8-8f66-9b34d720c553\") " pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:18:51.421487 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.420329    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/638d688d-d5fc-480d-8c53-08ab2f1864b1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tmtsr\" (UID: \"638d688d-d5fc-480d-8c53-08ab2f1864b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tmtsr"
Oct 31 19:18:51.422446 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.420392    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-ovnkube-script-lib\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:18:51.422446 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.420471    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/65c98317-00d2-4ba8-8f66-9b34d720c553-etc-modprobe-d\") pod \"tuned-kzv9d\" (UID: \"65c98317-00d2-4ba8-8f66-9b34d720c553\") " pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:18:51.422446 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.420499    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/65c98317-00d2-4ba8-8f66-9b34d720c553-etc-systemd\") pod \"tuned-kzv9d\" (UID: \"65c98317-00d2-4ba8-8f66-9b34d720c553\") " pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:18:51.422446 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.420505    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/65c98317-00d2-4ba8-8f66-9b34d720c553-etc-sysconfig\") pod \"tuned-kzv9d\" (UID: \"65c98317-00d2-4ba8-8f66-9b34d720c553\") " pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:18:51.422446 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.420014    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-run-ovn\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:18:51.422446 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.420526    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-var-lib-openvswitch\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:18:51.422446 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.420559    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-host-run-ovn-kubernetes\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:18:51.422446 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.420562    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-systemd-units\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:18:51.422446 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.420577    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4220431d-fa8a-4537-ad2e-e8ba6164f63d-cnibin\") pod \"multus-additional-cni-plugins-6s8ft\" (UID: \"4220431d-fa8a-4537-ad2e-e8ba6164f63d\") " pod="openshift-multus/multus-additional-cni-plugins-6s8ft"
Oct 31 19:18:51.422446 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.420704    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4220431d-fa8a-4537-ad2e-e8ba6164f63d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6s8ft\" (UID: \"4220431d-fa8a-4537-ad2e-e8ba6164f63d\") " pod="openshift-multus/multus-additional-cni-plugins-6s8ft"
Oct 31 19:18:51.422446 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.420744    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/638d688d-d5fc-480d-8c53-08ab2f1864b1-registration-dir\") pod \"aws-ebs-csi-driver-node-tmtsr\" (UID: \"638d688d-d5fc-480d-8c53-08ab2f1864b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tmtsr"
Oct 31 19:18:51.422446 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.420764    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/05ca774c-33f8-4368-a677-7c1a98b61bf7-konnectivity-ca\") pod \"konnectivity-agent-kc9mq\" (UID: \"05ca774c-33f8-4368-a677-7c1a98b61bf7\") " pod="kube-system/konnectivity-agent-kc9mq"
Oct 31 19:18:51.422446 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.420766    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/05ca774c-33f8-4368-a677-7c1a98b61bf7-agent-certs\") pod \"konnectivity-agent-kc9mq\" (UID: \"05ca774c-33f8-4368-a677-7c1a98b61bf7\") " pod="kube-system/konnectivity-agent-kc9mq"
Oct 31 19:18:51.422446 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.420834    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-node-log\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:18:51.422446 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.420851    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-ovnkube-config\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:18:51.422446 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.420861    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/638d688d-d5fc-480d-8c53-08ab2f1864b1-registration-dir\") pod \"aws-ebs-csi-driver-node-tmtsr\" (UID: \"638d688d-d5fc-480d-8c53-08ab2f1864b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tmtsr"
Oct 31 19:18:51.422446 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.420869    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4220431d-fa8a-4537-ad2e-e8ba6164f63d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6s8ft\" (UID: \"4220431d-fa8a-4537-ad2e-e8ba6164f63d\") " pod="openshift-multus/multus-additional-cni-plugins-6s8ft"
Oct 31 19:18:51.423047 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.420916    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-node-log\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:18:51.423047 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.420945    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4220431d-fa8a-4537-ad2e-e8ba6164f63d-cni-binary-copy\") pod \"multus-additional-cni-plugins-6s8ft\" (UID: \"4220431d-fa8a-4537-ad2e-e8ba6164f63d\") " pod="openshift-multus/multus-additional-cni-plugins-6s8ft"
Oct 31 19:18:51.423047 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.420975    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k8mg\" (UniqueName: \"kubernetes.io/projected/21dc10e8-ee61-46ae-83f7-67d1772f083c-kube-api-access-7k8mg\") pod \"node-resolver-58f52\" (UID: \"21dc10e8-ee61-46ae-83f7-67d1772f083c\") " pod="openshift-dns/node-resolver-58f52"
Oct 31 19:18:51.423047 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.421002    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk5qd\" (UniqueName: \"kubernetes.io/projected/638d688d-d5fc-480d-8c53-08ab2f1864b1-kube-api-access-gk5qd\") pod \"aws-ebs-csi-driver-node-tmtsr\" (UID: \"638d688d-d5fc-480d-8c53-08ab2f1864b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tmtsr"
Oct 31 19:18:51.423047 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.421029    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/65c98317-00d2-4ba8-8f66-9b34d720c553-etc-sysctl-conf\") pod \"tuned-kzv9d\" (UID: \"65c98317-00d2-4ba8-8f66-9b34d720c553\") " pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:18:51.423047 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.421051    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65c98317-00d2-4ba8-8f66-9b34d720c553-tmp\") pod \"tuned-kzv9d\" (UID: \"65c98317-00d2-4ba8-8f66-9b34d720c553\") " pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:18:51.423047 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.421077    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/638d688d-d5fc-480d-8c53-08ab2f1864b1-socket-dir\") pod \"aws-ebs-csi-driver-node-tmtsr\" (UID: \"638d688d-d5fc-480d-8c53-08ab2f1864b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tmtsr"
Oct 31 19:18:51.423047 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.421102    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/65c98317-00d2-4ba8-8f66-9b34d720c553-etc-sysctl-d\") pod \"tuned-kzv9d\" (UID: \"65c98317-00d2-4ba8-8f66-9b34d720c553\") " pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:18:51.423047 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.421127    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/65c98317-00d2-4ba8-8f66-9b34d720c553-run\") pod \"tuned-kzv9d\" (UID: \"65c98317-00d2-4ba8-8f66-9b34d720c553\") " pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:18:51.423047 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.421211    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/65c98317-00d2-4ba8-8f66-9b34d720c553-run\") pod \"tuned-kzv9d\" (UID: \"65c98317-00d2-4ba8-8f66-9b34d720c553\") " pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:18:51.423047 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.421359    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/65c98317-00d2-4ba8-8f66-9b34d720c553-etc-sysctl-conf\") pod \"tuned-kzv9d\" (UID: \"65c98317-00d2-4ba8-8f66-9b34d720c553\") " pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:18:51.423047 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.421704    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/65c98317-00d2-4ba8-8f66-9b34d720c553-etc-sysctl-d\") pod \"tuned-kzv9d\" (UID: \"65c98317-00d2-4ba8-8f66-9b34d720c553\") " pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:18:51.423047 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.421931    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4220431d-fa8a-4537-ad2e-e8ba6164f63d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6s8ft\" (UID: \"4220431d-fa8a-4537-ad2e-e8ba6164f63d\") " pod="openshift-multus/multus-additional-cni-plugins-6s8ft"
Oct 31 19:18:51.423047 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.422137    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/65c98317-00d2-4ba8-8f66-9b34d720c553-etc-tuned\") pod \"tuned-kzv9d\" (UID: \"65c98317-00d2-4ba8-8f66-9b34d720c553\") " pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:18:51.423047 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.422247    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-ovnkube-config\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:18:51.423047 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.422439    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/638d688d-d5fc-480d-8c53-08ab2f1864b1-socket-dir\") pod \"aws-ebs-csi-driver-node-tmtsr\" (UID: \"638d688d-d5fc-480d-8c53-08ab2f1864b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tmtsr"
Oct 31 19:18:51.423047 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.422714    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4220431d-fa8a-4537-ad2e-e8ba6164f63d-cni-binary-copy\") pod \"multus-additional-cni-plugins-6s8ft\" (UID: \"4220431d-fa8a-4537-ad2e-e8ba6164f63d\") " pod="openshift-multus/multus-additional-cni-plugins-6s8ft"
Oct 31 19:18:51.423595 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.422934    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-ovn-node-metrics-cert\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:18:51.423595 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.423433    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/05ca774c-33f8-4368-a677-7c1a98b61bf7-agent-certs\") pod \"konnectivity-agent-kc9mq\" (UID: \"05ca774c-33f8-4368-a677-7c1a98b61bf7\") " pod="kube-system/konnectivity-agent-kc9mq"
Oct 31 19:18:51.424325 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.424304    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65c98317-00d2-4ba8-8f66-9b34d720c553-tmp\") pod \"tuned-kzv9d\" (UID: \"65c98317-00d2-4ba8-8f66-9b34d720c553\") " pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:18:51.437232 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.437209    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq8nb\" (UniqueName: \"kubernetes.io/projected/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-kube-api-access-nq8nb\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:18:51.439124 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.439086    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk5qd\" (UniqueName: \"kubernetes.io/projected/638d688d-d5fc-480d-8c53-08ab2f1864b1-kube-api-access-gk5qd\") pod \"aws-ebs-csi-driver-node-tmtsr\" (UID: \"638d688d-d5fc-480d-8c53-08ab2f1864b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tmtsr"
Oct 31 19:18:51.439781 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.439762    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t62j5\" (UniqueName: \"kubernetes.io/projected/3382fc12-f376-4b15-95be-6cb371ae87bd-kube-api-access-t62j5\") pod \"network-metrics-daemon-j6fmb\" (UID: \"3382fc12-f376-4b15-95be-6cb371ae87bd\") " pod="openshift-multus/network-metrics-daemon-j6fmb"
Oct 31 19:18:51.442918 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:18:51.442887    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 19:18:51.442918 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:18:51.442911    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 19:18:51.443088 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:18:51.442925    2584 projected.go:196] Error preparing data for projected volume kube-api-access-npsmn for pod openshift-network-diagnostics/network-check-target-4j5p9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 19:18:51.443088 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:18:51.443007    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/534f9972-b54f-4713-8326-4808fdf08386-kube-api-access-npsmn podName:534f9972-b54f-4713-8326-4808fdf08386 nodeName:}" failed. No retries permitted until 2026-10-31 19:18:51.942987577 +0000 UTC m=+3.153832758 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-npsmn" (UniqueName: "kubernetes.io/projected/534f9972-b54f-4713-8326-4808fdf08386-kube-api-access-npsmn") pod "network-check-target-4j5p9" (UID: "534f9972-b54f-4713-8326-4808fdf08386") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 19:18:51.446083 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.445968    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k8mg\" (UniqueName: \"kubernetes.io/projected/21dc10e8-ee61-46ae-83f7-67d1772f083c-kube-api-access-7k8mg\") pod \"node-resolver-58f52\" (UID: \"21dc10e8-ee61-46ae-83f7-67d1772f083c\") " pod="openshift-dns/node-resolver-58f52"
Oct 31 19:18:51.446957 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.446932    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78hn4\" (UniqueName: \"kubernetes.io/projected/65c98317-00d2-4ba8-8f66-9b34d720c553-kube-api-access-78hn4\") pod \"tuned-kzv9d\" (UID: \"65c98317-00d2-4ba8-8f66-9b34d720c553\") " pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:18:51.454621 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.454592    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4lbw\" (UniqueName: \"kubernetes.io/projected/4220431d-fa8a-4537-ad2e-e8ba6164f63d-kube-api-access-v4lbw\") pod \"multus-additional-cni-plugins-6s8ft\" (UID: \"4220431d-fa8a-4537-ad2e-e8ba6164f63d\") " pod="openshift-multus/multus-additional-cni-plugins-6s8ft"
Oct 31 19:18:51.516839 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.516808    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-c5fnq"
Oct 31 19:18:51.526404 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.526375    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5mwk4"
Oct 31 19:18:51.527141 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:51.527109    2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod739472ea_5c22_44a3_9a81_acbf0755b3b4.slice/crio-3bd73889613314ef4bb6e74fcfb8cc40e91533df36fbe8525771038110a642ba WatchSource:0}: Error finding container 3bd73889613314ef4bb6e74fcfb8cc40e91533df36fbe8525771038110a642ba: Status 404 returned error can't find the container with id 3bd73889613314ef4bb6e74fcfb8cc40e91533df36fbe8525771038110a642ba
Oct 31 19:18:51.535763 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.535608    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hhnhg"
Oct 31 19:18:51.537381 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:51.537348    2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod944a350f_aa7e_4921_8ddc_b0001fa05086.slice/crio-be6e65fc286d720ef66bba2cd6c16bc5752194068f80469cda7aff0391b4731f WatchSource:0}: Error finding container be6e65fc286d720ef66bba2cd6c16bc5752194068f80469cda7aff0391b4731f: Status 404 returned error can't find the container with id be6e65fc286d720ef66bba2cd6c16bc5752194068f80469cda7aff0391b4731f
Oct 31 19:18:51.543551 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.543507    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kc9mq"
Oct 31 19:18:51.583182 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.583156    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-58f52"
Oct 31 19:18:51.587504 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.587450    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:18:51.594565 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:51.594536    2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21dc10e8_ee61_46ae_83f7_67d1772f083c.slice/crio-5f67e14a39e17e0a90f3920596f0f7ceae2f8b34d976c64f044c50e4aa5dced4 WatchSource:0}: Error finding container 5f67e14a39e17e0a90f3920596f0f7ceae2f8b34d976c64f044c50e4aa5dced4: Status 404 returned error can't find the container with id 5f67e14a39e17e0a90f3920596f0f7ceae2f8b34d976c64f044c50e4aa5dced4
Oct 31 19:18:51.596158 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.596129    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:18:51.597061 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:51.597039    2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14026ae1_aeee_4b0e_9fd9_67bd4ed7fc0c.slice/crio-dbc6469e8e9ba3ce4167e2a94399129dab2c64360e1194f48abbd68d32cdda0e WatchSource:0}: Error finding container dbc6469e8e9ba3ce4167e2a94399129dab2c64360e1194f48abbd68d32cdda0e: Status 404 returned error can't find the container with id dbc6469e8e9ba3ce4167e2a94399129dab2c64360e1194f48abbd68d32cdda0e
Oct 31 19:18:51.603682 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.603661    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6s8ft"
Oct 31 19:18:51.605299 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.605277    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tmtsr"
Oct 31 19:18:51.608130 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:51.607844    2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65c98317_00d2_4ba8_8f66_9b34d720c553.slice/crio-7de13ea034b6dc21f3c91b283b924fa8ab6efe36fbe932067a115a92f9a0ef3a WatchSource:0}: Error finding container 7de13ea034b6dc21f3c91b283b924fa8ab6efe36fbe932067a115a92f9a0ef3a: Status 404 returned error can't find the container with id 7de13ea034b6dc21f3c91b283b924fa8ab6efe36fbe932067a115a92f9a0ef3a
Oct 31 19:18:51.619092 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:18:51.619060    2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod638d688d_d5fc_480d_8c53_08ab2f1864b1.slice/crio-997e378d63e20d073440ec716dd280bc3a55c382b061102ea561d2549410d145 WatchSource:0}: Error finding container 997e378d63e20d073440ec716dd280bc3a55c382b061102ea561d2549410d145: Status 404 returned error can't find the container with id 997e378d63e20d073440ec716dd280bc3a55c382b061102ea561d2549410d145
Oct 31 19:18:51.824547 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.824507    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761-original-pull-secret\") pod \"global-pull-secret-syncer-4pntt\" (UID: \"4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761\") " pod="kube-system/global-pull-secret-syncer-4pntt"
Oct 31 19:18:51.824728 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:18:51.824695    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 19:18:51.824794 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:18:51.824756    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761-original-pull-secret podName:4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761 nodeName:}" failed. No retries permitted until 2026-10-31 19:18:52.824740319 +0000 UTC m=+4.035585485 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761-original-pull-secret") pod "global-pull-secret-syncer-4pntt" (UID: "4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761") : object "kube-system"/"original-pull-secret" not registered
Oct 31 19:18:51.925522 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:51.925403    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3382fc12-f376-4b15-95be-6cb371ae87bd-metrics-certs\") pod \"network-metrics-daemon-j6fmb\" (UID: \"3382fc12-f376-4b15-95be-6cb371ae87bd\") " pod="openshift-multus/network-metrics-daemon-j6fmb"
Oct 31 19:18:51.925685 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:18:51.925521    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 19:18:51.925685 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:18:51.925595    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3382fc12-f376-4b15-95be-6cb371ae87bd-metrics-certs podName:3382fc12-f376-4b15-95be-6cb371ae87bd nodeName:}" failed. No retries permitted until 2026-10-31 19:18:52.925576849 +0000 UTC m=+4.136422029 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3382fc12-f376-4b15-95be-6cb371ae87bd-metrics-certs") pod "network-metrics-daemon-j6fmb" (UID: "3382fc12-f376-4b15-95be-6cb371ae87bd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 19:18:52.026297 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:52.026266    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npsmn\" (UniqueName: \"kubernetes.io/projected/534f9972-b54f-4713-8326-4808fdf08386-kube-api-access-npsmn\") pod \"network-check-target-4j5p9\" (UID: \"534f9972-b54f-4713-8326-4808fdf08386\") " pod="openshift-network-diagnostics/network-check-target-4j5p9"
Oct 31 19:18:52.027239 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:18:52.026463    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 19:18:52.027239 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:18:52.026494    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 19:18:52.027239 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:18:52.026509    2584 projected.go:196] Error preparing data for projected volume kube-api-access-npsmn for pod openshift-network-diagnostics/network-check-target-4j5p9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 19:18:52.027239 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:18:52.026580    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/534f9972-b54f-4713-8326-4808fdf08386-kube-api-access-npsmn podName:534f9972-b54f-4713-8326-4808fdf08386 nodeName:}" failed. No retries permitted until 2026-10-31 19:18:53.026558782 +0000 UTC m=+4.237403968 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-npsmn" (UniqueName: "kubernetes.io/projected/534f9972-b54f-4713-8326-4808fdf08386-kube-api-access-npsmn") pod "network-check-target-4j5p9" (UID: "534f9972-b54f-4713-8326-4808fdf08386") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 19:18:52.295912 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:52.295750    2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 19:13:52 +0000 UTC" deadline="2028-05-28 03:10:04.014161311 +0000 UTC"
Oct 31 19:18:52.295912 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:52.295853    2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13783h51m11.718312832s"
Oct 31 19:18:52.384279 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:52.384082    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tmtsr" event={"ID":"638d688d-d5fc-480d-8c53-08ab2f1864b1","Type":"ContainerStarted","Data":"997e378d63e20d073440ec716dd280bc3a55c382b061102ea561d2549410d145"}
Oct 31 19:18:52.389309 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:52.389223    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6s8ft" event={"ID":"4220431d-fa8a-4537-ad2e-e8ba6164f63d","Type":"ContainerStarted","Data":"0467ace45db27d653773586f489f0e7107046a2129fdffc6696bce6e593dcd35"}
Oct 31 19:18:52.402373 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:52.402302    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-58f52" event={"ID":"21dc10e8-ee61-46ae-83f7-67d1772f083c","Type":"ContainerStarted","Data":"5f67e14a39e17e0a90f3920596f0f7ceae2f8b34d976c64f044c50e4aa5dced4"}
Oct 31 19:18:52.408810 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:52.408778    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hhnhg" event={"ID":"bb36e52c-ab3f-4c51-84e0-d943187ef234","Type":"ContainerStarted","Data":"d906967467612a30373fee26acf17e6cf18863f31120d951efa594bee7f3b425"}
Oct 31 19:18:52.411449 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:52.411390    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kzv9d" event={"ID":"65c98317-00d2-4ba8-8f66-9b34d720c553","Type":"ContainerStarted","Data":"7de13ea034b6dc21f3c91b283b924fa8ab6efe36fbe932067a115a92f9a0ef3a"}
Oct 31 19:18:52.417820 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:52.417791    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2" event={"ID":"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c","Type":"ContainerStarted","Data":"dbc6469e8e9ba3ce4167e2a94399129dab2c64360e1194f48abbd68d32cdda0e"}
Oct 31 19:18:52.434856 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:52.434828    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kc9mq" event={"ID":"05ca774c-33f8-4368-a677-7c1a98b61bf7","Type":"ContainerStarted","Data":"eab6b838c5183cc9562553ddea45990d0887348487d9cfd8fce770b50805e1a3"}
Oct 31 19:18:52.439777 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:52.439719    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5mwk4" event={"ID":"944a350f-aa7e-4921-8ddc-b0001fa05086","Type":"ContainerStarted","Data":"be6e65fc286d720ef66bba2cd6c16bc5752194068f80469cda7aff0391b4731f"}
Oct 31 19:18:52.446704 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:52.444604    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-c5fnq" event={"ID":"739472ea-5c22-44a3-9a81-acbf0755b3b4","Type":"ContainerStarted","Data":"3bd73889613314ef4bb6e74fcfb8cc40e91533df36fbe8525771038110a642ba"}
Oct 31 19:18:52.832363 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:52.832322    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761-original-pull-secret\") pod \"global-pull-secret-syncer-4pntt\" (UID: \"4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761\") " pod="kube-system/global-pull-secret-syncer-4pntt"
Oct 31 19:18:52.832545 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:18:52.832516    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 19:18:52.832623 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:18:52.832583    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761-original-pull-secret podName:4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761 nodeName:}" failed. No retries permitted until 2026-10-31 19:18:54.832563421 +0000 UTC m=+6.043408611 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761-original-pull-secret") pod "global-pull-secret-syncer-4pntt" (UID: "4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761") : object "kube-system"/"original-pull-secret" not registered
Oct 31 19:18:52.934272 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:52.932723    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3382fc12-f376-4b15-95be-6cb371ae87bd-metrics-certs\") pod \"network-metrics-daemon-j6fmb\" (UID: \"3382fc12-f376-4b15-95be-6cb371ae87bd\") " pod="openshift-multus/network-metrics-daemon-j6fmb"
Oct 31 19:18:52.934272 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:18:52.932884    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 19:18:52.934272 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:18:52.932944    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3382fc12-f376-4b15-95be-6cb371ae87bd-metrics-certs podName:3382fc12-f376-4b15-95be-6cb371ae87bd nodeName:}" failed. No retries permitted until 2026-10-31 19:18:54.932923995 +0000 UTC m=+6.143769178 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3382fc12-f376-4b15-95be-6cb371ae87bd-metrics-certs") pod "network-metrics-daemon-j6fmb" (UID: "3382fc12-f376-4b15-95be-6cb371ae87bd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 19:18:53.034698 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:53.033636    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npsmn\" (UniqueName: \"kubernetes.io/projected/534f9972-b54f-4713-8326-4808fdf08386-kube-api-access-npsmn\") pod \"network-check-target-4j5p9\" (UID: \"534f9972-b54f-4713-8326-4808fdf08386\") " pod="openshift-network-diagnostics/network-check-target-4j5p9"
Oct 31 19:18:53.034698 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:18:53.034157    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 19:18:53.034698 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:18:53.034181    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 19:18:53.034698 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:18:53.034196    2584 projected.go:196] Error preparing data for projected volume kube-api-access-npsmn for pod openshift-network-diagnostics/network-check-target-4j5p9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 19:18:53.034698 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:18:53.034255    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/534f9972-b54f-4713-8326-4808fdf08386-kube-api-access-npsmn podName:534f9972-b54f-4713-8326-4808fdf08386 nodeName:}" failed. No retries permitted until 2026-10-31 19:18:55.034236973 +0000 UTC m=+6.245082143 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-npsmn" (UniqueName: "kubernetes.io/projected/534f9972-b54f-4713-8326-4808fdf08386-kube-api-access-npsmn") pod "network-check-target-4j5p9" (UID: "534f9972-b54f-4713-8326-4808fdf08386") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 19:18:53.296278 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:53.296105    2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 19:13:52 +0000 UTC" deadline="2028-08-15 17:02:02.104384664 +0000 UTC"
Oct 31 19:18:53.296278 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:53.296218    2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15693h43m8.80818313s"
Oct 31 19:18:53.361650 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:53.360601    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j6fmb"
Oct 31 19:18:53.361650 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:18:53.360760    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j6fmb" podUID="3382fc12-f376-4b15-95be-6cb371ae87bd"
Oct 31 19:18:53.361650 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:53.361308    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4j5p9"
Oct 31 19:18:53.361650 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:18:53.361424    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4j5p9" podUID="534f9972-b54f-4713-8326-4808fdf08386"
Oct 31 19:18:53.361650 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:53.361502    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4pntt"
Oct 31 19:18:53.361650 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:18:53.361572    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4pntt" podUID="4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761"
Oct 31 19:18:53.472530 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:53.472448    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-94.us-east-2.compute.internal" event={"ID":"b8f50d0470923e3e6371360e86f8acbb","Type":"ContainerStarted","Data":"a90e822a8c267cc26e074ae0665ecaf6e6b07b83798f3059a45a59fdfe84a75f"}
Oct 31 19:18:53.483928 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:53.483263    2584 generic.go:356] "Generic (PLEG): container finished" podID="96d3baf5b18698617651d9af1ffbc2b2" containerID="97c4bdb66b920b35f4643083a28b5d46197e4d5b237e67dc5623c1cfce419d03" exitCode=0
Oct 31 19:18:53.483928 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:53.483315    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-94.us-east-2.compute.internal" event={"ID":"96d3baf5b18698617651d9af1ffbc2b2","Type":"ContainerDied","Data":"97c4bdb66b920b35f4643083a28b5d46197e4d5b237e67dc5623c1cfce419d03"}
Oct 31 19:18:53.527697 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:53.527305    2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-94.us-east-2.compute.internal" podStartSLOduration=2.527285303 podStartE2EDuration="2.527285303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 19:18:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:18:53.499513017 +0000 UTC m=+4.710358219" watchObservedRunningTime="2026-10-31 19:18:53.527285303 +0000 UTC m=+4.738130496"
Oct 31 19:18:54.506715 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:54.506629    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-94.us-east-2.compute.internal" event={"ID":"96d3baf5b18698617651d9af1ffbc2b2","Type":"ContainerStarted","Data":"990c841d8c81060041b3326cff15450569830cd6e0c119972e652af0688e0877"}
Oct 31 19:18:54.545335 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:54.545268    2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-94.us-east-2.compute.internal" podStartSLOduration=3.5452468870000002 podStartE2EDuration="3.545246887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 19:18:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:18:54.54403505 +0000 UTC m=+5.754880243" watchObservedRunningTime="2026-10-31 19:18:54.545246887 +0000 UTC m=+5.756092079"
Oct 31 19:18:54.849064 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:54.849029    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761-original-pull-secret\") pod \"global-pull-secret-syncer-4pntt\" (UID: \"4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761\") " pod="kube-system/global-pull-secret-syncer-4pntt"
Oct 31 19:18:54.849211 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:18:54.849184    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 19:18:54.849274 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:18:54.849239    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761-original-pull-secret podName:4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761 nodeName:}" failed. No retries permitted until 2026-10-31 19:18:58.849220795 +0000 UTC m=+10.060065978 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761-original-pull-secret") pod "global-pull-secret-syncer-4pntt" (UID: "4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761") : object "kube-system"/"original-pull-secret" not registered
Oct 31 19:18:54.950535 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:54.949886    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3382fc12-f376-4b15-95be-6cb371ae87bd-metrics-certs\") pod \"network-metrics-daemon-j6fmb\" (UID: \"3382fc12-f376-4b15-95be-6cb371ae87bd\") " pod="openshift-multus/network-metrics-daemon-j6fmb"
Oct 31 19:18:54.950535 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:18:54.950071    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 19:18:54.950535 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:18:54.950134    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3382fc12-f376-4b15-95be-6cb371ae87bd-metrics-certs podName:3382fc12-f376-4b15-95be-6cb371ae87bd nodeName:}" failed. No retries permitted until 2026-10-31 19:18:58.950116039 +0000 UTC m=+10.160961208 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3382fc12-f376-4b15-95be-6cb371ae87bd-metrics-certs") pod "network-metrics-daemon-j6fmb" (UID: "3382fc12-f376-4b15-95be-6cb371ae87bd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 19:18:55.050524 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:55.050333    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npsmn\" (UniqueName: \"kubernetes.io/projected/534f9972-b54f-4713-8326-4808fdf08386-kube-api-access-npsmn\") pod \"network-check-target-4j5p9\" (UID: \"534f9972-b54f-4713-8326-4808fdf08386\") " pod="openshift-network-diagnostics/network-check-target-4j5p9"
Oct 31 19:18:55.050693 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:18:55.050567    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 19:18:55.050693 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:18:55.050594    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 19:18:55.050693 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:18:55.050612    2584 projected.go:196] Error preparing data for projected volume kube-api-access-npsmn for pod openshift-network-diagnostics/network-check-target-4j5p9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 19:18:55.050693 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:18:55.050679    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/534f9972-b54f-4713-8326-4808fdf08386-kube-api-access-npsmn podName:534f9972-b54f-4713-8326-4808fdf08386 nodeName:}" failed. No retries permitted until 2026-10-31 19:18:59.050658906 +0000 UTC m=+10.261504096 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-npsmn" (UniqueName: "kubernetes.io/projected/534f9972-b54f-4713-8326-4808fdf08386-kube-api-access-npsmn") pod "network-check-target-4j5p9" (UID: "534f9972-b54f-4713-8326-4808fdf08386") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 19:18:55.360274 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:55.360196    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4j5p9"
Oct 31 19:18:55.360274 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:55.360252    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4pntt"
Oct 31 19:18:55.360586 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:18:55.360365    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4j5p9" podUID="534f9972-b54f-4713-8326-4808fdf08386"
Oct 31 19:18:55.360586 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:18:55.360431    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4pntt" podUID="4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761"
Oct 31 19:18:55.360586 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:55.360489    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j6fmb"
Oct 31 19:18:55.360752 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:18:55.360612    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j6fmb" podUID="3382fc12-f376-4b15-95be-6cb371ae87bd"
Oct 31 19:18:55.519871 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:55.517896    2584 generic.go:356] "Generic (PLEG): container finished" podID="4220431d-fa8a-4537-ad2e-e8ba6164f63d" containerID="82a00c2cc5a8d8da2428ecfe301c8db1cceee60d38dfc907a0042f1e886cb6e1" exitCode=0
Oct 31 19:18:55.519871 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:55.519502    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6s8ft" event={"ID":"4220431d-fa8a-4537-ad2e-e8ba6164f63d","Type":"ContainerDied","Data":"82a00c2cc5a8d8da2428ecfe301c8db1cceee60d38dfc907a0042f1e886cb6e1"}
Oct 31 19:18:56.523996 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:56.523167    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kc9mq" event={"ID":"05ca774c-33f8-4368-a677-7c1a98b61bf7","Type":"ContainerStarted","Data":"d093a0213c98076b0fe1722bdc01c18756c1bfe48ac294be207e29c52f5d7b7d"}
Oct 31 19:18:57.355552 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:57.355477    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j6fmb"
Oct 31 19:18:57.355711 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:18:57.355634    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j6fmb" podUID="3382fc12-f376-4b15-95be-6cb371ae87bd"
Oct 31 19:18:57.356182 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:57.356158    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4j5p9"
Oct 31 19:18:57.356291 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:18:57.356267    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4j5p9" podUID="534f9972-b54f-4713-8326-4808fdf08386"
Oct 31 19:18:57.356547 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:57.356530    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4pntt"
Oct 31 19:18:57.356730 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:18:57.356708    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4pntt" podUID="4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761"
Oct 31 19:18:57.529302 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:57.529266    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5mwk4" event={"ID":"944a350f-aa7e-4921-8ddc-b0001fa05086","Type":"ContainerStarted","Data":"a06bd00cfdb63c304106986f20ae83ad39e0779320fe816c5a6af739dc8b59e9"}
Oct 31 19:18:57.559197 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:57.559138    2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-kc9mq" podStartSLOduration=3.623864885 podStartE2EDuration="7.559118304s" totalImagesPullingTime="3.935253419s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:18:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:18:56.549544346 +0000 UTC m=+7.760389537" watchObservedRunningTime="2026-10-31 19:18:57.559118304 +0000 UTC m=+8.769963495"
Oct 31 19:18:58.535447 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:58.535387    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tmtsr" event={"ID":"638d688d-d5fc-480d-8c53-08ab2f1864b1","Type":"ContainerStarted","Data":"c97e7e3081ebe3da9b55296a6847d9890313a75d74693dc597ac45b6da400fc9"}
Oct 31 19:18:58.891557 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:58.891475    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761-original-pull-secret\") pod \"global-pull-secret-syncer-4pntt\" (UID: \"4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761\") " pod="kube-system/global-pull-secret-syncer-4pntt"
Oct 31 19:18:58.891698 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:18:58.891672    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 19:18:58.891760 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:18:58.891739    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761-original-pull-secret podName:4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761 nodeName:}" failed. No retries permitted until 2026-10-31 19:19:06.891719969 +0000 UTC m=+18.102565154 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761-original-pull-secret") pod "global-pull-secret-syncer-4pntt" (UID: "4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761") : object "kube-system"/"original-pull-secret" not registered
Oct 31 19:18:58.992163 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:58.992129    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3382fc12-f376-4b15-95be-6cb371ae87bd-metrics-certs\") pod \"network-metrics-daemon-j6fmb\" (UID: \"3382fc12-f376-4b15-95be-6cb371ae87bd\") " pod="openshift-multus/network-metrics-daemon-j6fmb"
Oct 31 19:18:58.992339 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:18:58.992258    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 19:18:58.992339 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:18:58.992312    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3382fc12-f376-4b15-95be-6cb371ae87bd-metrics-certs podName:3382fc12-f376-4b15-95be-6cb371ae87bd nodeName:}" failed. No retries permitted until 2026-10-31 19:19:06.992300448 +0000 UTC m=+18.203145615 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3382fc12-f376-4b15-95be-6cb371ae87bd-metrics-certs") pod "network-metrics-daemon-j6fmb" (UID: "3382fc12-f376-4b15-95be-6cb371ae87bd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 19:18:59.093245 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:59.092691    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npsmn\" (UniqueName: \"kubernetes.io/projected/534f9972-b54f-4713-8326-4808fdf08386-kube-api-access-npsmn\") pod \"network-check-target-4j5p9\" (UID: \"534f9972-b54f-4713-8326-4808fdf08386\") " pod="openshift-network-diagnostics/network-check-target-4j5p9"
Oct 31 19:18:59.093245 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:18:59.092833    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 19:18:59.093245 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:18:59.092847    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 19:18:59.093245 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:18:59.092859    2584 projected.go:196] Error preparing data for projected volume kube-api-access-npsmn for pod openshift-network-diagnostics/network-check-target-4j5p9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 19:18:59.093245 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:18:59.092905    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/534f9972-b54f-4713-8326-4808fdf08386-kube-api-access-npsmn podName:534f9972-b54f-4713-8326-4808fdf08386 nodeName:}" failed. No retries permitted until 2026-10-31 19:19:07.092892799 +0000 UTC m=+18.303737978 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-npsmn" (UniqueName: "kubernetes.io/projected/534f9972-b54f-4713-8326-4808fdf08386-kube-api-access-npsmn") pod "network-check-target-4j5p9" (UID: "534f9972-b54f-4713-8326-4808fdf08386") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 19:18:59.357899 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:59.357820    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4pntt"
Oct 31 19:18:59.358059 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:18:59.357982    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4pntt" podUID="4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761"
Oct 31 19:18:59.358117 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:59.358062    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j6fmb"
Oct 31 19:18:59.358181 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:18:59.358151    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j6fmb" podUID="3382fc12-f376-4b15-95be-6cb371ae87bd"
Oct 31 19:18:59.358242 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:59.358203    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4j5p9"
Oct 31 19:18:59.358295 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:18:59.358282    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4j5p9" podUID="534f9972-b54f-4713-8326-4808fdf08386"
Oct 31 19:18:59.690192 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:59.690154    2584 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 31 19:18:59.783982 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:59.783949    2584 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-kc9mq"
Oct 31 19:18:59.784885 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:59.784866    2584 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-kc9mq"
Oct 31 19:18:59.813632 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:18:59.813568    2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-5mwk4" podStartSLOduration=4.340418749 podStartE2EDuration="9.813548481s" totalImagesPullingTime="5.473129732s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 19:18:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:18:57.559538961 +0000 UTC m=+8.770384151" watchObservedRunningTime="2026-10-31 19:18:59.813548481 +0000 UTC m=+11.024393674"
Oct 31 19:19:00.274583 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:00.273779    2584 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T19:18:59.690183527Z","UUID":"e2f31c34-d339-492e-a175-91579923676c","Handler":null,"Name":"","Endpoint":""}
Oct 31 19:19:00.275552 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:00.275525    2584 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 31 19:19:00.275649 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:00.275562    2584 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 31 19:19:00.546933 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:00.546893    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tmtsr" event={"ID":"638d688d-d5fc-480d-8c53-08ab2f1864b1","Type":"ContainerStarted","Data":"f6355dde7ec1610e91464efbce958aaccf2065b6199a83993a18a5d7e58372e6"}
Oct 31 19:19:00.549293 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:00.549263    2584 generic.go:356] "Generic (PLEG): container finished" podID="4220431d-fa8a-4537-ad2e-e8ba6164f63d" containerID="4ea3f314bdcfc93fe084b6a9373d43c3f8ceae3e660077e8d5407373f441256b" exitCode=0
Oct 31 19:19:00.549394 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:00.549335    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6s8ft" event={"ID":"4220431d-fa8a-4537-ad2e-e8ba6164f63d","Type":"ContainerDied","Data":"4ea3f314bdcfc93fe084b6a9373d43c3f8ceae3e660077e8d5407373f441256b"}
Oct 31 19:19:00.552495 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:00.552463    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-58f52" event={"ID":"21dc10e8-ee61-46ae-83f7-67d1772f083c","Type":"ContainerStarted","Data":"0f640200da48bc15e8345fdaa038fa34ab3032f4c4a2f30ccfee3d6f71cdc5db"}
Oct 31 19:19:00.555179 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:00.555156    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hhnhg" event={"ID":"bb36e52c-ab3f-4c51-84e0-d943187ef234","Type":"ContainerStarted","Data":"4ed2397f7bc3e2a5ae8d99c5ca003f95c2813123d6807782d1a8189bf48f3f3c"}
Oct 31 19:19:00.634036 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:00.633355    2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-58f52" podStartSLOduration=2.028833248 podStartE2EDuration="10.633337014s" totalImagesPullingTime="8.604503766s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 19:18:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:19:00.608659366 +0000 UTC m=+11.819504557" watchObservedRunningTime="2026-10-31 19:19:00.633337014 +0000 UTC m=+11.844182205"
Oct 31 19:19:00.634036 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:00.633776    2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-hhnhg" podStartSLOduration=2.3003290180000002 podStartE2EDuration="10.63376587s" totalImagesPullingTime="8.333436852s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 19:18:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:19:00.632549558 +0000 UTC m=+11.843394774" watchObservedRunningTime="2026-10-31 19:19:00.63376587 +0000 UTC m=+11.844611076"
Oct 31 19:19:01.354857 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:01.354471    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4j5p9"
Oct 31 19:19:01.354857 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:01.354486    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4pntt"
Oct 31 19:19:01.354857 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:19:01.354583    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4j5p9" podUID="534f9972-b54f-4713-8326-4808fdf08386"
Oct 31 19:19:01.354857 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:01.354649    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j6fmb"
Oct 31 19:19:01.354857 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:19:01.354746    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j6fmb" podUID="3382fc12-f376-4b15-95be-6cb371ae87bd"
Oct 31 19:19:01.354857 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:19:01.354814    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4pntt" podUID="4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761"
Oct 31 19:19:01.560737 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:01.560652    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tmtsr" event={"ID":"638d688d-d5fc-480d-8c53-08ab2f1864b1","Type":"ContainerStarted","Data":"01df903a1b0d80238468405b74927f6ae1f1fbe4a93ad52a03a857698492a1bb"}
Oct 31 19:19:01.563522 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:01.563418    2584 generic.go:356] "Generic (PLEG): container finished" podID="4220431d-fa8a-4537-ad2e-e8ba6164f63d" containerID="2918bc81c9d36348f497b1d302fccfa18576a074400c9dd7bfac0e1ecc5beec7" exitCode=0
Oct 31 19:19:01.563522 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:01.563496    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6s8ft" event={"ID":"4220431d-fa8a-4537-ad2e-e8ba6164f63d","Type":"ContainerDied","Data":"2918bc81c9d36348f497b1d302fccfa18576a074400c9dd7bfac0e1ecc5beec7"}
Oct 31 19:19:01.568595 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:01.568154    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-c5fnq" event={"ID":"739472ea-5c22-44a3-9a81-acbf0755b3b4","Type":"ContainerStarted","Data":"4015f4519cbfce3725e89b00294ef385edc8ce5197fb7f517174700cfe68de6d"}
Oct 31 19:19:01.584457 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:01.584409    2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tmtsr" podStartSLOduration=2.799202482 podStartE2EDuration="11.584393854s" totalImagesPullingTime="8.785191372s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 19:18:50 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:19:01.584090815 +0000 UTC m=+12.794936003" watchObservedRunningTime="2026-10-31 19:19:01.584393854 +0000 UTC m=+12.795239041"
Oct 31 19:19:01.629224 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:01.629176    2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-c5fnq" podStartSLOduration=3.1271918579999998 podStartE2EDuration="11.629161555s" totalImagesPullingTime="8.501969697s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 19:18:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:19:01.628677809 +0000 UTC m=+12.839522996" watchObservedRunningTime="2026-10-31 19:19:01.629161555 +0000 UTC m=+12.840006742"
Oct 31 19:19:02.576128 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:02.576043    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kzv9d" event={"ID":"65c98317-00d2-4ba8-8f66-9b34d720c553","Type":"ContainerStarted","Data":"fc5bfc48c3cd9db7ffde3d941db7910718fed077b904d3e315af1c082a0d0909"}
Oct 31 19:19:03.354198 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:03.354152    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4j5p9"
Oct 31 19:19:03.354198 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:03.354191    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j6fmb"
Oct 31 19:19:03.354427 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:03.354156    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4pntt"
Oct 31 19:19:03.354427 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:19:03.354277    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4j5p9" podUID="534f9972-b54f-4713-8326-4808fdf08386"
Oct 31 19:19:03.354427 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:19:03.354368    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4pntt" podUID="4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761"
Oct 31 19:19:03.354534 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:19:03.354445    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j6fmb" podUID="3382fc12-f376-4b15-95be-6cb371ae87bd"
Oct 31 19:19:03.580000 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:03.579962    2584 generic.go:356] "Generic (PLEG): container finished" podID="4220431d-fa8a-4537-ad2e-e8ba6164f63d" containerID="cf68d9b3ce3e4d8b88576a14589470a4535237dc1f5613b600a360d6c24ddd0f" exitCode=0
Oct 31 19:19:03.580520 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:03.580026    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6s8ft" event={"ID":"4220431d-fa8a-4537-ad2e-e8ba6164f63d","Type":"ContainerDied","Data":"cf68d9b3ce3e4d8b88576a14589470a4535237dc1f5613b600a360d6c24ddd0f"}
Oct 31 19:19:03.610376 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:03.610289    2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-kzv9d" podStartSLOduration=2.724142874 podStartE2EDuration="13.610277204s" totalImagesPullingTime="10.88613433s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 19:18:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:19:02.601527735 +0000 UTC m=+13.812372939" watchObservedRunningTime="2026-10-31 19:19:03.610277204 +0000 UTC m=+14.821122390"
Oct 31 19:19:05.354819 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:05.354606    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4pntt"
Oct 31 19:19:05.355410 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:05.354627    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j6fmb"
Oct 31 19:19:05.355410 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:19:05.354902    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4pntt" podUID="4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761"
Oct 31 19:19:05.355410 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:19:05.355002    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j6fmb" podUID="3382fc12-f376-4b15-95be-6cb371ae87bd"
Oct 31 19:19:05.355410 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:05.354669    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4j5p9"
Oct 31 19:19:05.355410 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:19:05.355077    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4j5p9" podUID="534f9972-b54f-4713-8326-4808fdf08386"
Oct 31 19:19:05.586936 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:05.586821    2584 generic.go:356] "Generic (PLEG): container finished" podID="4220431d-fa8a-4537-ad2e-e8ba6164f63d" containerID="7c8996024040b83c5757bdcfc7f294a59d6eb2e1caf731697c9380ee34bbd28b" exitCode=0
Oct 31 19:19:05.586936 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:05.586883    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6s8ft" event={"ID":"4220431d-fa8a-4537-ad2e-e8ba6164f63d","Type":"ContainerDied","Data":"7c8996024040b83c5757bdcfc7f294a59d6eb2e1caf731697c9380ee34bbd28b"}
Oct 31 19:19:06.594912 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:06.594875    2584 generic.go:356] "Generic (PLEG): container finished" podID="4220431d-fa8a-4537-ad2e-e8ba6164f63d" containerID="882b36257883d1591901e16418d2f12d7867606430a4be6c04b1ce31654522bd" exitCode=0
Oct 31 19:19:06.595277 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:06.594920    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6s8ft" event={"ID":"4220431d-fa8a-4537-ad2e-e8ba6164f63d","Type":"ContainerDied","Data":"882b36257883d1591901e16418d2f12d7867606430a4be6c04b1ce31654522bd"}
Oct 31 19:19:06.953140 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:06.953110    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761-original-pull-secret\") pod \"global-pull-secret-syncer-4pntt\" (UID: \"4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761\") " pod="kube-system/global-pull-secret-syncer-4pntt"
Oct 31 19:19:06.953229 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:19:06.953206    2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 19:19:06.953274 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:19:06.953250    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761-original-pull-secret podName:4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761 nodeName:}" failed. No retries permitted until 2026-10-31 19:19:22.953238688 +0000 UTC m=+34.164083865 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761-original-pull-secret") pod "global-pull-secret-syncer-4pntt" (UID: "4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761") : object "kube-system"/"original-pull-secret" not registered
Oct 31 19:19:07.054354 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:07.054315    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3382fc12-f376-4b15-95be-6cb371ae87bd-metrics-certs\") pod \"network-metrics-daemon-j6fmb\" (UID: \"3382fc12-f376-4b15-95be-6cb371ae87bd\") " pod="openshift-multus/network-metrics-daemon-j6fmb"
Oct 31 19:19:07.054447 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:19:07.054434    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 19:19:07.054496 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:19:07.054487    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3382fc12-f376-4b15-95be-6cb371ae87bd-metrics-certs podName:3382fc12-f376-4b15-95be-6cb371ae87bd nodeName:}" failed. No retries permitted until 2026-10-31 19:19:23.054472308 +0000 UTC m=+34.265317476 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3382fc12-f376-4b15-95be-6cb371ae87bd-metrics-certs") pod "network-metrics-daemon-j6fmb" (UID: "3382fc12-f376-4b15-95be-6cb371ae87bd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 19:19:07.155624 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:07.155601    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npsmn\" (UniqueName: \"kubernetes.io/projected/534f9972-b54f-4713-8326-4808fdf08386-kube-api-access-npsmn\") pod \"network-check-target-4j5p9\" (UID: \"534f9972-b54f-4713-8326-4808fdf08386\") " pod="openshift-network-diagnostics/network-check-target-4j5p9"
Oct 31 19:19:07.155736 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:19:07.155724    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 19:19:07.155790 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:19:07.155738    2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 19:19:07.155790 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:19:07.155747    2584 projected.go:196] Error preparing data for projected volume kube-api-access-npsmn for pod openshift-network-diagnostics/network-check-target-4j5p9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 19:19:07.155790 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:19:07.155785    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/534f9972-b54f-4713-8326-4808fdf08386-kube-api-access-npsmn podName:534f9972-b54f-4713-8326-4808fdf08386 nodeName:}" failed. No retries permitted until 2026-10-31 19:19:23.155774167 +0000 UTC m=+34.366619331 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-npsmn" (UniqueName: "kubernetes.io/projected/534f9972-b54f-4713-8326-4808fdf08386-kube-api-access-npsmn") pod "network-check-target-4j5p9" (UID: "534f9972-b54f-4713-8326-4808fdf08386") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 19:19:07.354284 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:07.354251    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j6fmb"
Oct 31 19:19:07.354284 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:07.354272    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4pntt"
Oct 31 19:19:07.354485 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:07.354427    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4j5p9"
Oct 31 19:19:07.354485 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:19:07.354445    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j6fmb" podUID="3382fc12-f376-4b15-95be-6cb371ae87bd"
Oct 31 19:19:07.354607 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:19:07.354525    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4j5p9" podUID="534f9972-b54f-4713-8326-4808fdf08386"
Oct 31 19:19:07.354656 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:19:07.354616    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4pntt" podUID="4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761"
Oct 31 19:19:07.600124 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:07.600036    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2" event={"ID":"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c","Type":"ContainerStarted","Data":"1162f74f0cffd0d65dc7c5ab03276ec87bb5709fdc55bf49612c23bfa3fc0fd0"}
Oct 31 19:19:07.600124 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:07.600112    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2" event={"ID":"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c","Type":"ContainerStarted","Data":"8013dec97f1f615e7193d576668f9d1ee4e32d5e724504f16c599134d2dace45"}
Oct 31 19:19:07.600656 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:07.600153    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2" event={"ID":"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c","Type":"ContainerStarted","Data":"8aea823ffd98901d23c31a5001f5261392c1d19f01891394378b1663a04565dd"}
Oct 31 19:19:07.600656 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:07.600196    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2" event={"ID":"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c","Type":"ContainerStarted","Data":"3f44b2504fd311af44b418917a7819238328e50aeca9dfabba53cff88500df0f"}
Oct 31 19:19:07.600656 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:07.600250    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2" event={"ID":"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c","Type":"ContainerStarted","Data":"c8e5996b8d00ee828919691930e352a69ad374f1aea69e8b9eb23b4f70563625"}
Oct 31 19:19:07.600656 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:07.600304    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2" event={"ID":"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c","Type":"ContainerStarted","Data":"155f1cbe6e83806405240dc28d3c188bd4f7ec772a0db3e4f99cec5821621b89"}
Oct 31 19:19:07.603296 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:07.603274    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6s8ft" event={"ID":"4220431d-fa8a-4537-ad2e-e8ba6164f63d","Type":"ContainerStarted","Data":"01ed45eada72769383e6da7350c8015750aeb406fd853afc6836726d441704a5"}
Oct 31 19:19:07.634468 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:07.634430    2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-6s8ft" podStartSLOduration=6.639700836 podStartE2EDuration="17.634418908s" totalImagesPullingTime="10.994718072s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 19:18:50 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:19:07.634199382 +0000 UTC m=+18.845044567" watchObservedRunningTime="2026-10-31 19:19:07.634418908 +0000 UTC m=+18.845264115"
Oct 31 19:19:09.355591 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:09.355556    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4pntt"
Oct 31 19:19:09.356096 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:09.355657    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4j5p9"
Oct 31 19:19:09.356096 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:19:09.355658    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4pntt" podUID="4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761"
Oct 31 19:19:09.356096 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:09.355688    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j6fmb"
Oct 31 19:19:09.356096 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:19:09.355742    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4j5p9" podUID="534f9972-b54f-4713-8326-4808fdf08386"
Oct 31 19:19:09.356096 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:19:09.355811    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j6fmb" podUID="3382fc12-f376-4b15-95be-6cb371ae87bd"
Oct 31 19:19:09.610732 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:09.610655    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2" event={"ID":"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c","Type":"ContainerStarted","Data":"ded88167dd57a95f323ab59f21061d96bccbeac41d0da80e686948ad8f80e052"}
Oct 31 19:19:11.354721 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:11.354699    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4j5p9"
Oct 31 19:19:11.355134 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:11.354728    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4pntt"
Oct 31 19:19:11.355134 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:11.354757    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j6fmb"
Oct 31 19:19:11.355134 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:19:11.354825    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4pntt" podUID="4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761"
Oct 31 19:19:11.355134 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:19:11.354909    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j6fmb" podUID="3382fc12-f376-4b15-95be-6cb371ae87bd"
Oct 31 19:19:11.355134 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:19:11.354983    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4j5p9" podUID="534f9972-b54f-4713-8326-4808fdf08386"
Oct 31 19:19:11.620570 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:11.620485    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2" event={"ID":"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c","Type":"ContainerStarted","Data":"035891247d535f3c1e815afc20d29b7e7a02eb1ef7ad4e754acd21ee8fc723a9"}
Oct 31 19:19:11.620739 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:11.620717    2584 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:19:11.621468 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:11.620898    2584 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:19:11.621468 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:11.620973    2584 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:19:11.636950 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:11.636928    2584 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:19:11.637270 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:11.637255    2584 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:19:11.661740 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:11.661704    2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2" podStartSLOduration=6.396505595 podStartE2EDuration="21.661694273s" totalImagesPullingTime="15.265188678s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 19:18:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:19:11.661118895 +0000 UTC m=+22.871964083" watchObservedRunningTime="2026-10-31 19:19:11.661694273 +0000 UTC m=+22.872539458"
Oct 31 19:19:13.076118 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:13.075891    2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4pntt"]
Oct 31 19:19:13.076618 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:13.076145    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4pntt"
Oct 31 19:19:13.076618 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:19:13.076230    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4pntt" podUID="4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761"
Oct 31 19:19:13.078747 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:13.078716    2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4j5p9"]
Oct 31 19:19:13.078872 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:13.078820    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4j5p9"
Oct 31 19:19:13.078937 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:19:13.078907    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4j5p9" podUID="534f9972-b54f-4713-8326-4808fdf08386"
Oct 31 19:19:13.079605 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:13.079582    2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-j6fmb"]
Oct 31 19:19:13.079709 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:13.079693    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j6fmb"
Oct 31 19:19:13.079824 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:19:13.079803    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j6fmb" podUID="3382fc12-f376-4b15-95be-6cb371ae87bd"
Oct 31 19:19:13.652061 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:13.652034    2584 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-kc9mq"
Oct 31 19:19:13.652244 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:13.652148    2584 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 19:19:13.652586 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:13.652572    2584 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-kc9mq"
Oct 31 19:19:13.844454 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:13.844427    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:13.999929 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:13.999864    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:14.354508 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:14.354430    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j6fmb"
Oct 31 19:19:14.354508 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:14.354466    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4pntt"
Oct 31 19:19:14.354884 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:19:14.354553    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j6fmb" podUID="3382fc12-f376-4b15-95be-6cb371ae87bd"
Oct 31 19:19:14.354884 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:19:14.354668    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4pntt" podUID="4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761"
Oct 31 19:19:15.354630 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:15.354595    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4j5p9"
Oct 31 19:19:15.355029 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:19:15.354737    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4j5p9" podUID="534f9972-b54f-4713-8326-4808fdf08386"
Oct 31 19:19:15.424518 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:15.424486    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:15.524474 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:15.524441    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:16.354543 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:16.354468    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4pntt"
Oct 31 19:19:16.354686 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:19:16.354561    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4pntt" podUID="4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761"
Oct 31 19:19:16.354686 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:16.354614    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j6fmb"
Oct 31 19:19:16.354996 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:19:16.354713    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j6fmb" podUID="3382fc12-f376-4b15-95be-6cb371ae87bd"
Oct 31 19:19:17.355004 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:17.354971    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4j5p9"
Oct 31 19:19:17.355486 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:19:17.355090    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4j5p9" podUID="534f9972-b54f-4713-8326-4808fdf08386"
Oct 31 19:19:17.453557 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:17.453524    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:17.553962 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:17.553937    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:18.198779 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:18.198714    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:18.311827 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:18.311797    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:18.354234 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:18.354211    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j6fmb"
Oct 31 19:19:18.354234 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:18.354227    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4pntt"
Oct 31 19:19:18.354418 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:19:18.354304    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-j6fmb" podUID="3382fc12-f376-4b15-95be-6cb371ae87bd"
Oct 31 19:19:18.354463 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:19:18.354428    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4pntt" podUID="4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761"
Oct 31 19:19:19.355636 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:19.355605    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4j5p9"
Oct 31 19:19:19.356022 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:19:19.355700    2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4j5p9" podUID="534f9972-b54f-4713-8326-4808fdf08386"
Oct 31 19:19:19.641230 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:19.641176    2584 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-94.us-east-2.compute.internal" event="NodeReady"
Oct 31 19:19:19.641348 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:19.641273    2584 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 31 19:19:19.698665 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:19.698638    2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-2v47p"]
Oct 31 19:19:19.701782 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:19.701761    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2v47p"
Oct 31 19:19:19.702352 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:19.702318    2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-kjlt7"]
Oct 31 19:19:19.705091 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:19.705075    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kjlt7"
Oct 31 19:19:19.709857 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:19.709838    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 31 19:19:19.710008 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:19.709992    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-mrxc5\""
Oct 31 19:19:19.710090 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:19.710054    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 31 19:19:19.710154 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:19.710138    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 31 19:19:19.714141 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:19.714122    2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2v47p"]
Oct 31 19:19:19.716751 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:19.716730    2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kjlt7"]
Oct 31 19:19:19.811689 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:19.811662    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 31 19:19:19.811824 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:19.811709    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 31 19:19:19.811824 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:19.811725    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-q7xwb\""
Oct 31 19:19:19.853118 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:19.853094    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bc386a66-cdd2-4f4b-9666-88d48e42bd33-config-volume\") pod \"dns-default-kjlt7\" (UID: \"bc386a66-cdd2-4f4b-9666-88d48e42bd33\") " pod="openshift-dns/dns-default-kjlt7"
Oct 31 19:19:19.853118 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:19.853119    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7306a726-d4c7-4ef5-b496-726e0bd488a1-cert\") pod \"ingress-canary-2v47p\" (UID: \"7306a726-d4c7-4ef5-b496-726e0bd488a1\") " pod="openshift-ingress-canary/ingress-canary-2v47p"
Oct 31 19:19:19.853296 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:19.853226    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c64b\" (UniqueName: \"kubernetes.io/projected/7306a726-d4c7-4ef5-b496-726e0bd488a1-kube-api-access-9c64b\") pod \"ingress-canary-2v47p\" (UID: \"7306a726-d4c7-4ef5-b496-726e0bd488a1\") " pod="openshift-ingress-canary/ingress-canary-2v47p"
Oct 31 19:19:19.853296 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:19.853275    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc386a66-cdd2-4f4b-9666-88d48e42bd33-metrics-tls\") pod \"dns-default-kjlt7\" (UID: \"bc386a66-cdd2-4f4b-9666-88d48e42bd33\") " pod="openshift-dns/dns-default-kjlt7"
Oct 31 19:19:19.853391 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:19.853360    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc386a66-cdd2-4f4b-9666-88d48e42bd33-tmp-dir\") pod \"dns-default-kjlt7\" (UID: \"bc386a66-cdd2-4f4b-9666-88d48e42bd33\") " pod="openshift-dns/dns-default-kjlt7"
Oct 31 19:19:19.853391 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:19.853382    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxxt6\" (UniqueName: \"kubernetes.io/projected/bc386a66-cdd2-4f4b-9666-88d48e42bd33-kube-api-access-nxxt6\") pod \"dns-default-kjlt7\" (UID: \"bc386a66-cdd2-4f4b-9666-88d48e42bd33\") " pod="openshift-dns/dns-default-kjlt7"
Oct 31 19:19:19.857636 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:19.857604    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:19.954260 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:19.954204    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bc386a66-cdd2-4f4b-9666-88d48e42bd33-config-volume\") pod \"dns-default-kjlt7\" (UID: \"bc386a66-cdd2-4f4b-9666-88d48e42bd33\") " pod="openshift-dns/dns-default-kjlt7"
Oct 31 19:19:19.954260 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:19.954228    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7306a726-d4c7-4ef5-b496-726e0bd488a1-cert\") pod \"ingress-canary-2v47p\" (UID: \"7306a726-d4c7-4ef5-b496-726e0bd488a1\") " pod="openshift-ingress-canary/ingress-canary-2v47p"
Oct 31 19:19:19.954260 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:19.954252    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c64b\" (UniqueName: \"kubernetes.io/projected/7306a726-d4c7-4ef5-b496-726e0bd488a1-kube-api-access-9c64b\") pod \"ingress-canary-2v47p\" (UID: \"7306a726-d4c7-4ef5-b496-726e0bd488a1\") " pod="openshift-ingress-canary/ingress-canary-2v47p"
Oct 31 19:19:19.954492 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:19.954275    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc386a66-cdd2-4f4b-9666-88d48e42bd33-metrics-tls\") pod \"dns-default-kjlt7\" (UID: \"bc386a66-cdd2-4f4b-9666-88d48e42bd33\") " pod="openshift-dns/dns-default-kjlt7"
Oct 31 19:19:19.954492 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:19.954295    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc386a66-cdd2-4f4b-9666-88d48e42bd33-tmp-dir\") pod \"dns-default-kjlt7\" (UID: \"bc386a66-cdd2-4f4b-9666-88d48e42bd33\") " pod="openshift-dns/dns-default-kjlt7"
Oct 31 19:19:19.954492 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:19.954309    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxxt6\" (UniqueName: \"kubernetes.io/projected/bc386a66-cdd2-4f4b-9666-88d48e42bd33-kube-api-access-nxxt6\") pod \"dns-default-kjlt7\" (UID: \"bc386a66-cdd2-4f4b-9666-88d48e42bd33\") " pod="openshift-dns/dns-default-kjlt7"
Oct 31 19:19:19.954492 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:19:19.954408    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 19:19:19.954492 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:19:19.954472    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc386a66-cdd2-4f4b-9666-88d48e42bd33-metrics-tls podName:bc386a66-cdd2-4f4b-9666-88d48e42bd33 nodeName:}" failed. No retries permitted until 2026-10-31 19:19:20.454453409 +0000 UTC m=+31.665298591 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bc386a66-cdd2-4f4b-9666-88d48e42bd33-metrics-tls") pod "dns-default-kjlt7" (UID: "bc386a66-cdd2-4f4b-9666-88d48e42bd33") : secret "dns-default-metrics-tls" not found
Oct 31 19:19:19.954711 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:19.954659    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc386a66-cdd2-4f4b-9666-88d48e42bd33-tmp-dir\") pod \"dns-default-kjlt7\" (UID: \"bc386a66-cdd2-4f4b-9666-88d48e42bd33\") " pod="openshift-dns/dns-default-kjlt7"
Oct 31 19:19:19.955066 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:19.955046    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bc386a66-cdd2-4f4b-9666-88d48e42bd33-config-volume\") pod \"dns-default-kjlt7\" (UID: \"bc386a66-cdd2-4f4b-9666-88d48e42bd33\") " pod="openshift-dns/dns-default-kjlt7"
Oct 31 19:19:19.957815 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:19.957794    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7306a726-d4c7-4ef5-b496-726e0bd488a1-cert\") pod \"ingress-canary-2v47p\" (UID: \"7306a726-d4c7-4ef5-b496-726e0bd488a1\") " pod="openshift-ingress-canary/ingress-canary-2v47p"
Oct 31 19:19:19.967628 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:19.967600    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxxt6\" (UniqueName: \"kubernetes.io/projected/bc386a66-cdd2-4f4b-9666-88d48e42bd33-kube-api-access-nxxt6\") pod \"dns-default-kjlt7\" (UID: \"bc386a66-cdd2-4f4b-9666-88d48e42bd33\") " pod="openshift-dns/dns-default-kjlt7"
Oct 31 19:19:19.967939 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:19.967924    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c64b\" (UniqueName: \"kubernetes.io/projected/7306a726-d4c7-4ef5-b496-726e0bd488a1-kube-api-access-9c64b\") pod \"ingress-canary-2v47p\" (UID: \"7306a726-d4c7-4ef5-b496-726e0bd488a1\") " pod="openshift-ingress-canary/ingress-canary-2v47p"
Oct 31 19:19:19.970050 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:19.970028    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:20.013887 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:20.013866    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2v47p"
Oct 31 19:19:20.305728 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:20.305662    2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2v47p"]
Oct 31 19:19:20.307725 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:19:20.307703    2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7306a726_d4c7_4ef5_b496_726e0bd488a1.slice/crio-3aef875c6fddb789272d58c909a8ff0173c68d56db9a79372057f2ec2ef89af0 WatchSource:0}: Error finding container 3aef875c6fddb789272d58c909a8ff0173c68d56db9a79372057f2ec2ef89af0: Status 404 returned error can't find the container with id 3aef875c6fddb789272d58c909a8ff0173c68d56db9a79372057f2ec2ef89af0
Oct 31 19:19:20.354741 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:20.354714    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j6fmb"
Oct 31 19:19:20.354895 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:20.354875    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4pntt"
Oct 31 19:19:20.367703 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:20.367682    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 19:19:20.368011 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:20.367864    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jd7hq\""
Oct 31 19:19:20.369617 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:20.369596    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:20.369838 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:20.369821    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-4hvl4\""
Oct 31 19:19:20.369838 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:20.369834    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 19:19:20.390089 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:20.390066    2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-tztj8"]
Oct 31 19:19:20.393393 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:20.393379    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tztj8"
Oct 31 19:19:20.403596 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:20.403578    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 31 19:19:20.403596 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:20.403580    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 31 19:19:20.412160 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:20.412136    2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-tztj8"]
Oct 31 19:19:20.447853 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:20.447836    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-mpqbj\""
Oct 31 19:19:20.458083 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:20.458067    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc386a66-cdd2-4f4b-9666-88d48e42bd33-metrics-tls\") pod \"dns-default-kjlt7\" (UID: \"bc386a66-cdd2-4f4b-9666-88d48e42bd33\") " pod="openshift-dns/dns-default-kjlt7"
Oct 31 19:19:20.458215 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:19:20.458202    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 19:19:20.458296 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:19:20.458287    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc386a66-cdd2-4f4b-9666-88d48e42bd33-metrics-tls podName:bc386a66-cdd2-4f4b-9666-88d48e42bd33 nodeName:}" failed. No retries permitted until 2026-10-31 19:19:21.458273439 +0000 UTC m=+32.669118602 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bc386a66-cdd2-4f4b-9666-88d48e42bd33-metrics-tls") pod "dns-default-kjlt7" (UID: "bc386a66-cdd2-4f4b-9666-88d48e42bd33") : secret "dns-default-metrics-tls" not found
Oct 31 19:19:20.520339 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:20.520301    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:20.561750 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:20.559064    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86gj5\" (UniqueName: \"kubernetes.io/projected/2ae10cd8-0183-4aca-bf20-ecb1e375e420-kube-api-access-86gj5\") pod \"migrator-66df5d496b-tztj8\" (UID: \"2ae10cd8-0183-4aca-bf20-ecb1e375e420\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tztj8"
Oct 31 19:19:20.561750 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:20.559348    2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-kmlmk"]
Oct 31 19:19:20.566082 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:20.566063    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kmlmk"
Oct 31 19:19:20.576163 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:20.576129    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 31 19:19:20.576163 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:20.576145    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mxmqk\""
Oct 31 19:19:20.576428 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:20.576412    2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-j9l2c-hmgd8"]
Oct 31 19:19:20.579833 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:20.579811    2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kmlmk"]
Oct 31 19:19:20.579953 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:20.579938    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-j9l2c-hmgd8"
Oct 31 19:19:20.591318 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:20.591299    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-zz8zc\""
Oct 31 19:19:20.596496 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:20.596461    2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-j9l2c-hmgd8"]
Oct 31 19:19:20.607820 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:20.607801    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 31 19:19:20.607892 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:20.607858    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 31 19:19:20.608097 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:20.608083    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 31 19:19:20.608164 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:20.608153    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 31 19:19:20.616830 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:20.616815    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 31 19:19:20.639939 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:20.639916    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2v47p" event={"ID":"7306a726-d4c7-4ef5-b496-726e0bd488a1","Type":"ContainerStarted","Data":"3aef875c6fddb789272d58c909a8ff0173c68d56db9a79372057f2ec2ef89af0"}
Oct 31 19:19:20.660257 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:20.660239    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a7702c04-56e6-4349-97f8-cd8c2efc76aa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kmlmk\" (UID: \"a7702c04-56e6-4349-97f8-cd8c2efc76aa\") " pod="openshift-insights/insights-runtime-extractor-kmlmk"
Oct 31 19:19:20.660353 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:20.660266    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9x4n\" (UniqueName: \"kubernetes.io/projected/a7702c04-56e6-4349-97f8-cd8c2efc76aa-kube-api-access-c9x4n\") pod \"insights-runtime-extractor-kmlmk\" (UID: \"a7702c04-56e6-4349-97f8-cd8c2efc76aa\") " pod="openshift-insights/insights-runtime-extractor-kmlmk"
Oct 31 19:19:20.660353 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:20.660283    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/611ab431-d82c-46e9-a108-4b221cc8c2cc-trusted-ca-bundle\") pod \"periodic-gathering-j9l2c-hmgd8\" (UID: \"611ab431-d82c-46e9-a108-4b221cc8c2cc\") " pod="openshift-insights/periodic-gathering-j9l2c-hmgd8"
Oct 31 19:19:20.660443 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:20.660370    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a7702c04-56e6-4349-97f8-cd8c2efc76aa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kmlmk\" (UID: \"a7702c04-56e6-4349-97f8-cd8c2efc76aa\") " pod="openshift-insights/insights-runtime-extractor-kmlmk"
Oct 31 19:19:20.660486 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:20.660463    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a7702c04-56e6-4349-97f8-cd8c2efc76aa-data-volume\") pod \"insights-runtime-extractor-kmlmk\" (UID: \"a7702c04-56e6-4349-97f8-cd8c2efc76aa\") " pod="openshift-insights/insights-runtime-extractor-kmlmk"
Oct 31 19:19:20.660521 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:20.660488    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86gj5\" (UniqueName: \"kubernetes.io/projected/2ae10cd8-0183-4aca-bf20-ecb1e375e420-kube-api-access-86gj5\") pod \"migrator-66df5d496b-tztj8\" (UID: \"2ae10cd8-0183-4aca-bf20-ecb1e375e420\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tztj8"
Oct 31 19:19:20.660521 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:20.660504    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/611ab431-d82c-46e9-a108-4b221cc8c2cc-archives-path\") pod \"periodic-gathering-j9l2c-hmgd8\" (UID: \"611ab431-d82c-46e9-a108-4b221cc8c2cc\") " pod="openshift-insights/periodic-gathering-j9l2c-hmgd8"
Oct 31 19:19:20.660521 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:20.660519    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2jxb\" (UniqueName: \"kubernetes.io/projected/611ab431-d82c-46e9-a108-4b221cc8c2cc-kube-api-access-x2jxb\") pod \"periodic-gathering-j9l2c-hmgd8\" (UID: \"611ab431-d82c-46e9-a108-4b221cc8c2cc\") " pod="openshift-insights/periodic-gathering-j9l2c-hmgd8"
Oct 31 19:19:20.660657 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:20.660535    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a7702c04-56e6-4349-97f8-cd8c2efc76aa-crio-socket\") pod \"insights-runtime-extractor-kmlmk\" (UID: \"a7702c04-56e6-4349-97f8-cd8c2efc76aa\") " pod="openshift-insights/insights-runtime-extractor-kmlmk"
Oct 31 19:19:20.660657 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:20.660584    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/611ab431-d82c-46e9-a108-4b221cc8c2cc-service-ca-bundle\") pod \"periodic-gathering-j9l2c-hmgd8\" (UID: \"611ab431-d82c-46e9-a108-4b221cc8c2cc\") " pod="openshift-insights/periodic-gathering-j9l2c-hmgd8"
Oct 31 19:19:20.675635 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:20.675604    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86gj5\" (UniqueName: \"kubernetes.io/projected/2ae10cd8-0183-4aca-bf20-ecb1e375e420-kube-api-access-86gj5\") pod \"migrator-66df5d496b-tztj8\" (UID: \"2ae10cd8-0183-4aca-bf20-ecb1e375e420\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tztj8"
Oct 31 19:19:20.681899 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:20.681871    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:20.703776 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:20.703757    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tztj8"
Oct 31 19:19:20.761788 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:20.761760    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/611ab431-d82c-46e9-a108-4b221cc8c2cc-archives-path\") pod \"periodic-gathering-j9l2c-hmgd8\" (UID: \"611ab431-d82c-46e9-a108-4b221cc8c2cc\") " pod="openshift-insights/periodic-gathering-j9l2c-hmgd8"
Oct 31 19:19:20.761959 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:20.761792    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2jxb\" (UniqueName: \"kubernetes.io/projected/611ab431-d82c-46e9-a108-4b221cc8c2cc-kube-api-access-x2jxb\") pod \"periodic-gathering-j9l2c-hmgd8\" (UID: \"611ab431-d82c-46e9-a108-4b221cc8c2cc\") " pod="openshift-insights/periodic-gathering-j9l2c-hmgd8"
Oct 31 19:19:20.761959 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:20.761809    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a7702c04-56e6-4349-97f8-cd8c2efc76aa-crio-socket\") pod \"insights-runtime-extractor-kmlmk\" (UID: \"a7702c04-56e6-4349-97f8-cd8c2efc76aa\") " pod="openshift-insights/insights-runtime-extractor-kmlmk"
Oct 31 19:19:20.761959 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:20.761938    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/611ab431-d82c-46e9-a108-4b221cc8c2cc-service-ca-bundle\") pod \"periodic-gathering-j9l2c-hmgd8\" (UID: \"611ab431-d82c-46e9-a108-4b221cc8c2cc\") " pod="openshift-insights/periodic-gathering-j9l2c-hmgd8"
Oct 31 19:19:20.762115 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:20.761990    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a7702c04-56e6-4349-97f8-cd8c2efc76aa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kmlmk\" (UID: \"a7702c04-56e6-4349-97f8-cd8c2efc76aa\") " pod="openshift-insights/insights-runtime-extractor-kmlmk"
Oct 31 19:19:20.762115 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:20.762024    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9x4n\" (UniqueName: \"kubernetes.io/projected/a7702c04-56e6-4349-97f8-cd8c2efc76aa-kube-api-access-c9x4n\") pod \"insights-runtime-extractor-kmlmk\" (UID: \"a7702c04-56e6-4349-97f8-cd8c2efc76aa\") " pod="openshift-insights/insights-runtime-extractor-kmlmk"
Oct 31 19:19:20.762115 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:20.762049    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/611ab431-d82c-46e9-a108-4b221cc8c2cc-trusted-ca-bundle\") pod \"periodic-gathering-j9l2c-hmgd8\" (UID: \"611ab431-d82c-46e9-a108-4b221cc8c2cc\") " pod="openshift-insights/periodic-gathering-j9l2c-hmgd8"
Oct 31 19:19:20.762115 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:20.762095    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a7702c04-56e6-4349-97f8-cd8c2efc76aa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kmlmk\" (UID: \"a7702c04-56e6-4349-97f8-cd8c2efc76aa\") " pod="openshift-insights/insights-runtime-extractor-kmlmk"
Oct 31 19:19:20.762305 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:20.762129    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/611ab431-d82c-46e9-a108-4b221cc8c2cc-archives-path\") pod \"periodic-gathering-j9l2c-hmgd8\" (UID: \"611ab431-d82c-46e9-a108-4b221cc8c2cc\") " pod="openshift-insights/periodic-gathering-j9l2c-hmgd8"
Oct 31 19:19:20.762305 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:19:20.762141    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 19:19:20.762305 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:20.762152    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a7702c04-56e6-4349-97f8-cd8c2efc76aa-data-volume\") pod \"insights-runtime-extractor-kmlmk\" (UID: \"a7702c04-56e6-4349-97f8-cd8c2efc76aa\") " pod="openshift-insights/insights-runtime-extractor-kmlmk"
Oct 31 19:19:20.762305 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:20.762185    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a7702c04-56e6-4349-97f8-cd8c2efc76aa-crio-socket\") pod \"insights-runtime-extractor-kmlmk\" (UID: \"a7702c04-56e6-4349-97f8-cd8c2efc76aa\") " pod="openshift-insights/insights-runtime-extractor-kmlmk"
Oct 31 19:19:20.762305 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:19:20.762235    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a7702c04-56e6-4349-97f8-cd8c2efc76aa-kube-rbac-proxy-cm podName:a7702c04-56e6-4349-97f8-cd8c2efc76aa nodeName:}" failed. No retries permitted until 2026-10-31 19:19:21.262210123 +0000 UTC m=+32.473055356 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a7702c04-56e6-4349-97f8-cd8c2efc76aa-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kmlmk" (UID: "a7702c04-56e6-4349-97f8-cd8c2efc76aa") : configmap "kube-rbac-proxy" not found
Oct 31 19:19:20.762305 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:19:20.762276    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 19:19:20.762614 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:19:20.762324    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7702c04-56e6-4349-97f8-cd8c2efc76aa-insights-runtime-extractor-tls podName:a7702c04-56e6-4349-97f8-cd8c2efc76aa nodeName:}" failed. No retries permitted until 2026-10-31 19:19:21.262309125 +0000 UTC m=+32.473154302 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a7702c04-56e6-4349-97f8-cd8c2efc76aa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kmlmk" (UID: "a7702c04-56e6-4349-97f8-cd8c2efc76aa") : secret "insights-runtime-extractor-tls" not found
Oct 31 19:19:20.762614 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:20.762552    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/611ab431-d82c-46e9-a108-4b221cc8c2cc-service-ca-bundle\") pod \"periodic-gathering-j9l2c-hmgd8\" (UID: \"611ab431-d82c-46e9-a108-4b221cc8c2cc\") " pod="openshift-insights/periodic-gathering-j9l2c-hmgd8"
Oct 31 19:19:20.762614 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:20.762561    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a7702c04-56e6-4349-97f8-cd8c2efc76aa-data-volume\") pod \"insights-runtime-extractor-kmlmk\" (UID: \"a7702c04-56e6-4349-97f8-cd8c2efc76aa\") " pod="openshift-insights/insights-runtime-extractor-kmlmk"
Oct 31 19:19:20.762799 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:20.762776    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/611ab431-d82c-46e9-a108-4b221cc8c2cc-trusted-ca-bundle\") pod \"periodic-gathering-j9l2c-hmgd8\" (UID: \"611ab431-d82c-46e9-a108-4b221cc8c2cc\") " pod="openshift-insights/periodic-gathering-j9l2c-hmgd8"
Oct 31 19:19:20.787929 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:20.787908    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9x4n\" (UniqueName: \"kubernetes.io/projected/a7702c04-56e6-4349-97f8-cd8c2efc76aa-kube-api-access-c9x4n\") pod \"insights-runtime-extractor-kmlmk\" (UID: \"a7702c04-56e6-4349-97f8-cd8c2efc76aa\") " pod="openshift-insights/insights-runtime-extractor-kmlmk"
Oct 31 19:19:20.810131 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:20.810109    2584 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-hwvnh"]
Oct 31 19:19:20.824767 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:20.824721    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-hwvnh"
Oct 31 19:19:20.825852 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:20.825833    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2jxb\" (UniqueName: \"kubernetes.io/projected/611ab431-d82c-46e9-a108-4b221cc8c2cc-kube-api-access-x2jxb\") pod \"periodic-gathering-j9l2c-hmgd8\" (UID: \"611ab431-d82c-46e9-a108-4b221cc8c2cc\") " pod="openshift-insights/periodic-gathering-j9l2c-hmgd8"
Oct 31 19:19:20.838196 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:20.838180    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 31 19:19:20.843289 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:20.843268    2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-hwvnh"]
Oct 31 19:19:20.844302 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:20.844286    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 31 19:19:20.844404 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:20.844324    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 31 19:19:20.844563 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:20.844544    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 31 19:19:20.889370 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:20.889347    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-j9l2c-hmgd8"
Oct 31 19:19:20.938359 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:20.938323    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-rvvtz\""
Oct 31 19:19:20.963844 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:20.963817    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7gq5\" (UniqueName: \"kubernetes.io/projected/f2b26a37-2d01-495c-a3f3-12e5a0bbdca3-kube-api-access-s7gq5\") pod \"service-ca-5fffc97bdf-hwvnh\" (UID: \"f2b26a37-2d01-495c-a3f3-12e5a0bbdca3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hwvnh"
Oct 31 19:19:20.963963 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:20.963893    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f2b26a37-2d01-495c-a3f3-12e5a0bbdca3-signing-key\") pod \"service-ca-5fffc97bdf-hwvnh\" (UID: \"f2b26a37-2d01-495c-a3f3-12e5a0bbdca3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hwvnh"
Oct 31 19:19:20.963963 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:20.963952    2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f2b26a37-2d01-495c-a3f3-12e5a0bbdca3-signing-cabundle\") pod \"service-ca-5fffc97bdf-hwvnh\" (UID: \"f2b26a37-2d01-495c-a3f3-12e5a0bbdca3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hwvnh"
Oct 31 19:19:20.979019 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:20.978998    2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-tztj8"]
Oct 31 19:19:20.981714 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:19:20.981675    2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ae10cd8_0183_4aca_bf20_ecb1e375e420.slice/crio-6ee2e706596091c74ca0dd6f3acb765b21462c4df84997743d4e397e6b3a3a83 WatchSource:0}: Error finding container 6ee2e706596091c74ca0dd6f3acb765b21462c4df84997743d4e397e6b3a3a83: Status 404 returned error can't find the container with id 6ee2e706596091c74ca0dd6f3acb765b21462c4df84997743d4e397e6b3a3a83
Oct 31 19:19:21.035606 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:21.035558    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:21.064594 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:21.064569    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f2b26a37-2d01-495c-a3f3-12e5a0bbdca3-signing-key\") pod \"service-ca-5fffc97bdf-hwvnh\" (UID: \"f2b26a37-2d01-495c-a3f3-12e5a0bbdca3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hwvnh"
Oct 31 19:19:21.064717 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:21.064671    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f2b26a37-2d01-495c-a3f3-12e5a0bbdca3-signing-cabundle\") pod \"service-ca-5fffc97bdf-hwvnh\" (UID: \"f2b26a37-2d01-495c-a3f3-12e5a0bbdca3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hwvnh"
Oct 31 19:19:21.064771 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:21.064761    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7gq5\" (UniqueName: \"kubernetes.io/projected/f2b26a37-2d01-495c-a3f3-12e5a0bbdca3-kube-api-access-s7gq5\") pod \"service-ca-5fffc97bdf-hwvnh\" (UID: \"f2b26a37-2d01-495c-a3f3-12e5a0bbdca3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hwvnh"
Oct 31 19:19:21.065457 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:21.065435    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f2b26a37-2d01-495c-a3f3-12e5a0bbdca3-signing-cabundle\") pod \"service-ca-5fffc97bdf-hwvnh\" (UID: \"f2b26a37-2d01-495c-a3f3-12e5a0bbdca3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hwvnh"
Oct 31 19:19:21.067542 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:21.067524    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f2b26a37-2d01-495c-a3f3-12e5a0bbdca3-signing-key\") pod \"service-ca-5fffc97bdf-hwvnh\" (UID: \"f2b26a37-2d01-495c-a3f3-12e5a0bbdca3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hwvnh"
Oct 31 19:19:21.080456 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:21.080408    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7gq5\" (UniqueName: \"kubernetes.io/projected/f2b26a37-2d01-495c-a3f3-12e5a0bbdca3-kube-api-access-s7gq5\") pod \"service-ca-5fffc97bdf-hwvnh\" (UID: \"f2b26a37-2d01-495c-a3f3-12e5a0bbdca3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hwvnh"
Oct 31 19:19:21.147161 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:21.147138    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-hwvnh"
Oct 31 19:19:21.186114 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:21.186084    2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-j9l2c-hmgd8"]
Oct 31 19:19:21.193247 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:19:21.193223    2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod611ab431_d82c_46e9_a108_4b221cc8c2cc.slice/crio-a0c01424ea5f36aa2a34e7031ba579e446014499dcfb3069f58f82930562ff2c WatchSource:0}: Error finding container a0c01424ea5f36aa2a34e7031ba579e446014499dcfb3069f58f82930562ff2c: Status 404 returned error can't find the container with id a0c01424ea5f36aa2a34e7031ba579e446014499dcfb3069f58f82930562ff2c
Oct 31 19:19:21.198404 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:21.198366    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:21.266411 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:21.266376    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a7702c04-56e6-4349-97f8-cd8c2efc76aa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kmlmk\" (UID: \"a7702c04-56e6-4349-97f8-cd8c2efc76aa\") " pod="openshift-insights/insights-runtime-extractor-kmlmk"
Oct 31 19:19:21.266524 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:21.266496    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a7702c04-56e6-4349-97f8-cd8c2efc76aa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kmlmk\" (UID: \"a7702c04-56e6-4349-97f8-cd8c2efc76aa\") " pod="openshift-insights/insights-runtime-extractor-kmlmk"
Oct 31 19:19:21.266581 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:19:21.266527    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 19:19:21.266642 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:19:21.266584    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7702c04-56e6-4349-97f8-cd8c2efc76aa-insights-runtime-extractor-tls podName:a7702c04-56e6-4349-97f8-cd8c2efc76aa nodeName:}" failed. No retries permitted until 2026-10-31 19:19:22.266566595 +0000 UTC m=+33.477411759 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a7702c04-56e6-4349-97f8-cd8c2efc76aa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kmlmk" (UID: "a7702c04-56e6-4349-97f8-cd8c2efc76aa") : secret "insights-runtime-extractor-tls" not found
Oct 31 19:19:21.266642 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:19:21.266589    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 19:19:21.266642 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:19:21.266640    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a7702c04-56e6-4349-97f8-cd8c2efc76aa-kube-rbac-proxy-cm podName:a7702c04-56e6-4349-97f8-cd8c2efc76aa nodeName:}" failed. No retries permitted until 2026-10-31 19:19:22.26662556 +0000 UTC m=+33.477470741 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a7702c04-56e6-4349-97f8-cd8c2efc76aa-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kmlmk" (UID: "a7702c04-56e6-4349-97f8-cd8c2efc76aa") : configmap "kube-rbac-proxy" not found
Oct 31 19:19:21.358347 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:21.358271    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4j5p9"
Oct 31 19:19:21.364554 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:21.364518    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 19:19:21.364679 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:21.364580    2584 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 19:19:21.365377 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:21.365023    2584 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-nngmt\""
Oct 31 19:19:21.468579 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:21.468547    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc386a66-cdd2-4f4b-9666-88d48e42bd33-metrics-tls\") pod \"dns-default-kjlt7\" (UID: \"bc386a66-cdd2-4f4b-9666-88d48e42bd33\") " pod="openshift-dns/dns-default-kjlt7"
Oct 31 19:19:21.468947 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:19:21.468742    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 19:19:21.468947 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:19:21.468817    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc386a66-cdd2-4f4b-9666-88d48e42bd33-metrics-tls podName:bc386a66-cdd2-4f4b-9666-88d48e42bd33 nodeName:}" failed. No retries permitted until 2026-10-31 19:19:23.468794804 +0000 UTC m=+34.679639987 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bc386a66-cdd2-4f4b-9666-88d48e42bd33-metrics-tls") pod "dns-default-kjlt7" (UID: "bc386a66-cdd2-4f4b-9666-88d48e42bd33") : secret "dns-default-metrics-tls" not found
Oct 31 19:19:21.474934 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:21.474837    2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-hwvnh"]
Oct 31 19:19:21.477122 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:19:21.477098    2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2b26a37_2d01_495c_a3f3_12e5a0bbdca3.slice/crio-a2c94b5cfad7d3b7908931167a8f363624b774ce25a5f0926c51d913febcd9bc WatchSource:0}: Error finding container a2c94b5cfad7d3b7908931167a8f363624b774ce25a5f0926c51d913febcd9bc: Status 404 returned error can't find the container with id a2c94b5cfad7d3b7908931167a8f363624b774ce25a5f0926c51d913febcd9bc
Oct 31 19:19:21.643688 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:21.643611    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-j9l2c-hmgd8" event={"ID":"611ab431-d82c-46e9-a108-4b221cc8c2cc","Type":"ContainerStarted","Data":"a0c01424ea5f36aa2a34e7031ba579e446014499dcfb3069f58f82930562ff2c"}
Oct 31 19:19:21.644980 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:21.644934    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-hwvnh" event={"ID":"f2b26a37-2d01-495c-a3f3-12e5a0bbdca3","Type":"ContainerStarted","Data":"a2c94b5cfad7d3b7908931167a8f363624b774ce25a5f0926c51d913febcd9bc"}
Oct 31 19:19:21.646146 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:21.646124    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tztj8" event={"ID":"2ae10cd8-0183-4aca-bf20-ecb1e375e420","Type":"ContainerStarted","Data":"6ee2e706596091c74ca0dd6f3acb765b21462c4df84997743d4e397e6b3a3a83"}
Oct 31 19:19:22.017503 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:22.017406    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:22.278312 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:22.277380    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a7702c04-56e6-4349-97f8-cd8c2efc76aa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kmlmk\" (UID: \"a7702c04-56e6-4349-97f8-cd8c2efc76aa\") " pod="openshift-insights/insights-runtime-extractor-kmlmk"
Oct 31 19:19:22.278312 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:22.277462    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a7702c04-56e6-4349-97f8-cd8c2efc76aa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kmlmk\" (UID: \"a7702c04-56e6-4349-97f8-cd8c2efc76aa\") " pod="openshift-insights/insights-runtime-extractor-kmlmk"
Oct 31 19:19:22.278312 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:19:22.277672    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 19:19:22.278312 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:19:22.277746    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7702c04-56e6-4349-97f8-cd8c2efc76aa-insights-runtime-extractor-tls podName:a7702c04-56e6-4349-97f8-cd8c2efc76aa nodeName:}" failed. No retries permitted until 2026-10-31 19:19:24.277726472 +0000 UTC m=+35.488571641 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a7702c04-56e6-4349-97f8-cd8c2efc76aa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kmlmk" (UID: "a7702c04-56e6-4349-97f8-cd8c2efc76aa") : secret "insights-runtime-extractor-tls" not found
Oct 31 19:19:22.278312 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:19:22.278176    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 19:19:22.278312 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:19:22.278228    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a7702c04-56e6-4349-97f8-cd8c2efc76aa-kube-rbac-proxy-cm podName:a7702c04-56e6-4349-97f8-cd8c2efc76aa nodeName:}" failed. No retries permitted until 2026-10-31 19:19:24.278212354 +0000 UTC m=+35.489057522 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a7702c04-56e6-4349-97f8-cd8c2efc76aa-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kmlmk" (UID: "a7702c04-56e6-4349-97f8-cd8c2efc76aa") : configmap "kube-rbac-proxy" not found
Oct 31 19:19:22.983400 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:22.983374    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761-original-pull-secret\") pod \"global-pull-secret-syncer-4pntt\" (UID: \"4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761\") " pod="kube-system/global-pull-secret-syncer-4pntt"
Oct 31 19:19:22.986771 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:22.986748    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761-original-pull-secret\") pod \"global-pull-secret-syncer-4pntt\" (UID: \"4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761\") " pod="kube-system/global-pull-secret-syncer-4pntt"
Oct 31 19:19:23.071521 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:23.071447    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4pntt"
Oct 31 19:19:23.084681 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:23.084652    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3382fc12-f376-4b15-95be-6cb371ae87bd-metrics-certs\") pod \"network-metrics-daemon-j6fmb\" (UID: \"3382fc12-f376-4b15-95be-6cb371ae87bd\") " pod="openshift-multus/network-metrics-daemon-j6fmb"
Oct 31 19:19:23.084809 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:19:23.084794    2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 31 19:19:23.084873 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:19:23.084860    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3382fc12-f376-4b15-95be-6cb371ae87bd-metrics-certs podName:3382fc12-f376-4b15-95be-6cb371ae87bd nodeName:}" failed. No retries permitted until 2026-10-31 19:19:55.084842183 +0000 UTC m=+66.295687360 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3382fc12-f376-4b15-95be-6cb371ae87bd-metrics-certs") pod "network-metrics-daemon-j6fmb" (UID: "3382fc12-f376-4b15-95be-6cb371ae87bd") : secret "metrics-daemon-secret" not found
Oct 31 19:19:23.185996 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:23.185957    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npsmn\" (UniqueName: \"kubernetes.io/projected/534f9972-b54f-4713-8326-4808fdf08386-kube-api-access-npsmn\") pod \"network-check-target-4j5p9\" (UID: \"534f9972-b54f-4713-8326-4808fdf08386\") " pod="openshift-network-diagnostics/network-check-target-4j5p9"
Oct 31 19:19:23.189579 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:23.189546    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npsmn\" (UniqueName: \"kubernetes.io/projected/534f9972-b54f-4713-8326-4808fdf08386-kube-api-access-npsmn\") pod \"network-check-target-4j5p9\" (UID: \"534f9972-b54f-4713-8326-4808fdf08386\") " pod="openshift-network-diagnostics/network-check-target-4j5p9"
Oct 31 19:19:23.198141 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:23.198117    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4j5p9"
Oct 31 19:19:23.368110 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:23.368087    2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4pntt"]
Oct 31 19:19:23.488899 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:23.488871    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc386a66-cdd2-4f4b-9666-88d48e42bd33-metrics-tls\") pod \"dns-default-kjlt7\" (UID: \"bc386a66-cdd2-4f4b-9666-88d48e42bd33\") " pod="openshift-dns/dns-default-kjlt7"
Oct 31 19:19:23.489030 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:19:23.488991    2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 19:19:23.489075 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:19:23.489066    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc386a66-cdd2-4f4b-9666-88d48e42bd33-metrics-tls podName:bc386a66-cdd2-4f4b-9666-88d48e42bd33 nodeName:}" failed. No retries permitted until 2026-10-31 19:19:27.489048023 +0000 UTC m=+38.699893206 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bc386a66-cdd2-4f4b-9666-88d48e42bd33-metrics-tls") pod "dns-default-kjlt7" (UID: "bc386a66-cdd2-4f4b-9666-88d48e42bd33") : secret "dns-default-metrics-tls" not found
Oct 31 19:19:23.493485 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:23.493464    2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4j5p9"]
Oct 31 19:19:23.533971 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:19:23.533947    2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d2c9df4_c306_4f0b_b6a0_a4fafe9cc761.slice/crio-8a472eca945753737a9ab1ccdfc4f52f8c062ed3e92f413f175e77fc0861e6c9 WatchSource:0}: Error finding container 8a472eca945753737a9ab1ccdfc4f52f8c062ed3e92f413f175e77fc0861e6c9: Status 404 returned error can't find the container with id 8a472eca945753737a9ab1ccdfc4f52f8c062ed3e92f413f175e77fc0861e6c9
Oct 31 19:19:23.534405 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:19:23.534379    2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod534f9972_b54f_4713_8326_4808fdf08386.slice/crio-1bf8152bac50cc170567867a03d791329af8baa72a02537fbec3507f80ade260 WatchSource:0}: Error finding container 1bf8152bac50cc170567867a03d791329af8baa72a02537fbec3507f80ade260: Status 404 returned error can't find the container with id 1bf8152bac50cc170567867a03d791329af8baa72a02537fbec3507f80ade260
Oct 31 19:19:23.653735 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:23.653697    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tztj8" event={"ID":"2ae10cd8-0183-4aca-bf20-ecb1e375e420","Type":"ContainerStarted","Data":"b5a0ef551a912ebfffea3aa2f500a06a00de4124e1c3b3ce602f911321a98a93"}
Oct 31 19:19:23.655166 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:23.655130    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4pntt" event={"ID":"4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761","Type":"ContainerStarted","Data":"8a472eca945753737a9ab1ccdfc4f52f8c062ed3e92f413f175e77fc0861e6c9"}
Oct 31 19:19:23.656857 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:23.656806    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2v47p" event={"ID":"7306a726-d4c7-4ef5-b496-726e0bd488a1","Type":"ContainerStarted","Data":"ab9b20d3f08826415deee1270da00cdeb4b3ca8be3ec74a9b8eb1e1bba8a1944"}
Oct 31 19:19:23.657936 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:23.657898    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4j5p9" event={"ID":"534f9972-b54f-4713-8326-4808fdf08386","Type":"ContainerStarted","Data":"1bf8152bac50cc170567867a03d791329af8baa72a02537fbec3507f80ade260"}
Oct 31 19:19:23.676200 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:23.676148    2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tztj8" podStartSLOduration=1.788915319 podStartE2EDuration="3.676132531s" totalImagesPullingTime="1.887217212s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:19:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:19:23.67361201 +0000 UTC m=+34.884457197" watchObservedRunningTime="2026-10-31 19:19:23.676132531 +0000 UTC m=+34.886977736"
Oct 31 19:19:23.694873 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:23.694830    2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-2v47p" podStartSLOduration=1.443738422 podStartE2EDuration="4.694818712s" totalImagesPullingTime="3.25108029s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:19:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:19:23.693637471 +0000 UTC m=+34.904482658" watchObservedRunningTime="2026-10-31 19:19:23.694818712 +0000 UTC m=+34.905663901"
Oct 31 19:19:24.299512 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:24.299481    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a7702c04-56e6-4349-97f8-cd8c2efc76aa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kmlmk\" (UID: \"a7702c04-56e6-4349-97f8-cd8c2efc76aa\") " pod="openshift-insights/insights-runtime-extractor-kmlmk"
Oct 31 19:19:24.300202 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:24.299544    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a7702c04-56e6-4349-97f8-cd8c2efc76aa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kmlmk\" (UID: \"a7702c04-56e6-4349-97f8-cd8c2efc76aa\") " pod="openshift-insights/insights-runtime-extractor-kmlmk"
Oct 31 19:19:24.300202 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:19:24.299666    2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 19:19:24.300202 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:19:24.299731    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7702c04-56e6-4349-97f8-cd8c2efc76aa-insights-runtime-extractor-tls podName:a7702c04-56e6-4349-97f8-cd8c2efc76aa nodeName:}" failed. No retries permitted until 2026-10-31 19:19:28.299713644 +0000 UTC m=+39.510558829 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a7702c04-56e6-4349-97f8-cd8c2efc76aa-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kmlmk" (UID: "a7702c04-56e6-4349-97f8-cd8c2efc76aa") : secret "insights-runtime-extractor-tls" not found
Oct 31 19:19:24.300202 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:19:24.299862    2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 19:19:24.300202 ip-10-0-131-94 kubenswrapper[2584]: E1031 19:19:24.299926    2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a7702c04-56e6-4349-97f8-cd8c2efc76aa-kube-rbac-proxy-cm podName:a7702c04-56e6-4349-97f8-cd8c2efc76aa nodeName:}" failed. No retries permitted until 2026-10-31 19:19:28.299908858 +0000 UTC m=+39.510754046 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a7702c04-56e6-4349-97f8-cd8c2efc76aa-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kmlmk" (UID: "a7702c04-56e6-4349-97f8-cd8c2efc76aa") : configmap "kube-rbac-proxy" not found
Oct 31 19:19:24.535566 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:24.535474    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:25.665541 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:25.665502    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-j9l2c-hmgd8" event={"ID":"611ab431-d82c-46e9-a108-4b221cc8c2cc","Type":"ContainerStarted","Data":"db1b7118268ba3382a50788fbacb8d19aa0440e231841f2ad13fe0d90b1e3f47"}
Oct 31 19:19:25.667387 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:25.667353    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-hwvnh" event={"ID":"f2b26a37-2d01-495c-a3f3-12e5a0bbdca3","Type":"ContainerStarted","Data":"899077f1c8c0fc64e355175be946f1d111a48cb59449e06ffb665be22f30f27c"}
Oct 31 19:19:25.687614 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:25.687570    2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-j9l2c-hmgd8" podStartSLOduration=1.908607862 podStartE2EDuration="5.687555157s" totalImagesPullingTime="3.778947295s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:19:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:19:25.686003315 +0000 UTC m=+36.896848500" watchObservedRunningTime="2026-10-31 19:19:25.687555157 +0000 UTC m=+36.898400344"
Oct 31 19:19:25.760159 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:25.760110    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:26.025151 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:26.025034    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:26.197021 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:26.196935    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:26.351287 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:26.351199    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:26.551162 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:26.551126    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:26.659815 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:26.659738    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:27.030674 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:27.030509    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:27.532015 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:27.531990    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc386a66-cdd2-4f4b-9666-88d48e42bd33-metrics-tls\") pod \"dns-default-kjlt7\" (UID: \"bc386a66-cdd2-4f4b-9666-88d48e42bd33\") " pod="openshift-dns/dns-default-kjlt7"
Oct 31 19:19:27.534400 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:27.534381    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc386a66-cdd2-4f4b-9666-88d48e42bd33-metrics-tls\") pod \"dns-default-kjlt7\" (UID: \"bc386a66-cdd2-4f4b-9666-88d48e42bd33\") " pod="openshift-dns/dns-default-kjlt7"
Oct 31 19:19:27.631519 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:27.631464    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:27.674611 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:27.674549    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4j5p9" event={"ID":"534f9972-b54f-4713-8326-4808fdf08386","Type":"ContainerStarted","Data":"bd83c2c629818cb54cc5c8de7363c674a52a54b3463cd4e595a4dacdc5fd7a01"}
Oct 31 19:19:27.674739 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:27.674699    2584 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-4j5p9"
Oct 31 19:19:27.703658 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:27.703608    2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-hwvnh" podStartSLOduration=4.214680632 podStartE2EDuration="7.703593328s" totalImagesPullingTime="3.488912696s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:19:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:19:25.715618255 +0000 UTC m=+36.926463438" watchObservedRunningTime="2026-10-31 19:19:27.703593328 +0000 UTC m=+38.914438515"
Oct 31 19:19:27.703941 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:27.703916    2584 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-4j5p9" podStartSLOduration=33.850123758 podStartE2EDuration="37.703912s" totalImagesPullingTime="3.853788242s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:18:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:19:27.702562984 +0000 UTC m=+38.913408170" watchObservedRunningTime="2026-10-31 19:19:27.703912 +0000 UTC m=+38.914757187"
Oct 31 19:19:27.814155 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:27.814113    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:27.823102 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:27.822889    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kjlt7"
Oct 31 19:19:27.998803 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:27.998733    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-58f52_21dc10e8-ee61-46ae-83f7-67d1772f083c/dns-node-resolver/0.log"
Oct 31 19:19:28.117321 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:28.117297    2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kjlt7"]
Oct 31 19:19:28.122880 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:19:28.122438    2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc386a66_cdd2_4f4b_9666_88d48e42bd33.slice/crio-b393b80365b08f5f32594138538a31bd0c471050a98f165fc897dec2a6813381 WatchSource:0}: Error finding container b393b80365b08f5f32594138538a31bd0c471050a98f165fc897dec2a6813381: Status 404 returned error can't find the container with id b393b80365b08f5f32594138538a31bd0c471050a98f165fc897dec2a6813381
Oct 31 19:19:28.215195 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:28.215107    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:28.338353 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:28.338252    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a7702c04-56e6-4349-97f8-cd8c2efc76aa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kmlmk\" (UID: \"a7702c04-56e6-4349-97f8-cd8c2efc76aa\") " pod="openshift-insights/insights-runtime-extractor-kmlmk"
Oct 31 19:19:28.338353 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:28.338311    2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a7702c04-56e6-4349-97f8-cd8c2efc76aa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kmlmk\" (UID: \"a7702c04-56e6-4349-97f8-cd8c2efc76aa\") " pod="openshift-insights/insights-runtime-extractor-kmlmk"
Oct 31 19:19:28.338928 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:28.338900    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a7702c04-56e6-4349-97f8-cd8c2efc76aa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kmlmk\" (UID: \"a7702c04-56e6-4349-97f8-cd8c2efc76aa\") " pod="openshift-insights/insights-runtime-extractor-kmlmk"
Oct 31 19:19:28.340906 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:28.340883    2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a7702c04-56e6-4349-97f8-cd8c2efc76aa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kmlmk\" (UID: \"a7702c04-56e6-4349-97f8-cd8c2efc76aa\") " pod="openshift-insights/insights-runtime-extractor-kmlmk"
Oct 31 19:19:28.382822 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:28.382799    2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kmlmk"
Oct 31 19:19:28.413485 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:28.413454    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:28.679162 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:28.679087    2584 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kjlt7" event={"ID":"bc386a66-cdd2-4f4b-9666-88d48e42bd33","Type":"ContainerStarted","Data":"b393b80365b08f5f32594138538a31bd0c471050a98f165fc897dec2a6813381"}
Oct 31 19:19:28.691743 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:28.691718    2584 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kmlmk"]
Oct 31 19:19:28.704240 ip-10-0-131-94 kubenswrapper[2584]: W1031 19:19:28.704216    2584 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7702c04_56e6_4349_97f8_cd8c2efc76aa.slice/crio-0cc6092944f73c250c7507d1bf75f232ddcce10e190635a046ef1b4c3a94c244 WatchSource:0}: Error finding container 0cc6092944f73c250c7507d1bf75f232ddcce10e190635a046ef1b4c3a94c244: Status 404 returned error can't find the container with id 0cc6092944f73c250c7507d1bf75f232ddcce10e190635a046ef1b4c3a94c244
Oct 31 19:19:28.993740 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:28.993670    2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5mwk4_944a350f-aa7e-4921-8ddc-b0001fa05086/node-ca/0.log"
Oct 31 19:19:29.020349 ip-10-0-131-94 kubenswrapper[2584]: I1031 19:19:29.020294    2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:29.372864 ip-10-0-131-94 systemd[1]: Stopping Kubernetes Kubelet...
Oct 31 19:19:29.382867 ip-10-0-131-94 systemd[1]: kubelet.service: Deactivated successfully.
Oct 31 19:19:29.383057 ip-10-0-131-94 systemd[1]: Stopped Kubernetes Kubelet.
Oct 31 19:19:29.383120 ip-10-0-131-94 systemd[1]: kubelet.service: Consumed 2.417s CPU time.
Oct 31 19:19:29.394069 ip-10-0-131-94 systemd[1]: Starting Kubernetes Kubelet...
Oct 31 19:19:29.484462 ip-10-0-131-94 kubenswrapper[4645]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 19:19:29.484462 ip-10-0-131-94 kubenswrapper[4645]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 31 19:19:29.484462 ip-10-0-131-94 kubenswrapper[4645]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 19:19:29.484462 ip-10-0-131-94 kubenswrapper[4645]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 19:19:29.486994 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.486898    4645 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 19:19:29.486994 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.486991    4645 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 19:19:29.486994 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.486998    4645 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 19:19:29.486994 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487002    4645 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 19:19:29.487263 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487007    4645 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 19:19:29.487263 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487011    4645 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 19:19:29.487263 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487016    4645 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 19:19:29.487263 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487020    4645 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 19:19:29.487263 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487024    4645 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 19:19:29.487263 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487028    4645 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 19:19:29.487263 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487034    4645 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 19:19:29.487263 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487038    4645 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 19:19:29.487263 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487042    4645 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 19:19:29.487263 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487047    4645 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 19:19:29.487263 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487051    4645 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 19:19:29.487263 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487056    4645 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 19:19:29.487263 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487061    4645 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 19:19:29.487263 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487065    4645 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 19:19:29.487263 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487070    4645 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 19:19:29.487263 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487074    4645 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 19:19:29.487263 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487080    4645 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 19:19:29.487263 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487084    4645 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 19:19:29.487263 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487089    4645 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 19:19:29.487263 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487094    4645 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 19:19:29.488137 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487098    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 19:19:29.488137 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487102    4645 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 19:19:29.488137 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487107    4645 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 19:19:29.488137 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487112    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 19:19:29.488137 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487116    4645 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 19:19:29.488137 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487120    4645 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 19:19:29.488137 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487126    4645 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 19:19:29.488137 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487131    4645 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 19:19:29.488137 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487134    4645 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 19:19:29.488137 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487138    4645 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 19:19:29.488137 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487143    4645 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 19:19:29.488137 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487149    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 19:19:29.488137 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487153    4645 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 19:19:29.488137 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487158    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 19:19:29.488137 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487163    4645 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 19:19:29.488137 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487167    4645 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 19:19:29.488137 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487171    4645 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 19:19:29.488137 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487174    4645 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 19:19:29.488137 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487178    4645 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 19:19:29.488137 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487182    4645 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 19:19:29.489008 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487186    4645 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 19:19:29.489008 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487191    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 19:19:29.489008 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487195    4645 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 19:19:29.489008 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487199    4645 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 19:19:29.489008 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487203    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 19:19:29.489008 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487208    4645 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 19:19:29.489008 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487214    4645 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 19:19:29.489008 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487220    4645 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 19:19:29.489008 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487233    4645 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 19:19:29.489008 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487237    4645 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 19:19:29.489008 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487241    4645 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 19:19:29.489008 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487246    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 19:19:29.489008 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487250    4645 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 19:19:29.489008 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487255    4645 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 19:19:29.489008 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487259    4645 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 19:19:29.489008 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487262    4645 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 19:19:29.489008 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487265    4645 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 19:19:29.489008 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487268    4645 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 19:19:29.489812 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487271    4645 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 19:19:29.489812 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487274    4645 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 19:19:29.489812 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487277    4645 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 19:19:29.489812 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487280    4645 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 19:19:29.489812 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487282    4645 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 19:19:29.489812 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487286    4645 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 19:19:29.489812 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487290    4645 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 19:19:29.489812 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487295    4645 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 19:19:29.489812 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487299    4645 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 19:19:29.489812 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487303    4645 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 19:19:29.489812 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487309    4645 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 19:19:29.489812 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487313    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 19:19:29.489812 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487316    4645 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 19:19:29.489812 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487319    4645 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 19:19:29.489812 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487322    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 19:19:29.489812 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487338    4645 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 19:19:29.489812 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487343    4645 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 19:19:29.489812 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487348    4645 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 19:19:29.489812 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487354    4645 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 19:19:29.490636 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487358    4645 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 19:19:29.490636 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487362    4645 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 19:19:29.490636 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487366    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 19:19:29.490636 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487369    4645 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 19:19:29.490636 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487378    4645 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 19:19:29.490636 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487381    4645 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 19:19:29.490636 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487384    4645 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 19:19:29.490636 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487388    4645 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 19:19:29.490636 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487392    4645 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 19:19:29.490636 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487397    4645 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 19:19:29.490636 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487401    4645 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 19:19:29.490636 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487405    4645 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 19:19:29.490636 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487410    4645 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 19:19:29.490636 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487415    4645 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 19:19:29.490636 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487419    4645 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 19:19:29.490636 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487424    4645 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 19:19:29.490636 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487428    4645 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 19:19:29.490636 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487433    4645 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 19:19:29.490636 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487438    4645 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 19:19:29.490636 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487445    4645 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 19:19:29.491480 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487449    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 19:19:29.491480 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487453    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 19:19:29.491480 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487465    4645 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 19:19:29.491480 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487470    4645 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 19:19:29.491480 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.487474    4645 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 19:19:29.491480 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488583    4645 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 19:19:29.491480 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488595    4645 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 19:19:29.491480 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488599    4645 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 19:19:29.491480 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488604    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 19:19:29.491480 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488608    4645 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 19:19:29.491480 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488612    4645 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 19:19:29.491480 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488616    4645 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 19:19:29.491480 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488620    4645 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 19:19:29.491480 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488624    4645 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 19:19:29.491480 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488628    4645 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 19:19:29.491480 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488633    4645 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 19:19:29.491480 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488638    4645 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 19:19:29.491480 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488650    4645 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 19:19:29.491480 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488653    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 19:19:29.492296 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488656    4645 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 19:19:29.492296 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488658    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 19:19:29.492296 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488662    4645 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 19:19:29.492296 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488666    4645 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 19:19:29.492296 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488670    4645 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 19:19:29.492296 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488675    4645 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 19:19:29.492296 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488679    4645 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 19:19:29.492296 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488683    4645 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 19:19:29.492296 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488687    4645 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 19:19:29.492296 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488691    4645 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 19:19:29.492296 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488695    4645 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 19:19:29.492296 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488699    4645 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 19:19:29.492296 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488706    4645 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 19:19:29.492296 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488710    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 19:19:29.492296 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488714    4645 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 19:19:29.492296 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488717    4645 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 19:19:29.492296 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488719    4645 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 19:19:29.492296 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488732    4645 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 19:19:29.492296 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488737    4645 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 19:19:29.493118 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488741    4645 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 19:19:29.493118 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488745    4645 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 19:19:29.493118 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488749    4645 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 19:19:29.493118 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488753    4645 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 19:19:29.493118 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488756    4645 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 19:19:29.493118 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488760    4645 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 19:19:29.493118 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488764    4645 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 19:19:29.493118 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488769    4645 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 19:19:29.493118 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488772    4645 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 19:19:29.493118 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488777    4645 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 19:19:29.493118 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488782    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 19:19:29.493118 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488786    4645 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 19:19:29.493118 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488816    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 19:19:29.493118 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488823    4645 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 19:19:29.493118 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488828    4645 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 19:19:29.493118 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488834    4645 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 19:19:29.493118 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488838    4645 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 19:19:29.493118 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488842    4645 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 19:19:29.493118 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488854    4645 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 19:19:29.493118 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488858    4645 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 19:19:29.493978 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488863    4645 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 19:19:29.493978 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488867    4645 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 19:19:29.493978 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488871    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 19:19:29.493978 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488876    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 19:19:29.493978 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488885    4645 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 19:19:29.493978 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488889    4645 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 19:19:29.493978 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488893    4645 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 19:19:29.493978 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488897    4645 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 19:19:29.493978 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488902    4645 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 19:19:29.493978 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488907    4645 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 19:19:29.493978 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488912    4645 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 19:19:29.493978 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488916    4645 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 19:19:29.493978 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488921    4645 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 19:19:29.493978 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488923    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 19:19:29.493978 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488926    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 19:19:29.493978 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488929    4645 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 19:19:29.493978 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488931    4645 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 19:19:29.493978 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488934    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 19:19:29.493978 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488938    4645 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 19:19:29.493978 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488944    4645 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 19:19:29.494900 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488948    4645 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 19:19:29.494900 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488951    4645 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 19:19:29.494900 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488954    4645 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 19:19:29.494900 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488957    4645 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 19:19:29.494900 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488960    4645 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 19:19:29.494900 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488970    4645 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 19:19:29.494900 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488974    4645 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 19:19:29.494900 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488979    4645 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 19:19:29.494900 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488983    4645 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 19:19:29.494900 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488987    4645 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 19:19:29.494900 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488991    4645 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 19:19:29.494900 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.488996    4645 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 19:19:29.494900 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.489000    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 19:19:29.494900 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.489004    4645 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 19:19:29.494900 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.489006    4645 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 19:19:29.494900 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.489009    4645 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 19:19:29.494900 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.489011    4645 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 19:19:29.494900 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.489016    4645 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 19:19:29.494900 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.489020    4645 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 19:19:29.494900 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.489025    4645 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 19:19:29.495762 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.489028    4645 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 19:19:29.495762 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.489035    4645 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 19:19:29.495762 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.489042    4645 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 19:19:29.495762 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.489047    4645 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 19:19:29.495762 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.489051    4645 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 19:19:29.495762 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.489056    4645 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 19:19:29.495762 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.489060    4645 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 19:19:29.495762 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.489064    4645 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 19:19:29.495762 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.489068    4645 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 19:19:29.495762 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.489073    4645 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 19:19:29.495762 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.489076    4645 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 19:19:29.495762 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.489081    4645 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 19:19:29.495762 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.489085    4645 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 19:19:29.495762 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.489849    4645 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 19:19:29.495762 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.489860    4645 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 19:19:29.495762 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.489866    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 19:19:29.495762 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.489871    4645 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 19:19:29.495762 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.489883    4645 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 19:19:29.495762 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.489888    4645 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 19:19:29.496738 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.489892    4645 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 19:19:29.496738 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.489896    4645 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 19:19:29.496738 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.489900    4645 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 19:19:29.496738 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.489904    4645 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 19:19:29.496738 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.489908    4645 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 19:19:29.496738 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.489913    4645 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 19:19:29.496738 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.489918    4645 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 19:19:29.496738 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.489922    4645 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 19:19:29.496738 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.489926    4645 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 19:19:29.496738 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.489930    4645 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 19:19:29.496738 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.489934    4645 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 19:19:29.496738 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.489938    4645 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 19:19:29.496738 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.489943    4645 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 19:19:29.496738 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.489948    4645 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 19:19:29.496738 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.489953    4645 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 19:19:29.496738 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.489956    4645 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 19:19:29.496738 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.489959    4645 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 19:19:29.496738 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.489962    4645 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 19:19:29.496738 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.489964    4645 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 19:19:29.497566 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.489967    4645 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 19:19:29.497566 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.489970    4645 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 19:19:29.497566 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.489974    4645 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 19:19:29.497566 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.489978    4645 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 19:19:29.497566 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.489983    4645 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 19:19:29.497566 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.489987    4645 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 19:19:29.497566 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.489991    4645 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 19:19:29.497566 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.489996    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 19:19:29.497566 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.490000    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 19:19:29.497566 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.490004    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 19:19:29.497566 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.490009    4645 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 19:19:29.497566 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.490014    4645 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 19:19:29.497566 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.490027    4645 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 19:19:29.497566 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.490032    4645 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 19:19:29.497566 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.490037    4645 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 19:19:29.497566 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.490041    4645 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 19:19:29.497566 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.490045    4645 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 19:19:29.497566 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.490049    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 19:19:29.497566 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.490053    4645 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 19:19:29.497566 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.490056    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 19:19:29.498421 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.490061    4645 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 19:19:29.498421 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.490066    4645 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 19:19:29.498421 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.490070    4645 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 19:19:29.498421 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.490077    4645 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 19:19:29.498421 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.490082    4645 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 19:19:29.498421 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.490084    4645 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 19:19:29.498421 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.490087    4645 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 19:19:29.498421 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.490092    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 19:19:29.498421 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.490096    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 19:19:29.498421 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.490100    4645 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 19:19:29.498421 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.490104    4645 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 19:19:29.498421 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.490109    4645 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 19:19:29.498421 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.490113    4645 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 19:19:29.498421 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.490118    4645 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 19:19:29.498421 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.490123    4645 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 19:19:29.498421 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.490128    4645 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 19:19:29.498421 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.490132    4645 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 19:19:29.498421 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.490135    4645 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 19:19:29.498421 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.490138    4645 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 19:19:29.499286 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.490140    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 19:19:29.499286 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.490146    4645 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 19:19:29.499286 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.490148    4645 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 19:19:29.499286 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.490151    4645 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 19:19:29.499286 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.490154    4645 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 19:19:29.499286 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.490157    4645 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 19:19:29.499286 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.490166    4645 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 19:19:29.499286 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.490172    4645 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 19:19:29.499286 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.490176    4645 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 19:19:29.499286 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.490180    4645 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 19:19:29.499286 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.490185    4645 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 19:19:29.499286 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.490188    4645 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 19:19:29.499286 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.490190    4645 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 19:19:29.499286 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.490193    4645 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 19:19:29.499286 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.490195    4645 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 19:19:29.499286 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.490198    4645 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 19:19:29.499286 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.490210    4645 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 19:19:29.499286 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.490214    4645 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 19:19:29.499286 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.490217    4645 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 19:19:29.499286 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.490220    4645 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 19:19:29.500164 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.490222    4645 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 19:19:29.500164 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.490225    4645 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 19:19:29.500164 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.490227    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 19:19:29.500164 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.490231    4645 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 19:19:29.500164 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.490236    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 19:19:29.500164 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.490239    4645 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 19:19:29.500164 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.490244    4645 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 19:19:29.500164 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.490247    4645 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 19:19:29.500164 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.490252    4645 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 19:19:29.500164 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.490256    4645 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 19:19:29.500164 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.490261    4645 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 19:19:29.500164 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.490265    4645 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 19:19:29.500164 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.490269    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 19:19:29.500164 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.490274    4645 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 19:19:29.500164 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.490279    4645 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 19:19:29.500164 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.490282    4645 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 19:19:29.500164 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.490285    4645 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 19:19:29.500164 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.490288    4645 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 19:19:29.500164 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.490290    4645 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 19:19:29.501037 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.490299    4645 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 19:19:29.501037 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.490302    4645 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 19:19:29.501037 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.490305    4645 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 19:19:29.501037 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490447    4645 flags.go:64] FLAG: --address="0.0.0.0"
Oct 31 19:19:29.501037 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490458    4645 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 31 19:19:29.501037 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490470    4645 flags.go:64] FLAG: --anonymous-auth="true"
Oct 31 19:19:29.501037 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490479    4645 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 31 19:19:29.501037 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490486    4645 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 31 19:19:29.501037 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490492    4645 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 31 19:19:29.501037 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490499    4645 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 31 19:19:29.501037 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490506    4645 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 31 19:19:29.501037 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490511    4645 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 31 19:19:29.501037 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490516    4645 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 31 19:19:29.501037 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490521    4645 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 31 19:19:29.501037 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490524    4645 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 31 19:19:29.501037 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490528    4645 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 31 19:19:29.501037 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490531    4645 flags.go:64] FLAG: --cgroup-root=""
Oct 31 19:19:29.501037 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490534    4645 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 31 19:19:29.501037 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490537    4645 flags.go:64] FLAG: --client-ca-file=""
Oct 31 19:19:29.501037 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490540    4645 flags.go:64] FLAG: --cloud-provider="external"
Oct 31 19:19:29.501037 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490543    4645 flags.go:64] FLAG: --cluster-dns="[]"
Oct 31 19:19:29.501037 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490553    4645 flags.go:64] FLAG: --cluster-domain=""
Oct 31 19:19:29.501037 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490556    4645 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 31 19:19:29.501037 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490559    4645 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 31 19:19:29.502077 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490562    4645 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 31 19:19:29.502077 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490566    4645 flags.go:64] FLAG: --container-log-max-files="5"
Oct 31 19:19:29.502077 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490572    4645 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 31 19:19:29.502077 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490577    4645 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 31 19:19:29.502077 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490581    4645 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 31 19:19:29.502077 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490586    4645 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 31 19:19:29.502077 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490591    4645 flags.go:64] FLAG: --contention-profiling="false"
Oct 31 19:19:29.502077 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490598    4645 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 31 19:19:29.502077 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490602    4645 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 31 19:19:29.502077 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490614    4645 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 31 19:19:29.502077 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490619    4645 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 31 19:19:29.502077 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490626    4645 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 31 19:19:29.502077 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490630    4645 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 31 19:19:29.502077 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490635    4645 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 31 19:19:29.502077 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490639    4645 flags.go:64] FLAG: --enable-load-reader="false"
Oct 31 19:19:29.502077 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490642    4645 flags.go:64] FLAG: --enable-server="true"
Oct 31 19:19:29.502077 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490645    4645 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 31 19:19:29.502077 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490653    4645 flags.go:64] FLAG: --event-burst="100"
Oct 31 19:19:29.502077 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490659    4645 flags.go:64] FLAG: --event-qps="50"
Oct 31 19:19:29.502077 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490664    4645 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 31 19:19:29.502077 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490669    4645 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 31 19:19:29.502077 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490672    4645 flags.go:64] FLAG: --eviction-hard=""
Oct 31 19:19:29.502077 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490676    4645 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 31 19:19:29.502077 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490679    4645 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 31 19:19:29.503180 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490683    4645 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 31 19:19:29.503180 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490686    4645 flags.go:64] FLAG: --eviction-soft=""
Oct 31 19:19:29.503180 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490689    4645 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 31 19:19:29.503180 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490692    4645 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 31 19:19:29.503180 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490695    4645 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 31 19:19:29.503180 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490698    4645 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 31 19:19:29.503180 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490701    4645 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 31 19:19:29.503180 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490704    4645 flags.go:64] FLAG: --fail-swap-on="true"
Oct 31 19:19:29.503180 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490709    4645 flags.go:64] FLAG: --feature-gates=""
Oct 31 19:19:29.503180 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490719    4645 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 31 19:19:29.503180 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490729    4645 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 31 19:19:29.503180 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490735    4645 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 31 19:19:29.503180 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490741    4645 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 31 19:19:29.503180 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490746    4645 flags.go:64] FLAG: --healthz-port="10248"
Oct 31 19:19:29.503180 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490749    4645 flags.go:64] FLAG: --help="false"
Oct 31 19:19:29.503180 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490753    4645 flags.go:64] FLAG: --hostname-override="ip-10-0-131-94.us-east-2.compute.internal"
Oct 31 19:19:29.503180 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490757    4645 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 31 19:19:29.503180 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490761    4645 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 31 19:19:29.503180 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490775    4645 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 31 19:19:29.503180 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490782    4645 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 31 19:19:29.503180 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490788    4645 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 31 19:19:29.503180 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490793    4645 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 31 19:19:29.503180 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490798    4645 flags.go:64] FLAG: --image-service-endpoint=""
Oct 31 19:19:29.504167 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490803    4645 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 31 19:19:29.504167 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490808    4645 flags.go:64] FLAG: --kube-api-burst="100"
Oct 31 19:19:29.504167 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490813    4645 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 31 19:19:29.504167 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490818    4645 flags.go:64] FLAG: --kube-api-qps="50"
Oct 31 19:19:29.504167 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490823    4645 flags.go:64] FLAG: --kube-reserved=""
Oct 31 19:19:29.504167 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490829    4645 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 31 19:19:29.504167 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490833    4645 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 31 19:19:29.504167 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490838    4645 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 31 19:19:29.504167 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490843    4645 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 31 19:19:29.504167 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490849    4645 flags.go:64] FLAG: --lock-file=""
Oct 31 19:19:29.504167 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490854    4645 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 31 19:19:29.504167 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490859    4645 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 31 19:19:29.504167 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490863    4645 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 31 19:19:29.504167 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490869    4645 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 31 19:19:29.504167 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490872    4645 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 31 19:19:29.504167 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490875    4645 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 31 19:19:29.504167 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490878    4645 flags.go:64] FLAG: --logging-format="text"
Oct 31 19:19:29.504167 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490881    4645 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 31 19:19:29.504167 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490885    4645 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 31 19:19:29.504167 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490890    4645 flags.go:64] FLAG: --manifest-url=""
Oct 31 19:19:29.504167 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490893    4645 flags.go:64] FLAG: --manifest-url-header=""
Oct 31 19:19:29.504167 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490897    4645 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 31 19:19:29.504167 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490900    4645 flags.go:64] FLAG: --max-open-files="1000000"
Oct 31 19:19:29.504167 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490905    4645 flags.go:64] FLAG: --max-pods="110"
Oct 31 19:19:29.504167 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490909    4645 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 31 19:19:29.505252 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490913    4645 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 31 19:19:29.505252 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490916    4645 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 31 19:19:29.505252 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490919    4645 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 31 19:19:29.505252 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490929    4645 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 31 19:19:29.505252 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490932    4645 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 31 19:19:29.505252 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490935    4645 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 31 19:19:29.505252 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490943    4645 flags.go:64] FLAG: --node-status-max-images="50"
Oct 31 19:19:29.505252 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490946    4645 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 31 19:19:29.505252 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490949    4645 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 31 19:19:29.505252 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490952    4645 flags.go:64] FLAG: --pod-cidr=""
Oct 31 19:19:29.505252 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490955    4645 flags.go:64] FLAG: --pod-manifest-path=""
Oct 31 19:19:29.505252 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490958    4645 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 31 19:19:29.505252 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490961    4645 flags.go:64] FLAG: --pods-per-core="0"
Oct 31 19:19:29.505252 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490964    4645 flags.go:64] FLAG: --port="10250"
Oct 31 19:19:29.505252 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490967    4645 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 31 19:19:29.505252 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490970    4645 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-09d12504134ccfb62"
Oct 31 19:19:29.505252 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490973    4645 flags.go:64] FLAG: --qos-reserved=""
Oct 31 19:19:29.505252 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490976    4645 flags.go:64] FLAG: --read-only-port="10255"
Oct 31 19:19:29.505252 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490979    4645 flags.go:64] FLAG: --register-node="true"
Oct 31 19:19:29.505252 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490982    4645 flags.go:64] FLAG: --register-with-taints=""
Oct 31 19:19:29.505252 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490987    4645 flags.go:64] FLAG: --registry-burst="10"
Oct 31 19:19:29.505252 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490990    4645 flags.go:64] FLAG: --registry-qps="5"
Oct 31 19:19:29.505252 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490993    4645 flags.go:64] FLAG: --reserved-cpus=""
Oct 31 19:19:29.505252 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.490996    4645 flags.go:64] FLAG: --reserved-memory=""
Oct 31 19:19:29.505252 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.491002    4645 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 31 19:19:29.505252 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.491007    4645 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 31 19:19:29.506486 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.491016    4645 flags.go:64] FLAG: --rotate-certificates="false"
Oct 31 19:19:29.506486 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.491021    4645 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 31 19:19:29.506486 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.491027    4645 flags.go:64] FLAG: --runonce="false"
Oct 31 19:19:29.506486 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.491032    4645 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 31 19:19:29.506486 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.491036    4645 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 31 19:19:29.506486 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.491040    4645 flags.go:64] FLAG: --seccomp-default="false"
Oct 31 19:19:29.506486 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.491043    4645 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 31 19:19:29.506486 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.491046    4645 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 31 19:19:29.506486 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.491049    4645 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 31 19:19:29.506486 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.491052    4645 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 31 19:19:29.506486 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.491064    4645 flags.go:64] FLAG: --storage-driver-password="root"
Oct 31 19:19:29.506486 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.491069    4645 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 31 19:19:29.506486 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.491072    4645 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 31 19:19:29.506486 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.491075    4645 flags.go:64] FLAG: --storage-driver-user="root"
Oct 31 19:19:29.506486 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.491078    4645 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 31 19:19:29.506486 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.491082    4645 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 31 19:19:29.506486 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.491084    4645 flags.go:64] FLAG: --system-cgroups=""
Oct 31 19:19:29.506486 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.491087    4645 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 31 19:19:29.506486 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.491093    4645 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 31 19:19:29.506486 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.491096    4645 flags.go:64] FLAG: --tls-cert-file=""
Oct 31 19:19:29.506486 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.491099    4645 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 31 19:19:29.506486 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.491108    4645 flags.go:64] FLAG: --tls-min-version=""
Oct 31 19:19:29.506486 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.491112    4645 flags.go:64] FLAG: --tls-private-key-file=""
Oct 31 19:19:29.506486 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.491117    4645 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 31 19:19:29.506486 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.491122    4645 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 31 19:19:29.507548 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.491127    4645 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 31 19:19:29.507548 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.491131    4645 flags.go:64] FLAG: --v="2"
Oct 31 19:19:29.507548 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.491137    4645 flags.go:64] FLAG: --version="false"
Oct 31 19:19:29.507548 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.491144    4645 flags.go:64] FLAG: --vmodule=""
Oct 31 19:19:29.507548 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.491149    4645 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 31 19:19:29.507548 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.491155    4645 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 31 19:19:29.507548 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491411    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 19:19:29.507548 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491423    4645 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 19:19:29.507548 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491428    4645 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 19:19:29.507548 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491433    4645 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 19:19:29.507548 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491437    4645 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 19:19:29.507548 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491441    4645 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 19:19:29.507548 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491447    4645 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 19:19:29.507548 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491459    4645 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 19:19:29.507548 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491463    4645 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 19:19:29.507548 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491467    4645 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 19:19:29.507548 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491471    4645 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 19:19:29.507548 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491475    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 19:19:29.507548 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491487    4645 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 19:19:29.507548 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491492    4645 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 19:19:29.507548 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491497    4645 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 19:19:29.508434 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491503    4645 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 19:19:29.508434 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491507    4645 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 19:19:29.508434 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491512    4645 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 19:19:29.508434 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491515    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 19:19:29.508434 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491517    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 19:19:29.508434 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491520    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 19:19:29.508434 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491523    4645 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 19:19:29.508434 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491526    4645 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 19:19:29.508434 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491529    4645 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 19:19:29.508434 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491531    4645 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 19:19:29.508434 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491534    4645 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 19:19:29.508434 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491537    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 19:19:29.508434 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491539    4645 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 19:19:29.508434 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491542    4645 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 19:19:29.508434 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491545    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 19:19:29.508434 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491547    4645 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 19:19:29.508434 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491550    4645 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 19:19:29.508434 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491554    4645 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 19:19:29.508434 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491559    4645 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 19:19:29.508434 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491563    4645 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 19:19:29.509274 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491567    4645 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 19:19:29.509274 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491571    4645 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 19:19:29.509274 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491575    4645 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 19:19:29.509274 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491581    4645 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 19:19:29.509274 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491585    4645 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 19:19:29.509274 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491589    4645 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 19:19:29.509274 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491593    4645 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 19:19:29.509274 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491597    4645 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 19:19:29.509274 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491602    4645 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 19:19:29.509274 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491607    4645 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 19:19:29.509274 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491619    4645 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 19:19:29.509274 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491624    4645 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 19:19:29.509274 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491628    4645 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 19:19:29.509274 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491631    4645 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 19:19:29.509274 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491634    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 19:19:29.509274 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491638    4645 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 19:19:29.509274 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491642    4645 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 19:19:29.509274 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491646    4645 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 19:19:29.509274 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491651    4645 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 19:19:29.509274 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491655    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 19:19:29.510279 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491660    4645 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 19:19:29.510279 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491665    4645 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 19:19:29.510279 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491670    4645 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 19:19:29.510279 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491675    4645 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 19:19:29.510279 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491681    4645 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 19:19:29.510279 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491686    4645 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 19:19:29.510279 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491690    4645 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 19:19:29.510279 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491694    4645 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 19:19:29.510279 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491697    4645 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 19:19:29.510279 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491702    4645 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 19:19:29.510279 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491706    4645 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 19:19:29.510279 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491710    4645 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 19:19:29.510279 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491715    4645 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 19:19:29.510279 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491719    4645 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 19:19:29.510279 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491724    4645 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 19:19:29.510279 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491731    4645 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 19:19:29.510279 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491736    4645 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 19:19:29.510279 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491741    4645 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 19:19:29.510279 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491745    4645 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 19:19:29.511006 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491750    4645 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 19:19:29.511006 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491754    4645 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 19:19:29.511006 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491759    4645 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 19:19:29.511006 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491763    4645 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 19:19:29.511006 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491774    4645 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 19:19:29.511006 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491781    4645 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 19:19:29.511006 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491788    4645 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 19:19:29.511006 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491793    4645 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 19:19:29.511006 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491799    4645 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 19:19:29.511006 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491805    4645 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 19:19:29.511006 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491810    4645 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 19:19:29.511006 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491814    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 19:19:29.511006 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491818    4645 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 19:19:29.511006 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491823    4645 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 19:19:29.511006 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491828    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 19:19:29.511006 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491833    4645 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 19:19:29.511006 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491837    4645 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 19:19:29.511006 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491842    4645 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 19:19:29.511006 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491845    4645 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 19:19:29.511670 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491847    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 19:19:29.511670 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491850    4645 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 19:19:29.511670 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491853    4645 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 19:19:29.511670 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491856    4645 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 19:19:29.511670 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491859    4645 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 19:19:29.511670 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491862    4645 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 19:19:29.511670 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491864    4645 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 19:19:29.511670 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491867    4645 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 19:19:29.511670 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491870    4645 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 19:19:29.511670 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491873    4645 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 19:19:29.511670 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491876    4645 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 19:19:29.511670 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491879    4645 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 19:19:29.511670 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491881    4645 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 19:19:29.511670 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.491916    4645 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 19:19:29.511670 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491984    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 19:19:29.511670 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491990    4645 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 19:19:29.512450 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491994    4645 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 19:19:29.512450 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.491999    4645 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 19:19:29.512450 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492009    4645 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 19:19:29.512450 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492016    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 19:19:29.512450 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492020    4645 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 19:19:29.512450 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492027    4645 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 19:19:29.512450 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492032    4645 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 19:19:29.512450 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492035    4645 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 19:19:29.512450 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492038    4645 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 19:19:29.512450 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492041    4645 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 19:19:29.512450 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492045    4645 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 19:19:29.512450 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492050    4645 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 19:19:29.512450 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492054    4645 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 19:19:29.512450 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492069    4645 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 19:19:29.512450 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492074    4645 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 19:19:29.512450 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492078    4645 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 19:19:29.512450 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492083    4645 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 19:19:29.512450 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492087    4645 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 19:19:29.512450 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492091    4645 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 19:19:29.513204 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492095    4645 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 19:19:29.513204 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492099    4645 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 19:19:29.513204 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492104    4645 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 19:19:29.513204 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492107    4645 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 19:19:29.513204 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492112    4645 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 19:19:29.513204 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492115    4645 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 19:19:29.513204 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492118    4645 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 19:19:29.513204 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492122    4645 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 19:19:29.513204 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492127    4645 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 19:19:29.513204 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492133    4645 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 19:19:29.513204 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492138    4645 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 19:19:29.513204 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492142    4645 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 19:19:29.513204 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492148    4645 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 19:19:29.513204 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492154    4645 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 19:19:29.513204 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492158    4645 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 19:19:29.513204 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492163    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 19:19:29.513204 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492173    4645 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 19:19:29.513204 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492176    4645 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 19:19:29.513204 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492178    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 19:19:29.514050 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492188    4645 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 19:19:29.514050 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492192    4645 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 19:19:29.514050 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492194    4645 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 19:19:29.514050 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492197    4645 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 19:19:29.514050 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492200    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 19:19:29.514050 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492202    4645 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 19:19:29.514050 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492208    4645 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 19:19:29.514050 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492213    4645 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 19:19:29.514050 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492217    4645 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 19:19:29.514050 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492221    4645 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 19:19:29.514050 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492225    4645 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 19:19:29.514050 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492231    4645 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 19:19:29.514050 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492236    4645 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 19:19:29.514050 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492240    4645 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 19:19:29.514050 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492245    4645 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 19:19:29.514050 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492248    4645 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 19:19:29.514050 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492250    4645 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 19:19:29.514050 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492253    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 19:19:29.514050 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492256    4645 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 19:19:29.514050 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492258    4645 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 19:19:29.514737 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492263    4645 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 19:19:29.514737 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492265    4645 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 19:19:29.514737 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492268    4645 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 19:19:29.514737 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492271    4645 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 19:19:29.514737 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492273    4645 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 19:19:29.514737 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492276    4645 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 19:19:29.514737 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492278    4645 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 19:19:29.514737 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492281    4645 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 19:19:29.514737 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492284    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 19:19:29.514737 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492288    4645 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 19:19:29.514737 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492296    4645 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 19:19:29.514737 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492299    4645 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 19:19:29.514737 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492302    4645 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 19:19:29.514737 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492305    4645 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 19:19:29.514737 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492308    4645 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 19:19:29.514737 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492311    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 19:19:29.514737 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492314    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 19:19:29.514737 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492319    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 19:19:29.514737 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492323    4645 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 19:19:29.514737 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492341    4645 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 19:19:29.515406 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492347    4645 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 19:19:29.515406 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492351    4645 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 19:19:29.515406 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492355    4645 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 19:19:29.515406 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492359    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 19:19:29.515406 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492363    4645 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 19:19:29.515406 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492367    4645 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 19:19:29.515406 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492371    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 19:19:29.515406 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492375    4645 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 19:19:29.515406 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492380    4645 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 19:19:29.515406 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492384    4645 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 19:19:29.515406 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492388    4645 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 19:19:29.515406 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492392    4645 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 19:19:29.515406 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492399    4645 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 19:19:29.515406 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492403    4645 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 19:19:29.515406 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492407    4645 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 19:19:29.515406 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492411    4645 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 19:19:29.515406 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492415    4645 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 19:19:29.515406 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492419    4645 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 19:19:29.515406 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492422    4645 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 19:19:29.515406 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492425    4645 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 19:19:29.516147 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492427    4645 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 19:19:29.516147 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492433    4645 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 19:19:29.516147 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492437    4645 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 19:19:29.516147 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492447    4645 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 19:19:29.516147 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492451    4645 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 19:19:29.516147 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.492453    4645 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 19:19:29.516147 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.504273    4645 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 31 19:19:29.516147 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.504296    4645 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 31 19:19:29.516147 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.504320    4645 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 31 19:19:29.516147 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.504346    4645 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 31 19:19:29.516147 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.504709    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 19:19:29.516147 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.504718    4645 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 19:19:29.516147 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.504723    4645 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 19:19:29.516147 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.504728    4645 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 19:19:29.516147 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.504738    4645 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 19:19:29.516147 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.504743    4645 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 19:19:29.516147 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.504747    4645 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 19:19:29.516147 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.504752    4645 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 19:19:29.516147 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.504757    4645 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 19:19:29.516147 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.504762    4645 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 19:19:29.517057 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.504766    4645 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 19:19:29.517057 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.504771    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 19:19:29.517057 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.504775    4645 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 19:19:29.517057 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.504779    4645 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 19:19:29.517057 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.504785    4645 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 19:19:29.517057 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.504789    4645 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 19:19:29.517057 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.504794    4645 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 19:19:29.517057 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.504803    4645 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 19:19:29.517057 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.504808    4645 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 19:19:29.517057 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.504813    4645 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 19:19:29.517057 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.504817    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 19:19:29.517057 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.504822    4645 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 19:19:29.517057 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.504826    4645 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 19:19:29.517057 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.504831    4645 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 19:19:29.517057 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.504835    4645 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 19:19:29.517057 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.504839    4645 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 19:19:29.517057 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.504844    4645 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 19:19:29.517057 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.504849    4645 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 19:19:29.517057 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.504853    4645 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 19:19:29.517057 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.504869    4645 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 19:19:29.517878 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.504874    4645 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 19:19:29.517878 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.504878    4645 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 19:19:29.517878 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.504882    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 19:19:29.517878 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.504887    4645 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 19:19:29.517878 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.504891    4645 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 19:19:29.517878 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.504896    4645 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 19:19:29.517878 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.504901    4645 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 19:19:29.517878 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.504906    4645 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 19:19:29.517878 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.504910    4645 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 19:19:29.517878 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.504915    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 19:19:29.517878 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.504919    4645 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 19:19:29.517878 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.504926    4645 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 19:19:29.517878 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.504938    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 19:19:29.517878 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.504943    4645 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 19:19:29.517878 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.505023    4645 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 19:19:29.517878 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.505028    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 19:19:29.517878 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.505033    4645 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 19:19:29.517878 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.505037    4645 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 19:19:29.517878 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.505042    4645 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 19:19:29.518692 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.505047    4645 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 19:19:29.518692 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.505051    4645 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 19:19:29.518692 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.505056    4645 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 19:19:29.518692 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.505061    4645 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 19:19:29.518692 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.505065    4645 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 19:19:29.518692 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.505074    4645 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 19:19:29.518692 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.505079    4645 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 19:19:29.518692 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.505083    4645 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 19:19:29.518692 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.505087    4645 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 19:19:29.518692 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.505092    4645 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 19:19:29.518692 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.505096    4645 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 19:19:29.518692 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.505101    4645 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 19:19:29.518692 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.505106    4645 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 19:19:29.518692 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.505119    4645 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 19:19:29.518692 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.505123    4645 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 19:19:29.518692 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.505128    4645 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 19:19:29.518692 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.505132    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 19:19:29.518692 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.505136    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 19:19:29.518692 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.505145    4645 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 19:19:29.518692 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.505150    4645 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 19:19:29.519513 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.505154    4645 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 19:19:29.519513 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.505159    4645 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 19:19:29.519513 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.505163    4645 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 19:19:29.519513 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.505167    4645 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 19:19:29.519513 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.505172    4645 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 19:19:29.519513 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.505177    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 19:19:29.519513 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.505181    4645 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 19:19:29.519513 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.505186    4645 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 19:19:29.519513 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.505190    4645 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 19:19:29.519513 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.505194    4645 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 19:19:29.519513 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.505199    4645 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 19:19:29.519513 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.505208    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 19:19:29.519513 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.505212    4645 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 19:19:29.519513 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.505216    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 19:19:29.519513 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.505221    4645 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 19:19:29.519513 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.505226    4645 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 19:19:29.519513 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.505231    4645 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 19:19:29.519513 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.505236    4645 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 19:19:29.519513 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.505240    4645 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 19:19:29.519513 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.505244    4645 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 19:19:29.520176 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.505249    4645 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 19:19:29.520176 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.505253    4645 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 19:19:29.520176 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.505258    4645 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 19:19:29.520176 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.505267    4645 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 19:19:29.520176 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.505272    4645 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 19:19:29.520176 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.505276    4645 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 19:19:29.520176 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.505287    4645 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 19:19:29.520176 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.505292    4645 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 19:19:29.520176 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.505297    4645 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 19:19:29.520176 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.505303    4645 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 19:19:29.520176 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.505309    4645 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 19:19:29.520176 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.505313    4645 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 19:19:29.520176 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.505318    4645 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 19:19:29.520176 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.505322    4645 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 19:19:29.520176 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.505341    4645 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 19:19:29.520176 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.505351    4645 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 19:19:29.520176 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.505355    4645 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 19:19:29.520783 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.505405    4645 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 19:19:29.520783 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506210    4645 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 19:19:29.520783 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506219    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 19:19:29.520783 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506224    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 19:19:29.520783 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506229    4645 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 19:19:29.520783 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506234    4645 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 19:19:29.520783 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506239    4645 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 19:19:29.520783 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506249    4645 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 19:19:29.520783 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506253    4645 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 19:19:29.520783 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506258    4645 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 19:19:29.520783 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506262    4645 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 19:19:29.520783 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506267    4645 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 19:19:29.520783 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506272    4645 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 19:19:29.520783 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506277    4645 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 19:19:29.520783 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506281    4645 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 19:19:29.520783 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506285    4645 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 19:19:29.521419 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506289    4645 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 19:19:29.521419 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506294    4645 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 19:19:29.521419 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506298    4645 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 19:19:29.521419 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506307    4645 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 19:19:29.521419 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506312    4645 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 19:19:29.521419 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506316    4645 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 19:19:29.521419 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506342    4645 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 19:19:29.521419 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506347    4645 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 19:19:29.521419 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506351    4645 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 19:19:29.521419 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506356    4645 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 19:19:29.521419 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506360    4645 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 19:19:29.521419 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506365    4645 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 19:19:29.521419 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506369    4645 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 19:19:29.521419 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506373    4645 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 19:19:29.521419 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506378    4645 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 19:19:29.521419 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506382    4645 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 19:19:29.521419 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506392    4645 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 19:19:29.521419 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506396    4645 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 19:19:29.521419 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506400    4645 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 19:19:29.521419 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506405    4645 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 19:19:29.522033 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506411    4645 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 19:19:29.522033 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506419    4645 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 19:19:29.522033 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506424    4645 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 19:19:29.522033 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506428    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 19:19:29.522033 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506432    4645 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 19:19:29.522033 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506437    4645 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 19:19:29.522033 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506441    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 19:19:29.522033 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506445    4645 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 19:19:29.522033 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506529    4645 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 19:19:29.522033 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506534    4645 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 19:19:29.522033 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506538    4645 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 19:19:29.522033 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506542    4645 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 19:19:29.522033 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506547    4645 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 19:19:29.522033 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506551    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 19:19:29.522033 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506555    4645 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 19:19:29.522033 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506559    4645 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 19:19:29.522033 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506564    4645 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 19:19:29.522033 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506568    4645 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 19:19:29.522033 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506573    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 19:19:29.522624 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506584    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 19:19:29.522624 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506589    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 19:19:29.522624 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506598    4645 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 19:19:29.522624 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506602    4645 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 19:19:29.522624 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506606    4645 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 19:19:29.522624 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506611    4645 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 19:19:29.522624 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506615    4645 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 19:19:29.522624 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506619    4645 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 19:19:29.522624 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506624    4645 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 19:19:29.522624 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506628    4645 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 19:19:29.522624 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506632    4645 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 19:19:29.522624 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506636    4645 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 19:19:29.522624 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506640    4645 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 19:19:29.522624 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506645    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 19:19:29.522624 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506649    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 19:19:29.522624 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506659    4645 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 19:19:29.522624 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506663    4645 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 19:19:29.522624 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506668    4645 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 19:19:29.522624 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506672    4645 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 19:19:29.523189 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506676    4645 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 19:19:29.523189 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506680    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 19:19:29.523189 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506685    4645 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 19:19:29.523189 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506689    4645 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 19:19:29.523189 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506693    4645 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 19:19:29.523189 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506697    4645 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 19:19:29.523189 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506702    4645 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 19:19:29.523189 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506706    4645 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 19:19:29.523189 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506710    4645 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 19:19:29.523189 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506719    4645 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 19:19:29.523189 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506723    4645 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 19:19:29.523189 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506728    4645 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 19:19:29.523189 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506735    4645 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 19:19:29.523189 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506742    4645 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 19:19:29.523189 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506757    4645 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 19:19:29.523189 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506762    4645 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 19:19:29.523189 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506767    4645 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 19:19:29.523189 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506772    4645 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 19:19:29.523189 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506777    4645 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 19:19:29.523189 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506783    4645 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 19:19:29.523834 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506787    4645 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 19:19:29.523834 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506797    4645 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 19:19:29.523834 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506801    4645 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 19:19:29.523834 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506805    4645 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 19:19:29.523834 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506809    4645 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 19:19:29.523834 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506814    4645 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 19:19:29.523834 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506818    4645 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 19:19:29.523834 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506823    4645 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 19:19:29.523834 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506827    4645 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 19:19:29.523834 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506831    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 19:19:29.523834 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506836    4645 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 19:19:29.523834 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506840    4645 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 19:19:29.523834 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.506844    4645 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 19:19:29.523834 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.506899    4645 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 19:19:29.523834 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.507098    4645 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 19:19:29.523834 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.507109    4645 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 19:19:29.524323 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.507121    4645 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 19:19:29.524323 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.507127    4645 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 19:19:29.524323 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.507131    4645 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 19:19:29.524323 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.507137    4645 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 19:19:29.524323 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.507142    4645 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 19:19:29.524323 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.507147    4645 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 19:19:29.524323 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.507152    4645 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 19:19:29.524323 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.507156    4645 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 19:19:29.524323 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.507160    4645 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 19:19:29.524323 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.507165    4645 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 19:19:29.524323 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.507170    4645 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 19:19:29.524323 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.507183    4645 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 19:19:29.524323 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.507188    4645 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 19:19:29.524323 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.507198    4645 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 19:19:29.524323 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.507202    4645 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 19:19:29.524323 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.507206    4645 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 19:19:29.524323 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.507278    4645 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 19:19:29.524323 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.507283    4645 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 19:19:29.524323 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.507287    4645 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 19:19:29.524323 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.507293    4645 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 19:19:29.524998 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.507297    4645 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 19:19:29.524998 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.507302    4645 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 19:19:29.524998 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.507309    4645 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 19:19:29.524998 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.507315    4645 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 19:19:29.524998 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.507320    4645 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 19:19:29.524998 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.511791    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 19:19:29.524998 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.511910    4645 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 19:19:29.524998 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.511918    4645 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 19:19:29.524998 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.511924    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 19:19:29.524998 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.511928    4645 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 19:19:29.524998 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.511933    4645 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 19:19:29.524998 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.511938    4645 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 19:19:29.524998 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.511942    4645 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 19:19:29.524998 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.511947    4645 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 19:19:29.524998 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.511960    4645 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 19:19:29.524998 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.511964    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 19:19:29.524998 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.511969    4645 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 19:19:29.524998 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.511973    4645 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 19:19:29.524998 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.511978    4645 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 19:19:29.525735 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.511982    4645 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 19:19:29.525735 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.511987    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 19:19:29.525735 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.511992    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 19:19:29.525735 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.511997    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 19:19:29.525735 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.512003    4645 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 19:19:29.525735 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.512008    4645 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 19:19:29.525735 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.512013    4645 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 19:19:29.525735 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.512017    4645 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 19:19:29.525735 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.512022    4645 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 19:19:29.525735 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.512026    4645 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 19:19:29.525735 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.512031    4645 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 19:19:29.525735 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.512035    4645 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 19:19:29.525735 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.512039    4645 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 19:19:29.525735 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.512044    4645 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 19:19:29.525735 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.512048    4645 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 19:19:29.525735 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.512053    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 19:19:29.525735 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.512058    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 19:19:29.525735 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.512062    4645 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 19:19:29.525735 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.512066    4645 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 19:19:29.526361 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.512070    4645 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 19:19:29.526361 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.512075    4645 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 19:19:29.526361 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.512079    4645 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 19:19:29.526361 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.512083    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 19:19:29.526361 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.512088    4645 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 19:19:29.526361 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.512095    4645 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 19:19:29.526361 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.512099    4645 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 19:19:29.526361 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.512104    4645 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 19:19:29.526361 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.512109    4645 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 19:19:29.526361 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.512114    4645 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 19:19:29.526361 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.512121    4645 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 19:19:29.526361 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.512127    4645 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 19:19:29.526361 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.512132    4645 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 19:19:29.526361 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.512143    4645 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 19:19:29.526361 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.512151    4645 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 19:19:29.526361 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.512158    4645 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 19:19:29.526361 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.512162    4645 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 19:19:29.526361 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.512166    4645 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 19:19:29.526361 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.512169    4645 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 19:19:29.526959 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.512172    4645 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 19:19:29.526959 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.512175    4645 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 19:19:29.526959 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.512178    4645 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 19:19:29.526959 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.512187    4645 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 19:19:29.526959 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.512192    4645 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 19:19:29.526959 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.512196    4645 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 19:19:29.526959 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.512201    4645 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 19:19:29.526959 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.512205    4645 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 19:19:29.526959 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.512208    4645 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 19:19:29.526959 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.512211    4645 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 19:19:29.526959 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.512214    4645 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 19:19:29.526959 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.512219    4645 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 19:19:29.526959 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.512223    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 19:19:29.526959 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.512227    4645 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 19:19:29.526959 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.512231    4645 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 19:19:29.526959 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.512235    4645 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 19:19:29.526959 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.512237    4645 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 19:19:29.526959 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.512240    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 19:19:29.526959 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.512243    4645 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 19:19:29.526959 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.512246    4645 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 19:19:29.527611 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.512249    4645 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 19:19:29.527611 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.512252    4645 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 19:19:29.527611 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.512255    4645 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 19:19:29.527611 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.512258    4645 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 19:19:29.527611 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.512261    4645 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 19:19:29.527611 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.512267    4645 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 19:19:29.527611 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:29.512271    4645 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 19:19:29.527611 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.512372    4645 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 31 19:19:29.527611 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.513774    4645 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 31 19:19:29.527611 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.513835    4645 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 31 19:19:29.527611 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.514057    4645 server.go:1014] "Starting client certificate rotation"
Oct 31 19:19:29.527611 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.514697    4645 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 19:19:29.527611 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.515401    4645 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 19:19:29.527611 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.515492    4645 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 19:13:50 +0000 UTC" deadline="2028-07-18 13:39:36.870162998 +0000 UTC"
Oct 31 19:19:29.527611 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.515555    4645 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15018h20m7.354611426s"
Oct 31 19:19:29.527611 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.516198    4645 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 19:19:29.528134 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.518038    4645 log.go:25] "Validated CRI v1 runtime API"
Oct 31 19:19:29.528134 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.520111    4645 log.go:25] "Validated CRI v1 image API"
Oct 31 19:19:29.528134 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.520802    4645 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 31 19:19:29.528134 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.522791    4645 fs.go:135] Filesystem UUIDs: map[1a7924e8-5660-49f5-9f5f-4ca38580f698:/dev/nvme0n1p4 33970d72-81df-4c0e-bd75-9ac6392bcde7:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Oct 31 19:19:29.528298 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.522818    4645 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0467ace45db27d653773586f489f0e7107046a2129fdffc6696bce6e593dcd35/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0467ace45db27d653773586f489f0e7107046a2129fdffc6696bce6e593dcd35/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0cc6092944f73c250c7507d1bf75f232ddcce10e190635a046ef1b4c3a94c244/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0cc6092944f73c250c7507d1bf75f232ddcce10e190635a046ef1b4c3a94c244/userdata/shm major:0 minor:269 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1bf8152bac50cc170567867a03d791329af8baa72a02537fbec3507f80ade260/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1bf8152bac50cc170567867a03d791329af8baa72a02537fbec3507f80ade260/userdata/shm major:0 minor:233 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3aef875c6fddb789272d58c909a8ff0173c68d56db9a79372057f2ec2ef89af0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3aef875c6fddb789272d58c909a8ff0173c68d56db9a79372057f2ec2ef89af0/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3bd73889613314ef4bb6e74fcfb8cc40e91533df36fbe8525771038110a642ba/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3bd73889613314ef4bb6e74fcfb8cc40e91533df36fbe8525771038110a642ba/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5f67e14a39e17e0a90f3920596f0f7ceae2f8b34d976c64f044c50e4aa5dced4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5f67e14a39e17e0a90f3920596f0f7ceae2f8b34d976c64f044c50e4aa5dced4/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6ee2e706596091c74ca0dd6f3acb765b21462c4df84997743d4e397e6b3a3a83/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6ee2e706596091c74ca0dd6f3acb765b21462c4df84997743d4e397e6b3a3a83/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/876a4b2c55217a62653fbcffac89b3ff4f237029ca500213a384afde189b8b85/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/876a4b2c55217a62653fbcffac89b3ff4f237029ca500213a384afde189b8b85/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/87d4f3ec9faded6c0502d00c22586498bc6b8b44e0e3db741d6c1b45e313e9c6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/87d4f3ec9faded6c0502d00c22586498bc6b8b44e0e3db741d6c1b45e313e9c6/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8a472eca945753737a9ab1ccdfc4f52f8c062ed3e92f413f175e77fc0861e6c9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8a472eca945753737a9ab1ccdfc4f52f8c062ed3e92f413f175e77fc0861e6c9/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/997e378d63e20d073440ec716dd280bc3a55c382b061102ea561d2549410d145/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/997e378d63e20d073440ec716dd280bc3a55c382b061102ea561d2549410d145/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a0c01424ea5f36aa2a34e7031ba579e446014499dcfb3069f58f82930562ff2c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a0c01424ea5f36aa2a34e7031ba579e446014499dcfb3069f58f82930562ff2c/userdata/shm major:0 minor:211 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a2c94b5cfad7d3b7908931167a8f363624b774ce25a5f0926c51d913febcd9bc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a2c94b5cfad7d3b7908931167a8f363624b774ce25a5f0926c51d913febcd9bc/userdata/shm major:0 minor:217 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b393b80365b08f5f32594138538a31bd0c471050a98f165fc897dec2a6813381/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b393b80365b08f5f32594138538a31bd0c471050a98f165fc897dec2a6813381/userdata/shm major:0 minor:264 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/be6e65fc286d720ef66bba2cd6c16bc5752194068f80469cda7aff0391b4731f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/be6e65fc286d720ef66bba2cd6c16bc5752194068f80469cda7aff0391b4731f/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d906967467612a30373fee26acf17e6cf18863f31120d951efa594bee7f3b425/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d906967467612a30373fee26acf17e6cf18863f31120d951efa594bee7f3b425/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dbc6469e8e9ba3ce4167e2a94399129dab2c64360e1194f48abbd68d32cdda0e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dbc6469e8e9ba3ce4167e2a94399129dab2c64360e1194f48abbd68d32cdda0e/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/eab6b838c5183cc9562553ddea45990d0887348487d9cfd8fce770b50805e1a3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/eab6b838c5183cc9562553ddea45990d0887348487d9cfd8fce770b50805e1a3/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/05ca774c-33f8-4368-a677-7c1a98b61bf7/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/05ca774c-33f8-4368-a677-7c1a98b61bf7/volumes/kubernetes.io~secret/agent-certs major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c/volumes/kubernetes.io~projected/kube-api-access-nq8nb:{mountpoint:/var/lib/kubelet/pods/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c/volumes/kubernetes.io~projected/kube-api-access-nq8nb major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/21dc10e8-ee61-46ae-83f7-67d1772f083c/volumes/kubernetes.io~projected/kube-api-access-7k8mg:{mountpoint:/var/lib/kubelet/pods/21dc10e8-ee61-46ae-83f7-67d1772f083c/volumes/kubernetes.io~projected/kube-api-access-7k8mg major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ae10cd8-0183-4aca-bf20-ecb1e375e420/volumes/kubernetes.io~projected/kube-api-access-86gj5:{mountpoint:/var/lib/kubelet/pods/2ae10cd8-0183-4aca-bf20-ecb1e375e420/volumes/kubernetes.io~projected/kube-api-access-86gj5 major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3382fc12-f376-4b15-95be-6cb371ae87bd/volumes/kubernetes.io~projected/kube-api-access-t62j5:{mountpoint:/var/lib/kubelet/pods/3382fc12-f376-4b15-95be-6cb371ae87bd/volumes/kubernetes.io~projected/kube-api-access-t62j5 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4220431d-fa8a-4537-ad2e-e8ba6164f63d/volumes/kubernetes.io~projected/kube-api-access-v4lbw:{mountpoint:/var/lib/kubelet/pods/4220431d-fa8a-4537-ad2e-e8ba6164f63d/volumes/kubernetes.io~projected/kube-api-access-v4lbw major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/534f9972-b54f-4713-8326-4808fdf08386/volumes/kubernetes.io~projected/kube-api-access-npsmn:{mountpoint:/var/lib/kubelet/pods/534f9972-b54f-4713-8326-4808fdf08386/volumes/kubernetes.io~projected/kube-api-access-npsmn major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/611ab431-d82c-46e9-a108-4b221cc8c2cc/volumes/kubernetes.io~projected/kube-api-access-x2jxb:{mountpoint:/var/lib/kubelet/pods/611ab431-d82c-46e9-a108-4b221cc8c2cc/volumes/kubernetes.io~projected/kube-api-access-x2jxb major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/638d688d-d5fc-480d-8c53-08ab2f1864b1/volumes/kubernetes.io~projected/kube-api-access-gk5qd:{mountpoint:/var/lib/kubelet/pods/638d688d-d5fc-480d-8c53-08ab2f1864b1/volumes/kubernetes.io~projected/kube-api-access-gk5qd major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/65c98317-00d2-4ba8-8f66-9b34d720c553/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/65c98317-00d2-4ba8-8f66-9b34d720c553/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/65c98317-00d2-4ba8-8f66-9b34d720c553/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/65c98317-00d2-4ba8-8f66-9b34d720c553/volumes/kubernetes.io~empty-dir/tmp major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/65c98317-00d2-4ba8-8f66-9b34d720c553/volumes/kubernetes.io~projected/kube-api-access-78hn4:{mountpoint:/var/lib/kubelet/pods/65c98317-00d2-4ba8-8f66-9b34d720c553/volumes/kubernetes.io~projected/kube-api-access-78hn4 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7306a726-d4c7-4ef5-b496-726e0bd488a1/volumes/kubernetes.io~projected/kube-api-access-9c64b:{mountpoint:/var/lib/kubelet/pods/7306a726-d4c7-4ef5-b496-726e0bd488a1/volumes/kubernetes.io~projected/kube-api-access-9c64b major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7306a726-d4c7-4ef5-b496-726e0bd488a1/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/7306a726-d4c7-4ef5-b496-726e0bd488a1/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/739472ea-5c22-44a3-9a81-acbf0755b3b4/volumes/kubernetes.io~projected/kube-api-access-l6xws:{mountpoint:/var/lib/kubelet/pods/739472ea-5c22-44a3-9a81-acbf0755b3b4/volumes/kubernetes.io~projected/kube-api-access-l6xws major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/944a350f-aa7e-4921-8ddc-b0001fa05086/volumes/kubernetes.io~projected/kube-api-access-5lq8r:{mountpoint:/var/lib/kubelet/pods/944a350f-aa7e-4921-8ddc-b0001fa05086/volumes/kubernetes.io~projected/kube-api-access-5lq8r major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a7702c04-56e6-4349-97f8-cd8c2efc76aa/volumes/kubernetes.io~projected/kube-api-access-c9x4n:{mountpoint:/var/lib/kubelet/pods/a7702c04-56e6-4349-97f8-cd8c2efc76aa/volumes/kubernetes.io~projected/kube-api-access-c9x4n major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a7702c04-56e6-4349-97f8-cd8c2efc76aa/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/a7702c04-56e6-4349-97f8-cd8c2efc76aa/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:268 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bb36e52c-ab3f-4c51-84e0-d943187ef234/volumes/kubernetes.io~projected/kube-api-access-tfwph:{mountpoint:/var/lib/kubelet/pods/bb36e52c-ab3f-4c51-84e0-d943187ef234/volumes/kubernetes.io~projected/kube-api-access-tfwph major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bc386a66-cdd2-4f4b-9666-88d48e42bd33/volumes/kubernetes.io~projected/kube-api-access-nxxt6:{mountpoint:/var/lib/kubelet/pods/bc386a66-cdd2-4f4b-9666-88d48e42bd33/volumes/kubernetes.io~projected/kube-api-access-nxxt6 major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bc386a66-cdd2-4f4b-9666-88d48e42bd33/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/bc386a66-cdd2-4f4b-9666-88d48e42bd33/volumes/kubernetes.io~secret/metrics-tls major:0 minor:259 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f2b26a37-2d01-495c-a3f3-12e5a0bbdca3/volumes/kubernetes.io~projected/kube-api-access-s7gq5:{mountpoint:/var/lib/kubelet/pods/f2b26a37-2d01-495c-a3f3-12e5a0bbdca3/volumes/kubernetes.io~projected/kube-api-access-s7gq5 major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f2b26a37-2d01-495c-a3f3-12e5a0bbdca3/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/f2b26a37-2d01-495c-a3f3-12e5a0bbdca3/volumes/kubernetes.io~secret/signing-key major:0 minor:215 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/a7728e26751a57dd94e33e8a497874cd3b1bd1a174368c589b03f004392d71a6/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/07f4d9af21174ce1f90f6d366f15ea25978a660e3e658379e376d6c280867b66/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/99007a346bbeca9e06ef6ad1eed5a4ad23b064a3205fdbc5375ba17a59430e3b/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/9386242ec0ab5fba261aacca6c507d156e2265892178cab39ae9284f13ca0da8/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/2a3f751c93c0dd33b00d59b6dea7c5a64b22500ce24b928298b9ee6bab847c75/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/b1eabf0901124d325d29d8b382ab699521ccd605a9e4c70f9bfc70ac1859ae56/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/8b784fdef3cefa00c92f6c85b50a19078ea8efe51de1a0f2e5610c89c7bb33c7/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/5768a375aed90c0fb325002713847666f68d243c900b851b9bdecf34d7e58df7/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/6d761d08ce1811a8b048d2d0be78b6baad79bf5d4feda680e8f8f5cfc9d0ed26/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/8b9f3d61c82dc8c2fe0ee05bc6efb972e8ab97130c79cf0d6d3b2420990d4884/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/ac00cb3b5e0bba44dfa837bdf73cb26a93ad742957145fe5aff3ce5decf278bf/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-140:{mountpoint:/var/lib/containers/storage/overlay/8be2f5a145f17c8cb37a2677fc07e20e740c9efe5c12d34736176a4e618d6036/merged major:0 minor:140 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/f2f0b518ed207316651615fd31bc71a9c03ddcde4b484f9aed4bf5b12dea383e/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/098ef65d1896f71318bbe2559594fed8f140d893444dac1eb1d2aaa581ac3772/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-161:{mountpoint:/var/lib/containers/storage/overlay/b25ec1e582dc3f77330b6b7fd5dea27520257b8c6a562ecedd82a4c56d2ba082/merged major:0 minor:161 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/868df164f5acbbdcb198fa9164a69ba531a6bb4b7742b4624af380fde0223e14/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/12de3b456a65369d9108badcb3092fd251c5f9cc84ef79571719ff261c10d356/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/e843e097de7b197056e1efcaad22553890084a05e2eae14b7b39f1f77f80350a/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/3cdc767b6fa363fe3dca6d4fd0c1c0a3b0119442d3cd3ec54b969e35ee7aa362/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/f61b5bd0e10a754687962caf3ac538cc85c138ce57019b1793d411ac286395bd/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/73ee6567e2912bea2eb9cee9aec6cba3ce1dca22723880dfc5010b13456d5cf1/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/f8ba5d42adabcb1f49e08d3024cd380a2bcec17883d4bdf8623990fd34d6ee3a/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/164a1eb1e98808a7b13f4a88a78cbc147d93099990adb6df57371edcbdb15abf/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/1e69c3460bcc33c69ce66855db95484ea74836fb3b072edc9ae2f63b5b5b0dbb/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-221:{mountpoint:/var/lib/containers/storage/overlay/c4475039871cd526b20b6590740f7c974d0f044ea6cad14bc2a14de9efda14f1/merged major:0 minor:221 fsType:overlay blockSize:0} overlay_0-223:{mountpoint:/var/lib/containers/storage/overlay/b0005b6755f26ed946b029cbd09d6ff4c14e35ca581ee0bcb7812103a0cfb28c/merged major:0 minor:223 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/2c84c4cc3d4eaa320a0728524bb84627018064d26331173c8b4cda0cb52c6e0d/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/aec77e7b94e3a74f520e34464f4444450225501792bb7bafb8be4c98eb38830a/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-239:{mountpoint:/var/lib/containers/storage/overlay/d43a9f932edd6b7666185e2ee925be0843a02fbe999dff321065a56ea6a22fbe/merged major:0 minor:239 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/16dcec550e4d7c5e8bc782fad302a34dd7dd1dd9d544111c100ffcaf7fa0be6a/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/61f54f5d8517cfdf5003b66041e3dc3c7c1e85e94aaadd16df9f0bfe7ad97937/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-249:{mountpoint:/var/lib/containers/storage/overlay/2706aea83972196f181a5e244953ae6514297c3b8475cdc066b66356bea94253/merged major:0 minor:249 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/efb77023951e510d82eb58aee3c44ce01a5374c2b33b85bbb835974d7a53d07b/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/26e2fcdd45d1638761ce338825a5afc28a854953ccbf209d3bc7a31fbc6ec79d/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/b075adb09408165dc96298e3d866359bd65ccf838b4c677a0d19439d353efe54/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-275:{mountpoint:/var/lib/containers/storage/overlay/9b1598f2965f65d87354bc1590c79a2a96123c9beb780e9078edc6b1774cc70b/merged major:0 minor:275 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/121c7f276edf55651c3a5d6b717614641d23910d80ce9aad367819bb3f34c1c0/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/0faf3493902838bd82f7a32857cce5270e26cdad82ec026c2ee25ee7a92be4b9/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/24b0e179bac49599bd8484f00d5fbffdbca86bdb6e3c3a1cc7f7d2d134965fbf/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/3e603a8ce295b6175b66f64ebfa28beba67d9126d92cc7acc4b011c7d357f6de/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/a629799195f6972078cd565b65c535c9b647b099ece6c7131f8180fb90c0128e/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/ff5c15207ff9de5ebe27554287e6eac98b1381bba3e322ca9d60aff6590934cc/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/a5f63f862f4d4495dd60d0b8b30f6c37228f34b59bc311cdd64e52e7251236f6/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/d577284075eff887094158a10c4bdb6b48cdf4134432f2117f4d488c72f9f697/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/13d65374e71b3c62334ea7da8dadf5543bdc4f7644e877092acfc29ee0a79124/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/bbfeed756ea43201be8c609ca4d4c9bf27e042c99c03cfad013b5e2048115b55/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 31 19:19:29.532539 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.532179    4645 manager.go:217] Machine: {Timestamp:2026-10-31 19:19:29.531665142 +0000 UTC m=+0.104899262 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099698 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29c520b7507a46ec0dbcadf7ef2614 SystemUUID:ec29c520-b750-7a46-ec0d-bcadf7ef2614 BootID:ae04324e-8c8a-4491-92de-58aa3c832361 Filesystems:[{Device:/run/containers/storage/overlay-containers/a0c01424ea5f36aa2a34e7031ba579e446014499dcfb3069f58f82930562ff2c/userdata/shm DeviceMajor:0 DeviceMinor:211 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f2b26a37-2d01-495c-a3f3-12e5a0bbdca3/volumes/kubernetes.io~projected/kube-api-access-s7gq5 DeviceMajor:0 DeviceMinor:216 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-221 DeviceMajor:0 DeviceMinor:221 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:225 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/739472ea-5c22-44a3-9a81-acbf0755b3b4/volumes/kubernetes.io~projected/kube-api-access-l6xws DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dbc6469e8e9ba3ce4167e2a94399129dab2c64360e1194f48abbd68d32cdda0e/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bc386a66-cdd2-4f4b-9666-88d48e42bd33/volumes/kubernetes.io~projected/kube-api-access-nxxt6 DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/87d4f3ec9faded6c0502d00c22586498bc6b8b44e0e3db741d6c1b45e313e9c6/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/be6e65fc286d720ef66bba2cd6c16bc5752194068f80469cda7aff0391b4731f/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7306a726-d4c7-4ef5-b496-726e0bd488a1/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6ee2e706596091c74ca0dd6f3acb765b21462c4df84997743d4e397e6b3a3a83/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/534f9972-b54f-4713-8326-4808fdf08386/volumes/kubernetes.io~projected/kube-api-access-npsmn DeviceMajor:0 DeviceMinor:232 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/876a4b2c55217a62653fbcffac89b3ff4f237029ca500213a384afde189b8b85/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b393b80365b08f5f32594138538a31bd0c471050a98f165fc897dec2a6813381/userdata/shm DeviceMajor:0 DeviceMinor:264 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0cc6092944f73c250c7507d1bf75f232ddcce10e190635a046ef1b4c3a94c244/userdata/shm DeviceMajor:0 DeviceMinor:269 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/65c98317-00d2-4ba8-8f66-9b34d720c553/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/997e378d63e20d073440ec716dd280bc3a55c382b061102ea561d2549410d145/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a7702c04-56e6-4349-97f8-cd8c2efc76aa/volumes/kubernetes.io~projected/kube-api-access-c9x4n DeviceMajor:0 DeviceMinor:209 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/611ab431-d82c-46e9-a108-4b221cc8c2cc/volumes/kubernetes.io~projected/kube-api-access-x2jxb DeviceMajor:0 DeviceMinor:210 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0467ace45db27d653773586f489f0e7107046a2129fdffc6696bce6e593dcd35/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2ae10cd8-0183-4aca-bf20-ecb1e375e420/volumes/kubernetes.io~projected/kube-api-access-86gj5 DeviceMajor:0 DeviceMinor:206 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-249 DeviceMajor:0 DeviceMinor:249 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bc386a66-cdd2-4f4b-9666-88d48e42bd33/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:259 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4220431d-fa8a-4537-ad2e-e8ba6164f63d/volumes/kubernetes.io~projected/kube-api-access-v4lbw DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3aef875c6fddb789272d58c909a8ff0173c68d56db9a79372057f2ec2ef89af0/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1bf8152bac50cc170567867a03d791329af8baa72a02537fbec3507f80ade260/userdata/shm DeviceMajor:0 DeviceMinor:233 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/05ca774c-33f8-4368-a677-7c1a98b61bf7/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/21dc10e8-ee61-46ae-83f7-67d1772f083c/volumes/kubernetes.io~projected/kube-api-access-7k8mg DeviceMajor:0 DeviceMinor:65 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-140 DeviceMajor:0 DeviceMinor:140 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-161 DeviceMajor:0 DeviceMinor:161 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/65c98317-00d2-4ba8-8f66-9b34d720c553/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a2c94b5cfad7d3b7908931167a8f363624b774ce25a5f0926c51d913febcd9bc/userdata/shm DeviceMajor:0 DeviceMinor:217 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/eab6b838c5183cc9562553ddea45990d0887348487d9cfd8fce770b50805e1a3/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7306a726-d4c7-4ef5-b496-726e0bd488a1/volumes/kubernetes.io~projected/kube-api-access-9c64b DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f2b26a37-2d01-495c-a3f3-12e5a0bbdca3/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:215 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/bb36e52c-ab3f-4c51-84e0-d943187ef234/volumes/kubernetes.io~projected/kube-api-access-tfwph DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d906967467612a30373fee26acf17e6cf18863f31120d951efa594bee7f3b425/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-239 DeviceMajor:0 DeviceMinor:239 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3bd73889613314ef4bb6e74fcfb8cc40e91533df36fbe8525771038110a642ba/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-223 DeviceMajor:0 DeviceMinor:223 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c/volumes/kubernetes.io~projected/kube-api-access-nq8nb DeviceMajor:0 DeviceMinor:62 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/638d688d-d5fc-480d-8c53-08ab2f1864b1/volumes/kubernetes.io~projected/kube-api-access-gk5qd DeviceMajor:0 DeviceMinor:63 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3382fc12-f376-4b15-95be-6cb371ae87bd/volumes/kubernetes.io~projected/kube-api-access-t62j5 DeviceMajor:0 DeviceMinor:64 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/65c98317-00d2-4ba8-8f66-9b34d720c553/volumes/kubernetes.io~projected/kube-api-access-78hn4 DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8a472eca945753737a9ab1ccdfc4f52f8c062ed3e92f413f175e77fc0861e6c9/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a7702c04-56e6-4349-97f8-cd8c2efc76aa/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:268 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5f67e14a39e17e0a90f3920596f0f7ceae2f8b34d976c64f044c50e4aa5dced4/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-275 DeviceMajor:0 DeviceMinor:275 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/944a350f-aa7e-4921-8ddc-b0001fa05086/volumes/kubernetes.io~projected/kube-api-access-5lq8r DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0cc6092944f73c2 MacAddress:2a:57:b1:6a:c5:94 Speed:10000 Mtu:8901} {Name:1bf8152bac50cc1 MacAddress:62:99:a7:30:e8:27 Speed:10000 Mtu:8901} {Name:3aef875c6fddb78 MacAddress:62:a4:a6:e4:ca:c5 Speed:10000 Mtu:8901} {Name:6ee2e706596091c MacAddress:8a:f0:f9:a8:80:a9 Speed:10000 Mtu:8901} {Name:8a472eca9457537 MacAddress:b6:4f:63:9b:04:8b Speed:10000 Mtu:8901} {Name:a0c01424ea5f36a MacAddress:2e:0a:e0:e6:c5:49 Speed:10000 Mtu:8901} {Name:a2c94b5cfad7d3b MacAddress:a2:82:e5:6e:96:f4 Speed:10000 Mtu:8901} {Name:b393b80365b08f5 MacAddress:9e:de:3f:03:65:ef Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:e3:e7:bf:69 Speed:0 Mtu:9001} {Name:br-int MacAddress:6a:08:5f:28:e1:f4 Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:ff:e3:e7:bf:69 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:76:0b:8a:50:4a:c8 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:ba:6a:7b:69:ee:3e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 31 19:19:29.532779 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.532548    4645 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 31 19:19:29.532779 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.532616    4645 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 31 19:19:29.533119 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.533093    4645 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 31 19:19:29.533352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.533122    4645 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-94.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 31 19:19:29.533421 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.533359    4645 topology_manager.go:143] "Creating topology manager with none policy"
Oct 31 19:19:29.533421 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.533371    4645 container_manager_linux.go:311] "Creating device plugin manager"
Oct 31 19:19:29.533421 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.533386    4645 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 19:19:29.533421 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.533406    4645 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 19:19:29.533548 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.533504    4645 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 31 19:19:29.533760 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.533740    4645 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 31 19:19:29.533919 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.533908    4645 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 31 19:19:29.533992 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.533979    4645 kubelet.go:493] "Attempting to sync node with API server"
Oct 31 19:19:29.534059 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.534004    4645 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 31 19:19:29.534059 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.534019    4645 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 31 19:19:29.534059 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.534032    4645 kubelet.go:405] "Adding apiserver pod source"
Oct 31 19:19:29.534059 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.534043    4645 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 31 19:19:29.535533 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.535511    4645 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 31 19:19:29.535533 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.535534    4645 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 19:19:29.536157 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.536134    4645 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 31 19:19:29.550780 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.550751    4645 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 19:19:29.551865 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.551845    4645 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 19:19:29.563946 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.563929    4645 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 31 19:19:29.563946 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.563948    4645 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 19:19:29.564105 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.564080    4645 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 31 19:19:29.564313 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.564297    4645 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 31 19:19:29.564376 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.564357    4645 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 31 19:19:29.564474 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.564463    4645 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 31 19:19:29.564516 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.564478    4645 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 31 19:19:29.564516 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.564485    4645 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 31 19:19:29.564516 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.564491    4645 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 31 19:19:29.564516 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.564499    4645 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 31 19:19:29.564516 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.564509    4645 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 31 19:19:29.564516 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.564516    4645 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 31 19:19:29.564712 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.564521    4645 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 31 19:19:29.564712 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.564529    4645 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 31 19:19:29.564712 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.564534    4645 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 31 19:19:29.564712 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.564542    4645 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 31 19:19:29.564712 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.564553    4645 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 31 19:19:29.564712 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.564586    4645 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 31 19:19:29.564712 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.564596    4645 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 31 19:19:29.568272 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.568256    4645 server.go:1263] "Started kubelet"
Oct 31 19:19:29.568539 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.568487    4645 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 31 19:19:29.568624 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.568555    4645 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 31 19:19:29.568624 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.568594    4645 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 31 19:19:29.568845 ip-10-0-131-94 systemd[1]: Started Kubernetes Kubelet.
Oct 31 19:19:29.569790 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.568972    4645 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 31 19:19:29.576629 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.569881    4645 server.go:317] "Adding debug handlers to kubelet server"
Oct 31 19:19:29.576629 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.575941    4645 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 31 19:19:29.576629 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.576217    4645 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 31 19:19:29.576629 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.576230    4645 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 31 19:19:29.576629 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.576251    4645 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 31 19:19:29.577419 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.576229    4645 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 31 19:19:29.577419 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.577203    4645 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 19:13:52 +0000 UTC" deadline="2028-05-02 02:15:48.165194773 +0000 UTC"
Oct 31 19:19:29.577419 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.577239    4645 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13158h56m18.587959317s"
Oct 31 19:19:29.578890 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.578829    4645 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c" volumeName="kubernetes.io/configmap/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-ovnkube-config" seLinuxMountContext=""
Oct 31 19:19:29.578983 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.578888    4645 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4220431d-fa8a-4537-ad2e-e8ba6164f63d" volumeName="kubernetes.io/configmap/4220431d-fa8a-4537-ad2e-e8ba6164f63d-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 31 19:19:29.578983 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.578909    4645 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="611ab431-d82c-46e9-a108-4b221cc8c2cc" volumeName="kubernetes.io/configmap/611ab431-d82c-46e9-a108-4b221cc8c2cc-trusted-ca-bundle" seLinuxMountContext=""
Oct 31 19:19:29.578983 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.578925    4645 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="739472ea-5c22-44a3-9a81-acbf0755b3b4" volumeName="kubernetes.io/projected/739472ea-5c22-44a3-9a81-acbf0755b3b4-kube-api-access-l6xws" seLinuxMountContext=""
Oct 31 19:19:29.578983 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.578941    4645 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a7702c04-56e6-4349-97f8-cd8c2efc76aa" volumeName="kubernetes.io/secret/a7702c04-56e6-4349-97f8-cd8c2efc76aa-insights-runtime-extractor-tls" seLinuxMountContext=""
Oct 31 19:19:29.578983 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.578954    4645 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc386a66-cdd2-4f4b-9666-88d48e42bd33" volumeName="kubernetes.io/empty-dir/bc386a66-cdd2-4f4b-9666-88d48e42bd33-tmp-dir" seLinuxMountContext=""
Oct 31 19:19:29.578983 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.578965    4645 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc386a66-cdd2-4f4b-9666-88d48e42bd33" volumeName="kubernetes.io/secret/bc386a66-cdd2-4f4b-9666-88d48e42bd33-metrics-tls" seLinuxMountContext=""
Oct 31 19:19:29.578983 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.578977    4645 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c" volumeName="kubernetes.io/projected/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-kube-api-access-nq8nb" seLinuxMountContext=""
Oct 31 19:19:29.579288 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.578992    4645 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761" volumeName="kubernetes.io/secret/4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761-original-pull-secret" seLinuxMountContext=""
Oct 31 19:19:29.579288 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.579007    4645 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="534f9972-b54f-4713-8326-4808fdf08386" volumeName="kubernetes.io/projected/534f9972-b54f-4713-8326-4808fdf08386-kube-api-access-npsmn" seLinuxMountContext=""
Oct 31 19:19:29.579288 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.579017    4645 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65c98317-00d2-4ba8-8f66-9b34d720c553" volumeName="kubernetes.io/projected/65c98317-00d2-4ba8-8f66-9b34d720c553-kube-api-access-78hn4" seLinuxMountContext=""
Oct 31 19:19:29.579288 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.579028    4645 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a7702c04-56e6-4349-97f8-cd8c2efc76aa" volumeName="kubernetes.io/configmap/a7702c04-56e6-4349-97f8-cd8c2efc76aa-kube-rbac-proxy-cm" seLinuxMountContext=""
Oct 31 19:19:29.579288 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.579040    4645 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a7702c04-56e6-4349-97f8-cd8c2efc76aa" volumeName="kubernetes.io/empty-dir/a7702c04-56e6-4349-97f8-cd8c2efc76aa-data-volume" seLinuxMountContext=""
Oct 31 19:19:29.579288 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.579079    4645 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2b26a37-2d01-495c-a3f3-12e5a0bbdca3" volumeName="kubernetes.io/configmap/f2b26a37-2d01-495c-a3f3-12e5a0bbdca3-signing-cabundle" seLinuxMountContext=""
Oct 31 19:19:29.579288 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.579095    4645 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="05ca774c-33f8-4368-a677-7c1a98b61bf7" volumeName="kubernetes.io/configmap/05ca774c-33f8-4368-a677-7c1a98b61bf7-konnectivity-ca" seLinuxMountContext=""
Oct 31 19:19:29.579288 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.579118    4645 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c" volumeName="kubernetes.io/configmap/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-env-overrides" seLinuxMountContext=""
Oct 31 19:19:29.579288 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.579132    4645 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="611ab431-d82c-46e9-a108-4b221cc8c2cc" volumeName="kubernetes.io/empty-dir/611ab431-d82c-46e9-a108-4b221cc8c2cc-archives-path" seLinuxMountContext=""
Oct 31 19:19:29.579288 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.579150    4645 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="739472ea-5c22-44a3-9a81-acbf0755b3b4" volumeName="kubernetes.io/configmap/739472ea-5c22-44a3-9a81-acbf0755b3b4-iptables-alerter-script" seLinuxMountContext=""
Oct 31 19:19:29.579288 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.579161    4645 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="944a350f-aa7e-4921-8ddc-b0001fa05086" volumeName="kubernetes.io/projected/944a350f-aa7e-4921-8ddc-b0001fa05086-kube-api-access-5lq8r" seLinuxMountContext=""
Oct 31 19:19:29.579288 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.579172    4645 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2b26a37-2d01-495c-a3f3-12e5a0bbdca3" volumeName="kubernetes.io/projected/f2b26a37-2d01-495c-a3f3-12e5a0bbdca3-kube-api-access-s7gq5" seLinuxMountContext=""
Oct 31 19:19:29.579288 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.579184    4645 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7306a726-d4c7-4ef5-b496-726e0bd488a1" volumeName="kubernetes.io/projected/7306a726-d4c7-4ef5-b496-726e0bd488a1-kube-api-access-9c64b" seLinuxMountContext=""
Oct 31 19:19:29.579288 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.579198    4645 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21dc10e8-ee61-46ae-83f7-67d1772f083c" volumeName="kubernetes.io/projected/21dc10e8-ee61-46ae-83f7-67d1772f083c-kube-api-access-7k8mg" seLinuxMountContext=""
Oct 31 19:19:29.579288 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.579223    4645 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="638d688d-d5fc-480d-8c53-08ab2f1864b1" volumeName="kubernetes.io/projected/638d688d-d5fc-480d-8c53-08ab2f1864b1-kube-api-access-gk5qd" seLinuxMountContext=""
Oct 31 19:19:29.579288 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.579234    4645 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="944a350f-aa7e-4921-8ddc-b0001fa05086" volumeName="kubernetes.io/configmap/944a350f-aa7e-4921-8ddc-b0001fa05086-serviceca" seLinuxMountContext=""
Oct 31 19:19:29.579288 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.579245    4645 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bb36e52c-ab3f-4c51-84e0-d943187ef234" volumeName="kubernetes.io/projected/bb36e52c-ab3f-4c51-84e0-d943187ef234-kube-api-access-tfwph" seLinuxMountContext=""
Oct 31 19:19:29.579288 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.579259    4645 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="05ca774c-33f8-4368-a677-7c1a98b61bf7" volumeName="kubernetes.io/secret/05ca774c-33f8-4368-a677-7c1a98b61bf7-agent-certs" seLinuxMountContext=""
Oct 31 19:19:29.579288 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.579271    4645 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="611ab431-d82c-46e9-a108-4b221cc8c2cc" volumeName="kubernetes.io/projected/611ab431-d82c-46e9-a108-4b221cc8c2cc-kube-api-access-x2jxb" seLinuxMountContext=""
Oct 31 19:19:29.579288 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.579285    4645 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3382fc12-f376-4b15-95be-6cb371ae87bd" volumeName="kubernetes.io/projected/3382fc12-f376-4b15-95be-6cb371ae87bd-kube-api-access-t62j5" seLinuxMountContext=""
Oct 31 19:19:29.580164 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.579312    4645 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a7702c04-56e6-4349-97f8-cd8c2efc76aa" volumeName="kubernetes.io/projected/a7702c04-56e6-4349-97f8-cd8c2efc76aa-kube-api-access-c9x4n" seLinuxMountContext=""
Oct 31 19:19:29.580164 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.579349    4645 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c" volumeName="kubernetes.io/secret/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 31 19:19:29.580164 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.579365    4645 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ae10cd8-0183-4aca-bf20-ecb1e375e420" volumeName="kubernetes.io/projected/2ae10cd8-0183-4aca-bf20-ecb1e375e420-kube-api-access-86gj5" seLinuxMountContext=""
Oct 31 19:19:29.580164 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.579379    4645 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4220431d-fa8a-4537-ad2e-e8ba6164f63d" volumeName="kubernetes.io/projected/4220431d-fa8a-4537-ad2e-e8ba6164f63d-kube-api-access-v4lbw" seLinuxMountContext=""
Oct 31 19:19:29.580164 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.579391    4645 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65c98317-00d2-4ba8-8f66-9b34d720c553" volumeName="kubernetes.io/empty-dir/65c98317-00d2-4ba8-8f66-9b34d720c553-tmp" seLinuxMountContext=""
Oct 31 19:19:29.580164 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.579407    4645 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bb36e52c-ab3f-4c51-84e0-d943187ef234" volumeName="kubernetes.io/configmap/bb36e52c-ab3f-4c51-84e0-d943187ef234-cni-binary-copy" seLinuxMountContext=""
Oct 31 19:19:29.580164 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.579420    4645 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bb36e52c-ab3f-4c51-84e0-d943187ef234" volumeName="kubernetes.io/configmap/bb36e52c-ab3f-4c51-84e0-d943187ef234-multus-daemon-config" seLinuxMountContext=""
Oct 31 19:19:29.580164 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.579431    4645 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc386a66-cdd2-4f4b-9666-88d48e42bd33" volumeName="kubernetes.io/projected/bc386a66-cdd2-4f4b-9666-88d48e42bd33-kube-api-access-nxxt6" seLinuxMountContext=""
Oct 31 19:19:29.580164 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.579445    4645 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f2b26a37-2d01-495c-a3f3-12e5a0bbdca3" volumeName="kubernetes.io/secret/f2b26a37-2d01-495c-a3f3-12e5a0bbdca3-signing-key" seLinuxMountContext=""
Oct 31 19:19:29.580164 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.579460    4645 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7306a726-d4c7-4ef5-b496-726e0bd488a1" volumeName="kubernetes.io/secret/7306a726-d4c7-4ef5-b496-726e0bd488a1-cert" seLinuxMountContext=""
Oct 31 19:19:29.580164 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.579472    4645 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c" volumeName="kubernetes.io/configmap/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-ovnkube-script-lib" seLinuxMountContext=""
Oct 31 19:19:29.580164 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.579485    4645 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21dc10e8-ee61-46ae-83f7-67d1772f083c" volumeName="kubernetes.io/empty-dir/21dc10e8-ee61-46ae-83f7-67d1772f083c-tmp-dir" seLinuxMountContext=""
Oct 31 19:19:29.580164 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.579496    4645 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4220431d-fa8a-4537-ad2e-e8ba6164f63d" volumeName="kubernetes.io/configmap/4220431d-fa8a-4537-ad2e-e8ba6164f63d-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 31 19:19:29.580164 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.579506    4645 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4220431d-fa8a-4537-ad2e-e8ba6164f63d" volumeName="kubernetes.io/configmap/4220431d-fa8a-4537-ad2e-e8ba6164f63d-cni-binary-copy" seLinuxMountContext=""
Oct 31 19:19:29.580164 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.579518    4645 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="611ab431-d82c-46e9-a108-4b221cc8c2cc" volumeName="kubernetes.io/configmap/611ab431-d82c-46e9-a108-4b221cc8c2cc-service-ca-bundle" seLinuxMountContext=""
Oct 31 19:19:29.580164 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.579529    4645 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65c98317-00d2-4ba8-8f66-9b34d720c553" volumeName="kubernetes.io/empty-dir/65c98317-00d2-4ba8-8f66-9b34d720c553-etc-tuned" seLinuxMountContext=""
Oct 31 19:19:29.580164 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.579540    4645 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc386a66-cdd2-4f4b-9666-88d48e42bd33" volumeName="kubernetes.io/configmap/bc386a66-cdd2-4f4b-9666-88d48e42bd33-config-volume" seLinuxMountContext=""
Oct 31 19:19:29.580164 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.579554    4645 reconstruct.go:97] "Volume reconstruction finished"
Oct 31 19:19:29.580164 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.579561    4645 reconciler.go:29] "Reconciler: start to sync state"
Oct 31 19:19:29.580164 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.580074    4645 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 19:19:29.581059 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.580582    4645 factory.go:55] Registering systemd factory
Oct 31 19:19:29.581059 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.580601    4645 factory.go:223] Registration of the systemd container factory successfully
Oct 31 19:19:29.581199 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.581070    4645 factory.go:153] Registering CRI-O factory
Oct 31 19:19:29.581199 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.581085    4645 factory.go:223] Registration of the crio container factory successfully
Oct 31 19:19:29.581199 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.581153    4645 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 31 19:19:29.581199 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.581174    4645 factory.go:103] Registering Raw factory
Oct 31 19:19:29.581199 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.581191    4645 manager.go:1199] Started watching for new ooms in manager
Oct 31 19:19:29.581617 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.581601    4645 manager.go:319] Starting recovery of all containers
Oct 31 19:19:29.583504 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.583488    4645 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 31 19:19:29.603304 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.603259    4645 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 31 19:19:29.609895 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.609878    4645 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 31 19:19:29.609895 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.609896    4645 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 31 19:19:29.610038 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.609917    4645 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 31 19:19:29.610038 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:19:29.609952    4645 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 31 19:19:29.613008 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.612911    4645 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 19:19:29.638494 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.638421    4645 generic.go:356] "Generic (PLEG): container finished" podID="4220431d-fa8a-4537-ad2e-e8ba6164f63d" containerID="882b36257883d1591901e16418d2f12d7867606430a4be6c04b1ce31654522bd" exitCode=0
Oct 31 19:19:29.638494 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.638457    4645 generic.go:356] "Generic (PLEG): container finished" podID="4220431d-fa8a-4537-ad2e-e8ba6164f63d" containerID="7c8996024040b83c5757bdcfc7f294a59d6eb2e1caf731697c9380ee34bbd28b" exitCode=0
Oct 31 19:19:29.638494 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.638468    4645 generic.go:356] "Generic (PLEG): container finished" podID="4220431d-fa8a-4537-ad2e-e8ba6164f63d" containerID="cf68d9b3ce3e4d8b88576a14589470a4535237dc1f5613b600a360d6c24ddd0f" exitCode=0
Oct 31 19:19:29.638494 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.638477    4645 generic.go:356] "Generic (PLEG): container finished" podID="4220431d-fa8a-4537-ad2e-e8ba6164f63d" containerID="2918bc81c9d36348f497b1d302fccfa18576a074400c9dd7bfac0e1ecc5beec7" exitCode=0
Oct 31 19:19:29.638743 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.638486    4645 generic.go:356] "Generic (PLEG): container finished" podID="4220431d-fa8a-4537-ad2e-e8ba6164f63d" containerID="4ea3f314bdcfc93fe084b6a9373d43c3f8ceae3e660077e8d5407373f441256b" exitCode=0
Oct 31 19:19:29.638743 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.638516    4645 generic.go:356] "Generic (PLEG): container finished" podID="4220431d-fa8a-4537-ad2e-e8ba6164f63d" containerID="82a00c2cc5a8d8da2428ecfe301c8db1cceee60d38dfc907a0042f1e886cb6e1" exitCode=0
Oct 31 19:19:29.654638 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.654608    4645 generic.go:356] "Generic (PLEG): container finished" podID="96d3baf5b18698617651d9af1ffbc2b2" containerID="97c4bdb66b920b35f4643083a28b5d46197e4d5b237e67dc5623c1cfce419d03" exitCode=0
Oct 31 19:19:29.671012 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.670991    4645 manager.go:324] Recovery completed
Oct 31 19:19:29.692624 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.692601    4645 cpu_manager.go:226] "Starting" policy="none"
Oct 31 19:19:29.692624 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.692620    4645 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 31 19:19:29.692778 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.692641    4645 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 31 19:19:29.692825 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.692790    4645 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 31 19:19:29.693821 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.692807    4645 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 31 19:19:29.693917 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.693824    4645 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 31 19:19:29.693917 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.693834    4645 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 31 19:19:29.693917 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.693852    4645 policy_none.go:50] "Start"
Oct 31 19:19:29.693917 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.693876    4645 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 31 19:19:29.694078 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.693934    4645 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 31 19:19:29.707989 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.707969    4645 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 31 19:19:29.708070 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.707990    4645 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 31 19:19:29.708070 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.708011    4645 policy_none.go:44] "Start"
Oct 31 19:19:29.710041 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:19:29.710022    4645 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Oct 31 19:19:29.713190 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.713164    4645 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 31 19:19:29.716957 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.716938    4645 manager.go:341] "Starting Device Plugin manager"
Oct 31 19:19:29.717030 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:19:29.716969    4645 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 31 19:19:29.717030 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.716980    4645 server.go:87] "Starting device plugin registration server"
Oct 31 19:19:29.717312 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.717289    4645 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 31 19:19:29.717312 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.717302    4645 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 31 19:19:29.717466 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.717446    4645 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 31 19:19:29.717564 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.717532    4645 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 31 19:19:29.717605 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.717576    4645 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 31 19:19:29.717605 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.717581    4645 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 31 19:19:29.717671 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.717619    4645 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T19:19:29.717562625Z","UUID":"85bbf83e-a886-43e4-a59b-9cad8f65969d","Handler":null,"Name":"","Endpoint":""}
Oct 31 19:19:29.719181 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:19:29.719162    4645 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 31 19:19:29.719626 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.719608    4645 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 31 19:19:29.719729 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.719633    4645 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 31 19:19:29.817573 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.817545    4645 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 19:19:29.818788 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.818767    4645 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-94.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 19:19:29.818894 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.818799    4645 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-94.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 19:19:29.818894 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.818814    4645 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-94.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 19:19:29.818894 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.818858    4645 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-94.us-east-2.compute.internal"
Oct 31 19:19:29.830481 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.830457    4645 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-131-94.us-east-2.compute.internal"
Oct 31 19:19:29.830596 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.830525    4645 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-131-94.us-east-2.compute.internal"
Oct 31 19:19:29.910972 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.910889    4645 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-131-94.us-east-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-94.us-east-2.compute.internal"]
Oct 31 19:19:29.911456 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.911437    4645 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-94.us-east-2.compute.internal"
Oct 31 19:19:29.911510 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.911439    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-94.us-east-2.compute.internal" event={"ID":"b8f50d0470923e3e6371360e86f8acbb","Type":"ContainerStarted","Data":"a90e822a8c267cc26e074ae0665ecaf6e6b07b83798f3059a45a59fdfe84a75f"}
Oct 31 19:19:29.911510 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.911481    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-94.us-east-2.compute.internal" event={"ID":"b8f50d0470923e3e6371360e86f8acbb","Type":"ContainerStarted","Data":"87d4f3ec9faded6c0502d00c22586498bc6b8b44e0e3db741d6c1b45e313e9c6"}
Oct 31 19:19:29.911583 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.911571    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-94.us-east-2.compute.internal" event={"ID":"96d3baf5b18698617651d9af1ffbc2b2","Type":"ContainerStarted","Data":"990c841d8c81060041b3326cff15450569830cd6e0c119972e652af0688e0877"}
Oct 31 19:19:29.911613 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.911589    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-94.us-east-2.compute.internal" event={"ID":"96d3baf5b18698617651d9af1ffbc2b2","Type":"ContainerDied","Data":"97c4bdb66b920b35f4643083a28b5d46197e4d5b237e67dc5623c1cfce419d03"}
Oct 31 19:19:29.911613 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.911607    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-94.us-east-2.compute.internal" event={"ID":"96d3baf5b18698617651d9af1ffbc2b2","Type":"ContainerStarted","Data":"876a4b2c55217a62653fbcffac89b3ff4f237029ca500213a384afde189b8b85"}
Oct 31 19:19:29.911693 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.911679    4645 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-94.us-east-2.compute.internal"
Oct 31 19:19:29.923021 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.922997    4645 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 19:19:29.923092 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:19:29.923037    4645 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-131-94.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-94.us-east-2.compute.internal"
Oct 31 19:19:29.923277 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.923265    4645 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 19:19:29.923312 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:19:29.923292    4645 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-131-94.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-94.us-east-2.compute.internal"
Oct 31 19:19:29.986559 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.986532    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b8f50d0470923e3e6371360e86f8acbb-config\") pod \"kube-apiserver-proxy-ip-10-0-131-94.us-east-2.compute.internal\" (UID: \"b8f50d0470923e3e6371360e86f8acbb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-94.us-east-2.compute.internal"
Oct 31 19:19:29.986647 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.986571    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/96d3baf5b18698617651d9af1ffbc2b2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-94.us-east-2.compute.internal\" (UID: \"96d3baf5b18698617651d9af1ffbc2b2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-94.us-east-2.compute.internal"
Oct 31 19:19:29.986647 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:29.986595    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/96d3baf5b18698617651d9af1ffbc2b2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-94.us-east-2.compute.internal\" (UID: \"96d3baf5b18698617651d9af1ffbc2b2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-94.us-east-2.compute.internal"
Oct 31 19:19:30.087107 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.087084    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/96d3baf5b18698617651d9af1ffbc2b2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-94.us-east-2.compute.internal\" (UID: \"96d3baf5b18698617651d9af1ffbc2b2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-94.us-east-2.compute.internal"
Oct 31 19:19:30.087183 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.087110    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b8f50d0470923e3e6371360e86f8acbb-config\") pod \"kube-apiserver-proxy-ip-10-0-131-94.us-east-2.compute.internal\" (UID: \"b8f50d0470923e3e6371360e86f8acbb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-94.us-east-2.compute.internal"
Oct 31 19:19:30.087183 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.087127    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/96d3baf5b18698617651d9af1ffbc2b2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-94.us-east-2.compute.internal\" (UID: \"96d3baf5b18698617651d9af1ffbc2b2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-94.us-east-2.compute.internal"
Oct 31 19:19:30.087183 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.087171    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/96d3baf5b18698617651d9af1ffbc2b2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-94.us-east-2.compute.internal\" (UID: \"96d3baf5b18698617651d9af1ffbc2b2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-94.us-east-2.compute.internal"
Oct 31 19:19:30.087373 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.087186    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/96d3baf5b18698617651d9af1ffbc2b2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-94.us-east-2.compute.internal\" (UID: \"96d3baf5b18698617651d9af1ffbc2b2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-94.us-east-2.compute.internal"
Oct 31 19:19:30.087373 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.087195    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b8f50d0470923e3e6371360e86f8acbb-config\") pod \"kube-apiserver-proxy-ip-10-0-131-94.us-east-2.compute.internal\" (UID: \"b8f50d0470923e3e6371360e86f8acbb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-94.us-east-2.compute.internal"
Oct 31 19:19:30.535207 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.535174    4645 apiserver.go:52] "Watching apiserver"
Oct 31 19:19:30.541837 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.541815    4645 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 31 19:19:30.542308 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.542287    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-6s8ft","kube-system/kube-apiserver-proxy-ip-10-0-131-94.us-east-2.compute.internal","openshift-ingress-canary/ingress-canary-2v47p","openshift-network-operator/iptables-alerter-c5fnq","openshift-multus/multus-hhnhg","openshift-multus/network-metrics-daemon-j6fmb","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tmtsr","kube-system/global-pull-secret-syncer-4pntt","openshift-insights/insights-runtime-extractor-kmlmk","openshift-service-ca/service-ca-5fffc97bdf-hwvnh","kube-system/konnectivity-agent-kc9mq","openshift-image-registry/node-ca-5mwk4","openshift-kube-storage-version-migrator/migrator-66df5d496b-tztj8","openshift-cluster-node-tuning-operator/tuned-kzv9d","openshift-insights/periodic-gathering-j9l2c-hmgd8","openshift-dns/node-resolver-58f52","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-94.us-east-2.compute.internal","openshift-network-diagnostics/network-check-target-4j5p9","openshift-ovn-kubernetes/ovnkube-node-mr9n2","openshift-dns/dns-default-kjlt7"]
Oct 31 19:19:30.544304 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.544281    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j6fmb"
Oct 31 19:19:30.545824 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.545804    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 31 19:19:30.546099 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.546085    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 31 19:19:30.546502 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.546470    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 31 19:19:30.546502 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.546487    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-rs77w\""
Oct 31 19:19:30.546627 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.546574    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-zwg2q\""
Oct 31 19:19:30.550977 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.549911    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-kbgxr\""
Oct 31 19:19:30.550977 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.550589    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 31 19:19:30.551217 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.551198    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 31 19:19:30.551920 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.551897    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-w5n5s\""
Oct 31 19:19:30.552188 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.552158    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 31 19:19:30.552290 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.552253    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 31 19:19:30.552388 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.552369    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 31 19:19:30.552436 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.552385    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 19:19:30.552498 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.552452    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 19:19:30.552498 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.552475    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-j7nlh\""
Oct 31 19:19:30.552498 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.552484    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 31 19:19:30.552663 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.552499    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 31 19:19:30.552663 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.552517    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 19:19:30.552663 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.552530    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 31 19:19:30.552663 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.552548    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 31 19:19:30.552663 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.552559    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-dcmmt\""
Oct 31 19:19:30.552663 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.552589    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 31 19:19:30.552663 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.552596    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 31 19:19:30.552663 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.552621    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9gfbm\""
Oct 31 19:19:30.552663 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.552627    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 31 19:19:30.552663 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.552645    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 31 19:19:30.552954 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.552682    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 31 19:19:30.552954 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.552691    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 31 19:19:30.552954 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.552700    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jd7hq\""
Oct 31 19:19:30.552954 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.552690    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 31 19:19:30.552954 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.552738    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 31 19:19:30.552954 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.552744    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 31 19:19:30.552954 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.552749    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 31 19:19:30.552954 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.552765    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 31 19:19:30.552954 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.552782    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 31 19:19:30.552954 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.552886    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 31 19:19:30.552954 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.552921    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 31 19:19:30.552954 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.552951    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 31 19:19:30.553270 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.553034    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 31 19:19:30.553270 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.553092    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-4hvl4\""
Oct 31 19:19:30.553270 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.553122    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 31 19:19:30.553385 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.553290    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-cs8fb\""
Oct 31 19:19:30.553385 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.553353    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 19:19:30.553385 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.553379    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-mpqbj\""
Oct 31 19:19:30.553503 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.553382    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-nngmt\""
Oct 31 19:19:30.553503 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.553439    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 31 19:19:30.553503 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.553466    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-mrxc5\""
Oct 31 19:19:30.553503 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.553477    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 31 19:19:30.553503 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.553484    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-q7xwb\""
Oct 31 19:19:30.553684 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.553530    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 31 19:19:30.553684 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.553562    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 31 19:19:30.553684 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.553607    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 31 19:19:30.553838 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.553817    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-bbst6\""
Oct 31 19:19:30.554010 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.553997    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 31 19:19:30.554218 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.554204    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 31 19:19:30.554218 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.554213    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mxmqk\""
Oct 31 19:19:30.554304 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.554252    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 31 19:19:30.554426 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.554415    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-zz8zc\""
Oct 31 19:19:30.555792 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.555775    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 31 19:19:30.555859 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.555851    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 31 19:19:30.555923 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.555860    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 31 19:19:30.555923 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.555881    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 31 19:19:30.555923 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.555885    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 31 19:19:30.555923 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.555899    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 31 19:19:30.556093 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.556060    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-rvvtz\""
Oct 31 19:19:30.557917 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.557898    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 31 19:19:30.577870 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.577851    4645 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 31 19:19:30.589908 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.589889    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65c98317-00d2-4ba8-8f66-9b34d720c553-tmp\") pod \"tuned-kzv9d\" (UID: \"65c98317-00d2-4ba8-8f66-9b34d720c553\") " pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:19:30.589996 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.589915    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk5qd\" (UniqueName: \"kubernetes.io/projected/638d688d-d5fc-480d-8c53-08ab2f1864b1-kube-api-access-gk5qd\") pod \"aws-ebs-csi-driver-node-tmtsr\" (UID: \"638d688d-d5fc-480d-8c53-08ab2f1864b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tmtsr"
Oct 31 19:19:30.589996 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.589935    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-systemd-units\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:19:30.589996 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.589950    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-node-log\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:19:30.589996 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.589966    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-ovnkube-script-lib\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:19:30.589996 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.589986    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/739472ea-5c22-44a3-9a81-acbf0755b3b4-host-slash\") pod \"iptables-alerter-c5fnq\" (UID: \"739472ea-5c22-44a3-9a81-acbf0755b3b4\") " pod="openshift-network-operator/iptables-alerter-c5fnq"
Oct 31 19:19:30.590212 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.590005    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7gq5\" (UniqueName: \"kubernetes.io/projected/f2b26a37-2d01-495c-a3f3-12e5a0bbdca3-kube-api-access-s7gq5\") pod \"service-ca-5fffc97bdf-hwvnh\" (UID: \"f2b26a37-2d01-495c-a3f3-12e5a0bbdca3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hwvnh"
Oct 31 19:19:30.590212 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.590050    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k8mg\" (UniqueName: \"kubernetes.io/projected/21dc10e8-ee61-46ae-83f7-67d1772f083c-kube-api-access-7k8mg\") pod \"node-resolver-58f52\" (UID: \"21dc10e8-ee61-46ae-83f7-67d1772f083c\") " pod="openshift-dns/node-resolver-58f52"
Oct 31 19:19:30.590212 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.590130    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7306a726-d4c7-4ef5-b496-726e0bd488a1-cert\") pod \"ingress-canary-2v47p\" (UID: \"7306a726-d4c7-4ef5-b496-726e0bd488a1\") " pod="openshift-ingress-canary/ingress-canary-2v47p"
Oct 31 19:19:30.590212 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.590163    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65c98317-00d2-4ba8-8f66-9b34d720c553-tmp\") pod \"tuned-kzv9d\" (UID: \"65c98317-00d2-4ba8-8f66-9b34d720c553\") " pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:19:30.590212 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.590169    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a7702c04-56e6-4349-97f8-cd8c2efc76aa-data-volume\") pod \"insights-runtime-extractor-kmlmk\" (UID: \"a7702c04-56e6-4349-97f8-cd8c2efc76aa\") " pod="openshift-insights/insights-runtime-extractor-kmlmk"
Oct 31 19:19:30.590404 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.590215    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/65c98317-00d2-4ba8-8f66-9b34d720c553-etc-systemd\") pod \"tuned-kzv9d\" (UID: \"65c98317-00d2-4ba8-8f66-9b34d720c553\") " pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:19:30.590404 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.590259    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-ovnkube-script-lib\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:19:30.590404 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.590339    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/65c98317-00d2-4ba8-8f66-9b34d720c553-run\") pod \"tuned-kzv9d\" (UID: \"65c98317-00d2-4ba8-8f66-9b34d720c553\") " pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:19:30.590404 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.590373    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a7702c04-56e6-4349-97f8-cd8c2efc76aa-data-volume\") pod \"insights-runtime-extractor-kmlmk\" (UID: \"a7702c04-56e6-4349-97f8-cd8c2efc76aa\") " pod="openshift-insights/insights-runtime-extractor-kmlmk"
Oct 31 19:19:30.590404 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.590382    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bb36e52c-ab3f-4c51-84e0-d943187ef234-host-run-k8s-cni-cncf-io\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:19:30.590542 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.590429    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7306a726-d4c7-4ef5-b496-726e0bd488a1-cert\") pod \"ingress-canary-2v47p\" (UID: \"7306a726-d4c7-4ef5-b496-726e0bd488a1\") " pod="openshift-ingress-canary/ingress-canary-2v47p"
Oct 31 19:19:30.590542 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.590448    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bb36e52c-ab3f-4c51-84e0-d943187ef234-host-run-netns\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:19:30.590542 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.590469    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21dc10e8-ee61-46ae-83f7-67d1772f083c-tmp-dir\") pod \"node-resolver-58f52\" (UID: \"21dc10e8-ee61-46ae-83f7-67d1772f083c\") " pod="openshift-dns/node-resolver-58f52"
Oct 31 19:19:30.590542 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.590493    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-log-socket\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:19:30.590542 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.590539    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21dc10e8-ee61-46ae-83f7-67d1772f083c-tmp-dir\") pod \"node-resolver-58f52\" (UID: \"21dc10e8-ee61-46ae-83f7-67d1772f083c\") " pod="openshift-dns/node-resolver-58f52"
Oct 31 19:19:30.590676 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.590545    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-host-cni-netd\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:19:30.590676 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.590583    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-ovnkube-config\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:19:30.590676 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.590606    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4220431d-fa8a-4537-ad2e-e8ba6164f63d-system-cni-dir\") pod \"multus-additional-cni-plugins-6s8ft\" (UID: \"4220431d-fa8a-4537-ad2e-e8ba6164f63d\") " pod="openshift-multus/multus-additional-cni-plugins-6s8ft"
Oct 31 19:19:30.590676 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.590622    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4220431d-fa8a-4537-ad2e-e8ba6164f63d-cnibin\") pod \"multus-additional-cni-plugins-6s8ft\" (UID: \"4220431d-fa8a-4537-ad2e-e8ba6164f63d\") " pod="openshift-multus/multus-additional-cni-plugins-6s8ft"
Oct 31 19:19:30.590676 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.590661    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4220431d-fa8a-4537-ad2e-e8ba6164f63d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6s8ft\" (UID: \"4220431d-fa8a-4537-ad2e-e8ba6164f63d\") " pod="openshift-multus/multus-additional-cni-plugins-6s8ft"
Oct 31 19:19:30.590812 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.590699    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a7702c04-56e6-4349-97f8-cd8c2efc76aa-crio-socket\") pod \"insights-runtime-extractor-kmlmk\" (UID: \"a7702c04-56e6-4349-97f8-cd8c2efc76aa\") " pod="openshift-insights/insights-runtime-extractor-kmlmk"
Oct 31 19:19:30.590812 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.590732    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/05ca774c-33f8-4368-a677-7c1a98b61bf7-konnectivity-ca\") pod \"konnectivity-agent-kc9mq\" (UID: \"05ca774c-33f8-4368-a677-7c1a98b61bf7\") " pod="kube-system/konnectivity-agent-kc9mq"
Oct 31 19:19:30.590812 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.590765    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-ovnkube-config\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:19:30.590812 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.590770    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6xws\" (UniqueName: \"kubernetes.io/projected/739472ea-5c22-44a3-9a81-acbf0755b3b4-kube-api-access-l6xws\") pod \"iptables-alerter-c5fnq\" (UID: \"739472ea-5c22-44a3-9a81-acbf0755b3b4\") " pod="openshift-network-operator/iptables-alerter-c5fnq"
Oct 31 19:19:30.590988 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.590812    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/05ca774c-33f8-4368-a677-7c1a98b61bf7-agent-certs\") pod \"konnectivity-agent-kc9mq\" (UID: \"05ca774c-33f8-4368-a677-7c1a98b61bf7\") " pod="kube-system/konnectivity-agent-kc9mq"
Oct 31 19:19:30.590988 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.590829    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb36e52c-ab3f-4c51-84e0-d943187ef234-etc-kubernetes\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:19:30.590988 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.590868    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/638d688d-d5fc-480d-8c53-08ab2f1864b1-registration-dir\") pod \"aws-ebs-csi-driver-node-tmtsr\" (UID: \"638d688d-d5fc-480d-8c53-08ab2f1864b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tmtsr"
Oct 31 19:19:30.590988 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.590889    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3382fc12-f376-4b15-95be-6cb371ae87bd-metrics-certs\") pod \"network-metrics-daemon-j6fmb\" (UID: \"3382fc12-f376-4b15-95be-6cb371ae87bd\") " pod="openshift-multus/network-metrics-daemon-j6fmb"
Oct 31 19:19:30.590988 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.590911    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a7702c04-56e6-4349-97f8-cd8c2efc76aa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kmlmk\" (UID: \"a7702c04-56e6-4349-97f8-cd8c2efc76aa\") " pod="openshift-insights/insights-runtime-extractor-kmlmk"
Oct 31 19:19:30.590988 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.590937    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/611ab431-d82c-46e9-a108-4b221cc8c2cc-service-ca-bundle\") pod \"periodic-gathering-j9l2c-hmgd8\" (UID: \"611ab431-d82c-46e9-a108-4b221cc8c2cc\") " pod="openshift-insights/periodic-gathering-j9l2c-hmgd8"
Oct 31 19:19:30.590988 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.590962    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfwph\" (UniqueName: \"kubernetes.io/projected/bb36e52c-ab3f-4c51-84e0-d943187ef234-kube-api-access-tfwph\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:19:30.591381 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.591014    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4220431d-fa8a-4537-ad2e-e8ba6164f63d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6s8ft\" (UID: \"4220431d-fa8a-4537-ad2e-e8ba6164f63d\") " pod="openshift-multus/multus-additional-cni-plugins-6s8ft"
Oct 31 19:19:30.591381 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.591027    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/05ca774c-33f8-4368-a677-7c1a98b61bf7-konnectivity-ca\") pod \"konnectivity-agent-kc9mq\" (UID: \"05ca774c-33f8-4368-a677-7c1a98b61bf7\") " pod="kube-system/konnectivity-agent-kc9mq"
Oct 31 19:19:30.591381 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.591119    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npsmn\" (UniqueName: \"kubernetes.io/projected/534f9972-b54f-4713-8326-4808fdf08386-kube-api-access-npsmn\") pod \"network-check-target-4j5p9\" (UID: \"534f9972-b54f-4713-8326-4808fdf08386\") " pod="openshift-network-diagnostics/network-check-target-4j5p9"
Oct 31 19:19:30.591381 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.591154    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bb36e52c-ab3f-4c51-84e0-d943187ef234-multus-daemon-config\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:19:30.591381 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.591174    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/05ca774c-33f8-4368-a677-7c1a98b61bf7-agent-certs\") pod \"konnectivity-agent-kc9mq\" (UID: \"05ca774c-33f8-4368-a677-7c1a98b61bf7\") " pod="kube-system/konnectivity-agent-kc9mq"
Oct 31 19:19:30.591381 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.591267    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a7702c04-56e6-4349-97f8-cd8c2efc76aa-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kmlmk\" (UID: \"a7702c04-56e6-4349-97f8-cd8c2efc76aa\") " pod="openshift-insights/insights-runtime-extractor-kmlmk"
Oct 31 19:19:30.591381 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.591274    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/638d688d-d5fc-480d-8c53-08ab2f1864b1-device-dir\") pod \"aws-ebs-csi-driver-node-tmtsr\" (UID: \"638d688d-d5fc-480d-8c53-08ab2f1864b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tmtsr"
Oct 31 19:19:30.591381 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.591353    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lq8r\" (UniqueName: \"kubernetes.io/projected/944a350f-aa7e-4921-8ddc-b0001fa05086-kube-api-access-5lq8r\") pod \"node-ca-5mwk4\" (UID: \"944a350f-aa7e-4921-8ddc-b0001fa05086\") " pod="openshift-image-registry/node-ca-5mwk4"
Oct 31 19:19:30.591381 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.591373    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-host-run-netns\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:19:30.591770 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.591388    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-etc-openvswitch\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:19:30.591770 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.591404    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-host-cni-bin\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:19:30.591770 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.591472    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-env-overrides\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:19:30.591770 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.591508    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a7702c04-56e6-4349-97f8-cd8c2efc76aa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kmlmk\" (UID: \"a7702c04-56e6-4349-97f8-cd8c2efc76aa\") " pod="openshift-insights/insights-runtime-extractor-kmlmk"
Oct 31 19:19:30.591770 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.591527    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/65c98317-00d2-4ba8-8f66-9b34d720c553-etc-sysconfig\") pod \"tuned-kzv9d\" (UID: \"65c98317-00d2-4ba8-8f66-9b34d720c553\") " pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:19:30.591770 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.591549    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/65c98317-00d2-4ba8-8f66-9b34d720c553-etc-kubernetes\") pod \"tuned-kzv9d\" (UID: \"65c98317-00d2-4ba8-8f66-9b34d720c553\") " pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:19:30.591770 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.591548    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bb36e52c-ab3f-4c51-84e0-d943187ef234-multus-daemon-config\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:19:30.591770 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.591655    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/611ab431-d82c-46e9-a108-4b221cc8c2cc-service-ca-bundle\") pod \"periodic-gathering-j9l2c-hmgd8\" (UID: \"611ab431-d82c-46e9-a108-4b221cc8c2cc\") " pod="openshift-insights/periodic-gathering-j9l2c-hmgd8"
Oct 31 19:19:30.591770 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.591668    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bb36e52c-ab3f-4c51-84e0-d943187ef234-host-run-multus-certs\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:19:30.591770 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.591658    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-env-overrides\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:19:30.591770 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.591691    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bc386a66-cdd2-4f4b-9666-88d48e42bd33-config-volume\") pod \"dns-default-kjlt7\" (UID: \"bc386a66-cdd2-4f4b-9666-88d48e42bd33\") " pod="openshift-dns/dns-default-kjlt7"
Oct 31 19:19:30.591770 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.591724    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-var-lib-openvswitch\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:19:30.591770 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.591768    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a7702c04-56e6-4349-97f8-cd8c2efc76aa-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kmlmk\" (UID: \"a7702c04-56e6-4349-97f8-cd8c2efc76aa\") " pod="openshift-insights/insights-runtime-extractor-kmlmk"
Oct 31 19:19:30.592300 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.591862    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-run-openvswitch\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:19:30.592300 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.591891    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/65c98317-00d2-4ba8-8f66-9b34d720c553-host\") pod \"tuned-kzv9d\" (UID: \"65c98317-00d2-4ba8-8f66-9b34d720c553\") " pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:19:30.592300 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.591907    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/638d688d-d5fc-480d-8c53-08ab2f1864b1-socket-dir\") pod \"aws-ebs-csi-driver-node-tmtsr\" (UID: \"638d688d-d5fc-480d-8c53-08ab2f1864b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tmtsr"
Oct 31 19:19:30.592300 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.591944    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bc386a66-cdd2-4f4b-9666-88d48e42bd33-config-volume\") pod \"dns-default-kjlt7\" (UID: \"bc386a66-cdd2-4f4b-9666-88d48e42bd33\") " pod="openshift-dns/dns-default-kjlt7"
Oct 31 19:19:30.592300 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.591958    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/638d688d-d5fc-480d-8c53-08ab2f1864b1-etc-selinux\") pod \"aws-ebs-csi-driver-node-tmtsr\" (UID: \"638d688d-d5fc-480d-8c53-08ab2f1864b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tmtsr"
Oct 31 19:19:30.592300 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.591995    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/944a350f-aa7e-4921-8ddc-b0001fa05086-serviceca\") pod \"node-ca-5mwk4\" (UID: \"944a350f-aa7e-4921-8ddc-b0001fa05086\") " pod="openshift-image-registry/node-ca-5mwk4"
Oct 31 19:19:30.592300 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.592030    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761-original-pull-secret\") pod \"global-pull-secret-syncer-4pntt\" (UID: \"4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761\") " pod="kube-system/global-pull-secret-syncer-4pntt"
Oct 31 19:19:30.592300 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.592121    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-host-kubelet\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:19:30.592300 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.592169    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-ovn-node-metrics-cert\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:19:30.592300 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.592230    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4220431d-fa8a-4537-ad2e-e8ba6164f63d-cni-binary-copy\") pod \"multus-additional-cni-plugins-6s8ft\" (UID: \"4220431d-fa8a-4537-ad2e-e8ba6164f63d\") " pod="openshift-multus/multus-additional-cni-plugins-6s8ft"
Oct 31 19:19:30.592725 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.592316    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/944a350f-aa7e-4921-8ddc-b0001fa05086-host\") pod \"node-ca-5mwk4\" (UID: \"944a350f-aa7e-4921-8ddc-b0001fa05086\") " pod="openshift-image-registry/node-ca-5mwk4"
Oct 31 19:19:30.592725 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.592354    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761-original-pull-secret\") pod \"global-pull-secret-syncer-4pntt\" (UID: \"4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761\") " pod="kube-system/global-pull-secret-syncer-4pntt"
Oct 31 19:19:30.592725 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.592449    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4220431d-fa8a-4537-ad2e-e8ba6164f63d-cni-binary-copy\") pod \"multus-additional-cni-plugins-6s8ft\" (UID: \"4220431d-fa8a-4537-ad2e-e8ba6164f63d\") " pod="openshift-multus/multus-additional-cni-plugins-6s8ft"
Oct 31 19:19:30.592725 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.592456    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761-kubelet-config\") pod \"global-pull-secret-syncer-4pntt\" (UID: \"4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761\") " pod="kube-system/global-pull-secret-syncer-4pntt"
Oct 31 19:19:30.592725 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.592505    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t62j5\" (UniqueName: \"kubernetes.io/projected/3382fc12-f376-4b15-95be-6cb371ae87bd-kube-api-access-t62j5\") pod \"network-metrics-daemon-j6fmb\" (UID: \"3382fc12-f376-4b15-95be-6cb371ae87bd\") " pod="openshift-multus/network-metrics-daemon-j6fmb"
Oct 31 19:19:30.592725 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.592549    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9x4n\" (UniqueName: \"kubernetes.io/projected/a7702c04-56e6-4349-97f8-cd8c2efc76aa-kube-api-access-c9x4n\") pod \"insights-runtime-extractor-kmlmk\" (UID: \"a7702c04-56e6-4349-97f8-cd8c2efc76aa\") " pod="openshift-insights/insights-runtime-extractor-kmlmk"
Oct 31 19:19:30.592725 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.592595    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/65c98317-00d2-4ba8-8f66-9b34d720c553-sys\") pod \"tuned-kzv9d\" (UID: \"65c98317-00d2-4ba8-8f66-9b34d720c553\") " pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:19:30.592725 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.592602    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-ovn-node-metrics-cert\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:19:30.592725 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.592622    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bb36e52c-ab3f-4c51-84e0-d943187ef234-cni-binary-copy\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:19:30.592725 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.592654    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bb36e52c-ab3f-4c51-84e0-d943187ef234-multus-conf-dir\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:19:30.592725 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.592703    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/944a350f-aa7e-4921-8ddc-b0001fa05086-serviceca\") pod \"node-ca-5mwk4\" (UID: \"944a350f-aa7e-4921-8ddc-b0001fa05086\") " pod="openshift-image-registry/node-ca-5mwk4"
Oct 31 19:19:30.592725 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.592716    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-run-systemd\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:19:30.593167 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.592750    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4lbw\" (UniqueName: \"kubernetes.io/projected/4220431d-fa8a-4537-ad2e-e8ba6164f63d-kube-api-access-v4lbw\") pod \"multus-additional-cni-plugins-6s8ft\" (UID: \"4220431d-fa8a-4537-ad2e-e8ba6164f63d\") " pod="openshift-multus/multus-additional-cni-plugins-6s8ft"
Oct 31 19:19:30.593167 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.592773    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/611ab431-d82c-46e9-a108-4b221cc8c2cc-trusted-ca-bundle\") pod \"periodic-gathering-j9l2c-hmgd8\" (UID: \"611ab431-d82c-46e9-a108-4b221cc8c2cc\") " pod="openshift-insights/periodic-gathering-j9l2c-hmgd8"
Oct 31 19:19:30.593167 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.592789    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f2b26a37-2d01-495c-a3f3-12e5a0bbdca3-signing-key\") pod \"service-ca-5fffc97bdf-hwvnh\" (UID: \"f2b26a37-2d01-495c-a3f3-12e5a0bbdca3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hwvnh"
Oct 31 19:19:30.593167 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.592806    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bb36e52c-ab3f-4c51-84e0-d943187ef234-multus-socket-dir-parent\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:19:30.593167 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.592826    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/638d688d-d5fc-480d-8c53-08ab2f1864b1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tmtsr\" (UID: \"638d688d-d5fc-480d-8c53-08ab2f1864b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tmtsr"
Oct 31 19:19:30.593167 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.592850    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761-dbus\") pod \"global-pull-secret-syncer-4pntt\" (UID: \"4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761\") " pod="kube-system/global-pull-secret-syncer-4pntt"
Oct 31 19:19:30.593167 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.592853    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bb36e52c-ab3f-4c51-84e0-d943187ef234-cni-binary-copy\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:19:30.593167 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.592894    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-host-slash\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:19:30.593167 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.592921    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc386a66-cdd2-4f4b-9666-88d48e42bd33-metrics-tls\") pod \"dns-default-kjlt7\" (UID: \"bc386a66-cdd2-4f4b-9666-88d48e42bd33\") " pod="openshift-dns/dns-default-kjlt7"
Oct 31 19:19:30.593167 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.592951    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/611ab431-d82c-46e9-a108-4b221cc8c2cc-archives-path\") pod \"periodic-gathering-j9l2c-hmgd8\" (UID: \"611ab431-d82c-46e9-a108-4b221cc8c2cc\") " pod="openshift-insights/periodic-gathering-j9l2c-hmgd8"
Oct 31 19:19:30.593167 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.592977    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f2b26a37-2d01-495c-a3f3-12e5a0bbdca3-signing-key\") pod \"service-ca-5fffc97bdf-hwvnh\" (UID: \"f2b26a37-2d01-495c-a3f3-12e5a0bbdca3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hwvnh"
Oct 31 19:19:30.593167 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.592984    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/65c98317-00d2-4ba8-8f66-9b34d720c553-etc-modprobe-d\") pod \"tuned-kzv9d\" (UID: \"65c98317-00d2-4ba8-8f66-9b34d720c553\") " pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:19:30.593167 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.593011    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/65c98317-00d2-4ba8-8f66-9b34d720c553-etc-sysctl-d\") pod \"tuned-kzv9d\" (UID: \"65c98317-00d2-4ba8-8f66-9b34d720c553\") " pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:19:30.593167 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.593120    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/611ab431-d82c-46e9-a108-4b221cc8c2cc-archives-path\") pod \"periodic-gathering-j9l2c-hmgd8\" (UID: \"611ab431-d82c-46e9-a108-4b221cc8c2cc\") " pod="openshift-insights/periodic-gathering-j9l2c-hmgd8"
Oct 31 19:19:30.593167 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.593108    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/65c98317-00d2-4ba8-8f66-9b34d720c553-lib-modules\") pod \"tuned-kzv9d\" (UID: \"65c98317-00d2-4ba8-8f66-9b34d720c553\") " pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:19:30.593671 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.593170    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bb36e52c-ab3f-4c51-84e0-d943187ef234-hostroot\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:19:30.593671 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.593214    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/638d688d-d5fc-480d-8c53-08ab2f1864b1-sys-fs\") pod \"aws-ebs-csi-driver-node-tmtsr\" (UID: \"638d688d-d5fc-480d-8c53-08ab2f1864b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tmtsr"
Oct 31 19:19:30.593671 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.593237    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxxt6\" (UniqueName: \"kubernetes.io/projected/bc386a66-cdd2-4f4b-9666-88d48e42bd33-kube-api-access-nxxt6\") pod \"dns-default-kjlt7\" (UID: \"bc386a66-cdd2-4f4b-9666-88d48e42bd33\") " pod="openshift-dns/dns-default-kjlt7"
Oct 31 19:19:30.593671 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.593260    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc386a66-cdd2-4f4b-9666-88d48e42bd33-metrics-tls\") pod \"dns-default-kjlt7\" (UID: \"bc386a66-cdd2-4f4b-9666-88d48e42bd33\") " pod="openshift-dns/dns-default-kjlt7"
Oct 31 19:19:30.593671 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.593314    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-run-ovn\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:19:30.593671 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.593382    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/739472ea-5c22-44a3-9a81-acbf0755b3b4-iptables-alerter-script\") pod \"iptables-alerter-c5fnq\" (UID: \"739472ea-5c22-44a3-9a81-acbf0755b3b4\") " pod="openshift-network-operator/iptables-alerter-c5fnq"
Oct 31 19:19:30.593671 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.593453    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78hn4\" (UniqueName: \"kubernetes.io/projected/65c98317-00d2-4ba8-8f66-9b34d720c553-kube-api-access-78hn4\") pod \"tuned-kzv9d\" (UID: \"65c98317-00d2-4ba8-8f66-9b34d720c553\") " pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:19:30.593671 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.593471    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bb36e52c-ab3f-4c51-84e0-d943187ef234-system-cni-dir\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:19:30.593671 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.593494    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/611ab431-d82c-46e9-a108-4b221cc8c2cc-trusted-ca-bundle\") pod \"periodic-gathering-j9l2c-hmgd8\" (UID: \"611ab431-d82c-46e9-a108-4b221cc8c2cc\") " pod="openshift-insights/periodic-gathering-j9l2c-hmgd8"
Oct 31 19:19:30.593671 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.593500    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bb36e52c-ab3f-4c51-84e0-d943187ef234-cnibin\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:19:30.593671 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.593549    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bb36e52c-ab3f-4c51-84e0-d943187ef234-host-var-lib-cni-bin\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:19:30.593671 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.593567    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb36e52c-ab3f-4c51-84e0-d943187ef234-host-var-lib-kubelet\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:19:30.593671 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.593611    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c64b\" (UniqueName: \"kubernetes.io/projected/7306a726-d4c7-4ef5-b496-726e0bd488a1-kube-api-access-9c64b\") pod \"ingress-canary-2v47p\" (UID: \"7306a726-d4c7-4ef5-b496-726e0bd488a1\") " pod="openshift-ingress-canary/ingress-canary-2v47p"
Oct 31 19:19:30.593671 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.593640    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-host-run-ovn-kubernetes\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:19:30.593671 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.593635    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/739472ea-5c22-44a3-9a81-acbf0755b3b4-iptables-alerter-script\") pod \"iptables-alerter-c5fnq\" (UID: \"739472ea-5c22-44a3-9a81-acbf0755b3b4\") " pod="openshift-network-operator/iptables-alerter-c5fnq"
Oct 31 19:19:30.593671 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.593660    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:19:30.594322 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.593685    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4220431d-fa8a-4537-ad2e-e8ba6164f63d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6s8ft\" (UID: \"4220431d-fa8a-4537-ad2e-e8ba6164f63d\") " pod="openshift-multus/multus-additional-cni-plugins-6s8ft"
Oct 31 19:19:30.594322 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.593709    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2jxb\" (UniqueName: \"kubernetes.io/projected/611ab431-d82c-46e9-a108-4b221cc8c2cc-kube-api-access-x2jxb\") pod \"periodic-gathering-j9l2c-hmgd8\" (UID: \"611ab431-d82c-46e9-a108-4b221cc8c2cc\") " pod="openshift-insights/periodic-gathering-j9l2c-hmgd8"
Oct 31 19:19:30.594322 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.593740    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/21dc10e8-ee61-46ae-83f7-67d1772f083c-hosts-file\") pod \"node-resolver-58f52\" (UID: \"21dc10e8-ee61-46ae-83f7-67d1772f083c\") " pod="openshift-dns/node-resolver-58f52"
Oct 31 19:19:30.594322 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.593819    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq8nb\" (UniqueName: \"kubernetes.io/projected/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-kube-api-access-nq8nb\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:19:30.594322 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.593852    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4220431d-fa8a-4537-ad2e-e8ba6164f63d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6s8ft\" (UID: \"4220431d-fa8a-4537-ad2e-e8ba6164f63d\") " pod="openshift-multus/multus-additional-cni-plugins-6s8ft"
Oct 31 19:19:30.594322 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.593876    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/65c98317-00d2-4ba8-8f66-9b34d720c553-var-lib-kubelet\") pod \"tuned-kzv9d\" (UID: \"65c98317-00d2-4ba8-8f66-9b34d720c553\") " pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:19:30.594322 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.593904    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/65c98317-00d2-4ba8-8f66-9b34d720c553-etc-tuned\") pod \"tuned-kzv9d\" (UID: \"65c98317-00d2-4ba8-8f66-9b34d720c553\") " pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:19:30.594322 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.593950    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bb36e52c-ab3f-4c51-84e0-d943187ef234-multus-cni-dir\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:19:30.594322 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.593998    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bb36e52c-ab3f-4c51-84e0-d943187ef234-host-var-lib-cni-multus\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:19:30.594322 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.594005    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/65c98317-00d2-4ba8-8f66-9b34d720c553-etc-tuned\") pod \"tuned-kzv9d\" (UID: \"65c98317-00d2-4ba8-8f66-9b34d720c553\") " pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:19:30.594322 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.594043    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86gj5\" (UniqueName: \"kubernetes.io/projected/2ae10cd8-0183-4aca-bf20-ecb1e375e420-kube-api-access-86gj5\") pod \"migrator-66df5d496b-tztj8\" (UID: \"2ae10cd8-0183-4aca-bf20-ecb1e375e420\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tztj8"
Oct 31 19:19:30.594322 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.594059    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4220431d-fa8a-4537-ad2e-e8ba6164f63d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6s8ft\" (UID: \"4220431d-fa8a-4537-ad2e-e8ba6164f63d\") " pod="openshift-multus/multus-additional-cni-plugins-6s8ft"
Oct 31 19:19:30.594322 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.594069    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc386a66-cdd2-4f4b-9666-88d48e42bd33-tmp-dir\") pod \"dns-default-kjlt7\" (UID: \"bc386a66-cdd2-4f4b-9666-88d48e42bd33\") " pod="openshift-dns/dns-default-kjlt7"
Oct 31 19:19:30.594322 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.594114    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f2b26a37-2d01-495c-a3f3-12e5a0bbdca3-signing-cabundle\") pod \"service-ca-5fffc97bdf-hwvnh\" (UID: \"f2b26a37-2d01-495c-a3f3-12e5a0bbdca3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hwvnh"
Oct 31 19:19:30.594322 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.594128    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc386a66-cdd2-4f4b-9666-88d48e42bd33-tmp-dir\") pod \"dns-default-kjlt7\" (UID: \"bc386a66-cdd2-4f4b-9666-88d48e42bd33\") " pod="openshift-dns/dns-default-kjlt7"
Oct 31 19:19:30.594322 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.594146    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/65c98317-00d2-4ba8-8f66-9b34d720c553-etc-sysctl-conf\") pod \"tuned-kzv9d\" (UID: \"65c98317-00d2-4ba8-8f66-9b34d720c553\") " pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:19:30.594322 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.594318    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f2b26a37-2d01-495c-a3f3-12e5a0bbdca3-signing-cabundle\") pod \"service-ca-5fffc97bdf-hwvnh\" (UID: \"f2b26a37-2d01-495c-a3f3-12e5a0bbdca3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hwvnh"
Oct 31 19:19:30.601510 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.601489    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7gq5\" (UniqueName: \"kubernetes.io/projected/f2b26a37-2d01-495c-a3f3-12e5a0bbdca3-kube-api-access-s7gq5\") pod \"service-ca-5fffc97bdf-hwvnh\" (UID: \"f2b26a37-2d01-495c-a3f3-12e5a0bbdca3\") " pod="openshift-service-ca/service-ca-5fffc97bdf-hwvnh"
Oct 31 19:19:30.611036 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.611010    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk5qd\" (UniqueName: \"kubernetes.io/projected/638d688d-d5fc-480d-8c53-08ab2f1864b1-kube-api-access-gk5qd\") pod \"aws-ebs-csi-driver-node-tmtsr\" (UID: \"638d688d-d5fc-480d-8c53-08ab2f1864b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tmtsr"
Oct 31 19:19:30.611036 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.611029    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lq8r\" (UniqueName: \"kubernetes.io/projected/944a350f-aa7e-4921-8ddc-b0001fa05086-kube-api-access-5lq8r\") pod \"node-ca-5mwk4\" (UID: \"944a350f-aa7e-4921-8ddc-b0001fa05086\") " pod="openshift-image-registry/node-ca-5mwk4"
Oct 31 19:19:30.611395 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.611369    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k8mg\" (UniqueName: \"kubernetes.io/projected/21dc10e8-ee61-46ae-83f7-67d1772f083c-kube-api-access-7k8mg\") pod \"node-resolver-58f52\" (UID: \"21dc10e8-ee61-46ae-83f7-67d1772f083c\") " pod="openshift-dns/node-resolver-58f52"
Oct 31 19:19:30.612626 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.612607    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t62j5\" (UniqueName: \"kubernetes.io/projected/3382fc12-f376-4b15-95be-6cb371ae87bd-kube-api-access-t62j5\") pod \"network-metrics-daemon-j6fmb\" (UID: \"3382fc12-f376-4b15-95be-6cb371ae87bd\") " pod="openshift-multus/network-metrics-daemon-j6fmb"
Oct 31 19:19:30.618235 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.618216    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4lbw\" (UniqueName: \"kubernetes.io/projected/4220431d-fa8a-4537-ad2e-e8ba6164f63d-kube-api-access-v4lbw\") pod \"multus-additional-cni-plugins-6s8ft\" (UID: \"4220431d-fa8a-4537-ad2e-e8ba6164f63d\") " pod="openshift-multus/multus-additional-cni-plugins-6s8ft"
Oct 31 19:19:30.618443 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.618421    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6xws\" (UniqueName: \"kubernetes.io/projected/739472ea-5c22-44a3-9a81-acbf0755b3b4-kube-api-access-l6xws\") pod \"iptables-alerter-c5fnq\" (UID: \"739472ea-5c22-44a3-9a81-acbf0755b3b4\") " pod="openshift-network-operator/iptables-alerter-c5fnq"
Oct 31 19:19:30.618821 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.618789    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86gj5\" (UniqueName: \"kubernetes.io/projected/2ae10cd8-0183-4aca-bf20-ecb1e375e420-kube-api-access-86gj5\") pod \"migrator-66df5d496b-tztj8\" (UID: \"2ae10cd8-0183-4aca-bf20-ecb1e375e420\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-tztj8"
Oct 31 19:19:30.620144 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.619017    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2jxb\" (UniqueName: \"kubernetes.io/projected/611ab431-d82c-46e9-a108-4b221cc8c2cc-kube-api-access-x2jxb\") pod \"periodic-gathering-j9l2c-hmgd8\" (UID: \"611ab431-d82c-46e9-a108-4b221cc8c2cc\") " pod="openshift-insights/periodic-gathering-j9l2c-hmgd8"
Oct 31 19:19:30.620144 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.619065    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npsmn\" (UniqueName: \"kubernetes.io/projected/534f9972-b54f-4713-8326-4808fdf08386-kube-api-access-npsmn\") pod \"network-check-target-4j5p9\" (UID: \"534f9972-b54f-4713-8326-4808fdf08386\") " pod="openshift-network-diagnostics/network-check-target-4j5p9"
Oct 31 19:19:30.620144 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.619095    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78hn4\" (UniqueName: \"kubernetes.io/projected/65c98317-00d2-4ba8-8f66-9b34d720c553-kube-api-access-78hn4\") pod \"tuned-kzv9d\" (UID: \"65c98317-00d2-4ba8-8f66-9b34d720c553\") " pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:19:30.620144 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.619124    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c64b\" (UniqueName: \"kubernetes.io/projected/7306a726-d4c7-4ef5-b496-726e0bd488a1-kube-api-access-9c64b\") pod \"ingress-canary-2v47p\" (UID: \"7306a726-d4c7-4ef5-b496-726e0bd488a1\") " pod="openshift-ingress-canary/ingress-canary-2v47p"
Oct 31 19:19:30.620144 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.619360    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfwph\" (UniqueName: \"kubernetes.io/projected/bb36e52c-ab3f-4c51-84e0-d943187ef234-kube-api-access-tfwph\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:19:30.620144 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.619525    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxxt6\" (UniqueName: \"kubernetes.io/projected/bc386a66-cdd2-4f4b-9666-88d48e42bd33-kube-api-access-nxxt6\") pod \"dns-default-kjlt7\" (UID: \"bc386a66-cdd2-4f4b-9666-88d48e42bd33\") " pod="openshift-dns/dns-default-kjlt7"
Oct 31 19:19:30.622426 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.622398    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq8nb\" (UniqueName: \"kubernetes.io/projected/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-kube-api-access-nq8nb\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:19:30.623434 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.623418    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9x4n\" (UniqueName: \"kubernetes.io/projected/a7702c04-56e6-4349-97f8-cd8c2efc76aa-kube-api-access-c9x4n\") pod \"insights-runtime-extractor-kmlmk\" (UID: \"a7702c04-56e6-4349-97f8-cd8c2efc76aa\") " pod="openshift-insights/insights-runtime-extractor-kmlmk"
Oct 31 19:19:30.678263 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.678242    4645 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-94.us-east-2.compute.internal"
Oct 31 19:19:30.678383 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.678370    4645 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-94.us-east-2.compute.internal"
Oct 31 19:19:30.694704 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.694683    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/21dc10e8-ee61-46ae-83f7-67d1772f083c-hosts-file\") pod \"node-resolver-58f52\" (UID: \"21dc10e8-ee61-46ae-83f7-67d1772f083c\") " pod="openshift-dns/node-resolver-58f52"
Oct 31 19:19:30.694773 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.694708    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/65c98317-00d2-4ba8-8f66-9b34d720c553-var-lib-kubelet\") pod \"tuned-kzv9d\" (UID: \"65c98317-00d2-4ba8-8f66-9b34d720c553\") " pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:19:30.694773 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.694725    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bb36e52c-ab3f-4c51-84e0-d943187ef234-multus-cni-dir\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:19:30.694773 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.694740    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bb36e52c-ab3f-4c51-84e0-d943187ef234-host-var-lib-cni-multus\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:19:30.694882 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.694796    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bb36e52c-ab3f-4c51-84e0-d943187ef234-multus-cni-dir\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:19:30.694882 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.694810    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/65c98317-00d2-4ba8-8f66-9b34d720c553-var-lib-kubelet\") pod \"tuned-kzv9d\" (UID: \"65c98317-00d2-4ba8-8f66-9b34d720c553\") " pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:19:30.694882 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.694838    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/21dc10e8-ee61-46ae-83f7-67d1772f083c-hosts-file\") pod \"node-resolver-58f52\" (UID: \"21dc10e8-ee61-46ae-83f7-67d1772f083c\") " pod="openshift-dns/node-resolver-58f52"
Oct 31 19:19:30.694882 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.694837    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/65c98317-00d2-4ba8-8f66-9b34d720c553-etc-sysctl-conf\") pod \"tuned-kzv9d\" (UID: \"65c98317-00d2-4ba8-8f66-9b34d720c553\") " pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:19:30.694882 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.694864    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bb36e52c-ab3f-4c51-84e0-d943187ef234-host-var-lib-cni-multus\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:19:30.694882 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.694876    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-systemd-units\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:19:30.695065 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.694899    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-node-log\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:19:30.695065 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.694916    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/739472ea-5c22-44a3-9a81-acbf0755b3b4-host-slash\") pod \"iptables-alerter-c5fnq\" (UID: \"739472ea-5c22-44a3-9a81-acbf0755b3b4\") " pod="openshift-network-operator/iptables-alerter-c5fnq"
Oct 31 19:19:30.695065 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.694931    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/65c98317-00d2-4ba8-8f66-9b34d720c553-etc-systemd\") pod \"tuned-kzv9d\" (UID: \"65c98317-00d2-4ba8-8f66-9b34d720c553\") " pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:19:30.695065 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.694945    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/65c98317-00d2-4ba8-8f66-9b34d720c553-run\") pod \"tuned-kzv9d\" (UID: \"65c98317-00d2-4ba8-8f66-9b34d720c553\") " pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:19:30.695065 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.694965    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-node-log\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:19:30.695065 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.694966    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/65c98317-00d2-4ba8-8f66-9b34d720c553-etc-sysctl-conf\") pod \"tuned-kzv9d\" (UID: \"65c98317-00d2-4ba8-8f66-9b34d720c553\") " pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:19:30.695065 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.694991    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-systemd-units\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:19:30.695065 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.694991    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/739472ea-5c22-44a3-9a81-acbf0755b3b4-host-slash\") pod \"iptables-alerter-c5fnq\" (UID: \"739472ea-5c22-44a3-9a81-acbf0755b3b4\") " pod="openshift-network-operator/iptables-alerter-c5fnq"
Oct 31 19:19:30.695065 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.694997    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/65c98317-00d2-4ba8-8f66-9b34d720c553-etc-systemd\") pod \"tuned-kzv9d\" (UID: \"65c98317-00d2-4ba8-8f66-9b34d720c553\") " pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:19:30.695065 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.695035    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/65c98317-00d2-4ba8-8f66-9b34d720c553-run\") pod \"tuned-kzv9d\" (UID: \"65c98317-00d2-4ba8-8f66-9b34d720c553\") " pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:19:30.695525 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.695078    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bb36e52c-ab3f-4c51-84e0-d943187ef234-host-run-k8s-cni-cncf-io\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:19:30.695525 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.695115    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bb36e52c-ab3f-4c51-84e0-d943187ef234-host-run-netns\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:19:30.695525 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.695134    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bb36e52c-ab3f-4c51-84e0-d943187ef234-host-run-k8s-cni-cncf-io\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:19:30.695525 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.695141    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-log-socket\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:19:30.695525 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.695180    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-log-socket\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:19:30.695525 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.695187    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-host-cni-netd\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:19:30.695525 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.695154    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bb36e52c-ab3f-4c51-84e0-d943187ef234-host-run-netns\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:19:30.695525 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.695209    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4220431d-fa8a-4537-ad2e-e8ba6164f63d-system-cni-dir\") pod \"multus-additional-cni-plugins-6s8ft\" (UID: \"4220431d-fa8a-4537-ad2e-e8ba6164f63d\") " pod="openshift-multus/multus-additional-cni-plugins-6s8ft"
Oct 31 19:19:30.695525 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.695233    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4220431d-fa8a-4537-ad2e-e8ba6164f63d-cnibin\") pod \"multus-additional-cni-plugins-6s8ft\" (UID: \"4220431d-fa8a-4537-ad2e-e8ba6164f63d\") " pod="openshift-multus/multus-additional-cni-plugins-6s8ft"
Oct 31 19:19:30.695525 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.695244    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-host-cni-netd\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:19:30.695525 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.695252    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4220431d-fa8a-4537-ad2e-e8ba6164f63d-system-cni-dir\") pod \"multus-additional-cni-plugins-6s8ft\" (UID: \"4220431d-fa8a-4537-ad2e-e8ba6164f63d\") " pod="openshift-multus/multus-additional-cni-plugins-6s8ft"
Oct 31 19:19:30.695525 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.695256    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a7702c04-56e6-4349-97f8-cd8c2efc76aa-crio-socket\") pod \"insights-runtime-extractor-kmlmk\" (UID: \"a7702c04-56e6-4349-97f8-cd8c2efc76aa\") " pod="openshift-insights/insights-runtime-extractor-kmlmk"
Oct 31 19:19:30.695525 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.695283    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4220431d-fa8a-4537-ad2e-e8ba6164f63d-cnibin\") pod \"multus-additional-cni-plugins-6s8ft\" (UID: \"4220431d-fa8a-4537-ad2e-e8ba6164f63d\") " pod="openshift-multus/multus-additional-cni-plugins-6s8ft"
Oct 31 19:19:30.695525 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.695307    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb36e52c-ab3f-4c51-84e0-d943187ef234-etc-kubernetes\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:19:30.695525 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.695322    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a7702c04-56e6-4349-97f8-cd8c2efc76aa-crio-socket\") pod \"insights-runtime-extractor-kmlmk\" (UID: \"a7702c04-56e6-4349-97f8-cd8c2efc76aa\") " pod="openshift-insights/insights-runtime-extractor-kmlmk"
Oct 31 19:19:30.695525 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.695350    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/638d688d-d5fc-480d-8c53-08ab2f1864b1-registration-dir\") pod \"aws-ebs-csi-driver-node-tmtsr\" (UID: \"638d688d-d5fc-480d-8c53-08ab2f1864b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tmtsr"
Oct 31 19:19:30.695525 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.695381    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb36e52c-ab3f-4c51-84e0-d943187ef234-etc-kubernetes\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:19:30.695525 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.695388    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3382fc12-f376-4b15-95be-6cb371ae87bd-metrics-certs\") pod \"network-metrics-daemon-j6fmb\" (UID: \"3382fc12-f376-4b15-95be-6cb371ae87bd\") " pod="openshift-multus/network-metrics-daemon-j6fmb"
Oct 31 19:19:30.695525 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.695407    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/638d688d-d5fc-480d-8c53-08ab2f1864b1-device-dir\") pod \"aws-ebs-csi-driver-node-tmtsr\" (UID: \"638d688d-d5fc-480d-8c53-08ab2f1864b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tmtsr"
Oct 31 19:19:30.695525 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.695406    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/638d688d-d5fc-480d-8c53-08ab2f1864b1-registration-dir\") pod \"aws-ebs-csi-driver-node-tmtsr\" (UID: \"638d688d-d5fc-480d-8c53-08ab2f1864b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tmtsr"
Oct 31 19:19:30.695525 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.695441    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/638d688d-d5fc-480d-8c53-08ab2f1864b1-device-dir\") pod \"aws-ebs-csi-driver-node-tmtsr\" (UID: \"638d688d-d5fc-480d-8c53-08ab2f1864b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tmtsr"
Oct 31 19:19:30.695525 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.695434    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-host-run-netns\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:19:30.695525 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.695466    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-etc-openvswitch\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:19:30.695525 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.695470    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-host-run-netns\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:19:30.695525 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.695482    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-host-cni-bin\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:19:30.695525 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.695493    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-etc-openvswitch\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:19:30.695525 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.695469    4645 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 19:19:30.695525 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.695503    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-host-cni-bin\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:19:30.695525 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.695511    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/65c98317-00d2-4ba8-8f66-9b34d720c553-etc-sysconfig\") pod \"tuned-kzv9d\" (UID: \"65c98317-00d2-4ba8-8f66-9b34d720c553\") " pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:19:30.695525 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:19:30.695532    4645 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-131-94.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-94.us-east-2.compute.internal"
Oct 31 19:19:30.695525 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.695543    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/65c98317-00d2-4ba8-8f66-9b34d720c553-etc-kubernetes\") pod \"tuned-kzv9d\" (UID: \"65c98317-00d2-4ba8-8f66-9b34d720c553\") " pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:19:30.696750 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.695566    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bb36e52c-ab3f-4c51-84e0-d943187ef234-host-run-multus-certs\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:19:30.696750 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.695571    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/65c98317-00d2-4ba8-8f66-9b34d720c553-etc-kubernetes\") pod \"tuned-kzv9d\" (UID: \"65c98317-00d2-4ba8-8f66-9b34d720c553\") " pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:19:30.696750 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.695546    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/65c98317-00d2-4ba8-8f66-9b34d720c553-etc-sysconfig\") pod \"tuned-kzv9d\" (UID: \"65c98317-00d2-4ba8-8f66-9b34d720c553\") " pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:19:30.696750 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.695591    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-var-lib-openvswitch\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:19:30.696750 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.695601    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bb36e52c-ab3f-4c51-84e0-d943187ef234-host-run-multus-certs\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:19:30.696750 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.695614    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-run-openvswitch\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:19:30.696750 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.695631    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-var-lib-openvswitch\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:19:30.696750 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.695639    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/65c98317-00d2-4ba8-8f66-9b34d720c553-host\") pod \"tuned-kzv9d\" (UID: \"65c98317-00d2-4ba8-8f66-9b34d720c553\") " pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:19:30.696750 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.695661    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-run-openvswitch\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:19:30.696750 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.695665    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/638d688d-d5fc-480d-8c53-08ab2f1864b1-socket-dir\") pod \"aws-ebs-csi-driver-node-tmtsr\" (UID: \"638d688d-d5fc-480d-8c53-08ab2f1864b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tmtsr"
Oct 31 19:19:30.696750 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.695720    4645 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 31 19:19:30.696750 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.695732    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/638d688d-d5fc-480d-8c53-08ab2f1864b1-etc-selinux\") pod \"aws-ebs-csi-driver-node-tmtsr\" (UID: \"638d688d-d5fc-480d-8c53-08ab2f1864b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tmtsr"
Oct 31 19:19:30.696750 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.695766    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-host-kubelet\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:19:30.696750 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.695793    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/944a350f-aa7e-4921-8ddc-b0001fa05086-host\") pod \"node-ca-5mwk4\" (UID: \"944a350f-aa7e-4921-8ddc-b0001fa05086\") " pod="openshift-image-registry/node-ca-5mwk4"
Oct 31 19:19:30.696750 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.695816    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761-kubelet-config\") pod \"global-pull-secret-syncer-4pntt\" (UID: \"4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761\") " pod="kube-system/global-pull-secret-syncer-4pntt"
Oct 31 19:19:30.696750 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.695824    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/65c98317-00d2-4ba8-8f66-9b34d720c553-host\") pod \"tuned-kzv9d\" (UID: \"65c98317-00d2-4ba8-8f66-9b34d720c553\") " pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:19:30.696750 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.695827    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/638d688d-d5fc-480d-8c53-08ab2f1864b1-etc-selinux\") pod \"aws-ebs-csi-driver-node-tmtsr\" (UID: \"638d688d-d5fc-480d-8c53-08ab2f1864b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tmtsr"
Oct 31 19:19:30.696750 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.695838    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-host-kubelet\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:19:30.696750 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.695851    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/944a350f-aa7e-4921-8ddc-b0001fa05086-host\") pod \"node-ca-5mwk4\" (UID: \"944a350f-aa7e-4921-8ddc-b0001fa05086\") " pod="openshift-image-registry/node-ca-5mwk4"
Oct 31 19:19:30.696750 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.695840    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/65c98317-00d2-4ba8-8f66-9b34d720c553-sys\") pod \"tuned-kzv9d\" (UID: \"65c98317-00d2-4ba8-8f66-9b34d720c553\") " pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:19:30.696750 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.695864    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761-kubelet-config\") pod \"global-pull-secret-syncer-4pntt\" (UID: \"4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761\") " pod="kube-system/global-pull-secret-syncer-4pntt"
Oct 31 19:19:30.696750 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.695816    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/638d688d-d5fc-480d-8c53-08ab2f1864b1-socket-dir\") pod \"aws-ebs-csi-driver-node-tmtsr\" (UID: \"638d688d-d5fc-480d-8c53-08ab2f1864b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tmtsr"
Oct 31 19:19:30.696750 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.695876    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/65c98317-00d2-4ba8-8f66-9b34d720c553-sys\") pod \"tuned-kzv9d\" (UID: \"65c98317-00d2-4ba8-8f66-9b34d720c553\") " pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:19:30.696750 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.695890    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bb36e52c-ab3f-4c51-84e0-d943187ef234-multus-conf-dir\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:19:30.696750 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.695909    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-run-systemd\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:19:30.696750 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.695953    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-run-systemd\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:19:30.696750 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.695957    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bb36e52c-ab3f-4c51-84e0-d943187ef234-multus-socket-dir-parent\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:19:30.696750 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.695947    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bb36e52c-ab3f-4c51-84e0-d943187ef234-multus-conf-dir\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:19:30.696750 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.695985    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/638d688d-d5fc-480d-8c53-08ab2f1864b1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tmtsr\" (UID: \"638d688d-d5fc-480d-8c53-08ab2f1864b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tmtsr"
Oct 31 19:19:30.696750 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.696000    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bb36e52c-ab3f-4c51-84e0-d943187ef234-multus-socket-dir-parent\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:19:30.696750 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.696010    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761-dbus\") pod \"global-pull-secret-syncer-4pntt\" (UID: \"4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761\") " pod="kube-system/global-pull-secret-syncer-4pntt"
Oct 31 19:19:30.696750 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.696035    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-host-slash\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:19:30.696750 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.696041    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761-dbus\") pod \"global-pull-secret-syncer-4pntt\" (UID: \"4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761\") " pod="kube-system/global-pull-secret-syncer-4pntt"
Oct 31 19:19:30.696750 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.696043    4645 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 19:19:30.696750 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.696052    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/638d688d-d5fc-480d-8c53-08ab2f1864b1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tmtsr\" (UID: \"638d688d-d5fc-480d-8c53-08ab2f1864b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tmtsr"
Oct 31 19:19:30.696750 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.696058    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/65c98317-00d2-4ba8-8f66-9b34d720c553-etc-modprobe-d\") pod \"tuned-kzv9d\" (UID: \"65c98317-00d2-4ba8-8f66-9b34d720c553\") " pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:19:30.696750 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:19:30.696088    4645 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-131-94.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-94.us-east-2.compute.internal"
Oct 31 19:19:30.696750 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.696097    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-host-slash\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:19:30.696750 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.696107    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/65c98317-00d2-4ba8-8f66-9b34d720c553-etc-sysctl-d\") pod \"tuned-kzv9d\" (UID: \"65c98317-00d2-4ba8-8f66-9b34d720c553\") " pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:19:30.696750 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.696135    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/65c98317-00d2-4ba8-8f66-9b34d720c553-lib-modules\") pod \"tuned-kzv9d\" (UID: \"65c98317-00d2-4ba8-8f66-9b34d720c553\") " pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:19:30.696750 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.696139    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/65c98317-00d2-4ba8-8f66-9b34d720c553-etc-modprobe-d\") pod \"tuned-kzv9d\" (UID: \"65c98317-00d2-4ba8-8f66-9b34d720c553\") " pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:19:30.696750 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.696159    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bb36e52c-ab3f-4c51-84e0-d943187ef234-hostroot\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:19:30.696750 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.696188    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/638d688d-d5fc-480d-8c53-08ab2f1864b1-sys-fs\") pod \"aws-ebs-csi-driver-node-tmtsr\" (UID: \"638d688d-d5fc-480d-8c53-08ab2f1864b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tmtsr"
Oct 31 19:19:30.696750 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.696212    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-run-ovn\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:19:30.696750 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.696222    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/65c98317-00d2-4ba8-8f66-9b34d720c553-etc-sysctl-d\") pod \"tuned-kzv9d\" (UID: \"65c98317-00d2-4ba8-8f66-9b34d720c553\") " pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:19:30.696750 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.696223    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bb36e52c-ab3f-4c51-84e0-d943187ef234-hostroot\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:19:30.696750 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.696235    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bb36e52c-ab3f-4c51-84e0-d943187ef234-system-cni-dir\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:19:30.696750 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.696267    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/65c98317-00d2-4ba8-8f66-9b34d720c553-lib-modules\") pod \"tuned-kzv9d\" (UID: \"65c98317-00d2-4ba8-8f66-9b34d720c553\") " pod="openshift-cluster-node-tuning-operator/tuned-kzv9d"
Oct 31 19:19:30.696750 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.696311    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bb36e52c-ab3f-4c51-84e0-d943187ef234-system-cni-dir\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:19:30.696750 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.696319    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-run-ovn\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:19:30.696750 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.696347    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bb36e52c-ab3f-4c51-84e0-d943187ef234-cnibin\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:19:30.696750 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.696373    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/638d688d-d5fc-480d-8c53-08ab2f1864b1-sys-fs\") pod \"aws-ebs-csi-driver-node-tmtsr\" (UID: \"638d688d-d5fc-480d-8c53-08ab2f1864b1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tmtsr"
Oct 31 19:19:30.696750 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.696380    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bb36e52c-ab3f-4c51-84e0-d943187ef234-cnibin\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:19:30.696750 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.696373    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bb36e52c-ab3f-4c51-84e0-d943187ef234-host-var-lib-cni-bin\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:19:30.696750 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.696402    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb36e52c-ab3f-4c51-84e0-d943187ef234-host-var-lib-kubelet\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:19:30.696750 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.696407    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bb36e52c-ab3f-4c51-84e0-d943187ef234-host-var-lib-cni-bin\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:19:30.696750 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.696426    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-host-run-ovn-kubernetes\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:19:30.696750 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.696444    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:19:30.696750 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.696449    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb36e52c-ab3f-4c51-84e0-d943187ef234-host-var-lib-kubelet\") pod \"multus-hhnhg\" (UID: \"bb36e52c-ab3f-4c51-84e0-d943187ef234\") " pod="openshift-multus/multus-hhnhg"
Oct 31 19:19:30.696750 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.696459    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4220431d-fa8a-4537-ad2e-e8ba6164f63d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6s8ft\" (UID: \"4220431d-fa8a-4537-ad2e-e8ba6164f63d\") " pod="openshift-multus/multus-additional-cni-plugins-6s8ft"
Oct 31 19:19:30.696750 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.696493    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-host-run-ovn-kubernetes\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:19:30.696750 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.696498    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mr9n2\" (UID: \"14026ae1-aeee-4b0e-9fd9-67bd4ed7fc0c\") " pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:19:30.696750 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.696503    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4220431d-fa8a-4537-ad2e-e8ba6164f63d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6s8ft\" (UID: \"4220431d-fa8a-4537-ad2e-e8ba6164f63d\") " pod="openshift-multus/multus-additional-cni-plugins-6s8ft"
Oct 31 19:19:30.698405 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.698062    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3382fc12-f376-4b15-95be-6cb371ae87bd-metrics-certs\") pod \"network-metrics-daemon-j6fmb\" (UID: \"3382fc12-f376-4b15-95be-6cb371ae87bd\") " pod="openshift-multus/network-metrics-daemon-j6fmb"
Oct 31 19:19:30.810916 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.810848    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:30.844696 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.844675    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-j6fmb"
Oct 31 19:19:30.849150 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:30.849134    4645 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 19:19:31.014226 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:31.014196    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:31.131451 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:31.131385    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-j6fmb"]
Oct 31 19:19:31.134220 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:31.134188    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3382fc12_f376_4b15_95be_6cb371ae87bd.slice/crio-40ad04507f7a6b0a1154046cb554dfa7066d231dc684b6e20a277daa24f09e3f WatchSource:0}: Error finding container 40ad04507f7a6b0a1154046cb554dfa7066d231dc684b6e20a277daa24f09e3f: Status 404 returned error can't find the container with id 40ad04507f7a6b0a1154046cb554dfa7066d231dc684b6e20a277daa24f09e3f
Oct 31 19:19:31.615074 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:31.615040    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:31.685509 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:31.685477    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-j6fmb" event={"ID":"3382fc12-f376-4b15-95be-6cb371ae87bd","Type":"ContainerStarted","Data":"40ad04507f7a6b0a1154046cb554dfa7066d231dc684b6e20a277daa24f09e3f"}
Oct 31 19:19:31.821061 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:31.821024    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:32.211529 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:32.211496    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:32.414757 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:32.414706    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:32.519424 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:32.519352    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:32.691348 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:32.691294    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kmlmk" event={"ID":"a7702c04-56e6-4349-97f8-cd8c2efc76aa","Type":"ContainerStarted","Data":"d5317601a717dbf2c442607f54df6ba06c8a9bf0e104bc79bdf2c73b42976f42"}
Oct 31 19:19:32.693571 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:32.693546    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kjlt7" event={"ID":"bc386a66-cdd2-4f4b-9666-88d48e42bd33","Type":"ContainerStarted","Data":"72ae7df4fc8952db734041742d370d3884ad139c4c83f3a2d824166059715abc"}
Oct 31 19:19:32.693685 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:32.693595    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kjlt7" event={"ID":"bc386a66-cdd2-4f4b-9666-88d48e42bd33","Type":"ContainerStarted","Data":"aae4e9f6bb5394ce3c6a2de2d87ee2986ed8001867d3049c5dd663174eef1e30"}
Oct 31 19:19:32.977543 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:32.977382    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:33.076256 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:33.076176    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:33.176414 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:33.176372    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:33.272721 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:33.272694    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-kjlt7"
Oct 31 19:19:33.460531 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:33.460499    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:33.613927 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:33.613869    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:33.699227 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:33.699201    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kmlmk" event={"ID":"a7702c04-56e6-4349-97f8-cd8c2efc76aa","Type":"ContainerStarted","Data":"f7150f37923206af2d467c006734411008fbb946e5c995a2c297e17141bd7735"}
Oct 31 19:19:33.700970 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:33.700947    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-j6fmb" event={"ID":"3382fc12-f376-4b15-95be-6cb371ae87bd","Type":"ContainerStarted","Data":"bce1587c11176f68666a19f3a439827e8a777f8aa63a21112d057d95ab15f168"}
Oct 31 19:19:33.701047 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:33.700991    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-j6fmb" event={"ID":"3382fc12-f376-4b15-95be-6cb371ae87bd","Type":"ContainerStarted","Data":"2de5117b98261ca092e11a33413e6a785995a87c79d74cbc46bd4d81b3202325"}
Oct 31 19:19:33.813843 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:33.813817    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:34.013965 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:34.013939    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:34.413074 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:34.413048    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:35.215274 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:35.215239    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:35.522908 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:35.522842    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:35.622909 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:35.622881    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:35.811565 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:35.811502    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:36.012635 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:36.012604    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:36.247113 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:19:36.247084    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 19:19:36.351406 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:36.351381    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:19:36.423205 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:36.423156    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:36.812762 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:36.812725    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:37.012853 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:37.012820    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:37.214436 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:37.214410    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-4j5p9"
Oct 31 19:19:37.720007 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:37.719978    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:19:37.720523 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:37.720167    4645 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 19:19:38.397135 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:38.397096    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:38.504469 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:38.504440    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:38.604800 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:38.604771    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:38.705253 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:38.705227    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:38.804335 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:38.804314    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:38.937455 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:38.937424    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:39.038743 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:39.038639    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:39.340496 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:39.340426    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:39.442471 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:39.442450    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:41.338587 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:41.338550    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:41.444244 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:41.444200    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:41.641022 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:41.640956    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-kc9mq"
Oct 31 19:19:42.704162 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:42.704126    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-kjlt7"
Oct 31 19:19:43.992395 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:43.992361    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5zb27"]
Oct 31 19:19:43.995684 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:43.995662    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-fs6gw"]
Oct 31 19:19:43.995831 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:43.995789    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5zb27"
Oct 31 19:19:43.998717 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:43.998685    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-fs6gw"
Oct 31 19:19:44.014055 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:44.014033    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 31 19:19:44.015774 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:44.015752    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5zb27"]
Oct 31 19:19:44.016235 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:44.016215    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-fs6gw"]
Oct 31 19:19:44.023751 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:44.023712    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-vdd29\""
Oct 31 19:19:44.023854 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:44.023840    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-rk4kk\""
Oct 31 19:19:44.023930 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:44.023916    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Oct 31 19:19:44.039426 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:44.039407    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 31 19:19:44.082387 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:44.082358    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a4864737-8f10-4291-8529-57acff0fe5c2-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-5zb27\" (UID: \"a4864737-8f10-4291-8529-57acff0fe5c2\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5zb27"
Oct 31 19:19:44.082516 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:44.082406    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c62a9b4b-1882-4d88-b486-ec5f3a6d18db-tmp\") pod \"downloads-6dd789c8b7-fs6gw\" (UID: \"c62a9b4b-1882-4d88-b486-ec5f3a6d18db\") " pod="openshift-console/downloads-6dd789c8b7-fs6gw"
Oct 31 19:19:44.082561 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:44.082509    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckc95\" (UniqueName: \"kubernetes.io/projected/c62a9b4b-1882-4d88-b486-ec5f3a6d18db-kube-api-access-ckc95\") pod \"downloads-6dd789c8b7-fs6gw\" (UID: \"c62a9b4b-1882-4d88-b486-ec5f3a6d18db\") " pod="openshift-console/downloads-6dd789c8b7-fs6gw"
Oct 31 19:19:44.089893 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:44.089866    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-779c8588cc-ns2s4"]
Oct 31 19:19:44.092964 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:44.092946    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-779c8588cc-ns2s4"
Oct 31 19:19:44.099257 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:44.099234    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 31 19:19:44.110940 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:44.110903    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-779c8588cc-ns2s4"]
Oct 31 19:19:44.113045 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:44.113019    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:44.116223 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:44.116194    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 31 19:19:44.116317 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:44.116241    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 31 19:19:44.116317 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:44.116278    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 31 19:19:44.116317 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:44.116282    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-vq9kc\""
Oct 31 19:19:44.182985 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:44.182962    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0f7455d5-9b9d-47f7-a0ad-df85dea8139c-image-registry-private-configuration\") pod \"image-registry-779c8588cc-ns2s4\" (UID: \"0f7455d5-9b9d-47f7-a0ad-df85dea8139c\") " pod="openshift-image-registry/image-registry-779c8588cc-ns2s4"
Oct 31 19:19:44.183094 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:44.182993    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0f7455d5-9b9d-47f7-a0ad-df85dea8139c-bound-sa-token\") pod \"image-registry-779c8588cc-ns2s4\" (UID: \"0f7455d5-9b9d-47f7-a0ad-df85dea8139c\") " pod="openshift-image-registry/image-registry-779c8588cc-ns2s4"
Oct 31 19:19:44.183094 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:44.183018    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a4864737-8f10-4291-8529-57acff0fe5c2-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-5zb27\" (UID: \"a4864737-8f10-4291-8529-57acff0fe5c2\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5zb27"
Oct 31 19:19:44.183230 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:44.183208    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0f7455d5-9b9d-47f7-a0ad-df85dea8139c-registry-tls\") pod \"image-registry-779c8588cc-ns2s4\" (UID: \"0f7455d5-9b9d-47f7-a0ad-df85dea8139c\") " pod="openshift-image-registry/image-registry-779c8588cc-ns2s4"
Oct 31 19:19:44.183285 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:44.183275    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c62a9b4b-1882-4d88-b486-ec5f3a6d18db-tmp\") pod \"downloads-6dd789c8b7-fs6gw\" (UID: \"c62a9b4b-1882-4d88-b486-ec5f3a6d18db\") " pod="openshift-console/downloads-6dd789c8b7-fs6gw"
Oct 31 19:19:44.183360 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:44.183294    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5qzn\" (UniqueName: \"kubernetes.io/projected/0f7455d5-9b9d-47f7-a0ad-df85dea8139c-kube-api-access-b5qzn\") pod \"image-registry-779c8588cc-ns2s4\" (UID: \"0f7455d5-9b9d-47f7-a0ad-df85dea8139c\") " pod="openshift-image-registry/image-registry-779c8588cc-ns2s4"
Oct 31 19:19:44.183360 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:44.183322    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckc95\" (UniqueName: \"kubernetes.io/projected/c62a9b4b-1882-4d88-b486-ec5f3a6d18db-kube-api-access-ckc95\") pod \"downloads-6dd789c8b7-fs6gw\" (UID: \"c62a9b4b-1882-4d88-b486-ec5f3a6d18db\") " pod="openshift-console/downloads-6dd789c8b7-fs6gw"
Oct 31 19:19:44.183471 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:44.183421    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0f7455d5-9b9d-47f7-a0ad-df85dea8139c-registry-certificates\") pod \"image-registry-779c8588cc-ns2s4\" (UID: \"0f7455d5-9b9d-47f7-a0ad-df85dea8139c\") " pod="openshift-image-registry/image-registry-779c8588cc-ns2s4"
Oct 31 19:19:44.183471 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:44.183454    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0f7455d5-9b9d-47f7-a0ad-df85dea8139c-ca-trust-extracted\") pod \"image-registry-779c8588cc-ns2s4\" (UID: \"0f7455d5-9b9d-47f7-a0ad-df85dea8139c\") " pod="openshift-image-registry/image-registry-779c8588cc-ns2s4"
Oct 31 19:19:44.183579 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:44.183486    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0f7455d5-9b9d-47f7-a0ad-df85dea8139c-installation-pull-secrets\") pod \"image-registry-779c8588cc-ns2s4\" (UID: \"0f7455d5-9b9d-47f7-a0ad-df85dea8139c\") " pod="openshift-image-registry/image-registry-779c8588cc-ns2s4"
Oct 31 19:19:44.183579 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:44.183510    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0f7455d5-9b9d-47f7-a0ad-df85dea8139c-trusted-ca\") pod \"image-registry-779c8588cc-ns2s4\" (UID: \"0f7455d5-9b9d-47f7-a0ad-df85dea8139c\") " pod="openshift-image-registry/image-registry-779c8588cc-ns2s4"
Oct 31 19:19:44.183686 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:44.183625    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c62a9b4b-1882-4d88-b486-ec5f3a6d18db-tmp\") pod \"downloads-6dd789c8b7-fs6gw\" (UID: \"c62a9b4b-1882-4d88-b486-ec5f3a6d18db\") " pod="openshift-console/downloads-6dd789c8b7-fs6gw"
Oct 31 19:19:44.185307 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:44.185284    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a4864737-8f10-4291-8529-57acff0fe5c2-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-5zb27\" (UID: \"a4864737-8f10-4291-8529-57acff0fe5c2\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5zb27"
Oct 31 19:19:44.196907 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:44.196888    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckc95\" (UniqueName: \"kubernetes.io/projected/c62a9b4b-1882-4d88-b486-ec5f3a6d18db-kube-api-access-ckc95\") pod \"downloads-6dd789c8b7-fs6gw\" (UID: \"c62a9b4b-1882-4d88-b486-ec5f3a6d18db\") " pod="openshift-console/downloads-6dd789c8b7-fs6gw"
Oct 31 19:19:44.284798 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:44.284726    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0f7455d5-9b9d-47f7-a0ad-df85dea8139c-registry-certificates\") pod \"image-registry-779c8588cc-ns2s4\" (UID: \"0f7455d5-9b9d-47f7-a0ad-df85dea8139c\") " pod="openshift-image-registry/image-registry-779c8588cc-ns2s4"
Oct 31 19:19:44.284798 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:44.284764    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0f7455d5-9b9d-47f7-a0ad-df85dea8139c-ca-trust-extracted\") pod \"image-registry-779c8588cc-ns2s4\" (UID: \"0f7455d5-9b9d-47f7-a0ad-df85dea8139c\") " pod="openshift-image-registry/image-registry-779c8588cc-ns2s4"
Oct 31 19:19:44.284798 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:44.284786    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0f7455d5-9b9d-47f7-a0ad-df85dea8139c-installation-pull-secrets\") pod \"image-registry-779c8588cc-ns2s4\" (UID: \"0f7455d5-9b9d-47f7-a0ad-df85dea8139c\") " pod="openshift-image-registry/image-registry-779c8588cc-ns2s4"
Oct 31 19:19:44.285069 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:44.284801    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0f7455d5-9b9d-47f7-a0ad-df85dea8139c-trusted-ca\") pod \"image-registry-779c8588cc-ns2s4\" (UID: \"0f7455d5-9b9d-47f7-a0ad-df85dea8139c\") " pod="openshift-image-registry/image-registry-779c8588cc-ns2s4"
Oct 31 19:19:44.285069 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:44.284824    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0f7455d5-9b9d-47f7-a0ad-df85dea8139c-image-registry-private-configuration\") pod \"image-registry-779c8588cc-ns2s4\" (UID: \"0f7455d5-9b9d-47f7-a0ad-df85dea8139c\") " pod="openshift-image-registry/image-registry-779c8588cc-ns2s4"
Oct 31 19:19:44.285069 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:44.284848    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0f7455d5-9b9d-47f7-a0ad-df85dea8139c-bound-sa-token\") pod \"image-registry-779c8588cc-ns2s4\" (UID: \"0f7455d5-9b9d-47f7-a0ad-df85dea8139c\") " pod="openshift-image-registry/image-registry-779c8588cc-ns2s4"
Oct 31 19:19:44.285069 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:44.284875    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0f7455d5-9b9d-47f7-a0ad-df85dea8139c-registry-tls\") pod \"image-registry-779c8588cc-ns2s4\" (UID: \"0f7455d5-9b9d-47f7-a0ad-df85dea8139c\") " pod="openshift-image-registry/image-registry-779c8588cc-ns2s4"
Oct 31 19:19:44.285264 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:44.285198    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5qzn\" (UniqueName: \"kubernetes.io/projected/0f7455d5-9b9d-47f7-a0ad-df85dea8139c-kube-api-access-b5qzn\") pod \"image-registry-779c8588cc-ns2s4\" (UID: \"0f7455d5-9b9d-47f7-a0ad-df85dea8139c\") " pod="openshift-image-registry/image-registry-779c8588cc-ns2s4"
Oct 31 19:19:44.285369 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:44.285320    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0f7455d5-9b9d-47f7-a0ad-df85dea8139c-ca-trust-extracted\") pod \"image-registry-779c8588cc-ns2s4\" (UID: \"0f7455d5-9b9d-47f7-a0ad-df85dea8139c\") " pod="openshift-image-registry/image-registry-779c8588cc-ns2s4"
Oct 31 19:19:44.285675 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:44.285652    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0f7455d5-9b9d-47f7-a0ad-df85dea8139c-registry-certificates\") pod \"image-registry-779c8588cc-ns2s4\" (UID: \"0f7455d5-9b9d-47f7-a0ad-df85dea8139c\") " pod="openshift-image-registry/image-registry-779c8588cc-ns2s4"
Oct 31 19:19:44.285939 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:44.285907    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0f7455d5-9b9d-47f7-a0ad-df85dea8139c-trusted-ca\") pod \"image-registry-779c8588cc-ns2s4\" (UID: \"0f7455d5-9b9d-47f7-a0ad-df85dea8139c\") " pod="openshift-image-registry/image-registry-779c8588cc-ns2s4"
Oct 31 19:19:44.287662 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:44.287639    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0f7455d5-9b9d-47f7-a0ad-df85dea8139c-installation-pull-secrets\") pod \"image-registry-779c8588cc-ns2s4\" (UID: \"0f7455d5-9b9d-47f7-a0ad-df85dea8139c\") " pod="openshift-image-registry/image-registry-779c8588cc-ns2s4"
Oct 31 19:19:44.287849 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:44.287825    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0f7455d5-9b9d-47f7-a0ad-df85dea8139c-image-registry-private-configuration\") pod \"image-registry-779c8588cc-ns2s4\" (UID: \"0f7455d5-9b9d-47f7-a0ad-df85dea8139c\") " pod="openshift-image-registry/image-registry-779c8588cc-ns2s4"
Oct 31 19:19:44.289034 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:44.289008    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0f7455d5-9b9d-47f7-a0ad-df85dea8139c-registry-tls\") pod \"image-registry-779c8588cc-ns2s4\" (UID: \"0f7455d5-9b9d-47f7-a0ad-df85dea8139c\") " pod="openshift-image-registry/image-registry-779c8588cc-ns2s4"
Oct 31 19:19:44.305199 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:44.305175    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0f7455d5-9b9d-47f7-a0ad-df85dea8139c-bound-sa-token\") pod \"image-registry-779c8588cc-ns2s4\" (UID: \"0f7455d5-9b9d-47f7-a0ad-df85dea8139c\") " pod="openshift-image-registry/image-registry-779c8588cc-ns2s4"
Oct 31 19:19:44.305573 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:44.305548    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5qzn\" (UniqueName: \"kubernetes.io/projected/0f7455d5-9b9d-47f7-a0ad-df85dea8139c-kube-api-access-b5qzn\") pod \"image-registry-779c8588cc-ns2s4\" (UID: \"0f7455d5-9b9d-47f7-a0ad-df85dea8139c\") " pod="openshift-image-registry/image-registry-779c8588cc-ns2s4"
Oct 31 19:19:44.310272 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:44.310226    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5zb27"
Oct 31 19:19:44.312204 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:44.312174    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:44.316275 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:44.316256    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-fs6gw"
Oct 31 19:19:44.412922 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:44.407459    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-779c8588cc-ns2s4"
Oct 31 19:19:44.597005 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:44.596978    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5zb27"]
Oct 31 19:19:44.597742 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:44.597706    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4864737_8f10_4291_8529_57acff0fe5c2.slice/crio-a5e67be322d30411e1e6bbf8e6b2fbf5e811a869cc0d604a2725fc12ac6a5c62 WatchSource:0}: Error finding container a5e67be322d30411e1e6bbf8e6b2fbf5e811a869cc0d604a2725fc12ac6a5c62: Status 404 returned error can't find the container with id a5e67be322d30411e1e6bbf8e6b2fbf5e811a869cc0d604a2725fc12ac6a5c62
Oct 31 19:19:44.611645 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:44.611623    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-fs6gw"]
Oct 31 19:19:44.614559 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:44.614526    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc62a9b4b_1882_4d88_b486_ec5f3a6d18db.slice/crio-36aeb517d95e82bfd552b330930de094695b5ca4da8e4b1e156d48d7094dc85b WatchSource:0}: Error finding container 36aeb517d95e82bfd552b330930de094695b5ca4da8e4b1e156d48d7094dc85b: Status 404 returned error can't find the container with id 36aeb517d95e82bfd552b330930de094695b5ca4da8e4b1e156d48d7094dc85b
Oct 31 19:19:44.697696 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:44.697664    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-779c8588cc-ns2s4"]
Oct 31 19:19:44.700341 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:44.700308    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f7455d5_9b9d_47f7_a0ad_df85dea8139c.slice/crio-be7dbd5d179a298307ff69dbfe02ff8bfca54515876a8923e4aee8321d20b2a6 WatchSource:0}: Error finding container be7dbd5d179a298307ff69dbfe02ff8bfca54515876a8923e4aee8321d20b2a6: Status 404 returned error can't find the container with id be7dbd5d179a298307ff69dbfe02ff8bfca54515876a8923e4aee8321d20b2a6
Oct 31 19:19:44.742700 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:44.742671    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-779c8588cc-ns2s4" event={"ID":"0f7455d5-9b9d-47f7-a0ad-df85dea8139c","Type":"ContainerStarted","Data":"be7dbd5d179a298307ff69dbfe02ff8bfca54515876a8923e4aee8321d20b2a6"}
Oct 31 19:19:44.743803 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:44.743782    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-fs6gw" event={"ID":"c62a9b4b-1882-4d88-b486-ec5f3a6d18db","Type":"ContainerStarted","Data":"36aeb517d95e82bfd552b330930de094695b5ca4da8e4b1e156d48d7094dc85b"}
Oct 31 19:19:44.745013 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:44.744988    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5zb27" event={"ID":"a4864737-8f10-4291-8529-57acff0fe5c2","Type":"ContainerStarted","Data":"a5e67be322d30411e1e6bbf8e6b2fbf5e811a869cc0d604a2725fc12ac6a5c62"}
Oct 31 19:19:45.751291 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:45.751248    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-779c8588cc-ns2s4" event={"ID":"0f7455d5-9b9d-47f7-a0ad-df85dea8139c","Type":"ContainerStarted","Data":"ca45d941cf63ba81d2413436b5e16a2b32e919d8eccd27a40d11d8d54a51359a"}
Oct 31 19:19:45.751738 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:45.751584    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-779c8588cc-ns2s4"
Oct 31 19:19:45.774553 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:45.774491    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-779c8588cc-ns2s4" podStartSLOduration=1.774474796 podStartE2EDuration="1.774474796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:19:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:19:45.773703754 +0000 UTC m=+16.346937882" watchObservedRunningTime="2026-10-31 19:19:45.774474796 +0000 UTC m=+16.347709130"
Oct 31 19:19:46.755964 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:46.755924    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5zb27" event={"ID":"a4864737-8f10-4291-8529-57acff0fe5c2","Type":"ContainerStarted","Data":"78beabda464260fbb82798623a4fa66ebc02c476d2a175c5507bd98813567b93"}
Oct 31 19:19:46.756522 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:46.756357    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5zb27"
Oct 31 19:19:46.761358 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:46.761320    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5zb27"
Oct 31 19:19:46.780375 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:46.780314    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5zb27" podStartSLOduration=2.568089068 podStartE2EDuration="3.780293476s" totalImagesPullingTime="1.212204408s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:19:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:19:46.778943021 +0000 UTC m=+17.352177150" watchObservedRunningTime="2026-10-31 19:19:46.780293476 +0000 UTC m=+17.353527604"
Oct 31 19:19:47.206828 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:47.206786    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:47.309070 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:47.309042    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:47.433321 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:47.433240    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:47.533148 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:47.533064    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:49.535621 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:49.535583    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:50.344286 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:50.344247    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:50.463725 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:50.463666    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:50.536793 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:50.536767    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mr9n2"
Oct 31 19:19:51.412833 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:51.412789    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:51.521709 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:51.521670    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:51.631147 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:51.631110    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-tfk67"]
Oct 31 19:19:51.636518 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:51.636482    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tfk67"
Oct 31 19:19:51.641655 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:51.641624    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 31 19:19:51.641936 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:51.641805    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 31 19:19:51.641936 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:51.641882    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 31 19:19:51.642032 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:51.641937    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 31 19:19:51.646887 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:51.646736    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 31 19:19:51.646887 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:51.646847    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 31 19:19:51.647090 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:51.646948    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-mxbfw\""
Oct 31 19:19:51.746432 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:51.746343    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3e0e15bd-a239-4a71-8f6c-2fbe061f03fd-node-exporter-wtmp\") pod \"node-exporter-tfk67\" (UID: \"3e0e15bd-a239-4a71-8f6c-2fbe061f03fd\") " pod="openshift-monitoring/node-exporter-tfk67"
Oct 31 19:19:51.746432 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:51.746393    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59nxm\" (UniqueName: \"kubernetes.io/projected/3e0e15bd-a239-4a71-8f6c-2fbe061f03fd-kube-api-access-59nxm\") pod \"node-exporter-tfk67\" (UID: \"3e0e15bd-a239-4a71-8f6c-2fbe061f03fd\") " pod="openshift-monitoring/node-exporter-tfk67"
Oct 31 19:19:51.746664 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:51.746431    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3e0e15bd-a239-4a71-8f6c-2fbe061f03fd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tfk67\" (UID: \"3e0e15bd-a239-4a71-8f6c-2fbe061f03fd\") " pod="openshift-monitoring/node-exporter-tfk67"
Oct 31 19:19:51.746664 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:51.746486    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3e0e15bd-a239-4a71-8f6c-2fbe061f03fd-node-exporter-accelerators-collector-config\") pod \"node-exporter-tfk67\" (UID: \"3e0e15bd-a239-4a71-8f6c-2fbe061f03fd\") " pod="openshift-monitoring/node-exporter-tfk67"
Oct 31 19:19:51.746664 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:51.746522    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3e0e15bd-a239-4a71-8f6c-2fbe061f03fd-node-exporter-textfile\") pod \"node-exporter-tfk67\" (UID: \"3e0e15bd-a239-4a71-8f6c-2fbe061f03fd\") " pod="openshift-monitoring/node-exporter-tfk67"
Oct 31 19:19:51.746664 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:51.746544    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3e0e15bd-a239-4a71-8f6c-2fbe061f03fd-root\") pod \"node-exporter-tfk67\" (UID: \"3e0e15bd-a239-4a71-8f6c-2fbe061f03fd\") " pod="openshift-monitoring/node-exporter-tfk67"
Oct 31 19:19:51.746664 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:51.746571    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3e0e15bd-a239-4a71-8f6c-2fbe061f03fd-node-exporter-tls\") pod \"node-exporter-tfk67\" (UID: \"3e0e15bd-a239-4a71-8f6c-2fbe061f03fd\") " pod="openshift-monitoring/node-exporter-tfk67"
Oct 31 19:19:51.746664 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:51.746607    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3e0e15bd-a239-4a71-8f6c-2fbe061f03fd-sys\") pod \"node-exporter-tfk67\" (UID: \"3e0e15bd-a239-4a71-8f6c-2fbe061f03fd\") " pod="openshift-monitoring/node-exporter-tfk67"
Oct 31 19:19:51.746664 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:51.746628    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3e0e15bd-a239-4a71-8f6c-2fbe061f03fd-metrics-client-ca\") pod \"node-exporter-tfk67\" (UID: \"3e0e15bd-a239-4a71-8f6c-2fbe061f03fd\") " pod="openshift-monitoring/node-exporter-tfk67"
Oct 31 19:19:51.783849 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:51.783805    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:51.847681 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:51.847643    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3e0e15bd-a239-4a71-8f6c-2fbe061f03fd-node-exporter-wtmp\") pod \"node-exporter-tfk67\" (UID: \"3e0e15bd-a239-4a71-8f6c-2fbe061f03fd\") " pod="openshift-monitoring/node-exporter-tfk67"
Oct 31 19:19:51.847681 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:51.847692    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59nxm\" (UniqueName: \"kubernetes.io/projected/3e0e15bd-a239-4a71-8f6c-2fbe061f03fd-kube-api-access-59nxm\") pod \"node-exporter-tfk67\" (UID: \"3e0e15bd-a239-4a71-8f6c-2fbe061f03fd\") " pod="openshift-monitoring/node-exporter-tfk67"
Oct 31 19:19:51.847946 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:51.847821    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3e0e15bd-a239-4a71-8f6c-2fbe061f03fd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tfk67\" (UID: \"3e0e15bd-a239-4a71-8f6c-2fbe061f03fd\") " pod="openshift-monitoring/node-exporter-tfk67"
Oct 31 19:19:51.847946 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:51.847875    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3e0e15bd-a239-4a71-8f6c-2fbe061f03fd-node-exporter-wtmp\") pod \"node-exporter-tfk67\" (UID: \"3e0e15bd-a239-4a71-8f6c-2fbe061f03fd\") " pod="openshift-monitoring/node-exporter-tfk67"
Oct 31 19:19:51.847946 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:51.847894    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3e0e15bd-a239-4a71-8f6c-2fbe061f03fd-node-exporter-accelerators-collector-config\") pod \"node-exporter-tfk67\" (UID: \"3e0e15bd-a239-4a71-8f6c-2fbe061f03fd\") " pod="openshift-monitoring/node-exporter-tfk67"
Oct 31 19:19:51.847946 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:51.847937    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3e0e15bd-a239-4a71-8f6c-2fbe061f03fd-node-exporter-textfile\") pod \"node-exporter-tfk67\" (UID: \"3e0e15bd-a239-4a71-8f6c-2fbe061f03fd\") " pod="openshift-monitoring/node-exporter-tfk67"
Oct 31 19:19:51.848124 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:51.847965    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3e0e15bd-a239-4a71-8f6c-2fbe061f03fd-root\") pod \"node-exporter-tfk67\" (UID: \"3e0e15bd-a239-4a71-8f6c-2fbe061f03fd\") " pod="openshift-monitoring/node-exporter-tfk67"
Oct 31 19:19:51.848124 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:51.847996    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3e0e15bd-a239-4a71-8f6c-2fbe061f03fd-node-exporter-tls\") pod \"node-exporter-tfk67\" (UID: \"3e0e15bd-a239-4a71-8f6c-2fbe061f03fd\") " pod="openshift-monitoring/node-exporter-tfk67"
Oct 31 19:19:51.848124 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:51.848036    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3e0e15bd-a239-4a71-8f6c-2fbe061f03fd-sys\") pod \"node-exporter-tfk67\" (UID: \"3e0e15bd-a239-4a71-8f6c-2fbe061f03fd\") " pod="openshift-monitoring/node-exporter-tfk67"
Oct 31 19:19:51.848124 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:51.848057    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3e0e15bd-a239-4a71-8f6c-2fbe061f03fd-metrics-client-ca\") pod \"node-exporter-tfk67\" (UID: \"3e0e15bd-a239-4a71-8f6c-2fbe061f03fd\") " pod="openshift-monitoring/node-exporter-tfk67"
Oct 31 19:19:51.848355 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:51.848267    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3e0e15bd-a239-4a71-8f6c-2fbe061f03fd-root\") pod \"node-exporter-tfk67\" (UID: \"3e0e15bd-a239-4a71-8f6c-2fbe061f03fd\") " pod="openshift-monitoring/node-exporter-tfk67"
Oct 31 19:19:51.848355 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:51.848272    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3e0e15bd-a239-4a71-8f6c-2fbe061f03fd-sys\") pod \"node-exporter-tfk67\" (UID: \"3e0e15bd-a239-4a71-8f6c-2fbe061f03fd\") " pod="openshift-monitoring/node-exporter-tfk67"
Oct 31 19:19:51.848626 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:51.848599    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3e0e15bd-a239-4a71-8f6c-2fbe061f03fd-node-exporter-textfile\") pod \"node-exporter-tfk67\" (UID: \"3e0e15bd-a239-4a71-8f6c-2fbe061f03fd\") " pod="openshift-monitoring/node-exporter-tfk67"
Oct 31 19:19:51.849005 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:51.848959    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3e0e15bd-a239-4a71-8f6c-2fbe061f03fd-metrics-client-ca\") pod \"node-exporter-tfk67\" (UID: \"3e0e15bd-a239-4a71-8f6c-2fbe061f03fd\") " pod="openshift-monitoring/node-exporter-tfk67"
Oct 31 19:19:51.849005 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:51.848974    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3e0e15bd-a239-4a71-8f6c-2fbe061f03fd-node-exporter-accelerators-collector-config\") pod \"node-exporter-tfk67\" (UID: \"3e0e15bd-a239-4a71-8f6c-2fbe061f03fd\") " pod="openshift-monitoring/node-exporter-tfk67"
Oct 31 19:19:51.850596 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:51.850574    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3e0e15bd-a239-4a71-8f6c-2fbe061f03fd-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tfk67\" (UID: \"3e0e15bd-a239-4a71-8f6c-2fbe061f03fd\") " pod="openshift-monitoring/node-exporter-tfk67"
Oct 31 19:19:51.852831 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:51.852792    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3e0e15bd-a239-4a71-8f6c-2fbe061f03fd-node-exporter-tls\") pod \"node-exporter-tfk67\" (UID: \"3e0e15bd-a239-4a71-8f6c-2fbe061f03fd\") " pod="openshift-monitoring/node-exporter-tfk67"
Oct 31 19:19:51.860276 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:51.860245    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59nxm\" (UniqueName: \"kubernetes.io/projected/3e0e15bd-a239-4a71-8f6c-2fbe061f03fd-kube-api-access-59nxm\") pod \"node-exporter-tfk67\" (UID: \"3e0e15bd-a239-4a71-8f6c-2fbe061f03fd\") " pod="openshift-monitoring/node-exporter-tfk67"
Oct 31 19:19:51.898063 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:51.898016    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:51.959231 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:51.958960    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tfk67"
Oct 31 19:19:51.975616 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:51.975574    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e0e15bd_a239_4a71_8f6c_2fbe061f03fd.slice/crio-76bf466613ed68c2e665e27d5c5e0e26faa56f3f9e54b29a053179897ffb1eac WatchSource:0}: Error finding container 76bf466613ed68c2e665e27d5c5e0e26faa56f3f9e54b29a053179897ffb1eac: Status 404 returned error can't find the container with id 76bf466613ed68c2e665e27d5c5e0e26faa56f3f9e54b29a053179897ffb1eac
Oct 31 19:19:52.020097 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:52.020010    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:52.138184 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:52.138135    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:52.640409 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:52.640371    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:52.757100 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:52.756946    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:52.781699 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:52.781662    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tfk67" event={"ID":"3e0e15bd-a239-4a71-8f6c-2fbe061f03fd","Type":"ContainerStarted","Data":"76bf466613ed68c2e665e27d5c5e0e26faa56f3f9e54b29a053179897ffb1eac"}
Oct 31 19:19:52.920448 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:52.920413    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:53.096583 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:53.096490    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:53.198160 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:53.198089    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:53.786865 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:53.786831    4645 generic.go:356] "Generic (PLEG): container finished" podID="3e0e15bd-a239-4a71-8f6c-2fbe061f03fd" containerID="555165e279ee36ba55f4069a8d9133d23f0dc94f9d685ee75452b912f6a7bab3" exitCode=0
Oct 31 19:19:53.787276 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:53.786896    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tfk67" event={"ID":"3e0e15bd-a239-4a71-8f6c-2fbe061f03fd","Type":"ContainerDied","Data":"555165e279ee36ba55f4069a8d9133d23f0dc94f9d685ee75452b912f6a7bab3"}
Oct 31 19:19:54.707865 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:54.707833    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:54.792756 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:54.792452    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tfk67" event={"ID":"3e0e15bd-a239-4a71-8f6c-2fbe061f03fd","Type":"ContainerStarted","Data":"2065ac55f1dedfdf8c58ddce91aadf3a2bdc026b88a4bef1cc27b1eb9b196743"}
Oct 31 19:19:54.793204 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:54.792806    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tfk67" event={"ID":"3e0e15bd-a239-4a71-8f6c-2fbe061f03fd","Type":"ContainerStarted","Data":"8d07ff5db1d573969f6c422fb4bf5fc6c401bb9f7c67548971a5b9fa54f860ef"}
Oct 31 19:19:54.811405 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:54.811354    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:54.815457 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:54.815390    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-tfk67" podStartSLOduration=2.74595824 podStartE2EDuration="3.815376548s" totalImagesPullingTime="994.134831ms" totalInitContainerRuntime="75.283477ms" isStatefulPod=true podCreationTimestamp="2026-10-31 19:19:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:19:54.813673689 +0000 UTC m=+25.386907830" watchObservedRunningTime="2026-10-31 19:19:54.815376548 +0000 UTC m=+25.388610676"
Oct 31 19:19:55.341797 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:55.341745    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:55.442830 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:55.442784    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:56.423614 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:56.423584    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-hwpwv"]
Oct 31 19:19:56.442494 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:56.442465    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:56.451444 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:56.451421    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-hwpwv"]
Oct 31 19:19:56.451573 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:56.451531    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-hwpwv"
Oct 31 19:19:56.454889 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:56.454858    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Oct 31 19:19:56.455134 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:56.455109    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-l5w6w\""
Oct 31 19:19:56.542755 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:56.542668    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:56.585217 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:56.585185    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b4b8cb87-d398-43fd-a6da-b56a4d3d8e9b-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-hwpwv\" (UID: \"b4b8cb87-d398-43fd-a6da-b56a4d3d8e9b\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-hwpwv"
Oct 31 19:19:56.585395 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:56.585303    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rbsm\" (UniqueName: \"kubernetes.io/projected/b4b8cb87-d398-43fd-a6da-b56a4d3d8e9b-kube-api-access-5rbsm\") pod \"monitoring-plugin-77f4f76594-hwpwv\" (UID: \"b4b8cb87-d398-43fd-a6da-b56a4d3d8e9b\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-hwpwv"
Oct 31 19:19:56.686511 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:56.686443    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rbsm\" (UniqueName: \"kubernetes.io/projected/b4b8cb87-d398-43fd-a6da-b56a4d3d8e9b-kube-api-access-5rbsm\") pod \"monitoring-plugin-77f4f76594-hwpwv\" (UID: \"b4b8cb87-d398-43fd-a6da-b56a4d3d8e9b\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-hwpwv"
Oct 31 19:19:56.686631 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:56.686519    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b4b8cb87-d398-43fd-a6da-b56a4d3d8e9b-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-hwpwv\" (UID: \"b4b8cb87-d398-43fd-a6da-b56a4d3d8e9b\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-hwpwv"
Oct 31 19:19:56.689144 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:56.689119    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b4b8cb87-d398-43fd-a6da-b56a4d3d8e9b-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-hwpwv\" (UID: \"b4b8cb87-d398-43fd-a6da-b56a4d3d8e9b\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-hwpwv"
Oct 31 19:19:56.697912 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:56.697889    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rbsm\" (UniqueName: \"kubernetes.io/projected/b4b8cb87-d398-43fd-a6da-b56a4d3d8e9b-kube-api-access-5rbsm\") pod \"monitoring-plugin-77f4f76594-hwpwv\" (UID: \"b4b8cb87-d398-43fd-a6da-b56a4d3d8e9b\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-hwpwv"
Oct 31 19:19:56.747045 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:56.746994    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:56.775883 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:56.775855    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-hwpwv"
Oct 31 19:19:57.063565 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:57.063492    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-hwpwv"]
Oct 31 19:19:57.067707 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:19:57.067673    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4b8cb87_d398_43fd_a6da_b56a4d3d8e9b.slice/crio-cbfe8b84727b6106ef381729b44bf74e0c645956d16caeabc9c7ce81808bf3df WatchSource:0}: Error finding container cbfe8b84727b6106ef381729b44bf74e0c645956d16caeabc9c7ce81808bf3df: Status 404 returned error can't find the container with id cbfe8b84727b6106ef381729b44bf74e0c645956d16caeabc9c7ce81808bf3df
Oct 31 19:19:57.217631 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:57.217605    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-4j5p9"
Oct 31 19:19:57.805679 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:57.805639    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-hwpwv" event={"ID":"b4b8cb87-d398-43fd-a6da-b56a4d3d8e9b","Type":"ContainerStarted","Data":"cbfe8b84727b6106ef381729b44bf74e0c645956d16caeabc9c7ce81808bf3df"}
Oct 31 19:19:57.927605 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:57.927544    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:58.236016 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:58.235990    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:58.335796 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:58.335760    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:19:59.815574 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:59.815528    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-hwpwv" event={"ID":"b4b8cb87-d398-43fd-a6da-b56a4d3d8e9b","Type":"ContainerStarted","Data":"292e474c03bc6bc79884554f35b225eea70142ca3bfa412df1ddbb20e2b38946"}
Oct 31 19:19:59.816011 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:59.815848    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-hwpwv"
Oct 31 19:19:59.821816 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:59.821794    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-hwpwv"
Oct 31 19:19:59.832965 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:19:59.832916    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-hwpwv" podStartSLOduration=1.888898164 podStartE2EDuration="3.832901252s" totalImagesPullingTime="1.944003088s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:19:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:19:59.832003109 +0000 UTC m=+30.405237255" watchObservedRunningTime="2026-10-31 19:19:59.832901252 +0000 UTC m=+30.406135384"
Oct 31 19:20:00.040740 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:00.040697    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:20:00.139956 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:00.139879    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:20:01.824108 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:01.824068    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-fs6gw" event={"ID":"c62a9b4b-1882-4d88-b486-ec5f3a6d18db","Type":"ContainerStarted","Data":"82f322a3639b2c5033b69e3aeabcc825cde40a5f30e13658c56740ed3de9dc91"}
Oct 31 19:20:01.824623 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:01.824319    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-fs6gw"
Oct 31 19:20:01.838221 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:01.838195    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-fs6gw"
Oct 31 19:20:01.842969 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:01.842923    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-fs6gw" podStartSLOduration=2.683995588 podStartE2EDuration="18.842909469s" totalImagesPullingTime="16.158913881s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:19:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:20:01.84234955 +0000 UTC m=+32.415583715" watchObservedRunningTime="2026-10-31 19:20:01.842909469 +0000 UTC m=+32.416143597"
Oct 31 19:20:02.431361 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:02.431307    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:20:02.551315 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:02.551232    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:20:03.552793 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:03.552731    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:20:03.670251 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:03.670216    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:20:05.207994 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:05.207955    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:20:05.350285 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:05.350257    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:20:06.161621 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:06.161588    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-68969f7467-lpqh4"]
Oct 31 19:20:06.185851 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:06.185817    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:20:06.188627 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:06.188602    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68969f7467-lpqh4"]
Oct 31 19:20:06.188759 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:06.188706    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68969f7467-lpqh4"
Oct 31 19:20:06.194449 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:06.194395    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 31 19:20:06.194449 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:06.194399    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 31 19:20:06.194594 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:06.194470    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 31 19:20:06.200161 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:06.200138    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 31 19:20:06.226803 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:06.226780    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 31 19:20:06.227109 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:06.226863    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-dw4k4\""
Oct 31 19:20:06.227109 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:06.226884    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 31 19:20:06.249119 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:20:06.249091    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 19:20:06.262560 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:06.262535    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/207efcfa-0ed5-4b16-b94c-b881ac06b9f6-tmp\") pod \"console-68969f7467-lpqh4\" (UID: \"207efcfa-0ed5-4b16-b94c-b881ac06b9f6\") " pod="openshift-console/console-68969f7467-lpqh4"
Oct 31 19:20:06.262655 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:06.262578    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fts6\" (UniqueName: \"kubernetes.io/projected/207efcfa-0ed5-4b16-b94c-b881ac06b9f6-kube-api-access-6fts6\") pod \"console-68969f7467-lpqh4\" (UID: \"207efcfa-0ed5-4b16-b94c-b881ac06b9f6\") " pod="openshift-console/console-68969f7467-lpqh4"
Oct 31 19:20:06.262655 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:06.262607    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/207efcfa-0ed5-4b16-b94c-b881ac06b9f6-console-config\") pod \"console-68969f7467-lpqh4\" (UID: \"207efcfa-0ed5-4b16-b94c-b881ac06b9f6\") " pod="openshift-console/console-68969f7467-lpqh4"
Oct 31 19:20:06.262791 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:06.262667    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/207efcfa-0ed5-4b16-b94c-b881ac06b9f6-trusted-ca-bundle\") pod \"console-68969f7467-lpqh4\" (UID: \"207efcfa-0ed5-4b16-b94c-b881ac06b9f6\") " pod="openshift-console/console-68969f7467-lpqh4"
Oct 31 19:20:06.262791 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:06.262781    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/207efcfa-0ed5-4b16-b94c-b881ac06b9f6-console-serving-cert\") pod \"console-68969f7467-lpqh4\" (UID: \"207efcfa-0ed5-4b16-b94c-b881ac06b9f6\") " pod="openshift-console/console-68969f7467-lpqh4"
Oct 31 19:20:06.262871 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:06.262802    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/207efcfa-0ed5-4b16-b94c-b881ac06b9f6-service-ca\") pod \"console-68969f7467-lpqh4\" (UID: \"207efcfa-0ed5-4b16-b94c-b881ac06b9f6\") " pod="openshift-console/console-68969f7467-lpqh4"
Oct 31 19:20:06.262871 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:06.262829    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/207efcfa-0ed5-4b16-b94c-b881ac06b9f6-console-oauth-config\") pod \"console-68969f7467-lpqh4\" (UID: \"207efcfa-0ed5-4b16-b94c-b881ac06b9f6\") " pod="openshift-console/console-68969f7467-lpqh4"
Oct 31 19:20:06.262972 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:06.262953    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/207efcfa-0ed5-4b16-b94c-b881ac06b9f6-oauth-serving-cert\") pod \"console-68969f7467-lpqh4\" (UID: \"207efcfa-0ed5-4b16-b94c-b881ac06b9f6\") " pod="openshift-console/console-68969f7467-lpqh4"
Oct 31 19:20:06.313193 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:06.313166    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:20:06.363948 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:06.363923    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/207efcfa-0ed5-4b16-b94c-b881ac06b9f6-console-serving-cert\") pod \"console-68969f7467-lpqh4\" (UID: \"207efcfa-0ed5-4b16-b94c-b881ac06b9f6\") " pod="openshift-console/console-68969f7467-lpqh4"
Oct 31 19:20:06.364068 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:06.363966    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/207efcfa-0ed5-4b16-b94c-b881ac06b9f6-service-ca\") pod \"console-68969f7467-lpqh4\" (UID: \"207efcfa-0ed5-4b16-b94c-b881ac06b9f6\") " pod="openshift-console/console-68969f7467-lpqh4"
Oct 31 19:20:06.364068 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:06.363998    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/207efcfa-0ed5-4b16-b94c-b881ac06b9f6-console-oauth-config\") pod \"console-68969f7467-lpqh4\" (UID: \"207efcfa-0ed5-4b16-b94c-b881ac06b9f6\") " pod="openshift-console/console-68969f7467-lpqh4"
Oct 31 19:20:06.364068 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:06.364035    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/207efcfa-0ed5-4b16-b94c-b881ac06b9f6-oauth-serving-cert\") pod \"console-68969f7467-lpqh4\" (UID: \"207efcfa-0ed5-4b16-b94c-b881ac06b9f6\") " pod="openshift-console/console-68969f7467-lpqh4"
Oct 31 19:20:06.364068 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:06.364066    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/207efcfa-0ed5-4b16-b94c-b881ac06b9f6-tmp\") pod \"console-68969f7467-lpqh4\" (UID: \"207efcfa-0ed5-4b16-b94c-b881ac06b9f6\") " pod="openshift-console/console-68969f7467-lpqh4"
Oct 31 19:20:06.364293 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:06.364103    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fts6\" (UniqueName: \"kubernetes.io/projected/207efcfa-0ed5-4b16-b94c-b881ac06b9f6-kube-api-access-6fts6\") pod \"console-68969f7467-lpqh4\" (UID: \"207efcfa-0ed5-4b16-b94c-b881ac06b9f6\") " pod="openshift-console/console-68969f7467-lpqh4"
Oct 31 19:20:06.364363 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:06.364312    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/207efcfa-0ed5-4b16-b94c-b881ac06b9f6-console-config\") pod \"console-68969f7467-lpqh4\" (UID: \"207efcfa-0ed5-4b16-b94c-b881ac06b9f6\") " pod="openshift-console/console-68969f7467-lpqh4"
Oct 31 19:20:06.364423 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:06.364381    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/207efcfa-0ed5-4b16-b94c-b881ac06b9f6-trusted-ca-bundle\") pod \"console-68969f7467-lpqh4\" (UID: \"207efcfa-0ed5-4b16-b94c-b881ac06b9f6\") " pod="openshift-console/console-68969f7467-lpqh4"
Oct 31 19:20:06.364823 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:06.364800    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/207efcfa-0ed5-4b16-b94c-b881ac06b9f6-tmp\") pod \"console-68969f7467-lpqh4\" (UID: \"207efcfa-0ed5-4b16-b94c-b881ac06b9f6\") " pod="openshift-console/console-68969f7467-lpqh4"
Oct 31 19:20:06.365319 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:06.365294    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/207efcfa-0ed5-4b16-b94c-b881ac06b9f6-service-ca\") pod \"console-68969f7467-lpqh4\" (UID: \"207efcfa-0ed5-4b16-b94c-b881ac06b9f6\") " pod="openshift-console/console-68969f7467-lpqh4"
Oct 31 19:20:06.365573 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:06.365488    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/207efcfa-0ed5-4b16-b94c-b881ac06b9f6-trusted-ca-bundle\") pod \"console-68969f7467-lpqh4\" (UID: \"207efcfa-0ed5-4b16-b94c-b881ac06b9f6\") " pod="openshift-console/console-68969f7467-lpqh4"
Oct 31 19:20:06.366008 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:06.365988    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/207efcfa-0ed5-4b16-b94c-b881ac06b9f6-oauth-serving-cert\") pod \"console-68969f7467-lpqh4\" (UID: \"207efcfa-0ed5-4b16-b94c-b881ac06b9f6\") " pod="openshift-console/console-68969f7467-lpqh4"
Oct 31 19:20:06.366696 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:06.366670    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/207efcfa-0ed5-4b16-b94c-b881ac06b9f6-console-config\") pod \"console-68969f7467-lpqh4\" (UID: \"207efcfa-0ed5-4b16-b94c-b881ac06b9f6\") " pod="openshift-console/console-68969f7467-lpqh4"
Oct 31 19:20:06.366805 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:06.366737    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/207efcfa-0ed5-4b16-b94c-b881ac06b9f6-console-serving-cert\") pod \"console-68969f7467-lpqh4\" (UID: \"207efcfa-0ed5-4b16-b94c-b881ac06b9f6\") " pod="openshift-console/console-68969f7467-lpqh4"
Oct 31 19:20:06.366866 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:06.366849    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/207efcfa-0ed5-4b16-b94c-b881ac06b9f6-console-oauth-config\") pod \"console-68969f7467-lpqh4\" (UID: \"207efcfa-0ed5-4b16-b94c-b881ac06b9f6\") " pod="openshift-console/console-68969f7467-lpqh4"
Oct 31 19:20:06.376077 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:06.376057    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fts6\" (UniqueName: \"kubernetes.io/projected/207efcfa-0ed5-4b16-b94c-b881ac06b9f6-kube-api-access-6fts6\") pod \"console-68969f7467-lpqh4\" (UID: \"207efcfa-0ed5-4b16-b94c-b881ac06b9f6\") " pod="openshift-console/console-68969f7467-lpqh4"
Oct 31 19:20:06.522477 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:06.522411    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68969f7467-lpqh4"
Oct 31 19:20:06.761994 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:06.761961    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-779c8588cc-ns2s4"
Oct 31 19:20:06.849057 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:06.848985    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68969f7467-lpqh4"]
Oct 31 19:20:06.856168 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:20:06.856125    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod207efcfa_0ed5_4b16_b94c_b881ac06b9f6.slice/crio-f303559aa2951dd88927d850a106c7200d384b065002ee60685c6f84563fdfcb WatchSource:0}: Error finding container f303559aa2951dd88927d850a106c7200d384b065002ee60685c6f84563fdfcb: Status 404 returned error can't find the container with id f303559aa2951dd88927d850a106c7200d384b065002ee60685c6f84563fdfcb
Oct 31 19:20:06.963511 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:06.963473    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:20:07.076459 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:07.076432    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:20:07.848695 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:07.848648    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68969f7467-lpqh4" event={"ID":"207efcfa-0ed5-4b16-b94c-b881ac06b9f6","Type":"ContainerStarted","Data":"f303559aa2951dd88927d850a106c7200d384b065002ee60685c6f84563fdfcb"}
Oct 31 19:20:08.808161 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:08.808128    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:20:08.919202 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:08.919164    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:20:09.244217 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:09.244131    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:20:09.360627 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:09.360592    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:20:10.860974 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:10.860934    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68969f7467-lpqh4" event={"ID":"207efcfa-0ed5-4b16-b94c-b881ac06b9f6","Type":"ContainerStarted","Data":"490204aa1aafa9b7e55e2208ddb5ad412efe4713476e4f6fd73da8d301bd02e9"}
Oct 31 19:20:10.883123 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:10.883066    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-68969f7467-lpqh4" podStartSLOduration=1.409987152 podStartE2EDuration="4.883054074s" totalImagesPullingTime="3.473066922s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:20:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:20:10.882657395 +0000 UTC m=+41.455891524" watchObservedRunningTime="2026-10-31 19:20:10.883054074 +0000 UTC m=+41.456288203"
Oct 31 19:20:11.355861 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:11.355833    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:20:11.456881 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:11.456848    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:20:14.168566 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:14.168473    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:20:14.268588 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:14.268550    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:20:14.394497 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:14.394460    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:20:14.521480 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:14.521413    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:20:16.522565 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:16.522535    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-68969f7467-lpqh4"
Oct 31 19:20:16.523034 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:16.522587    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-68969f7467-lpqh4"
Oct 31 19:20:16.527280 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:16.527259    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-68969f7467-lpqh4"
Oct 31 19:20:16.886724 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:16.886658    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-68969f7467-lpqh4"
Oct 31 19:20:17.436479 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:17.436438    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:20:17.536012 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:17.535971    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:20:20.843233 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:20.843199    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:20:20.944013 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:20.943982    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:20:21.203439 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:21.203401    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:20:21.310858 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:21.310824    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:20:30.967422 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:30.967377    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:20:31.075718 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:31.075681    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:20:32.435233 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:32.435193    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:20:32.524124 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:32.524088    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-54cfdcd6c9-wvfn5"]
Oct 31 19:20:32.540967 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:32.540938    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:20:32.604602 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:32.604578    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54cfdcd6c9-wvfn5"]
Oct 31 19:20:32.604708 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:32.604670    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54cfdcd6c9-wvfn5"
Oct 31 19:20:32.740787 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:32.740728    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3d63fb6a-ab64-41a9-add1-c05801100442-console-serving-cert\") pod \"console-54cfdcd6c9-wvfn5\" (UID: \"3d63fb6a-ab64-41a9-add1-c05801100442\") " pod="openshift-console/console-54cfdcd6c9-wvfn5"
Oct 31 19:20:32.740787 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:32.740756    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d63fb6a-ab64-41a9-add1-c05801100442-trusted-ca-bundle\") pod \"console-54cfdcd6c9-wvfn5\" (UID: \"3d63fb6a-ab64-41a9-add1-c05801100442\") " pod="openshift-console/console-54cfdcd6c9-wvfn5"
Oct 31 19:20:32.740787 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:32.740781    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3d63fb6a-ab64-41a9-add1-c05801100442-console-oauth-config\") pod \"console-54cfdcd6c9-wvfn5\" (UID: \"3d63fb6a-ab64-41a9-add1-c05801100442\") " pod="openshift-console/console-54cfdcd6c9-wvfn5"
Oct 31 19:20:32.740950 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:32.740848    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3d63fb6a-ab64-41a9-add1-c05801100442-service-ca\") pod \"console-54cfdcd6c9-wvfn5\" (UID: \"3d63fb6a-ab64-41a9-add1-c05801100442\") " pod="openshift-console/console-54cfdcd6c9-wvfn5"
Oct 31 19:20:32.740950 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:32.740938    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3d63fb6a-ab64-41a9-add1-c05801100442-console-config\") pod \"console-54cfdcd6c9-wvfn5\" (UID: \"3d63fb6a-ab64-41a9-add1-c05801100442\") " pod="openshift-console/console-54cfdcd6c9-wvfn5"
Oct 31 19:20:32.741012 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:32.740959    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mxjx\" (UniqueName: \"kubernetes.io/projected/3d63fb6a-ab64-41a9-add1-c05801100442-kube-api-access-4mxjx\") pod \"console-54cfdcd6c9-wvfn5\" (UID: \"3d63fb6a-ab64-41a9-add1-c05801100442\") " pod="openshift-console/console-54cfdcd6c9-wvfn5"
Oct 31 19:20:32.741012 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:32.740982    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d63fb6a-ab64-41a9-add1-c05801100442-tmp\") pod \"console-54cfdcd6c9-wvfn5\" (UID: \"3d63fb6a-ab64-41a9-add1-c05801100442\") " pod="openshift-console/console-54cfdcd6c9-wvfn5"
Oct 31 19:20:32.741012 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:32.740996    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3d63fb6a-ab64-41a9-add1-c05801100442-oauth-serving-cert\") pod \"console-54cfdcd6c9-wvfn5\" (UID: \"3d63fb6a-ab64-41a9-add1-c05801100442\") " pod="openshift-console/console-54cfdcd6c9-wvfn5"
Oct 31 19:20:32.842339 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:32.842309    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d63fb6a-ab64-41a9-add1-c05801100442-trusted-ca-bundle\") pod \"console-54cfdcd6c9-wvfn5\" (UID: \"3d63fb6a-ab64-41a9-add1-c05801100442\") " pod="openshift-console/console-54cfdcd6c9-wvfn5"
Oct 31 19:20:32.842436 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:32.842369    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3d63fb6a-ab64-41a9-add1-c05801100442-console-oauth-config\") pod \"console-54cfdcd6c9-wvfn5\" (UID: \"3d63fb6a-ab64-41a9-add1-c05801100442\") " pod="openshift-console/console-54cfdcd6c9-wvfn5"
Oct 31 19:20:32.842436 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:32.842403    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3d63fb6a-ab64-41a9-add1-c05801100442-service-ca\") pod \"console-54cfdcd6c9-wvfn5\" (UID: \"3d63fb6a-ab64-41a9-add1-c05801100442\") " pod="openshift-console/console-54cfdcd6c9-wvfn5"
Oct 31 19:20:32.842436 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:32.842426    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3d63fb6a-ab64-41a9-add1-c05801100442-console-config\") pod \"console-54cfdcd6c9-wvfn5\" (UID: \"3d63fb6a-ab64-41a9-add1-c05801100442\") " pod="openshift-console/console-54cfdcd6c9-wvfn5"
Oct 31 19:20:32.842599 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:32.842448    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mxjx\" (UniqueName: \"kubernetes.io/projected/3d63fb6a-ab64-41a9-add1-c05801100442-kube-api-access-4mxjx\") pod \"console-54cfdcd6c9-wvfn5\" (UID: \"3d63fb6a-ab64-41a9-add1-c05801100442\") " pod="openshift-console/console-54cfdcd6c9-wvfn5"
Oct 31 19:20:32.842599 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:32.842487    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d63fb6a-ab64-41a9-add1-c05801100442-tmp\") pod \"console-54cfdcd6c9-wvfn5\" (UID: \"3d63fb6a-ab64-41a9-add1-c05801100442\") " pod="openshift-console/console-54cfdcd6c9-wvfn5"
Oct 31 19:20:32.842838 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:32.842757    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3d63fb6a-ab64-41a9-add1-c05801100442-oauth-serving-cert\") pod \"console-54cfdcd6c9-wvfn5\" (UID: \"3d63fb6a-ab64-41a9-add1-c05801100442\") " pod="openshift-console/console-54cfdcd6c9-wvfn5"
Oct 31 19:20:32.842955 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:32.842891    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3d63fb6a-ab64-41a9-add1-c05801100442-console-serving-cert\") pod \"console-54cfdcd6c9-wvfn5\" (UID: \"3d63fb6a-ab64-41a9-add1-c05801100442\") " pod="openshift-console/console-54cfdcd6c9-wvfn5"
Oct 31 19:20:32.842955 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:32.842900    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d63fb6a-ab64-41a9-add1-c05801100442-tmp\") pod \"console-54cfdcd6c9-wvfn5\" (UID: \"3d63fb6a-ab64-41a9-add1-c05801100442\") " pod="openshift-console/console-54cfdcd6c9-wvfn5"
Oct 31 19:20:32.843426 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:32.843391    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3d63fb6a-ab64-41a9-add1-c05801100442-console-config\") pod \"console-54cfdcd6c9-wvfn5\" (UID: \"3d63fb6a-ab64-41a9-add1-c05801100442\") " pod="openshift-console/console-54cfdcd6c9-wvfn5"
Oct 31 19:20:32.843539 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:32.843458    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3d63fb6a-ab64-41a9-add1-c05801100442-service-ca\") pod \"console-54cfdcd6c9-wvfn5\" (UID: \"3d63fb6a-ab64-41a9-add1-c05801100442\") " pod="openshift-console/console-54cfdcd6c9-wvfn5"
Oct 31 19:20:32.843539 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:32.843528    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3d63fb6a-ab64-41a9-add1-c05801100442-oauth-serving-cert\") pod \"console-54cfdcd6c9-wvfn5\" (UID: \"3d63fb6a-ab64-41a9-add1-c05801100442\") " pod="openshift-console/console-54cfdcd6c9-wvfn5"
Oct 31 19:20:32.843656 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:32.843552    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d63fb6a-ab64-41a9-add1-c05801100442-trusted-ca-bundle\") pod \"console-54cfdcd6c9-wvfn5\" (UID: \"3d63fb6a-ab64-41a9-add1-c05801100442\") " pod="openshift-console/console-54cfdcd6c9-wvfn5"
Oct 31 19:20:32.845449 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:32.845416    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3d63fb6a-ab64-41a9-add1-c05801100442-console-oauth-config\") pod \"console-54cfdcd6c9-wvfn5\" (UID: \"3d63fb6a-ab64-41a9-add1-c05801100442\") " pod="openshift-console/console-54cfdcd6c9-wvfn5"
Oct 31 19:20:32.845751 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:32.845728    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3d63fb6a-ab64-41a9-add1-c05801100442-console-serving-cert\") pod \"console-54cfdcd6c9-wvfn5\" (UID: \"3d63fb6a-ab64-41a9-add1-c05801100442\") " pod="openshift-console/console-54cfdcd6c9-wvfn5"
Oct 31 19:20:32.854660 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:32.854632    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mxjx\" (UniqueName: \"kubernetes.io/projected/3d63fb6a-ab64-41a9-add1-c05801100442-kube-api-access-4mxjx\") pod \"console-54cfdcd6c9-wvfn5\" (UID: \"3d63fb6a-ab64-41a9-add1-c05801100442\") " pod="openshift-console/console-54cfdcd6c9-wvfn5"
Oct 31 19:20:32.935612 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:32.935575    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54cfdcd6c9-wvfn5"
Oct 31 19:20:33.243755 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:33.243725    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54cfdcd6c9-wvfn5"]
Oct 31 19:20:33.246735 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:20:33.246695    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d63fb6a_ab64_41a9_add1_c05801100442.slice/crio-4ff5289e62ee6093a91d0f7081db09ad4bd561d9844f2c4819e073b60be2b8fe WatchSource:0}: Error finding container 4ff5289e62ee6093a91d0f7081db09ad4bd561d9844f2c4819e073b60be2b8fe: Status 404 returned error can't find the container with id 4ff5289e62ee6093a91d0f7081db09ad4bd561d9844f2c4819e073b60be2b8fe
Oct 31 19:20:33.575315 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:33.575240    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:20:33.672970 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:33.672936    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:20:33.942883 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:33.942854    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54cfdcd6c9-wvfn5" event={"ID":"3d63fb6a-ab64-41a9-add1-c05801100442","Type":"ContainerStarted","Data":"2959a2c79d2003b719d3945af47a4c8369b6110256b0ccdf5f7434a3a31ddf88"}
Oct 31 19:20:33.942994 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:33.942900    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54cfdcd6c9-wvfn5" event={"ID":"3d63fb6a-ab64-41a9-add1-c05801100442","Type":"ContainerStarted","Data":"4ff5289e62ee6093a91d0f7081db09ad4bd561d9844f2c4819e073b60be2b8fe"}
Oct 31 19:20:33.962834 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:33.962777    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-54cfdcd6c9-wvfn5" podStartSLOduration=1.962765039 podStartE2EDuration="1.962765039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:20:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:20:33.962392295 +0000 UTC m=+64.535626423" watchObservedRunningTime="2026-10-31 19:20:33.962765039 +0000 UTC m=+64.535999168"
Oct 31 19:20:35.379809 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:35.379764    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:20:35.477580 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:35.477546    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:20:36.247203 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:20:36.247156    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 19:20:42.707209 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:42.707170    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:20:42.812579 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:42.812544    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:20:42.911708 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:42.911680    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:20:42.936671 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:42.936646    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-54cfdcd6c9-wvfn5"
Oct 31 19:20:42.936774 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:42.936697    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-54cfdcd6c9-wvfn5"
Oct 31 19:20:42.941191 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:42.941169    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-54cfdcd6c9-wvfn5"
Oct 31 19:20:42.974996 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:42.974945    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-54cfdcd6c9-wvfn5"
Oct 31 19:20:43.037658 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:43.037633    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68969f7467-lpqh4"]
Oct 31 19:20:47.438663 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:47.438618    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:20:47.542443 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:47.542402    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:20:47.990059 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:47.990027    4645 generic.go:356] "Generic (PLEG): container finished" podID="611ab431-d82c-46e9-a108-4b221cc8c2cc" containerID="db1b7118268ba3382a50788fbacb8d19aa0440e231841f2ad13fe0d90b1e3f47" exitCode=0
Oct 31 19:20:47.990208 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:47.990084    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-j9l2c-hmgd8" event={"ID":"611ab431-d82c-46e9-a108-4b221cc8c2cc","Type":"ContainerDied","Data":"db1b7118268ba3382a50788fbacb8d19aa0440e231841f2ad13fe0d90b1e3f47"}
Oct 31 19:20:49.120995 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:49.120972    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-j9l2c-hmgd8"
Oct 31 19:20:49.251757 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:49.251701    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/611ab431-d82c-46e9-a108-4b221cc8c2cc-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/611ab431-d82c-46e9-a108-4b221cc8c2cc-archives-path\") pod \"611ab431-d82c-46e9-a108-4b221cc8c2cc\" (UID: \"611ab431-d82c-46e9-a108-4b221cc8c2cc\") "
Oct 31 19:20:49.251757 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:49.251744    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/611ab431-d82c-46e9-a108-4b221cc8c2cc-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/611ab431-d82c-46e9-a108-4b221cc8c2cc-service-ca-bundle\") pod \"611ab431-d82c-46e9-a108-4b221cc8c2cc\" (UID: \"611ab431-d82c-46e9-a108-4b221cc8c2cc\") "
Oct 31 19:20:49.251930 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:49.251785    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/611ab431-d82c-46e9-a108-4b221cc8c2cc-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/611ab431-d82c-46e9-a108-4b221cc8c2cc-trusted-ca-bundle\") pod \"611ab431-d82c-46e9-a108-4b221cc8c2cc\" (UID: \"611ab431-d82c-46e9-a108-4b221cc8c2cc\") "
Oct 31 19:20:49.251930 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:49.251813    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/611ab431-d82c-46e9-a108-4b221cc8c2cc-kube-api-access-x2jxb\" (UniqueName: \"kubernetes.io/projected/611ab431-d82c-46e9-a108-4b221cc8c2cc-kube-api-access-x2jxb\") pod \"611ab431-d82c-46e9-a108-4b221cc8c2cc\" (UID: \"611ab431-d82c-46e9-a108-4b221cc8c2cc\") "
Oct 31 19:20:49.252030 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:49.252003    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/611ab431-d82c-46e9-a108-4b221cc8c2cc-archives-path" pod "611ab431-d82c-46e9-a108-4b221cc8c2cc" (UID: "611ab431-d82c-46e9-a108-4b221cc8c2cc"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:20:49.252135 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:49.252113    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/611ab431-d82c-46e9-a108-4b221cc8c2cc-trusted-ca-bundle" pod "611ab431-d82c-46e9-a108-4b221cc8c2cc" (UID: "611ab431-d82c-46e9-a108-4b221cc8c2cc"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:20:49.252169 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:49.252153    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/611ab431-d82c-46e9-a108-4b221cc8c2cc-service-ca-bundle" pod "611ab431-d82c-46e9-a108-4b221cc8c2cc" (UID: "611ab431-d82c-46e9-a108-4b221cc8c2cc"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:20:49.254109 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:49.254084    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/611ab431-d82c-46e9-a108-4b221cc8c2cc-kube-api-access-x2jxb" pod "611ab431-d82c-46e9-a108-4b221cc8c2cc" (UID: "611ab431-d82c-46e9-a108-4b221cc8c2cc"). InnerVolumeSpecName "kube-api-access-x2jxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:20:49.352313 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:49.352288    4645 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/611ab431-d82c-46e9-a108-4b221cc8c2cc-archives-path\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:20:49.352313 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:49.352309    4645 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/611ab431-d82c-46e9-a108-4b221cc8c2cc-service-ca-bundle\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:20:49.352465 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:49.352318    4645 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/611ab431-d82c-46e9-a108-4b221cc8c2cc-trusted-ca-bundle\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:20:49.352465 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:49.352340    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2jxb\" (UniqueName: \"kubernetes.io/projected/611ab431-d82c-46e9-a108-4b221cc8c2cc-kube-api-access-x2jxb\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:20:49.997951 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:49.997921    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-j9l2c-hmgd8" event={"ID":"611ab431-d82c-46e9-a108-4b221cc8c2cc","Type":"ContainerDied","Data":"a0c01424ea5f36aa2a34e7031ba579e446014499dcfb3069f58f82930562ff2c"}
Oct 31 19:20:49.998079 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:49.997957    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-j9l2c-hmgd8"
Oct 31 19:20:49.998079 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:49.998000    4645 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0c01424ea5f36aa2a34e7031ba579e446014499dcfb3069f58f82930562ff2c"
Oct 31 19:20:50.341114 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:50.341041    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:20:50.451977 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:20:50.451946    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:21:00.773988 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:00.773954    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-s8p5b-qtsxv"]
Oct 31 19:21:00.774548 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:00.774527    4645 cpu_manager.go:414] "containerMap: removing container" podUID="611ab431-d82c-46e9-a108-4b221cc8c2cc" containerName="insights-gathering"
Oct 31 19:21:00.774597 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:00.774551    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="611ab431-d82c-46e9-a108-4b221cc8c2cc" containerName="insights-gathering"
Oct 31 19:21:00.774651 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:00.774639    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="611ab431-d82c-46e9-a108-4b221cc8c2cc" containerName="insights-gathering"
Oct 31 19:21:00.776367 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:00.776350    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-s8p5b-qtsxv"
Oct 31 19:21:00.781006 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:00.780972    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 31 19:21:00.781124 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:00.781009    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-zz8zc\""
Oct 31 19:21:00.783013 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:00.782941    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 31 19:21:00.787425 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:00.787390    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-s8p5b-qtsxv"]
Oct 31 19:21:00.825465 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:00.825439    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srh7m\" (UniqueName: \"kubernetes.io/projected/0d87fa56-1043-431f-9304-6ce9c42bfafb-kube-api-access-srh7m\") pod \"periodic-gathering-s8p5b-qtsxv\" (UID: \"0d87fa56-1043-431f-9304-6ce9c42bfafb\") " pod="openshift-insights/periodic-gathering-s8p5b-qtsxv"
Oct 31 19:21:00.825465 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:00.825467    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d87fa56-1043-431f-9304-6ce9c42bfafb-trusted-ca-bundle\") pod \"periodic-gathering-s8p5b-qtsxv\" (UID: \"0d87fa56-1043-431f-9304-6ce9c42bfafb\") " pod="openshift-insights/periodic-gathering-s8p5b-qtsxv"
Oct 31 19:21:00.825612 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:00.825492    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d87fa56-1043-431f-9304-6ce9c42bfafb-service-ca-bundle\") pod \"periodic-gathering-s8p5b-qtsxv\" (UID: \"0d87fa56-1043-431f-9304-6ce9c42bfafb\") " pod="openshift-insights/periodic-gathering-s8p5b-qtsxv"
Oct 31 19:21:00.825649 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:00.825625    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/0d87fa56-1043-431f-9304-6ce9c42bfafb-archives-path\") pod \"periodic-gathering-s8p5b-qtsxv\" (UID: \"0d87fa56-1043-431f-9304-6ce9c42bfafb\") " pod="openshift-insights/periodic-gathering-s8p5b-qtsxv"
Oct 31 19:21:00.926071 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:00.926048    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srh7m\" (UniqueName: \"kubernetes.io/projected/0d87fa56-1043-431f-9304-6ce9c42bfafb-kube-api-access-srh7m\") pod \"periodic-gathering-s8p5b-qtsxv\" (UID: \"0d87fa56-1043-431f-9304-6ce9c42bfafb\") " pod="openshift-insights/periodic-gathering-s8p5b-qtsxv"
Oct 31 19:21:00.926159 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:00.926075    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d87fa56-1043-431f-9304-6ce9c42bfafb-trusted-ca-bundle\") pod \"periodic-gathering-s8p5b-qtsxv\" (UID: \"0d87fa56-1043-431f-9304-6ce9c42bfafb\") " pod="openshift-insights/periodic-gathering-s8p5b-qtsxv"
Oct 31 19:21:00.926159 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:00.926098    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d87fa56-1043-431f-9304-6ce9c42bfafb-service-ca-bundle\") pod \"periodic-gathering-s8p5b-qtsxv\" (UID: \"0d87fa56-1043-431f-9304-6ce9c42bfafb\") " pod="openshift-insights/periodic-gathering-s8p5b-qtsxv"
Oct 31 19:21:00.926268 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:00.926231    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/0d87fa56-1043-431f-9304-6ce9c42bfafb-archives-path\") pod \"periodic-gathering-s8p5b-qtsxv\" (UID: \"0d87fa56-1043-431f-9304-6ce9c42bfafb\") " pod="openshift-insights/periodic-gathering-s8p5b-qtsxv"
Oct 31 19:21:00.926560 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:00.926539    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/0d87fa56-1043-431f-9304-6ce9c42bfafb-archives-path\") pod \"periodic-gathering-s8p5b-qtsxv\" (UID: \"0d87fa56-1043-431f-9304-6ce9c42bfafb\") " pod="openshift-insights/periodic-gathering-s8p5b-qtsxv"
Oct 31 19:21:00.926703 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:00.926687    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d87fa56-1043-431f-9304-6ce9c42bfafb-service-ca-bundle\") pod \"periodic-gathering-s8p5b-qtsxv\" (UID: \"0d87fa56-1043-431f-9304-6ce9c42bfafb\") " pod="openshift-insights/periodic-gathering-s8p5b-qtsxv"
Oct 31 19:21:00.926804 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:00.926785    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d87fa56-1043-431f-9304-6ce9c42bfafb-trusted-ca-bundle\") pod \"periodic-gathering-s8p5b-qtsxv\" (UID: \"0d87fa56-1043-431f-9304-6ce9c42bfafb\") " pod="openshift-insights/periodic-gathering-s8p5b-qtsxv"
Oct 31 19:21:00.938175 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:00.938156    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srh7m\" (UniqueName: \"kubernetes.io/projected/0d87fa56-1043-431f-9304-6ce9c42bfafb-kube-api-access-srh7m\") pod \"periodic-gathering-s8p5b-qtsxv\" (UID: \"0d87fa56-1043-431f-9304-6ce9c42bfafb\") " pod="openshift-insights/periodic-gathering-s8p5b-qtsxv"
Oct 31 19:21:01.086254 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:01.086200    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-s8p5b-qtsxv"
Oct 31 19:21:01.354684 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:01.354580    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-s8p5b-qtsxv"]
Oct 31 19:21:01.357573 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:21:01.357542    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d87fa56_1043_431f_9304_6ce9c42bfafb.slice/crio-7ba5258ea348bcbdaba1c754936549ecd2bbb37519b09c647e31f0ffc893909e WatchSource:0}: Error finding container 7ba5258ea348bcbdaba1c754936549ecd2bbb37519b09c647e31f0ffc893909e: Status 404 returned error can't find the container with id 7ba5258ea348bcbdaba1c754936549ecd2bbb37519b09c647e31f0ffc893909e
Oct 31 19:21:02.040891 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:02.040850    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-s8p5b-qtsxv" event={"ID":"0d87fa56-1043-431f-9304-6ce9c42bfafb","Type":"ContainerStarted","Data":"978eb233e510ac769b984758391c6dec212b639eea9598f146e585e3d9b21cfb"}
Oct 31 19:21:02.041424 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:02.040914    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-s8p5b-qtsxv" event={"ID":"0d87fa56-1043-431f-9304-6ce9c42bfafb","Type":"ContainerStarted","Data":"7ba5258ea348bcbdaba1c754936549ecd2bbb37519b09c647e31f0ffc893909e"}
Oct 31 19:21:02.057836 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:02.057789    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-s8p5b-qtsxv" podStartSLOduration=2.057773594 podStartE2EDuration="2.057773594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:21:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:21:02.057247975 +0000 UTC m=+92.630482104" watchObservedRunningTime="2026-10-31 19:21:02.057773594 +0000 UTC m=+92.631007724"
Oct 31 19:21:06.246669 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:21:06.246627    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 19:21:08.057337 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:08.057283    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-68969f7467-lpqh4" podUID="207efcfa-0ed5-4b16-b94c-b881ac06b9f6" containerName="console" containerID="cri-o://490204aa1aafa9b7e55e2208ddb5ad412efe4713476e4f6fd73da8d301bd02e9" gracePeriod=15
Oct 31 19:21:08.292606 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:08.292585    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68969f7467-lpqh4"
Oct 31 19:21:08.371590 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:08.371531    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/207efcfa-0ed5-4b16-b94c-b881ac06b9f6-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/207efcfa-0ed5-4b16-b94c-b881ac06b9f6-trusted-ca-bundle\") pod \"207efcfa-0ed5-4b16-b94c-b881ac06b9f6\" (UID: \"207efcfa-0ed5-4b16-b94c-b881ac06b9f6\") "
Oct 31 19:21:08.371590 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:08.371567    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/207efcfa-0ed5-4b16-b94c-b881ac06b9f6-kube-api-access-6fts6\" (UniqueName: \"kubernetes.io/projected/207efcfa-0ed5-4b16-b94c-b881ac06b9f6-kube-api-access-6fts6\") pod \"207efcfa-0ed5-4b16-b94c-b881ac06b9f6\" (UID: \"207efcfa-0ed5-4b16-b94c-b881ac06b9f6\") "
Oct 31 19:21:08.371590 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:08.371590    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/207efcfa-0ed5-4b16-b94c-b881ac06b9f6-tmp\" (UniqueName: \"kubernetes.io/empty-dir/207efcfa-0ed5-4b16-b94c-b881ac06b9f6-tmp\") pod \"207efcfa-0ed5-4b16-b94c-b881ac06b9f6\" (UID: \"207efcfa-0ed5-4b16-b94c-b881ac06b9f6\") "
Oct 31 19:21:08.371802 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:08.371608    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/207efcfa-0ed5-4b16-b94c-b881ac06b9f6-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/207efcfa-0ed5-4b16-b94c-b881ac06b9f6-oauth-serving-cert\") pod \"207efcfa-0ed5-4b16-b94c-b881ac06b9f6\" (UID: \"207efcfa-0ed5-4b16-b94c-b881ac06b9f6\") "
Oct 31 19:21:08.371802 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:08.371742    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/207efcfa-0ed5-4b16-b94c-b881ac06b9f6-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/207efcfa-0ed5-4b16-b94c-b881ac06b9f6-console-oauth-config\") pod \"207efcfa-0ed5-4b16-b94c-b881ac06b9f6\" (UID: \"207efcfa-0ed5-4b16-b94c-b881ac06b9f6\") "
Oct 31 19:21:08.371892 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:08.371810    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/207efcfa-0ed5-4b16-b94c-b881ac06b9f6-service-ca\" (UniqueName: \"kubernetes.io/configmap/207efcfa-0ed5-4b16-b94c-b881ac06b9f6-service-ca\") pod \"207efcfa-0ed5-4b16-b94c-b881ac06b9f6\" (UID: \"207efcfa-0ed5-4b16-b94c-b881ac06b9f6\") "
Oct 31 19:21:08.371892 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:08.371867    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/207efcfa-0ed5-4b16-b94c-b881ac06b9f6-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/207efcfa-0ed5-4b16-b94c-b881ac06b9f6-console-serving-cert\") pod \"207efcfa-0ed5-4b16-b94c-b881ac06b9f6\" (UID: \"207efcfa-0ed5-4b16-b94c-b881ac06b9f6\") "
Oct 31 19:21:08.372000 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:08.371919    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/207efcfa-0ed5-4b16-b94c-b881ac06b9f6-console-config\" (UniqueName: \"kubernetes.io/configmap/207efcfa-0ed5-4b16-b94c-b881ac06b9f6-console-config\") pod \"207efcfa-0ed5-4b16-b94c-b881ac06b9f6\" (UID: \"207efcfa-0ed5-4b16-b94c-b881ac06b9f6\") "
Oct 31 19:21:08.372000 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:08.371927    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/207efcfa-0ed5-4b16-b94c-b881ac06b9f6-tmp" pod "207efcfa-0ed5-4b16-b94c-b881ac06b9f6" (UID: "207efcfa-0ed5-4b16-b94c-b881ac06b9f6"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:21:08.372000 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:08.371959    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/207efcfa-0ed5-4b16-b94c-b881ac06b9f6-trusted-ca-bundle" pod "207efcfa-0ed5-4b16-b94c-b881ac06b9f6" (UID: "207efcfa-0ed5-4b16-b94c-b881ac06b9f6"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:21:08.372153 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:08.372003    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/207efcfa-0ed5-4b16-b94c-b881ac06b9f6-oauth-serving-cert" pod "207efcfa-0ed5-4b16-b94c-b881ac06b9f6" (UID: "207efcfa-0ed5-4b16-b94c-b881ac06b9f6"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:21:08.372210 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:08.372151    4645 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/207efcfa-0ed5-4b16-b94c-b881ac06b9f6-trusted-ca-bundle\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:21:08.372210 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:08.372171    4645 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/207efcfa-0ed5-4b16-b94c-b881ac06b9f6-tmp\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:21:08.372210 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:08.372186    4645 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/207efcfa-0ed5-4b16-b94c-b881ac06b9f6-oauth-serving-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:21:08.372379 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:08.372241    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/207efcfa-0ed5-4b16-b94c-b881ac06b9f6-service-ca" pod "207efcfa-0ed5-4b16-b94c-b881ac06b9f6" (UID: "207efcfa-0ed5-4b16-b94c-b881ac06b9f6"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:21:08.372462 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:08.372444    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/207efcfa-0ed5-4b16-b94c-b881ac06b9f6-console-config" pod "207efcfa-0ed5-4b16-b94c-b881ac06b9f6" (UID: "207efcfa-0ed5-4b16-b94c-b881ac06b9f6"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:21:08.373847 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:08.373829    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/207efcfa-0ed5-4b16-b94c-b881ac06b9f6-kube-api-access-6fts6" pod "207efcfa-0ed5-4b16-b94c-b881ac06b9f6" (UID: "207efcfa-0ed5-4b16-b94c-b881ac06b9f6"). InnerVolumeSpecName "kube-api-access-6fts6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:21:08.374288 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:08.374260    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/207efcfa-0ed5-4b16-b94c-b881ac06b9f6-console-oauth-config" pod "207efcfa-0ed5-4b16-b94c-b881ac06b9f6" (UID: "207efcfa-0ed5-4b16-b94c-b881ac06b9f6"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:21:08.374393 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:08.374285    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/207efcfa-0ed5-4b16-b94c-b881ac06b9f6-console-serving-cert" pod "207efcfa-0ed5-4b16-b94c-b881ac06b9f6" (UID: "207efcfa-0ed5-4b16-b94c-b881ac06b9f6"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:21:08.472629 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:08.472606    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fts6\" (UniqueName: \"kubernetes.io/projected/207efcfa-0ed5-4b16-b94c-b881ac06b9f6-kube-api-access-6fts6\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:21:08.472629 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:08.472628    4645 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/207efcfa-0ed5-4b16-b94c-b881ac06b9f6-console-oauth-config\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:21:08.472807 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:08.472640    4645 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/207efcfa-0ed5-4b16-b94c-b881ac06b9f6-service-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:21:08.472807 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:08.472650    4645 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/207efcfa-0ed5-4b16-b94c-b881ac06b9f6-console-serving-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:21:08.472807 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:08.472658    4645 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/207efcfa-0ed5-4b16-b94c-b881ac06b9f6-console-config\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:21:09.066675 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:09.066644    4645 generic.go:356] "Generic (PLEG): container finished" podID="207efcfa-0ed5-4b16-b94c-b881ac06b9f6" containerID="490204aa1aafa9b7e55e2208ddb5ad412efe4713476e4f6fd73da8d301bd02e9" exitCode=0
Oct 31 19:21:09.067120 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:09.066703    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68969f7467-lpqh4" event={"ID":"207efcfa-0ed5-4b16-b94c-b881ac06b9f6","Type":"ContainerDied","Data":"490204aa1aafa9b7e55e2208ddb5ad412efe4713476e4f6fd73da8d301bd02e9"}
Oct 31 19:21:09.067120 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:09.066742    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68969f7467-lpqh4" event={"ID":"207efcfa-0ed5-4b16-b94c-b881ac06b9f6","Type":"ContainerDied","Data":"f303559aa2951dd88927d850a106c7200d384b065002ee60685c6f84563fdfcb"}
Oct 31 19:21:09.067120 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:09.066766    4645 scope.go:122] "RemoveContainer" containerID="490204aa1aafa9b7e55e2208ddb5ad412efe4713476e4f6fd73da8d301bd02e9"
Oct 31 19:21:09.067120 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:09.066705    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68969f7467-lpqh4"
Oct 31 19:21:09.075918 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:09.075894    4645 scope.go:122] "RemoveContainer" containerID="490204aa1aafa9b7e55e2208ddb5ad412efe4713476e4f6fd73da8d301bd02e9"
Oct 31 19:21:09.076241 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:21:09.076215    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"490204aa1aafa9b7e55e2208ddb5ad412efe4713476e4f6fd73da8d301bd02e9\": container with ID starting with 490204aa1aafa9b7e55e2208ddb5ad412efe4713476e4f6fd73da8d301bd02e9 not found: ID does not exist" containerID="490204aa1aafa9b7e55e2208ddb5ad412efe4713476e4f6fd73da8d301bd02e9"
Oct 31 19:21:09.076295 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:09.076250    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"490204aa1aafa9b7e55e2208ddb5ad412efe4713476e4f6fd73da8d301bd02e9"} err="failed to get container status \"490204aa1aafa9b7e55e2208ddb5ad412efe4713476e4f6fd73da8d301bd02e9\": rpc error: code = NotFound desc = could not find container \"490204aa1aafa9b7e55e2208ddb5ad412efe4713476e4f6fd73da8d301bd02e9\": container with ID starting with 490204aa1aafa9b7e55e2208ddb5ad412efe4713476e4f6fd73da8d301bd02e9 not found: ID does not exist"
Oct 31 19:21:09.087671 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:09.087639    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68969f7467-lpqh4"]
Oct 31 19:21:09.092213 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:09.092184    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-68969f7467-lpqh4"]
Oct 31 19:21:09.614866 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:09.614834    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="207efcfa-0ed5-4b16-b94c-b881ac06b9f6" path="/var/lib/kubelet/pods/207efcfa-0ed5-4b16-b94c-b881ac06b9f6/volumes"
Oct 31 19:21:14.484044 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:14.483985    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:21:16.100847 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:16.100799    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:21:16.203375 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:16.203342    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:21:18.480219 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:18.480177    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-68496cf448-m7fgc"]
Oct 31 19:21:18.480800 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:18.480647    4645 cpu_manager.go:414] "containerMap: removing container" podUID="207efcfa-0ed5-4b16-b94c-b881ac06b9f6" containerName="console"
Oct 31 19:21:18.480800 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:18.480670    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="207efcfa-0ed5-4b16-b94c-b881ac06b9f6" containerName="console"
Oct 31 19:21:18.480800 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:18.480743    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="207efcfa-0ed5-4b16-b94c-b881ac06b9f6" containerName="console"
Oct 31 19:21:18.483882 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:18.483852    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68496cf448-m7fgc"
Oct 31 19:21:18.504411 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:18.504381    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68496cf448-m7fgc"]
Oct 31 19:21:18.541912 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:18.541887    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0e752859-51b7-481b-a5ba-81681289e311-console-serving-cert\") pod \"console-68496cf448-m7fgc\" (UID: \"0e752859-51b7-481b-a5ba-81681289e311\") " pod="openshift-console/console-68496cf448-m7fgc"
Oct 31 19:21:18.542040 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:18.541919    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0e752859-51b7-481b-a5ba-81681289e311-console-oauth-config\") pod \"console-68496cf448-m7fgc\" (UID: \"0e752859-51b7-481b-a5ba-81681289e311\") " pod="openshift-console/console-68496cf448-m7fgc"
Oct 31 19:21:18.542040 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:18.541941    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0e752859-51b7-481b-a5ba-81681289e311-oauth-serving-cert\") pod \"console-68496cf448-m7fgc\" (UID: \"0e752859-51b7-481b-a5ba-81681289e311\") " pod="openshift-console/console-68496cf448-m7fgc"
Oct 31 19:21:18.542040 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:18.541957    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e752859-51b7-481b-a5ba-81681289e311-trusted-ca-bundle\") pod \"console-68496cf448-m7fgc\" (UID: \"0e752859-51b7-481b-a5ba-81681289e311\") " pod="openshift-console/console-68496cf448-m7fgc"
Oct 31 19:21:18.542040 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:18.542023    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0e752859-51b7-481b-a5ba-81681289e311-service-ca\") pod \"console-68496cf448-m7fgc\" (UID: \"0e752859-51b7-481b-a5ba-81681289e311\") " pod="openshift-console/console-68496cf448-m7fgc"
Oct 31 19:21:18.542213 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:18.542101    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0e752859-51b7-481b-a5ba-81681289e311-tmp\") pod \"console-68496cf448-m7fgc\" (UID: \"0e752859-51b7-481b-a5ba-81681289e311\") " pod="openshift-console/console-68496cf448-m7fgc"
Oct 31 19:21:18.542213 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:18.542138    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0e752859-51b7-481b-a5ba-81681289e311-console-config\") pod \"console-68496cf448-m7fgc\" (UID: \"0e752859-51b7-481b-a5ba-81681289e311\") " pod="openshift-console/console-68496cf448-m7fgc"
Oct 31 19:21:18.542213 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:18.542168    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7shj5\" (UniqueName: \"kubernetes.io/projected/0e752859-51b7-481b-a5ba-81681289e311-kube-api-access-7shj5\") pod \"console-68496cf448-m7fgc\" (UID: \"0e752859-51b7-481b-a5ba-81681289e311\") " pod="openshift-console/console-68496cf448-m7fgc"
Oct 31 19:21:18.572502 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:18.572471    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:21:18.642632 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:18.642592    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0e752859-51b7-481b-a5ba-81681289e311-console-serving-cert\") pod \"console-68496cf448-m7fgc\" (UID: \"0e752859-51b7-481b-a5ba-81681289e311\") " pod="openshift-console/console-68496cf448-m7fgc"
Oct 31 19:21:18.642742 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:18.642645    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0e752859-51b7-481b-a5ba-81681289e311-console-oauth-config\") pod \"console-68496cf448-m7fgc\" (UID: \"0e752859-51b7-481b-a5ba-81681289e311\") " pod="openshift-console/console-68496cf448-m7fgc"
Oct 31 19:21:18.642742 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:18.642676    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0e752859-51b7-481b-a5ba-81681289e311-oauth-serving-cert\") pod \"console-68496cf448-m7fgc\" (UID: \"0e752859-51b7-481b-a5ba-81681289e311\") " pod="openshift-console/console-68496cf448-m7fgc"
Oct 31 19:21:18.642742 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:18.642701    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e752859-51b7-481b-a5ba-81681289e311-trusted-ca-bundle\") pod \"console-68496cf448-m7fgc\" (UID: \"0e752859-51b7-481b-a5ba-81681289e311\") " pod="openshift-console/console-68496cf448-m7fgc"
Oct 31 19:21:18.642742 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:18.642734    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0e752859-51b7-481b-a5ba-81681289e311-service-ca\") pod \"console-68496cf448-m7fgc\" (UID: \"0e752859-51b7-481b-a5ba-81681289e311\") " pod="openshift-console/console-68496cf448-m7fgc"
Oct 31 19:21:18.642954 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:18.642762    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0e752859-51b7-481b-a5ba-81681289e311-tmp\") pod \"console-68496cf448-m7fgc\" (UID: \"0e752859-51b7-481b-a5ba-81681289e311\") " pod="openshift-console/console-68496cf448-m7fgc"
Oct 31 19:21:18.642954 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:18.642798    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0e752859-51b7-481b-a5ba-81681289e311-console-config\") pod \"console-68496cf448-m7fgc\" (UID: \"0e752859-51b7-481b-a5ba-81681289e311\") " pod="openshift-console/console-68496cf448-m7fgc"
Oct 31 19:21:18.642954 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:18.642845    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7shj5\" (UniqueName: \"kubernetes.io/projected/0e752859-51b7-481b-a5ba-81681289e311-kube-api-access-7shj5\") pod \"console-68496cf448-m7fgc\" (UID: \"0e752859-51b7-481b-a5ba-81681289e311\") " pod="openshift-console/console-68496cf448-m7fgc"
Oct 31 19:21:18.643296 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:18.643265    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0e752859-51b7-481b-a5ba-81681289e311-tmp\") pod \"console-68496cf448-m7fgc\" (UID: \"0e752859-51b7-481b-a5ba-81681289e311\") " pod="openshift-console/console-68496cf448-m7fgc"
Oct 31 19:21:18.643601 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:18.643570    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0e752859-51b7-481b-a5ba-81681289e311-oauth-serving-cert\") pod \"console-68496cf448-m7fgc\" (UID: \"0e752859-51b7-481b-a5ba-81681289e311\") " pod="openshift-console/console-68496cf448-m7fgc"
Oct 31 19:21:18.643754 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:18.643732    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0e752859-51b7-481b-a5ba-81681289e311-console-config\") pod \"console-68496cf448-m7fgc\" (UID: \"0e752859-51b7-481b-a5ba-81681289e311\") " pod="openshift-console/console-68496cf448-m7fgc"
Oct 31 19:21:18.644010 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:18.643985    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0e752859-51b7-481b-a5ba-81681289e311-service-ca\") pod \"console-68496cf448-m7fgc\" (UID: \"0e752859-51b7-481b-a5ba-81681289e311\") " pod="openshift-console/console-68496cf448-m7fgc"
Oct 31 19:21:18.644053 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:18.644017    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e752859-51b7-481b-a5ba-81681289e311-trusted-ca-bundle\") pod \"console-68496cf448-m7fgc\" (UID: \"0e752859-51b7-481b-a5ba-81681289e311\") " pod="openshift-console/console-68496cf448-m7fgc"
Oct 31 19:21:18.645261 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:18.645238    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0e752859-51b7-481b-a5ba-81681289e311-console-serving-cert\") pod \"console-68496cf448-m7fgc\" (UID: \"0e752859-51b7-481b-a5ba-81681289e311\") " pod="openshift-console/console-68496cf448-m7fgc"
Oct 31 19:21:18.645404 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:18.645384    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0e752859-51b7-481b-a5ba-81681289e311-console-oauth-config\") pod \"console-68496cf448-m7fgc\" (UID: \"0e752859-51b7-481b-a5ba-81681289e311\") " pod="openshift-console/console-68496cf448-m7fgc"
Oct 31 19:21:18.654249 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:18.654224    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7shj5\" (UniqueName: \"kubernetes.io/projected/0e752859-51b7-481b-a5ba-81681289e311-kube-api-access-7shj5\") pod \"console-68496cf448-m7fgc\" (UID: \"0e752859-51b7-481b-a5ba-81681289e311\") " pod="openshift-console/console-68496cf448-m7fgc"
Oct 31 19:21:18.671079 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:18.671048    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:21:18.796472 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:18.796409    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68496cf448-m7fgc"
Oct 31 19:21:18.927530 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:18.927497    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:21:19.024857 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:19.024823    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:21:19.071097 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:19.071077    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68496cf448-m7fgc"]
Oct 31 19:21:19.073608 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:21:19.073579    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e752859_51b7_481b_a5ba_81681289e311.slice/crio-1dfbc2f344ca7e1f02dc21eda9b781bdd1fa64f9fc9af30ab4a3994260e4d3e5 WatchSource:0}: Error finding container 1dfbc2f344ca7e1f02dc21eda9b781bdd1fa64f9fc9af30ab4a3994260e4d3e5: Status 404 returned error can't find the container with id 1dfbc2f344ca7e1f02dc21eda9b781bdd1fa64f9fc9af30ab4a3994260e4d3e5
Oct 31 19:21:19.103506 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:19.103485    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68496cf448-m7fgc" event={"ID":"0e752859-51b7-481b-a5ba-81681289e311","Type":"ContainerStarted","Data":"1dfbc2f344ca7e1f02dc21eda9b781bdd1fa64f9fc9af30ab4a3994260e4d3e5"}
Oct 31 19:21:20.113975 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:20.113946    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68496cf448-m7fgc" event={"ID":"0e752859-51b7-481b-a5ba-81681289e311","Type":"ContainerStarted","Data":"b8f4c12a17b77312c2d6ee8f2f9ce1e72378e7b68cb8f444fb6b8783cbc377e9"}
Oct 31 19:21:20.133170 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:20.133119    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-68496cf448-m7fgc" podStartSLOduration=2.133103103 podStartE2EDuration="2.133103103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:21:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:21:20.132966337 +0000 UTC m=+110.706200466" watchObservedRunningTime="2026-10-31 19:21:20.133103103 +0000 UTC m=+110.706337232"
Oct 31 19:21:28.771092 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:28.771045    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:21:28.796916 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:28.796885    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-68496cf448-m7fgc"
Oct 31 19:21:28.797231 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:28.797203    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-68496cf448-m7fgc"
Oct 31 19:21:28.802936 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:28.802908    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-68496cf448-m7fgc"
Oct 31 19:21:28.871567 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:28.871539    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:21:29.150356 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:29.150278    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-68496cf448-m7fgc"
Oct 31 19:21:29.201989 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:29.201959    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54cfdcd6c9-wvfn5"]
Oct 31 19:21:32.438261 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:32.438219    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:21:32.538549 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:32.538517    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:21:32.638204 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:32.638173    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:21:35.158053 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:35.158011    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:21:36.245522 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:21:36.245484    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 19:21:38.306592 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:38.306551    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:21:54.221482 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:54.221381    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-54cfdcd6c9-wvfn5" podUID="3d63fb6a-ab64-41a9-add1-c05801100442" containerName="console" containerID="cri-o://2959a2c79d2003b719d3945af47a4c8369b6110256b0ccdf5f7434a3a31ddf88" gracePeriod=15
Oct 31 19:21:54.459208 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:54.459186    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54cfdcd6c9-wvfn5"
Oct 31 19:21:54.590313 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:54.590251    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d63fb6a-ab64-41a9-add1-c05801100442-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d63fb6a-ab64-41a9-add1-c05801100442-service-ca\") pod \"3d63fb6a-ab64-41a9-add1-c05801100442\" (UID: \"3d63fb6a-ab64-41a9-add1-c05801100442\") "
Oct 31 19:21:54.590313 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:54.590293    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d63fb6a-ab64-41a9-add1-c05801100442-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3d63fb6a-ab64-41a9-add1-c05801100442-console-serving-cert\") pod \"3d63fb6a-ab64-41a9-add1-c05801100442\" (UID: \"3d63fb6a-ab64-41a9-add1-c05801100442\") "
Oct 31 19:21:54.590472 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:54.590316    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d63fb6a-ab64-41a9-add1-c05801100442-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d63fb6a-ab64-41a9-add1-c05801100442-trusted-ca-bundle\") pod \"3d63fb6a-ab64-41a9-add1-c05801100442\" (UID: \"3d63fb6a-ab64-41a9-add1-c05801100442\") "
Oct 31 19:21:54.590536 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:54.590511    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d63fb6a-ab64-41a9-add1-c05801100442-tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d63fb6a-ab64-41a9-add1-c05801100442-tmp\") pod \"3d63fb6a-ab64-41a9-add1-c05801100442\" (UID: \"3d63fb6a-ab64-41a9-add1-c05801100442\") "
Oct 31 19:21:54.590586 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:54.590564    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d63fb6a-ab64-41a9-add1-c05801100442-kube-api-access-4mxjx\" (UniqueName: \"kubernetes.io/projected/3d63fb6a-ab64-41a9-add1-c05801100442-kube-api-access-4mxjx\") pod \"3d63fb6a-ab64-41a9-add1-c05801100442\" (UID: \"3d63fb6a-ab64-41a9-add1-c05801100442\") "
Oct 31 19:21:54.590641 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:54.590606    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d63fb6a-ab64-41a9-add1-c05801100442-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3d63fb6a-ab64-41a9-add1-c05801100442-oauth-serving-cert\") pod \"3d63fb6a-ab64-41a9-add1-c05801100442\" (UID: \"3d63fb6a-ab64-41a9-add1-c05801100442\") "
Oct 31 19:21:54.590641 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:54.590612    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d63fb6a-ab64-41a9-add1-c05801100442-service-ca" pod "3d63fb6a-ab64-41a9-add1-c05801100442" (UID: "3d63fb6a-ab64-41a9-add1-c05801100442"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:21:54.590745 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:54.590667    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d63fb6a-ab64-41a9-add1-c05801100442-console-config\" (UniqueName: \"kubernetes.io/configmap/3d63fb6a-ab64-41a9-add1-c05801100442-console-config\") pod \"3d63fb6a-ab64-41a9-add1-c05801100442\" (UID: \"3d63fb6a-ab64-41a9-add1-c05801100442\") "
Oct 31 19:21:54.590797 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:54.590701    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d63fb6a-ab64-41a9-add1-c05801100442-trusted-ca-bundle" pod "3d63fb6a-ab64-41a9-add1-c05801100442" (UID: "3d63fb6a-ab64-41a9-add1-c05801100442"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:21:54.590848 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:54.590820    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d63fb6a-ab64-41a9-add1-c05801100442-tmp" pod "3d63fb6a-ab64-41a9-add1-c05801100442" (UID: "3d63fb6a-ab64-41a9-add1-c05801100442"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:21:54.590890 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:54.590847    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d63fb6a-ab64-41a9-add1-c05801100442-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3d63fb6a-ab64-41a9-add1-c05801100442-console-oauth-config\") pod \"3d63fb6a-ab64-41a9-add1-c05801100442\" (UID: \"3d63fb6a-ab64-41a9-add1-c05801100442\") "
Oct 31 19:21:54.590994 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:54.590971    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d63fb6a-ab64-41a9-add1-c05801100442-oauth-serving-cert" pod "3d63fb6a-ab64-41a9-add1-c05801100442" (UID: "3d63fb6a-ab64-41a9-add1-c05801100442"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:21:54.591062 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:54.590987    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d63fb6a-ab64-41a9-add1-c05801100442-console-config" pod "3d63fb6a-ab64-41a9-add1-c05801100442" (UID: "3d63fb6a-ab64-41a9-add1-c05801100442"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:21:54.591113 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:54.591067    4645 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d63fb6a-ab64-41a9-add1-c05801100442-tmp\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:21:54.591113 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:54.591089    4645 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3d63fb6a-ab64-41a9-add1-c05801100442-oauth-serving-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:21:54.591113 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:54.591104    4645 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3d63fb6a-ab64-41a9-add1-c05801100442-console-config\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:21:54.591210 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:54.591117    4645 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3d63fb6a-ab64-41a9-add1-c05801100442-service-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:21:54.591210 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:54.591131    4645 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3d63fb6a-ab64-41a9-add1-c05801100442-trusted-ca-bundle\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:21:54.592806 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:54.592779    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d63fb6a-ab64-41a9-add1-c05801100442-kube-api-access-4mxjx" pod "3d63fb6a-ab64-41a9-add1-c05801100442" (UID: "3d63fb6a-ab64-41a9-add1-c05801100442"). InnerVolumeSpecName "kube-api-access-4mxjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:21:54.592906 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:54.592794    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d63fb6a-ab64-41a9-add1-c05801100442-console-oauth-config" pod "3d63fb6a-ab64-41a9-add1-c05801100442" (UID: "3d63fb6a-ab64-41a9-add1-c05801100442"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:21:54.592906 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:54.592826    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d63fb6a-ab64-41a9-add1-c05801100442-console-serving-cert" pod "3d63fb6a-ab64-41a9-add1-c05801100442" (UID: "3d63fb6a-ab64-41a9-add1-c05801100442"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:21:54.691500 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:54.691480    4645 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3d63fb6a-ab64-41a9-add1-c05801100442-console-serving-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:21:54.691500 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:54.691499    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mxjx\" (UniqueName: \"kubernetes.io/projected/3d63fb6a-ab64-41a9-add1-c05801100442-kube-api-access-4mxjx\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:21:54.691623 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:54.691509    4645 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3d63fb6a-ab64-41a9-add1-c05801100442-console-oauth-config\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:21:55.239089 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:55.239050    4645 generic.go:356] "Generic (PLEG): container finished" podID="3d63fb6a-ab64-41a9-add1-c05801100442" containerID="2959a2c79d2003b719d3945af47a4c8369b6110256b0ccdf5f7434a3a31ddf88" exitCode=0
Oct 31 19:21:55.239437 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:55.239107    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54cfdcd6c9-wvfn5"
Oct 31 19:21:55.239437 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:55.239124    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54cfdcd6c9-wvfn5" event={"ID":"3d63fb6a-ab64-41a9-add1-c05801100442","Type":"ContainerDied","Data":"2959a2c79d2003b719d3945af47a4c8369b6110256b0ccdf5f7434a3a31ddf88"}
Oct 31 19:21:55.239437 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:55.239173    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54cfdcd6c9-wvfn5" event={"ID":"3d63fb6a-ab64-41a9-add1-c05801100442","Type":"ContainerDied","Data":"4ff5289e62ee6093a91d0f7081db09ad4bd561d9844f2c4819e073b60be2b8fe"}
Oct 31 19:21:55.239437 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:55.239197    4645 scope.go:122] "RemoveContainer" containerID="2959a2c79d2003b719d3945af47a4c8369b6110256b0ccdf5f7434a3a31ddf88"
Oct 31 19:21:55.248576 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:55.248561    4645 scope.go:122] "RemoveContainer" containerID="2959a2c79d2003b719d3945af47a4c8369b6110256b0ccdf5f7434a3a31ddf88"
Oct 31 19:21:55.248899 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:21:55.248878    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2959a2c79d2003b719d3945af47a4c8369b6110256b0ccdf5f7434a3a31ddf88\": container with ID starting with 2959a2c79d2003b719d3945af47a4c8369b6110256b0ccdf5f7434a3a31ddf88 not found: ID does not exist" containerID="2959a2c79d2003b719d3945af47a4c8369b6110256b0ccdf5f7434a3a31ddf88"
Oct 31 19:21:55.248946 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:55.248909    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2959a2c79d2003b719d3945af47a4c8369b6110256b0ccdf5f7434a3a31ddf88"} err="failed to get container status \"2959a2c79d2003b719d3945af47a4c8369b6110256b0ccdf5f7434a3a31ddf88\": rpc error: code = NotFound desc = could not find container \"2959a2c79d2003b719d3945af47a4c8369b6110256b0ccdf5f7434a3a31ddf88\": container with ID starting with 2959a2c79d2003b719d3945af47a4c8369b6110256b0ccdf5f7434a3a31ddf88 not found: ID does not exist"
Oct 31 19:21:55.259463 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:55.259439    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54cfdcd6c9-wvfn5"]
Oct 31 19:21:55.266019 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:55.265990    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-54cfdcd6c9-wvfn5"]
Oct 31 19:21:55.615224 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:21:55.615161    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d63fb6a-ab64-41a9-add1-c05801100442" path="/var/lib/kubelet/pods/3d63fb6a-ab64-41a9-add1-c05801100442/volumes"
Oct 31 19:22:06.247414 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:22:06.247378    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 19:22:14.535347 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:22:14.535293    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:18.758586 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:22:18.758547    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:22:25.344996 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:22:25.344925    4645 generic.go:356] "Generic (PLEG): container finished" podID="0d87fa56-1043-431f-9304-6ce9c42bfafb" containerID="978eb233e510ac769b984758391c6dec212b639eea9598f146e585e3d9b21cfb" exitCode=0
Oct 31 19:22:25.345439 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:22:25.345002    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-s8p5b-qtsxv" event={"ID":"0d87fa56-1043-431f-9304-6ce9c42bfafb","Type":"ContainerDied","Data":"978eb233e510ac769b984758391c6dec212b639eea9598f146e585e3d9b21cfb"}
Oct 31 19:22:26.475108 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:22:26.475085    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-s8p5b-qtsxv"
Oct 31 19:22:26.606864 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:22:26.606801    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d87fa56-1043-431f-9304-6ce9c42bfafb-kube-api-access-srh7m\" (UniqueName: \"kubernetes.io/projected/0d87fa56-1043-431f-9304-6ce9c42bfafb-kube-api-access-srh7m\") pod \"0d87fa56-1043-431f-9304-6ce9c42bfafb\" (UID: \"0d87fa56-1043-431f-9304-6ce9c42bfafb\") "
Oct 31 19:22:26.606984 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:22:26.606869    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d87fa56-1043-431f-9304-6ce9c42bfafb-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/0d87fa56-1043-431f-9304-6ce9c42bfafb-archives-path\") pod \"0d87fa56-1043-431f-9304-6ce9c42bfafb\" (UID: \"0d87fa56-1043-431f-9304-6ce9c42bfafb\") "
Oct 31 19:22:26.606984 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:22:26.606900    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d87fa56-1043-431f-9304-6ce9c42bfafb-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d87fa56-1043-431f-9304-6ce9c42bfafb-service-ca-bundle\") pod \"0d87fa56-1043-431f-9304-6ce9c42bfafb\" (UID: \"0d87fa56-1043-431f-9304-6ce9c42bfafb\") "
Oct 31 19:22:26.606984 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:22:26.606968    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d87fa56-1043-431f-9304-6ce9c42bfafb-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d87fa56-1043-431f-9304-6ce9c42bfafb-trusted-ca-bundle\") pod \"0d87fa56-1043-431f-9304-6ce9c42bfafb\" (UID: \"0d87fa56-1043-431f-9304-6ce9c42bfafb\") "
Oct 31 19:22:26.607237 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:22:26.607170    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d87fa56-1043-431f-9304-6ce9c42bfafb-archives-path" pod "0d87fa56-1043-431f-9304-6ce9c42bfafb" (UID: "0d87fa56-1043-431f-9304-6ce9c42bfafb"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:22:26.607349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:22:26.607311    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d87fa56-1043-431f-9304-6ce9c42bfafb-service-ca-bundle" pod "0d87fa56-1043-431f-9304-6ce9c42bfafb" (UID: "0d87fa56-1043-431f-9304-6ce9c42bfafb"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:22:26.607418 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:22:26.607397    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d87fa56-1043-431f-9304-6ce9c42bfafb-trusted-ca-bundle" pod "0d87fa56-1043-431f-9304-6ce9c42bfafb" (UID: "0d87fa56-1043-431f-9304-6ce9c42bfafb"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:22:26.608782 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:22:26.608762    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d87fa56-1043-431f-9304-6ce9c42bfafb-kube-api-access-srh7m" pod "0d87fa56-1043-431f-9304-6ce9c42bfafb" (UID: "0d87fa56-1043-431f-9304-6ce9c42bfafb"). InnerVolumeSpecName "kube-api-access-srh7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:22:26.707358 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:22:26.707323    4645 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d87fa56-1043-431f-9304-6ce9c42bfafb-trusted-ca-bundle\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:22:26.707358 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:22:26.707355    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srh7m\" (UniqueName: \"kubernetes.io/projected/0d87fa56-1043-431f-9304-6ce9c42bfafb-kube-api-access-srh7m\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:22:26.707483 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:22:26.707365    4645 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/0d87fa56-1043-431f-9304-6ce9c42bfafb-archives-path\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:22:26.707483 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:22:26.707374    4645 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d87fa56-1043-431f-9304-6ce9c42bfafb-service-ca-bundle\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:22:27.353781 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:22:27.353749    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-s8p5b-qtsxv" event={"ID":"0d87fa56-1043-431f-9304-6ce9c42bfafb","Type":"ContainerDied","Data":"7ba5258ea348bcbdaba1c754936549ecd2bbb37519b09c647e31f0ffc893909e"}
Oct 31 19:22:27.353908 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:22:27.353799    4645 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ba5258ea348bcbdaba1c754936549ecd2bbb37519b09c647e31f0ffc893909e"
Oct 31 19:22:27.353908 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:22:27.353800    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-s8p5b-qtsxv"
Oct 31 19:22:36.247145 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:22:36.247091    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 19:22:37.315826 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:22:37.315789    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:00.821263 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:00.821227    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:00.921953 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:00.921921    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:06.246824 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:23:06.246782    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 19:23:11.622589 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:11.622558    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:11.800917 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:11.800882    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:11.989623 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:11.989583    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:13.328679 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:13.328647    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tt4vs"]
Oct 31 19:23:13.329040 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:13.328888    4645 cpu_manager.go:414] "containerMap: removing container" podUID="3d63fb6a-ab64-41a9-add1-c05801100442" containerName="console"
Oct 31 19:23:13.329040 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:13.328898    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d63fb6a-ab64-41a9-add1-c05801100442" containerName="console"
Oct 31 19:23:13.329040 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:13.328907    4645 cpu_manager.go:414] "containerMap: removing container" podUID="0d87fa56-1043-431f-9304-6ce9c42bfafb" containerName="insights-gathering"
Oct 31 19:23:13.329040 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:13.328913    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d87fa56-1043-431f-9304-6ce9c42bfafb" containerName="insights-gathering"
Oct 31 19:23:13.329040 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:13.328974    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d63fb6a-ab64-41a9-add1-c05801100442" containerName="console"
Oct 31 19:23:13.329040 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:13.328981    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d87fa56-1043-431f-9304-6ce9c42bfafb" containerName="insights-gathering"
Oct 31 19:23:13.331865 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:13.331847    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tt4vs"
Oct 31 19:23:13.348892 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:13.348871    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-hgwh2\""
Oct 31 19:23:13.349015 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:13.348946    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 31 19:23:13.349226 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:13.349212    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 31 19:23:13.349979 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:13.349960    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tt4vs"]
Oct 31 19:23:13.415961 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:13.415933    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/453e8d30-04fe-48d7-9f5d-70cd516b1ef6-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tt4vs\" (UID: \"453e8d30-04fe-48d7-9f5d-70cd516b1ef6\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tt4vs"
Oct 31 19:23:13.416070 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:13.415969    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzf94\" (UniqueName: \"kubernetes.io/projected/453e8d30-04fe-48d7-9f5d-70cd516b1ef6-kube-api-access-rzf94\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tt4vs\" (UID: \"453e8d30-04fe-48d7-9f5d-70cd516b1ef6\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tt4vs"
Oct 31 19:23:13.416113 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:13.416079    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/453e8d30-04fe-48d7-9f5d-70cd516b1ef6-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tt4vs\" (UID: \"453e8d30-04fe-48d7-9f5d-70cd516b1ef6\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tt4vs"
Oct 31 19:23:13.516491 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:13.516466    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/453e8d30-04fe-48d7-9f5d-70cd516b1ef6-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tt4vs\" (UID: \"453e8d30-04fe-48d7-9f5d-70cd516b1ef6\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tt4vs"
Oct 31 19:23:13.516606 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:13.516509    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/453e8d30-04fe-48d7-9f5d-70cd516b1ef6-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tt4vs\" (UID: \"453e8d30-04fe-48d7-9f5d-70cd516b1ef6\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tt4vs"
Oct 31 19:23:13.516606 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:13.516528    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzf94\" (UniqueName: \"kubernetes.io/projected/453e8d30-04fe-48d7-9f5d-70cd516b1ef6-kube-api-access-rzf94\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tt4vs\" (UID: \"453e8d30-04fe-48d7-9f5d-70cd516b1ef6\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tt4vs"
Oct 31 19:23:13.516848 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:13.516829    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/453e8d30-04fe-48d7-9f5d-70cd516b1ef6-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tt4vs\" (UID: \"453e8d30-04fe-48d7-9f5d-70cd516b1ef6\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tt4vs"
Oct 31 19:23:13.516887 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:13.516871    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/453e8d30-04fe-48d7-9f5d-70cd516b1ef6-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tt4vs\" (UID: \"453e8d30-04fe-48d7-9f5d-70cd516b1ef6\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tt4vs"
Oct 31 19:23:13.540920 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:13.540889    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzf94\" (UniqueName: \"kubernetes.io/projected/453e8d30-04fe-48d7-9f5d-70cd516b1ef6-kube-api-access-rzf94\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tt4vs\" (UID: \"453e8d30-04fe-48d7-9f5d-70cd516b1ef6\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tt4vs"
Oct 31 19:23:13.631798 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:13.631729    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-mnrh7"]
Oct 31 19:23:13.634869 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:13.634854    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mnrh7"
Oct 31 19:23:13.641626 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:13.641605    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-7z5vd\""
Oct 31 19:23:13.646080 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:13.646058    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 31 19:23:13.646169 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:13.646085    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tt4vs"
Oct 31 19:23:13.646856 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:13.646841    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 31 19:23:13.646903 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:13.646856    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 31 19:23:13.647135 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:13.647116    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 31 19:23:13.653071 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:13.653050    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-mnrh7"]
Oct 31 19:23:13.660507 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:13.660476    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 31 19:23:13.660596 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:13.660547    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 31 19:23:13.677881 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:13.677854    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:13.717392 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:13.717368    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/28c727cb-1a33-4557-a0a0-76377739bddb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-mnrh7\" (UID: \"28c727cb-1a33-4557-a0a0-76377739bddb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mnrh7"
Oct 31 19:23:13.717533 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:13.717403    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/28c727cb-1a33-4557-a0a0-76377739bddb-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-mnrh7\" (UID: \"28c727cb-1a33-4557-a0a0-76377739bddb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mnrh7"
Oct 31 19:23:13.717533 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:13.717423    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/28c727cb-1a33-4557-a0a0-76377739bddb-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-mnrh7\" (UID: \"28c727cb-1a33-4557-a0a0-76377739bddb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mnrh7"
Oct 31 19:23:13.717533 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:13.717444    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28c727cb-1a33-4557-a0a0-76377739bddb-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-mnrh7\" (UID: \"28c727cb-1a33-4557-a0a0-76377739bddb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mnrh7"
Oct 31 19:23:13.717533 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:13.717504    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/28c727cb-1a33-4557-a0a0-76377739bddb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-mnrh7\" (UID: \"28c727cb-1a33-4557-a0a0-76377739bddb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mnrh7"
Oct 31 19:23:13.717818 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:13.717635    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjxln\" (UniqueName: \"kubernetes.io/projected/28c727cb-1a33-4557-a0a0-76377739bddb-kube-api-access-sjxln\") pod \"istiod-openshift-gateway-7876bfcb74-mnrh7\" (UID: \"28c727cb-1a33-4557-a0a0-76377739bddb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mnrh7"
Oct 31 19:23:13.717818 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:13.717654    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/28c727cb-1a33-4557-a0a0-76377739bddb-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-mnrh7\" (UID: \"28c727cb-1a33-4557-a0a0-76377739bddb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mnrh7"
Oct 31 19:23:13.818511 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:13.818488    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjxln\" (UniqueName: \"kubernetes.io/projected/28c727cb-1a33-4557-a0a0-76377739bddb-kube-api-access-sjxln\") pod \"istiod-openshift-gateway-7876bfcb74-mnrh7\" (UID: \"28c727cb-1a33-4557-a0a0-76377739bddb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mnrh7"
Oct 31 19:23:13.818634 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:13.818518    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/28c727cb-1a33-4557-a0a0-76377739bddb-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-mnrh7\" (UID: \"28c727cb-1a33-4557-a0a0-76377739bddb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mnrh7"
Oct 31 19:23:13.818634 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:13.818541    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/28c727cb-1a33-4557-a0a0-76377739bddb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-mnrh7\" (UID: \"28c727cb-1a33-4557-a0a0-76377739bddb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mnrh7"
Oct 31 19:23:13.818634 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:13.818577    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/28c727cb-1a33-4557-a0a0-76377739bddb-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-mnrh7\" (UID: \"28c727cb-1a33-4557-a0a0-76377739bddb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mnrh7"
Oct 31 19:23:13.818634 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:13.818605    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/28c727cb-1a33-4557-a0a0-76377739bddb-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-mnrh7\" (UID: \"28c727cb-1a33-4557-a0a0-76377739bddb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mnrh7"
Oct 31 19:23:13.818634 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:13.818630    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28c727cb-1a33-4557-a0a0-76377739bddb-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-mnrh7\" (UID: \"28c727cb-1a33-4557-a0a0-76377739bddb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mnrh7"
Oct 31 19:23:13.818879 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:13.818651    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/28c727cb-1a33-4557-a0a0-76377739bddb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-mnrh7\" (UID: \"28c727cb-1a33-4557-a0a0-76377739bddb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mnrh7"
Oct 31 19:23:13.819298 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:13.819268    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/28c727cb-1a33-4557-a0a0-76377739bddb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-mnrh7\" (UID: \"28c727cb-1a33-4557-a0a0-76377739bddb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mnrh7"
Oct 31 19:23:13.820976 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:13.820899    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/28c727cb-1a33-4557-a0a0-76377739bddb-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-mnrh7\" (UID: \"28c727cb-1a33-4557-a0a0-76377739bddb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mnrh7"
Oct 31 19:23:13.821085 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:13.821017    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/28c727cb-1a33-4557-a0a0-76377739bddb-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-mnrh7\" (UID: \"28c727cb-1a33-4557-a0a0-76377739bddb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mnrh7"
Oct 31 19:23:13.821085 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:13.821047    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/28c727cb-1a33-4557-a0a0-76377739bddb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-mnrh7\" (UID: \"28c727cb-1a33-4557-a0a0-76377739bddb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mnrh7"
Oct 31 19:23:13.821166 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:13.821146    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/28c727cb-1a33-4557-a0a0-76377739bddb-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-mnrh7\" (UID: \"28c727cb-1a33-4557-a0a0-76377739bddb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mnrh7"
Oct 31 19:23:13.822273 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:13.822252    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:13.830021 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:13.829995    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28c727cb-1a33-4557-a0a0-76377739bddb-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-mnrh7\" (UID: \"28c727cb-1a33-4557-a0a0-76377739bddb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mnrh7"
Oct 31 19:23:13.830644 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:13.830617    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjxln\" (UniqueName: \"kubernetes.io/projected/28c727cb-1a33-4557-a0a0-76377739bddb-kube-api-access-sjxln\") pod \"istiod-openshift-gateway-7876bfcb74-mnrh7\" (UID: \"28c727cb-1a33-4557-a0a0-76377739bddb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mnrh7"
Oct 31 19:23:13.917842 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:13.917819    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tt4vs"]
Oct 31 19:23:13.921135 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:23:13.921114    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod453e8d30_04fe_48d7_9f5d_70cd516b1ef6.slice/crio-8c64767c2a8d8065fd65d230853423a63578558b85b7dfd649c753b7f0a39cfe WatchSource:0}: Error finding container 8c64767c2a8d8065fd65d230853423a63578558b85b7dfd649c753b7f0a39cfe: Status 404 returned error can't find the container with id 8c64767c2a8d8065fd65d230853423a63578558b85b7dfd649c753b7f0a39cfe
Oct 31 19:23:13.946606 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:13.946589    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mnrh7"
Oct 31 19:23:14.231062 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:14.230985    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-mnrh7"]
Oct 31 19:23:14.233862 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:23:14.233839    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28c727cb_1a33_4557_a0a0_76377739bddb.slice/crio-5c906284cfa759797ef7cb4df276f5f2d129fd2a9994ad74db00f2fe8d779205 WatchSource:0}: Error finding container 5c906284cfa759797ef7cb4df276f5f2d129fd2a9994ad74db00f2fe8d779205: Status 404 returned error can't find the container with id 5c906284cfa759797ef7cb4df276f5f2d129fd2a9994ad74db00f2fe8d779205
Oct 31 19:23:14.521524 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:14.521439    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tt4vs" event={"ID":"453e8d30-04fe-48d7-9f5d-70cd516b1ef6","Type":"ContainerStarted","Data":"8c64767c2a8d8065fd65d230853423a63578558b85b7dfd649c753b7f0a39cfe"}
Oct 31 19:23:14.522858 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:14.522834    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mnrh7" event={"ID":"28c727cb-1a33-4557-a0a0-76377739bddb","Type":"ContainerStarted","Data":"5c906284cfa759797ef7cb4df276f5f2d129fd2a9994ad74db00f2fe8d779205"}
Oct 31 19:23:14.610022 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:14.609980    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:15.196681 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:15.196633    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:17.684271 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:17.684231    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:23:17.684709 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:17.684314    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:23:18.266449 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:18.266410    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:18.418631 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:18.418550    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:18.542563 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:18.542473    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mnrh7" event={"ID":"28c727cb-1a33-4557-a0a0-76377739bddb","Type":"ContainerStarted","Data":"620b5bad2940bd1d9c3c7bae47583c282672a1b2a6a9232139fa4211e9901f85"}
Oct 31 19:23:18.542722 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:18.542684    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mnrh7"
Oct 31 19:23:18.544004 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:18.543972    4645 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-mnrh7 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 19:23:18.544174 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:18.544027    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mnrh7" podUID="28c727cb-1a33-4557-a0a0-76377739bddb" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 19:23:18.624191 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:18.624148    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:18.867209 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:18.867126    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:19.014433 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:19.014392    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:19.548123 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:19.548096    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mnrh7"
Oct 31 19:23:19.573214 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:19.573009    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mnrh7" podStartSLOduration=3.1253954090000002 podStartE2EDuration="6.572991194s" totalImagesPullingTime="3.447595785s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:23:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:23:18.583086431 +0000 UTC m=+229.156320564" watchObservedRunningTime="2026-10-31 19:23:19.572991194 +0000 UTC m=+230.146225325"
Oct 31 19:23:20.552286 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:20.552253    4645 generic.go:356] "Generic (PLEG): container finished" podID="453e8d30-04fe-48d7-9f5d-70cd516b1ef6" containerID="c6efe370ff0a5dea800febedf2ec0f32344e02d38d2b33703cb9e76096d3089a" exitCode=0
Oct 31 19:23:20.552775 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:20.552353    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tt4vs" event={"ID":"453e8d30-04fe-48d7-9f5d-70cd516b1ef6","Type":"ContainerDied","Data":"c6efe370ff0a5dea800febedf2ec0f32344e02d38d2b33703cb9e76096d3089a"}
Oct 31 19:23:21.526341 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:21.526300    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:23.564824 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:23.564793    4645 generic.go:356] "Generic (PLEG): container finished" podID="453e8d30-04fe-48d7-9f5d-70cd516b1ef6" containerID="8f891cee57edd7563caeb8e0defb6a064ebf38b726a621993dba7e0658588566" exitCode=0
Oct 31 19:23:23.565175 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:23.564874    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tt4vs" event={"ID":"453e8d30-04fe-48d7-9f5d-70cd516b1ef6","Type":"ContainerDied","Data":"8f891cee57edd7563caeb8e0defb6a064ebf38b726a621993dba7e0658588566"}
Oct 31 19:23:29.588232 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:29.588203    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tt4vs" event={"ID":"453e8d30-04fe-48d7-9f5d-70cd516b1ef6","Type":"ContainerStarted","Data":"cd84e5ae76cc0293852c7f53f8183c56ac00a8cd6b393efd20d25e7f42289625"}
Oct 31 19:23:29.610160 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:29.610092    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tt4vs" podStartSLOduration=2.387536592 podStartE2EDuration="16.610078028s" totalImagesPullingTime="14.222541436s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:23:13 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:23:29.609976234 +0000 UTC m=+240.183210354" watchObservedRunningTime="2026-10-31 19:23:29.610078028 +0000 UTC m=+240.183312156"
Oct 31 19:23:30.594025 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:30.593980    4645 generic.go:356] "Generic (PLEG): container finished" podID="453e8d30-04fe-48d7-9f5d-70cd516b1ef6" containerID="cd84e5ae76cc0293852c7f53f8183c56ac00a8cd6b393efd20d25e7f42289625" exitCode=0
Oct 31 19:23:30.594439 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:30.594067    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tt4vs" event={"ID":"453e8d30-04fe-48d7-9f5d-70cd516b1ef6","Type":"ContainerDied","Data":"cd84e5ae76cc0293852c7f53f8183c56ac00a8cd6b393efd20d25e7f42289625"}
Oct 31 19:23:31.720513 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:31.720486    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tt4vs"
Oct 31 19:23:31.759999 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:31.759979    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/453e8d30-04fe-48d7-9f5d-70cd516b1ef6-kube-api-access-rzf94\" (UniqueName: \"kubernetes.io/projected/453e8d30-04fe-48d7-9f5d-70cd516b1ef6-kube-api-access-rzf94\") pod \"453e8d30-04fe-48d7-9f5d-70cd516b1ef6\" (UID: \"453e8d30-04fe-48d7-9f5d-70cd516b1ef6\") "
Oct 31 19:23:31.760112 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:31.760017    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/453e8d30-04fe-48d7-9f5d-70cd516b1ef6-util\" (UniqueName: \"kubernetes.io/empty-dir/453e8d30-04fe-48d7-9f5d-70cd516b1ef6-util\") pod \"453e8d30-04fe-48d7-9f5d-70cd516b1ef6\" (UID: \"453e8d30-04fe-48d7-9f5d-70cd516b1ef6\") "
Oct 31 19:23:31.760112 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:31.760071    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/453e8d30-04fe-48d7-9f5d-70cd516b1ef6-bundle\" (UniqueName: \"kubernetes.io/empty-dir/453e8d30-04fe-48d7-9f5d-70cd516b1ef6-bundle\") pod \"453e8d30-04fe-48d7-9f5d-70cd516b1ef6\" (UID: \"453e8d30-04fe-48d7-9f5d-70cd516b1ef6\") "
Oct 31 19:23:31.760475 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:31.760450    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/453e8d30-04fe-48d7-9f5d-70cd516b1ef6-bundle" pod "453e8d30-04fe-48d7-9f5d-70cd516b1ef6" (UID: "453e8d30-04fe-48d7-9f5d-70cd516b1ef6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:23:31.762043 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:31.762023    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/453e8d30-04fe-48d7-9f5d-70cd516b1ef6-kube-api-access-rzf94" pod "453e8d30-04fe-48d7-9f5d-70cd516b1ef6" (UID: "453e8d30-04fe-48d7-9f5d-70cd516b1ef6"). InnerVolumeSpecName "kube-api-access-rzf94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:23:31.764300 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:31.764275    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/453e8d30-04fe-48d7-9f5d-70cd516b1ef6-util" pod "453e8d30-04fe-48d7-9f5d-70cd516b1ef6" (UID: "453e8d30-04fe-48d7-9f5d-70cd516b1ef6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:23:31.860840 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:31.860786    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzf94\" (UniqueName: \"kubernetes.io/projected/453e8d30-04fe-48d7-9f5d-70cd516b1ef6-kube-api-access-rzf94\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:23:31.860840 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:31.860818    4645 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/453e8d30-04fe-48d7-9f5d-70cd516b1ef6-util\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:23:31.860840 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:31.860829    4645 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/453e8d30-04fe-48d7-9f5d-70cd516b1ef6-bundle\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:23:32.602994 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:32.602962    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tt4vs"
Oct 31 19:23:32.603099 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:32.602958    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644tt4vs" event={"ID":"453e8d30-04fe-48d7-9f5d-70cd516b1ef6","Type":"ContainerDied","Data":"8c64767c2a8d8065fd65d230853423a63578558b85b7dfd649c753b7f0a39cfe"}
Oct 31 19:23:32.603099 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:32.603092    4645 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c64767c2a8d8065fd65d230853423a63578558b85b7dfd649c753b7f0a39cfe"
Oct 31 19:23:36.246779 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:23:36.246737    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 19:23:40.371529 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:40.371486    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:41.379858 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:41.379823    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:44.769519 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:44.769469    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:44.904916 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:44.904875    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:47.047780 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:47.047728    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:48.355882 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:48.355833    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:49.591612 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:49.591577    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:49.727009 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:49.726976    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:49.844174 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:49.844098    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:55.588858 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:55.588825    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:55.731819 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:55.731786    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:56.497296 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:56.497240    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:56.678879 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:56.678845    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-mbfk5"]
Oct 31 19:23:56.679246 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:56.679098    4645 cpu_manager.go:414] "containerMap: removing container" podUID="453e8d30-04fe-48d7-9f5d-70cd516b1ef6" containerName="pull"
Oct 31 19:23:56.679246 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:56.679109    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="453e8d30-04fe-48d7-9f5d-70cd516b1ef6" containerName="pull"
Oct 31 19:23:56.679246 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:56.679136    4645 cpu_manager.go:414] "containerMap: removing container" podUID="453e8d30-04fe-48d7-9f5d-70cd516b1ef6" containerName="util"
Oct 31 19:23:56.679246 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:56.679141    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="453e8d30-04fe-48d7-9f5d-70cd516b1ef6" containerName="util"
Oct 31 19:23:56.679246 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:56.679149    4645 cpu_manager.go:414] "containerMap: removing container" podUID="453e8d30-04fe-48d7-9f5d-70cd516b1ef6" containerName="extract"
Oct 31 19:23:56.679246 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:56.679154    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="453e8d30-04fe-48d7-9f5d-70cd516b1ef6" containerName="extract"
Oct 31 19:23:56.679246 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:56.679195    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="453e8d30-04fe-48d7-9f5d-70cd516b1ef6" containerName="extract"
Oct 31 19:23:56.682103 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:56.682086    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-mbfk5"
Oct 31 19:23:56.706423 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:56.706399    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kbtm\" (UniqueName: \"kubernetes.io/projected/e2ab4650-3c3b-4754-a1a8-f99673da37ac-kube-api-access-5kbtm\") pod \"opendatahub-operator-controller-manager-6bb8c5759f-mbfk5\" (UID: \"e2ab4650-3c3b-4754-a1a8-f99673da37ac\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-mbfk5"
Oct 31 19:23:56.706541 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:56.706436    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e2ab4650-3c3b-4754-a1a8-f99673da37ac-manifests\") pod \"opendatahub-operator-controller-manager-6bb8c5759f-mbfk5\" (UID: \"e2ab4650-3c3b-4754-a1a8-f99673da37ac\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-mbfk5"
Oct 31 19:23:56.706541 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:56.706465    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e2ab4650-3c3b-4754-a1a8-f99673da37ac-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bb8c5759f-mbfk5\" (UID: \"e2ab4650-3c3b-4754-a1a8-f99673da37ac\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-mbfk5"
Oct 31 19:23:56.706614 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:56.706555    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e2ab4650-3c3b-4754-a1a8-f99673da37ac-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bb8c5759f-mbfk5\" (UID: \"e2ab4650-3c3b-4754-a1a8-f99673da37ac\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-mbfk5"
Oct 31 19:23:56.706614 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:56.706587    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e2ab4650-3c3b-4754-a1a8-f99673da37ac-metrics-cert\") pod \"opendatahub-operator-controller-manager-6bb8c5759f-mbfk5\" (UID: \"e2ab4650-3c3b-4754-a1a8-f99673da37ac\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-mbfk5"
Oct 31 19:23:56.706614 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:56.706610    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e2ab4650-3c3b-4754-a1a8-f99673da37ac-tmp\") pod \"opendatahub-operator-controller-manager-6bb8c5759f-mbfk5\" (UID: \"e2ab4650-3c3b-4754-a1a8-f99673da37ac\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-mbfk5"
Oct 31 19:23:56.730310 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:56.730283    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-mbfk5"]
Oct 31 19:23:56.750313 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:56.750262    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 31 19:23:56.750412 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:56.750339    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 31 19:23:56.751253 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:56.751237    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 31 19:23:56.751452 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:56.751433    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 31 19:23:56.751544 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:56.751512    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 31 19:23:56.751544 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:56.751524    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-hzfp2\""
Oct 31 19:23:56.807171 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:56.807144    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kbtm\" (UniqueName: \"kubernetes.io/projected/e2ab4650-3c3b-4754-a1a8-f99673da37ac-kube-api-access-5kbtm\") pod \"opendatahub-operator-controller-manager-6bb8c5759f-mbfk5\" (UID: \"e2ab4650-3c3b-4754-a1a8-f99673da37ac\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-mbfk5"
Oct 31 19:23:56.807279 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:56.807180    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e2ab4650-3c3b-4754-a1a8-f99673da37ac-manifests\") pod \"opendatahub-operator-controller-manager-6bb8c5759f-mbfk5\" (UID: \"e2ab4650-3c3b-4754-a1a8-f99673da37ac\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-mbfk5"
Oct 31 19:23:56.807279 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:56.807204    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e2ab4650-3c3b-4754-a1a8-f99673da37ac-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bb8c5759f-mbfk5\" (UID: \"e2ab4650-3c3b-4754-a1a8-f99673da37ac\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-mbfk5"
Oct 31 19:23:56.807279 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:56.807222    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e2ab4650-3c3b-4754-a1a8-f99673da37ac-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bb8c5759f-mbfk5\" (UID: \"e2ab4650-3c3b-4754-a1a8-f99673da37ac\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-mbfk5"
Oct 31 19:23:56.807279 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:56.807239    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e2ab4650-3c3b-4754-a1a8-f99673da37ac-metrics-cert\") pod \"opendatahub-operator-controller-manager-6bb8c5759f-mbfk5\" (UID: \"e2ab4650-3c3b-4754-a1a8-f99673da37ac\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-mbfk5"
Oct 31 19:23:56.807279 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:56.807260    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e2ab4650-3c3b-4754-a1a8-f99673da37ac-tmp\") pod \"opendatahub-operator-controller-manager-6bb8c5759f-mbfk5\" (UID: \"e2ab4650-3c3b-4754-a1a8-f99673da37ac\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-mbfk5"
Oct 31 19:23:56.807632 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:56.807612    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e2ab4650-3c3b-4754-a1a8-f99673da37ac-manifests\") pod \"opendatahub-operator-controller-manager-6bb8c5759f-mbfk5\" (UID: \"e2ab4650-3c3b-4754-a1a8-f99673da37ac\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-mbfk5"
Oct 31 19:23:56.809741 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:56.809718    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e2ab4650-3c3b-4754-a1a8-f99673da37ac-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bb8c5759f-mbfk5\" (UID: \"e2ab4650-3c3b-4754-a1a8-f99673da37ac\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-mbfk5"
Oct 31 19:23:56.809741 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:56.809734    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e2ab4650-3c3b-4754-a1a8-f99673da37ac-tmp\") pod \"opendatahub-operator-controller-manager-6bb8c5759f-mbfk5\" (UID: \"e2ab4650-3c3b-4754-a1a8-f99673da37ac\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-mbfk5"
Oct 31 19:23:56.809953 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:56.809924    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e2ab4650-3c3b-4754-a1a8-f99673da37ac-metrics-cert\") pod \"opendatahub-operator-controller-manager-6bb8c5759f-mbfk5\" (UID: \"e2ab4650-3c3b-4754-a1a8-f99673da37ac\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-mbfk5"
Oct 31 19:23:56.810047 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:56.809924    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e2ab4650-3c3b-4754-a1a8-f99673da37ac-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bb8c5759f-mbfk5\" (UID: \"e2ab4650-3c3b-4754-a1a8-f99673da37ac\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-mbfk5"
Oct 31 19:23:56.821961 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:56.821939    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kbtm\" (UniqueName: \"kubernetes.io/projected/e2ab4650-3c3b-4754-a1a8-f99673da37ac-kube-api-access-5kbtm\") pod \"opendatahub-operator-controller-manager-6bb8c5759f-mbfk5\" (UID: \"e2ab4650-3c3b-4754-a1a8-f99673da37ac\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-mbfk5"
Oct 31 19:23:56.897901 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:56.897870    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:56.997092 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:56.997065    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-mbfk5"
Oct 31 19:23:57.050359 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:57.049826    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:57.190387 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:57.190351    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:23:57.458082 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:57.458056    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-mbfk5"]
Oct 31 19:23:57.461385 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:23:57.461358    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2ab4650_3c3b_4754_a1a8_f99673da37ac.slice/crio-13f45300dcba6fd1d00e9469399cf60a9eb494bcec00a311e90a50e93f131ea8 WatchSource:0}: Error finding container 13f45300dcba6fd1d00e9469399cf60a9eb494bcec00a311e90a50e93f131ea8: Status 404 returned error can't find the container with id 13f45300dcba6fd1d00e9469399cf60a9eb494bcec00a311e90a50e93f131ea8
Oct 31 19:23:57.696595 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:57.696564    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-mbfk5" event={"ID":"e2ab4650-3c3b-4754-a1a8-f99673da37ac","Type":"ContainerStarted","Data":"13f45300dcba6fd1d00e9469399cf60a9eb494bcec00a311e90a50e93f131ea8"}
Oct 31 19:23:59.677151 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:23:59.677084    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:00.640410 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:00.640343    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:00.712065 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:00.712030    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-mbfk5" event={"ID":"e2ab4650-3c3b-4754-a1a8-f99673da37ac","Type":"ContainerStarted","Data":"05625aadcac622767deccf4437c467940e7537e76bf919af4139376a6237ae7c"}
Oct 31 19:24:01.245210 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:01.245166    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:01.716707 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:01.716676    4645 generic.go:356] "Generic (PLEG): container finished" podID="e2ab4650-3c3b-4754-a1a8-f99673da37ac" containerID="05625aadcac622767deccf4437c467940e7537e76bf919af4139376a6237ae7c" exitCode=0
Oct 31 19:24:01.717097 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:01.716714    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-mbfk5" event={"ID":"e2ab4650-3c3b-4754-a1a8-f99673da37ac","Type":"ContainerDied","Data":"05625aadcac622767deccf4437c467940e7537e76bf919af4139376a6237ae7c"}
Oct 31 19:24:02.723003 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:02.722960    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-mbfk5" event={"ID":"e2ab4650-3c3b-4754-a1a8-f99673da37ac","Type":"ContainerStarted","Data":"6c2cedeca0f660f08b0b36e34b80e22b1a2697da9ca267fe9916fbbc847ecdd8"}
Oct 31 19:24:02.723541 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:02.723107    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-mbfk5"
Oct 31 19:24:02.781153 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:02.781098    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-mbfk5" podStartSLOduration=3.167030477 podStartE2EDuration="6.781081373s" totalImagesPullingTime="3.178357517s" totalInitContainerRuntime="435.693379ms" isStatefulPod=false podCreationTimestamp="2026-10-31 19:23:56 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:24:02.780111575 +0000 UTC m=+273.353345704" watchObservedRunningTime="2026-10-31 19:24:02.781081373 +0000 UTC m=+273.354315504"
Oct 31 19:24:03.588252 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:03.588213    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:04.409137 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:04.409103    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:05.110918 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:05.110882    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:05.271493 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:05.271451    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:06.243627 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:24:06.243594    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 19:24:09.182769 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:09.182720    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:09.519369 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:09.519280    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:10.635426 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:10.635382    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:10.816303 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:10.816269    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:11.598975 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:11.598950    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:11.936022 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:11.935993    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:12.116543 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:12.116512    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:13.368184 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:13.368154    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:13.729401 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:13.729372    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6bb8c5759f-mbfk5"
Oct 31 19:24:14.093211 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:14.093124    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:14.206728 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:14.206689    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:14.621096 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:14.621063    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:14.946488 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:14.946453    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:15.643902 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:15.643872    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:15.807773 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:15.807739    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:22.239235 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:22.239200    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:22.376582 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:22.376547    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:24.403616 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:24.403584    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:24.527567 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:24.527541    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:24.708389 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:24.708358    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-849d94dfbb-l28r8"]
Oct 31 19:24:24.713645 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:24.713626    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-849d94dfbb-l28r8"
Oct 31 19:24:24.732976 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:24.732953    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-849d94dfbb-l28r8"]
Oct 31 19:24:24.744181 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:24.744153    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:24.750129 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:24.750112    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 31 19:24:24.750290 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:24.750262    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-skfzl\""
Oct 31 19:24:24.750364 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:24.750289    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 31 19:24:24.800885 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:24.800855    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae933cbf-384e-4052-881d-800d405f8e7b-tls-certs\") pod \"kube-auth-proxy-849d94dfbb-l28r8\" (UID: \"ae933cbf-384e-4052-881d-800d405f8e7b\") " pod="openshift-ingress/kube-auth-proxy-849d94dfbb-l28r8"
Oct 31 19:24:24.801039 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:24.800905    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae933cbf-384e-4052-881d-800d405f8e7b-tmp\") pod \"kube-auth-proxy-849d94dfbb-l28r8\" (UID: \"ae933cbf-384e-4052-881d-800d405f8e7b\") " pod="openshift-ingress/kube-auth-proxy-849d94dfbb-l28r8"
Oct 31 19:24:24.801039 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:24.801025    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgkbj\" (UniqueName: \"kubernetes.io/projected/ae933cbf-384e-4052-881d-800d405f8e7b-kube-api-access-kgkbj\") pod \"kube-auth-proxy-849d94dfbb-l28r8\" (UID: \"ae933cbf-384e-4052-881d-800d405f8e7b\") " pod="openshift-ingress/kube-auth-proxy-849d94dfbb-l28r8"
Oct 31 19:24:24.902320 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:24.902289    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae933cbf-384e-4052-881d-800d405f8e7b-tls-certs\") pod \"kube-auth-proxy-849d94dfbb-l28r8\" (UID: \"ae933cbf-384e-4052-881d-800d405f8e7b\") " pod="openshift-ingress/kube-auth-proxy-849d94dfbb-l28r8"
Oct 31 19:24:24.902484 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:24.902343    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae933cbf-384e-4052-881d-800d405f8e7b-tmp\") pod \"kube-auth-proxy-849d94dfbb-l28r8\" (UID: \"ae933cbf-384e-4052-881d-800d405f8e7b\") " pod="openshift-ingress/kube-auth-proxy-849d94dfbb-l28r8"
Oct 31 19:24:24.902484 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:24.902372    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgkbj\" (UniqueName: \"kubernetes.io/projected/ae933cbf-384e-4052-881d-800d405f8e7b-kube-api-access-kgkbj\") pod \"kube-auth-proxy-849d94dfbb-l28r8\" (UID: \"ae933cbf-384e-4052-881d-800d405f8e7b\") " pod="openshift-ingress/kube-auth-proxy-849d94dfbb-l28r8"
Oct 31 19:24:24.904497 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:24.904477    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae933cbf-384e-4052-881d-800d405f8e7b-tmp\") pod \"kube-auth-proxy-849d94dfbb-l28r8\" (UID: \"ae933cbf-384e-4052-881d-800d405f8e7b\") " pod="openshift-ingress/kube-auth-proxy-849d94dfbb-l28r8"
Oct 31 19:24:24.904829 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:24.904809    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae933cbf-384e-4052-881d-800d405f8e7b-tls-certs\") pod \"kube-auth-proxy-849d94dfbb-l28r8\" (UID: \"ae933cbf-384e-4052-881d-800d405f8e7b\") " pod="openshift-ingress/kube-auth-proxy-849d94dfbb-l28r8"
Oct 31 19:24:24.916796 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:24.916768    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgkbj\" (UniqueName: \"kubernetes.io/projected/ae933cbf-384e-4052-881d-800d405f8e7b-kube-api-access-kgkbj\") pod \"kube-auth-proxy-849d94dfbb-l28r8\" (UID: \"ae933cbf-384e-4052-881d-800d405f8e7b\") " pod="openshift-ingress/kube-auth-proxy-849d94dfbb-l28r8"
Oct 31 19:24:24.927278 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:24.927255    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:25.025936 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:25.025887    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-849d94dfbb-l28r8"
Oct 31 19:24:25.064467 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:25.064434    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:25.180083 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:25.180050    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:25.312025 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:25.311966    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-849d94dfbb-l28r8"]
Oct 31 19:24:25.314886 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:24:25.314860    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae933cbf_384e_4052_881d_800d405f8e7b.slice/crio-198e3b626b090f2fea887675a0f2bcca3a07fac21d09cbeb417ffe5e9d2007d8 WatchSource:0}: Error finding container 198e3b626b090f2fea887675a0f2bcca3a07fac21d09cbeb417ffe5e9d2007d8: Status 404 returned error can't find the container with id 198e3b626b090f2fea887675a0f2bcca3a07fac21d09cbeb417ffe5e9d2007d8
Oct 31 19:24:25.808312 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:25.808273    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-849d94dfbb-l28r8" event={"ID":"ae933cbf-384e-4052-881d-800d405f8e7b","Type":"ContainerStarted","Data":"198e3b626b090f2fea887675a0f2bcca3a07fac21d09cbeb417ffe5e9d2007d8"}
Oct 31 19:24:26.406473 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:26.406443    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:26.554717 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:26.554681    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:26.694101 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:26.694065    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:26.836628 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:26.836593    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:27.011940 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:27.011867    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:27.447422 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:27.447387    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:27.631186 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:27.631153    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:27.710517 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:27.710464    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-vkdkl"]
Oct 31 19:24:27.713501 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:27.713485    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-vkdkl"
Oct 31 19:24:27.722451 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:27.722431    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 31 19:24:27.727098 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:27.727078    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-42cfq\""
Oct 31 19:24:27.731619 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:27.731599    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-vkdkl"]
Oct 31 19:24:27.798957 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:27.798917    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:27.821656 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:27.821618    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-849d94dfbb-l28r8" event={"ID":"ae933cbf-384e-4052-881d-800d405f8e7b","Type":"ContainerStarted","Data":"7791d26a2507e2736cc4ce3985110f29d1b9575d1783fa977b1e13636d3f4b11"}
Oct 31 19:24:27.821818 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:27.821799    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 31 19:24:27.828118 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:27.828098    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgdqt\" (UniqueName: \"kubernetes.io/projected/70ec4dc6-89c2-4004-9c4b-42375f9bddb3-kube-api-access-jgdqt\") pod \"authorino-57f7f8659f-vkdkl\" (UID: \"70ec4dc6-89c2-4004-9c4b-42375f9bddb3\") " pod="kuadrant-system/authorino-57f7f8659f-vkdkl"
Oct 31 19:24:27.928432 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:27.928404    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgdqt\" (UniqueName: \"kubernetes.io/projected/70ec4dc6-89c2-4004-9c4b-42375f9bddb3-kube-api-access-jgdqt\") pod \"authorino-57f7f8659f-vkdkl\" (UID: \"70ec4dc6-89c2-4004-9c4b-42375f9bddb3\") " pod="kuadrant-system/authorino-57f7f8659f-vkdkl"
Oct 31 19:24:27.943186 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:27.943159    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgdqt\" (UniqueName: \"kubernetes.io/projected/70ec4dc6-89c2-4004-9c4b-42375f9bddb3-kube-api-access-jgdqt\") pod \"authorino-57f7f8659f-vkdkl\" (UID: \"70ec4dc6-89c2-4004-9c4b-42375f9bddb3\") " pod="kuadrant-system/authorino-57f7f8659f-vkdkl"
Oct 31 19:24:27.952567 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:27.952542    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:28.037857 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:28.037801    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-vkdkl"
Oct 31 19:24:28.312211 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:28.312122    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-849d94dfbb-l28r8" podStartSLOduration=1.993395602 podStartE2EDuration="4.312106198s" totalImagesPullingTime="2.318710596s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:24:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:24:27.85036585 +0000 UTC m=+298.423599979" watchObservedRunningTime="2026-10-31 19:24:28.312106198 +0000 UTC m=+298.885340326"
Oct 31 19:24:28.312382 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:28.312245    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-vkdkl"]
Oct 31 19:24:28.315185 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:24:28.315159    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod70ec4dc6_89c2_4004_9c4b_42375f9bddb3.slice/crio-cb42d7e5dc5a69be8bb409f2d954187a200afd9fd47badeda4030d4d2af9ee18 WatchSource:0}: Error finding container cb42d7e5dc5a69be8bb409f2d954187a200afd9fd47badeda4030d4d2af9ee18: Status 404 returned error can't find the container with id cb42d7e5dc5a69be8bb409f2d954187a200afd9fd47badeda4030d4d2af9ee18
Oct 31 19:24:28.443018 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:28.442996    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-c9tc7"]
Oct 31 19:24:28.447238 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:28.447223    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-c9tc7"
Oct 31 19:24:28.455504 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:28.455484    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-gn576\""
Oct 31 19:24:28.457621 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:28.457602    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\""
Oct 31 19:24:28.457982 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:28.457967    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 31 19:24:28.460209 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:28.460187    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-c9tc7"]
Oct 31 19:24:28.534462 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:28.534441    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9f12248a-17ea-4f35-9de4-d45b71d238cb-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-c9tc7\" (UID: \"9f12248a-17ea-4f35-9de4-d45b71d238cb\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-c9tc7"
Oct 31 19:24:28.534556 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:28.534484    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9f12248a-17ea-4f35-9de4-d45b71d238cb-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-c9tc7\" (UID: \"9f12248a-17ea-4f35-9de4-d45b71d238cb\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-c9tc7"
Oct 31 19:24:28.534599 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:28.534580    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f12248a-17ea-4f35-9de4-d45b71d238cb-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-c9tc7\" (UID: \"9f12248a-17ea-4f35-9de4-d45b71d238cb\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-c9tc7"
Oct 31 19:24:28.534641 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:28.534627    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw2c9\" (UniqueName: \"kubernetes.io/projected/9f12248a-17ea-4f35-9de4-d45b71d238cb-kube-api-access-rw2c9\") pod \"kserve-module-controller-manager-7566cf5fc8-c9tc7\" (UID: \"9f12248a-17ea-4f35-9de4-d45b71d238cb\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-c9tc7"
Oct 31 19:24:28.555002 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:28.554980    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 31 19:24:28.635923 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:28.635874    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw2c9\" (UniqueName: \"kubernetes.io/projected/9f12248a-17ea-4f35-9de4-d45b71d238cb-kube-api-access-rw2c9\") pod \"kserve-module-controller-manager-7566cf5fc8-c9tc7\" (UID: \"9f12248a-17ea-4f35-9de4-d45b71d238cb\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-c9tc7"
Oct 31 19:24:28.635923 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:28.635906    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9f12248a-17ea-4f35-9de4-d45b71d238cb-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-c9tc7\" (UID: \"9f12248a-17ea-4f35-9de4-d45b71d238cb\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-c9tc7"
Oct 31 19:24:28.636042 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:28.635930    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9f12248a-17ea-4f35-9de4-d45b71d238cb-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-c9tc7\" (UID: \"9f12248a-17ea-4f35-9de4-d45b71d238cb\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-c9tc7"
Oct 31 19:24:28.636042 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:28.635960    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f12248a-17ea-4f35-9de4-d45b71d238cb-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-c9tc7\" (UID: \"9f12248a-17ea-4f35-9de4-d45b71d238cb\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-c9tc7"
Oct 31 19:24:28.636304 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:28.636288    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9f12248a-17ea-4f35-9de4-d45b71d238cb-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-c9tc7\" (UID: \"9f12248a-17ea-4f35-9de4-d45b71d238cb\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-c9tc7"
Oct 31 19:24:28.636404 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:28.636390    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f12248a-17ea-4f35-9de4-d45b71d238cb-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-c9tc7\" (UID: \"9f12248a-17ea-4f35-9de4-d45b71d238cb\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-c9tc7"
Oct 31 19:24:28.636800 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:28.636784    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9f12248a-17ea-4f35-9de4-d45b71d238cb-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-c9tc7\" (UID: \"9f12248a-17ea-4f35-9de4-d45b71d238cb\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-c9tc7"
Oct 31 19:24:28.648379 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:28.648353    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw2c9\" (UniqueName: \"kubernetes.io/projected/9f12248a-17ea-4f35-9de4-d45b71d238cb-kube-api-access-rw2c9\") pod \"kserve-module-controller-manager-7566cf5fc8-c9tc7\" (UID: \"9f12248a-17ea-4f35-9de4-d45b71d238cb\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-c9tc7"
Oct 31 19:24:28.759232 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:28.759212    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-c9tc7"
Oct 31 19:24:28.826413 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:28.826371    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-vkdkl" event={"ID":"70ec4dc6-89c2-4004-9c4b-42375f9bddb3","Type":"ContainerStarted","Data":"cb42d7e5dc5a69be8bb409f2d954187a200afd9fd47badeda4030d4d2af9ee18"}
Oct 31 19:24:29.039390 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:29.039367    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-c9tc7"]
Oct 31 19:24:29.041526 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:24:29.041499    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f12248a_17ea_4f35_9de4_d45b71d238cb.slice/crio-6aee5e7747bcd3b7e9a67f1167e0d53fbd401fb0aeac0631b5e8128438c1e096 WatchSource:0}: Error finding container 6aee5e7747bcd3b7e9a67f1167e0d53fbd401fb0aeac0631b5e8128438c1e096: Status 404 returned error can't find the container with id 6aee5e7747bcd3b7e9a67f1167e0d53fbd401fb0aeac0631b5e8128438c1e096
Oct 31 19:24:29.821472 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:29.821440    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:29.831352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:29.831317    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-c9tc7" event={"ID":"9f12248a-17ea-4f35-9de4-d45b71d238cb","Type":"ContainerStarted","Data":"6aee5e7747bcd3b7e9a67f1167e0d53fbd401fb0aeac0631b5e8128438c1e096"}
Oct 31 19:24:29.974605 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:29.974570    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:30.838058 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:30.837976    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-c9tc7" event={"ID":"9f12248a-17ea-4f35-9de4-d45b71d238cb","Type":"ContainerStarted","Data":"6a686bd43de9cd4eeb40ec9496dfb15a741979ac2d1e713fe1f52f2d98d31cea"}
Oct 31 19:24:30.838520 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:30.838106    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-c9tc7"
Oct 31 19:24:30.860389 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:30.859380    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-c9tc7" podStartSLOduration=1.291798813 podStartE2EDuration="2.859364497s" totalImagesPullingTime="1.567565684s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:24:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:24:30.858277795 +0000 UTC m=+301.431511934" watchObservedRunningTime="2026-10-31 19:24:30.859364497 +0000 UTC m=+301.432598627"
Oct 31 19:24:30.977418 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:30.977383    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:31.083875 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:31.083837    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:31.454531 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:31.454492    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-vkdkl"]
Oct 31 19:24:31.668972 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:31.668944    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-6c8c6c9d8f-rjxtl"]
Oct 31 19:24:31.672300 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:31.672277    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-6c8c6c9d8f-rjxtl"
Oct 31 19:24:31.677222 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:31.677198    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\""
Oct 31 19:24:31.677360 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:31.677241    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\""
Oct 31 19:24:31.678736 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:31.678713    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\""
Oct 31 19:24:31.679866 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:31.679845    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Oct 31 19:24:31.679983 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:31.679904    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Oct 31 19:24:31.680048 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:31.680027    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-svkk9\""
Oct 31 19:24:31.685751 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:31.685730    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-6c8c6c9d8f-rjxtl"]
Oct 31 19:24:31.763651 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:31.763588    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2f80883e-35e9-46d6-aee0-490b3a0d8e05-cert\") pod \"jobset-controller-manager-6c8c6c9d8f-rjxtl\" (UID: \"2f80883e-35e9-46d6-aee0-490b3a0d8e05\") " pod="openshift-jobset-operator/jobset-controller-manager-6c8c6c9d8f-rjxtl"
Oct 31 19:24:31.763807 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:31.763651    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2f80883e-35e9-46d6-aee0-490b3a0d8e05-manager-config\") pod \"jobset-controller-manager-6c8c6c9d8f-rjxtl\" (UID: \"2f80883e-35e9-46d6-aee0-490b3a0d8e05\") " pod="openshift-jobset-operator/jobset-controller-manager-6c8c6c9d8f-rjxtl"
Oct 31 19:24:31.763807 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:31.763705    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjsqq\" (UniqueName: \"kubernetes.io/projected/2f80883e-35e9-46d6-aee0-490b3a0d8e05-kube-api-access-rjsqq\") pod \"jobset-controller-manager-6c8c6c9d8f-rjxtl\" (UID: \"2f80883e-35e9-46d6-aee0-490b3a0d8e05\") " pod="openshift-jobset-operator/jobset-controller-manager-6c8c6c9d8f-rjxtl"
Oct 31 19:24:31.763807 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:31.763792    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f80883e-35e9-46d6-aee0-490b3a0d8e05-metrics-certs\") pod \"jobset-controller-manager-6c8c6c9d8f-rjxtl\" (UID: \"2f80883e-35e9-46d6-aee0-490b3a0d8e05\") " pod="openshift-jobset-operator/jobset-controller-manager-6c8c6c9d8f-rjxtl"
Oct 31 19:24:31.864944 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:31.864915    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2f80883e-35e9-46d6-aee0-490b3a0d8e05-cert\") pod \"jobset-controller-manager-6c8c6c9d8f-rjxtl\" (UID: \"2f80883e-35e9-46d6-aee0-490b3a0d8e05\") " pod="openshift-jobset-operator/jobset-controller-manager-6c8c6c9d8f-rjxtl"
Oct 31 19:24:31.865432 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:31.864990    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2f80883e-35e9-46d6-aee0-490b3a0d8e05-manager-config\") pod \"jobset-controller-manager-6c8c6c9d8f-rjxtl\" (UID: \"2f80883e-35e9-46d6-aee0-490b3a0d8e05\") " pod="openshift-jobset-operator/jobset-controller-manager-6c8c6c9d8f-rjxtl"
Oct 31 19:24:31.865432 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:31.865017    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjsqq\" (UniqueName: \"kubernetes.io/projected/2f80883e-35e9-46d6-aee0-490b3a0d8e05-kube-api-access-rjsqq\") pod \"jobset-controller-manager-6c8c6c9d8f-rjxtl\" (UID: \"2f80883e-35e9-46d6-aee0-490b3a0d8e05\") " pod="openshift-jobset-operator/jobset-controller-manager-6c8c6c9d8f-rjxtl"
Oct 31 19:24:31.865432 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:31.865055    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f80883e-35e9-46d6-aee0-490b3a0d8e05-metrics-certs\") pod \"jobset-controller-manager-6c8c6c9d8f-rjxtl\" (UID: \"2f80883e-35e9-46d6-aee0-490b3a0d8e05\") " pod="openshift-jobset-operator/jobset-controller-manager-6c8c6c9d8f-rjxtl"
Oct 31 19:24:31.865792 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:31.865771    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2f80883e-35e9-46d6-aee0-490b3a0d8e05-manager-config\") pod \"jobset-controller-manager-6c8c6c9d8f-rjxtl\" (UID: \"2f80883e-35e9-46d6-aee0-490b3a0d8e05\") " pod="openshift-jobset-operator/jobset-controller-manager-6c8c6c9d8f-rjxtl"
Oct 31 19:24:31.867587 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:31.867563    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f80883e-35e9-46d6-aee0-490b3a0d8e05-metrics-certs\") pod \"jobset-controller-manager-6c8c6c9d8f-rjxtl\" (UID: \"2f80883e-35e9-46d6-aee0-490b3a0d8e05\") " pod="openshift-jobset-operator/jobset-controller-manager-6c8c6c9d8f-rjxtl"
Oct 31 19:24:31.867950 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:31.867932    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2f80883e-35e9-46d6-aee0-490b3a0d8e05-cert\") pod \"jobset-controller-manager-6c8c6c9d8f-rjxtl\" (UID: \"2f80883e-35e9-46d6-aee0-490b3a0d8e05\") " pod="openshift-jobset-operator/jobset-controller-manager-6c8c6c9d8f-rjxtl"
Oct 31 19:24:31.883566 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:31.883539    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjsqq\" (UniqueName: \"kubernetes.io/projected/2f80883e-35e9-46d6-aee0-490b3a0d8e05-kube-api-access-rjsqq\") pod \"jobset-controller-manager-6c8c6c9d8f-rjxtl\" (UID: \"2f80883e-35e9-46d6-aee0-490b3a0d8e05\") " pod="openshift-jobset-operator/jobset-controller-manager-6c8c6c9d8f-rjxtl"
Oct 31 19:24:31.984547 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:31.984313    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-6c8c6c9d8f-rjxtl"
Oct 31 19:24:32.277379 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:32.277352    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-6c8c6c9d8f-rjxtl"]
Oct 31 19:24:32.278004 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:24:32.277982    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f80883e_35e9_46d6_aee0_490b3a0d8e05.slice/crio-70b77310ea0d06b7f7771df2fc9ea2c305db3febf7f6141c25f966d4be31caa0 WatchSource:0}: Error finding container 70b77310ea0d06b7f7771df2fc9ea2c305db3febf7f6141c25f966d4be31caa0: Status 404 returned error can't find the container with id 70b77310ea0d06b7f7771df2fc9ea2c305db3febf7f6141c25f966d4be31caa0
Oct 31 19:24:32.279837 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:32.279818    4645 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 19:24:32.848630 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:32.848596    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-vkdkl" event={"ID":"70ec4dc6-89c2-4004-9c4b-42375f9bddb3","Type":"ContainerStarted","Data":"a6ee63acbc6a5c62f3973f1386a44363a48aeeb04f82187943f66fbabec7cf29"}
Oct 31 19:24:32.848804 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:32.848651    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-vkdkl" podUID="70ec4dc6-89c2-4004-9c4b-42375f9bddb3" containerName="authorino" containerID="cri-o://a6ee63acbc6a5c62f3973f1386a44363a48aeeb04f82187943f66fbabec7cf29" gracePeriod=30
Oct 31 19:24:32.849928 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:32.849904    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-6c8c6c9d8f-rjxtl" event={"ID":"2f80883e-35e9-46d6-aee0-490b3a0d8e05","Type":"ContainerStarted","Data":"70b77310ea0d06b7f7771df2fc9ea2c305db3febf7f6141c25f966d4be31caa0"}
Oct 31 19:24:32.869071 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:32.869032    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-vkdkl" podStartSLOduration=2.018601713 podStartE2EDuration="5.869020794s" totalImagesPullingTime="3.850419081s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:24:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:24:32.867986218 +0000 UTC m=+303.441220345" watchObservedRunningTime="2026-10-31 19:24:32.869020794 +0000 UTC m=+303.442254917"
Oct 31 19:24:33.081722 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:33.081702    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-vkdkl"
Oct 31 19:24:33.175877 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:33.175848    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70ec4dc6-89c2-4004-9c4b-42375f9bddb3-kube-api-access-jgdqt\" (UniqueName: \"kubernetes.io/projected/70ec4dc6-89c2-4004-9c4b-42375f9bddb3-kube-api-access-jgdqt\") pod \"70ec4dc6-89c2-4004-9c4b-42375f9bddb3\" (UID: \"70ec4dc6-89c2-4004-9c4b-42375f9bddb3\") "
Oct 31 19:24:33.177897 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:33.177877    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70ec4dc6-89c2-4004-9c4b-42375f9bddb3-kube-api-access-jgdqt" pod "70ec4dc6-89c2-4004-9c4b-42375f9bddb3" (UID: "70ec4dc6-89c2-4004-9c4b-42375f9bddb3"). InnerVolumeSpecName "kube-api-access-jgdqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:24:33.277368 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:33.277345    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgdqt\" (UniqueName: \"kubernetes.io/projected/70ec4dc6-89c2-4004-9c4b-42375f9bddb3-kube-api-access-jgdqt\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:24:33.856933 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:33.856693    4645 generic.go:356] "Generic (PLEG): container finished" podID="70ec4dc6-89c2-4004-9c4b-42375f9bddb3" containerID="a6ee63acbc6a5c62f3973f1386a44363a48aeeb04f82187943f66fbabec7cf29" exitCode=0
Oct 31 19:24:33.856933 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:33.856750    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-vkdkl"
Oct 31 19:24:33.856933 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:33.856775    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-vkdkl" event={"ID":"70ec4dc6-89c2-4004-9c4b-42375f9bddb3","Type":"ContainerDied","Data":"a6ee63acbc6a5c62f3973f1386a44363a48aeeb04f82187943f66fbabec7cf29"}
Oct 31 19:24:33.856933 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:33.856808    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-vkdkl" event={"ID":"70ec4dc6-89c2-4004-9c4b-42375f9bddb3","Type":"ContainerDied","Data":"cb42d7e5dc5a69be8bb409f2d954187a200afd9fd47badeda4030d4d2af9ee18"}
Oct 31 19:24:33.856933 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:33.856827    4645 scope.go:122] "RemoveContainer" containerID="a6ee63acbc6a5c62f3973f1386a44363a48aeeb04f82187943f66fbabec7cf29"
Oct 31 19:24:33.866857 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:33.866835    4645 scope.go:122] "RemoveContainer" containerID="a6ee63acbc6a5c62f3973f1386a44363a48aeeb04f82187943f66fbabec7cf29"
Oct 31 19:24:33.867182 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:24:33.867154    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6ee63acbc6a5c62f3973f1386a44363a48aeeb04f82187943f66fbabec7cf29\": container with ID starting with a6ee63acbc6a5c62f3973f1386a44363a48aeeb04f82187943f66fbabec7cf29 not found: ID does not exist" containerID="a6ee63acbc6a5c62f3973f1386a44363a48aeeb04f82187943f66fbabec7cf29"
Oct 31 19:24:33.867270 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:33.867194    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6ee63acbc6a5c62f3973f1386a44363a48aeeb04f82187943f66fbabec7cf29"} err="failed to get container status \"a6ee63acbc6a5c62f3973f1386a44363a48aeeb04f82187943f66fbabec7cf29\": rpc error: code = NotFound desc = could not find container \"a6ee63acbc6a5c62f3973f1386a44363a48aeeb04f82187943f66fbabec7cf29\": container with ID starting with a6ee63acbc6a5c62f3973f1386a44363a48aeeb04f82187943f66fbabec7cf29 not found: ID does not exist"
Oct 31 19:24:33.872013 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:33.871988    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-vkdkl"]
Oct 31 19:24:33.878153 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:33.878099    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-vkdkl"]
Oct 31 19:24:34.713409 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:34.713373    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:34.860987 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:34.860958    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:35.615130 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:35.615056    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70ec4dc6-89c2-4004-9c4b-42375f9bddb3" path="/var/lib/kubelet/pods/70ec4dc6-89c2-4004-9c4b-42375f9bddb3/volumes"
Oct 31 19:24:35.868431 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:35.868357    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-6c8c6c9d8f-rjxtl" event={"ID":"2f80883e-35e9-46d6-aee0-490b3a0d8e05","Type":"ContainerStarted","Data":"24ef0742a97c7837daa115854ff7d9027293f1836d3cb9dd3f342739d771cbbd"}
Oct 31 19:24:35.868570 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:35.868515    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-6c8c6c9d8f-rjxtl"
Oct 31 19:24:35.908103 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:35.908055    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-6c8c6c9d8f-rjxtl" podStartSLOduration=1.996665038 podStartE2EDuration="4.908042299s" totalImagesPullingTime="2.911377261s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:24:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:24:35.905613396 +0000 UTC m=+306.478847524" watchObservedRunningTime="2026-10-31 19:24:35.908042299 +0000 UTC m=+306.481276478"
Oct 31 19:24:36.242845 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:24:36.242815    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 19:24:38.450660 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:38.450628    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:38.584836 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:38.584806    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:41.425423 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:41.425393    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:41.564908 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:41.564880    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:46.819975 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:46.819945    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:46.876830 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:46.876806    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-6c8c6c9d8f-rjxtl"
Oct 31 19:24:46.923128 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:46.923101    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:47.111369 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:47.111273    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:47.242466 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:47.242436    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:47.357291 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:47.357262    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:47.994744 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:47.994713    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:51.035862 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:51.035823    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:51.150973 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:51.150933    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:51.844019 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:51.843989    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-c9tc7"
Oct 31 19:24:54.824634 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:54.824593    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:24:54.930801 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:24:54.930759    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:25:06.245205 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:25:06.245166    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 19:25:06.295131 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:06.295096    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:25:06.402601 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:06.402565    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:25:08.071801 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:08.071765    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:25:08.181996 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:08.181959    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:25:08.291993 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:08.291958    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:25:08.398984 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:08.398903    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:25:09.181651 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:09.181610    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:25:09.281696 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:09.281668    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:25:11.794311 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:11.794269    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:25:11.916360 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:11.916314    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:25:13.935200 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:13.935162    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:25:14.088618 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:14.088580    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:25:14.746293 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:14.746259    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:25:20.714195 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:20.714152    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:25:20.829055 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:20.829024    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:25:20.952480 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:20.952447    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:25:21.070609 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:21.070521    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:25:21.194396 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:21.194360    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:25:22.103198 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:22.103165    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:25:22.210779 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:22.210748    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:25:22.331393 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:22.331359    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:25:27.314169 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:27.314136    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-mnrh7"]
Oct 31 19:25:27.314595 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:27.314319    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mnrh7" podUID="28c727cb-1a33-4557-a0a0-76377739bddb" containerName="discovery" containerID="cri-o://620b5bad2940bd1d9c3c7bae47583c282672a1b2a6a9232139fa4211e9901f85" gracePeriod=30
Oct 31 19:25:27.549081 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:27.549059    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mnrh7"
Oct 31 19:25:27.598546 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:27.598493    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28c727cb-1a33-4557-a0a0-76377739bddb-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/28c727cb-1a33-4557-a0a0-76377739bddb-istio-kubeconfig\") pod \"28c727cb-1a33-4557-a0a0-76377739bddb\" (UID: \"28c727cb-1a33-4557-a0a0-76377739bddb\") "
Oct 31 19:25:27.598546 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:27.598525    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28c727cb-1a33-4557-a0a0-76377739bddb-kube-api-access-sjxln\" (UniqueName: \"kubernetes.io/projected/28c727cb-1a33-4557-a0a0-76377739bddb-kube-api-access-sjxln\") pod \"28c727cb-1a33-4557-a0a0-76377739bddb\" (UID: \"28c727cb-1a33-4557-a0a0-76377739bddb\") "
Oct 31 19:25:27.598674 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:27.598624    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28c727cb-1a33-4557-a0a0-76377739bddb-cacerts\" (UniqueName: \"kubernetes.io/secret/28c727cb-1a33-4557-a0a0-76377739bddb-cacerts\") pod \"28c727cb-1a33-4557-a0a0-76377739bddb\" (UID: \"28c727cb-1a33-4557-a0a0-76377739bddb\") "
Oct 31 19:25:27.598719 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:27.598669    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28c727cb-1a33-4557-a0a0-76377739bddb-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/28c727cb-1a33-4557-a0a0-76377739bddb-local-certs\") pod \"28c727cb-1a33-4557-a0a0-76377739bddb\" (UID: \"28c727cb-1a33-4557-a0a0-76377739bddb\") "
Oct 31 19:25:27.598719 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:27.598702    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28c727cb-1a33-4557-a0a0-76377739bddb-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/28c727cb-1a33-4557-a0a0-76377739bddb-istio-csr-dns-cert\") pod \"28c727cb-1a33-4557-a0a0-76377739bddb\" (UID: \"28c727cb-1a33-4557-a0a0-76377739bddb\") "
Oct 31 19:25:27.598811 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:27.598731    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28c727cb-1a33-4557-a0a0-76377739bddb-istio-token\" (UniqueName: \"kubernetes.io/projected/28c727cb-1a33-4557-a0a0-76377739bddb-istio-token\") pod \"28c727cb-1a33-4557-a0a0-76377739bddb\" (UID: \"28c727cb-1a33-4557-a0a0-76377739bddb\") "
Oct 31 19:25:27.598811 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:27.598777    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28c727cb-1a33-4557-a0a0-76377739bddb-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/28c727cb-1a33-4557-a0a0-76377739bddb-istio-csr-ca-configmap\") pod \"28c727cb-1a33-4557-a0a0-76377739bddb\" (UID: \"28c727cb-1a33-4557-a0a0-76377739bddb\") "
Oct 31 19:25:27.599219 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:27.599170    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28c727cb-1a33-4557-a0a0-76377739bddb-istio-csr-ca-configmap" pod "28c727cb-1a33-4557-a0a0-76377739bddb" (UID: "28c727cb-1a33-4557-a0a0-76377739bddb"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:25:27.600957 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:27.600905    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28c727cb-1a33-4557-a0a0-76377739bddb-istio-token" pod "28c727cb-1a33-4557-a0a0-76377739bddb" (UID: "28c727cb-1a33-4557-a0a0-76377739bddb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:25:27.601058 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:27.601006    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28c727cb-1a33-4557-a0a0-76377739bddb-local-certs" pod "28c727cb-1a33-4557-a0a0-76377739bddb" (UID: "28c727cb-1a33-4557-a0a0-76377739bddb"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:25:27.601114 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:27.601093    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28c727cb-1a33-4557-a0a0-76377739bddb-istio-kubeconfig" pod "28c727cb-1a33-4557-a0a0-76377739bddb" (UID: "28c727cb-1a33-4557-a0a0-76377739bddb"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:25:27.601150 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:27.601108    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28c727cb-1a33-4557-a0a0-76377739bddb-kube-api-access-sjxln" pod "28c727cb-1a33-4557-a0a0-76377739bddb" (UID: "28c727cb-1a33-4557-a0a0-76377739bddb"). InnerVolumeSpecName "kube-api-access-sjxln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:25:27.601150 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:27.601127    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28c727cb-1a33-4557-a0a0-76377739bddb-cacerts" pod "28c727cb-1a33-4557-a0a0-76377739bddb" (UID: "28c727cb-1a33-4557-a0a0-76377739bddb"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:25:27.601224 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:27.601185    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28c727cb-1a33-4557-a0a0-76377739bddb-istio-csr-dns-cert" pod "28c727cb-1a33-4557-a0a0-76377739bddb" (UID: "28c727cb-1a33-4557-a0a0-76377739bddb"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:25:27.699931 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:27.699910    4645 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/28c727cb-1a33-4557-a0a0-76377739bddb-istio-kubeconfig\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:25:27.699931 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:27.699931    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjxln\" (UniqueName: \"kubernetes.io/projected/28c727cb-1a33-4557-a0a0-76377739bddb-kube-api-access-sjxln\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:25:27.700048 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:27.699940    4645 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/28c727cb-1a33-4557-a0a0-76377739bddb-cacerts\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:25:27.700048 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:27.699950    4645 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/28c727cb-1a33-4557-a0a0-76377739bddb-local-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:25:27.700048 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:27.699958    4645 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/28c727cb-1a33-4557-a0a0-76377739bddb-istio-csr-dns-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:25:27.700048 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:27.699966    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28c727cb-1a33-4557-a0a0-76377739bddb-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:25:27.700048 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:27.699976    4645 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/28c727cb-1a33-4557-a0a0-76377739bddb-istio-csr-ca-configmap\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:25:28.079379 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:28.079348    4645 generic.go:356] "Generic (PLEG): container finished" podID="28c727cb-1a33-4557-a0a0-76377739bddb" containerID="620b5bad2940bd1d9c3c7bae47583c282672a1b2a6a9232139fa4211e9901f85" exitCode=0
Oct 31 19:25:28.079519 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:28.079394    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mnrh7"
Oct 31 19:25:28.079519 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:28.079422    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mnrh7" event={"ID":"28c727cb-1a33-4557-a0a0-76377739bddb","Type":"ContainerDied","Data":"620b5bad2940bd1d9c3c7bae47583c282672a1b2a6a9232139fa4211e9901f85"}
Oct 31 19:25:28.079519 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:28.079477    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-mnrh7" event={"ID":"28c727cb-1a33-4557-a0a0-76377739bddb","Type":"ContainerDied","Data":"5c906284cfa759797ef7cb4df276f5f2d129fd2a9994ad74db00f2fe8d779205"}
Oct 31 19:25:28.079519 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:28.079499    4645 scope.go:122] "RemoveContainer" containerID="620b5bad2940bd1d9c3c7bae47583c282672a1b2a6a9232139fa4211e9901f85"
Oct 31 19:25:28.087627 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:28.087602    4645 scope.go:122] "RemoveContainer" containerID="620b5bad2940bd1d9c3c7bae47583c282672a1b2a6a9232139fa4211e9901f85"
Oct 31 19:25:28.087886 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:25:28.087867    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"620b5bad2940bd1d9c3c7bae47583c282672a1b2a6a9232139fa4211e9901f85\": container with ID starting with 620b5bad2940bd1d9c3c7bae47583c282672a1b2a6a9232139fa4211e9901f85 not found: ID does not exist" containerID="620b5bad2940bd1d9c3c7bae47583c282672a1b2a6a9232139fa4211e9901f85"
Oct 31 19:25:28.087944 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:28.087896    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"620b5bad2940bd1d9c3c7bae47583c282672a1b2a6a9232139fa4211e9901f85"} err="failed to get container status \"620b5bad2940bd1d9c3c7bae47583c282672a1b2a6a9232139fa4211e9901f85\": rpc error: code = NotFound desc = could not find container \"620b5bad2940bd1d9c3c7bae47583c282672a1b2a6a9232139fa4211e9901f85\": container with ID starting with 620b5bad2940bd1d9c3c7bae47583c282672a1b2a6a9232139fa4211e9901f85 not found: ID does not exist"
Oct 31 19:25:28.095687 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:28.095667    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-mnrh7"]
Oct 31 19:25:28.099431 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:28.099408    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-mnrh7"]
Oct 31 19:25:29.614689 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:29.614658    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28c727cb-1a33-4557-a0a0-76377739bddb" path="/var/lib/kubelet/pods/28c727cb-1a33-4557-a0a0-76377739bddb/volumes"
Oct 31 19:25:36.242850 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:25:36.242811    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 19:25:39.342126 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:39.342085    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:25:39.454077 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:39.454043    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:25:40.636740 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:40.636697    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:25:40.740688 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:40.740655    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-g6klw"]
Oct 31 19:25:40.740972 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:40.740960    4645 cpu_manager.go:414] "containerMap: removing container" podUID="70ec4dc6-89c2-4004-9c4b-42375f9bddb3" containerName="authorino"
Oct 31 19:25:40.741023 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:40.740973    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70ec4dc6-89c2-4004-9c4b-42375f9bddb3" containerName="authorino"
Oct 31 19:25:40.741023 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:40.740987    4645 cpu_manager.go:414] "containerMap: removing container" podUID="28c727cb-1a33-4557-a0a0-76377739bddb" containerName="discovery"
Oct 31 19:25:40.741023 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:40.740992    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28c727cb-1a33-4557-a0a0-76377739bddb" containerName="discovery"
Oct 31 19:25:40.741123 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:40.741048    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="70ec4dc6-89c2-4004-9c4b-42375f9bddb3" containerName="authorino"
Oct 31 19:25:40.741123 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:40.741055    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="28c727cb-1a33-4557-a0a0-76377739bddb" containerName="discovery"
Oct 31 19:25:40.745406 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:40.745387    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-g6klw"
Oct 31 19:25:40.747593 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:40.747565    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:25:40.753043 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:40.753025    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 19:25:40.755897 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:40.755882    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 19:25:40.757598 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:40.757578    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 31 19:25:40.757722 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:40.757709    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 31 19:25:40.757785 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:40.757720    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9dsr5\""
Oct 31 19:25:40.758582 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:40.758539    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-g6klw"]
Oct 31 19:25:40.800755 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:40.800723    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c451c5c-f091-446f-944a-f76d0749b6dc-metrics-certs\") pod \"maas-api-6d669c7fcd-g6klw\" (UID: \"1c451c5c-f091-446f-944a-f76d0749b6dc\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-g6klw"
Oct 31 19:25:40.800880 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:40.800771    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c451c5c-f091-446f-944a-f76d0749b6dc-maas-api-tls\") pod \"maas-api-6d669c7fcd-g6klw\" (UID: \"1c451c5c-f091-446f-944a-f76d0749b6dc\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-g6klw"
Oct 31 19:25:40.800924 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:40.800899    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v259r\" (UniqueName: \"kubernetes.io/projected/1c451c5c-f091-446f-944a-f76d0749b6dc-kube-api-access-v259r\") pod \"maas-api-6d669c7fcd-g6klw\" (UID: \"1c451c5c-f091-446f-944a-f76d0749b6dc\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-g6klw"
Oct 31 19:25:40.800959 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:40.800937    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c451c5c-f091-446f-944a-f76d0749b6dc-openshift-service-ca\") pod \"maas-api-6d669c7fcd-g6klw\" (UID: \"1c451c5c-f091-446f-944a-f76d0749b6dc\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-g6klw"
Oct 31 19:25:40.902042 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:40.901978    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v259r\" (UniqueName: \"kubernetes.io/projected/1c451c5c-f091-446f-944a-f76d0749b6dc-kube-api-access-v259r\") pod \"maas-api-6d669c7fcd-g6klw\" (UID: \"1c451c5c-f091-446f-944a-f76d0749b6dc\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-g6klw"
Oct 31 19:25:40.902042 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:40.902013    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c451c5c-f091-446f-944a-f76d0749b6dc-openshift-service-ca\") pod \"maas-api-6d669c7fcd-g6klw\" (UID: \"1c451c5c-f091-446f-944a-f76d0749b6dc\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-g6klw"
Oct 31 19:25:40.902042 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:40.902032    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c451c5c-f091-446f-944a-f76d0749b6dc-metrics-certs\") pod \"maas-api-6d669c7fcd-g6klw\" (UID: \"1c451c5c-f091-446f-944a-f76d0749b6dc\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-g6klw"
Oct 31 19:25:40.902285 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:40.902057    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c451c5c-f091-446f-944a-f76d0749b6dc-maas-api-tls\") pod \"maas-api-6d669c7fcd-g6klw\" (UID: \"1c451c5c-f091-446f-944a-f76d0749b6dc\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-g6klw"
Oct 31 19:25:40.902652 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:40.902629    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c451c5c-f091-446f-944a-f76d0749b6dc-openshift-service-ca\") pod \"maas-api-6d669c7fcd-g6klw\" (UID: \"1c451c5c-f091-446f-944a-f76d0749b6dc\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-g6klw"
Oct 31 19:25:40.904596 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:40.904568    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c451c5c-f091-446f-944a-f76d0749b6dc-metrics-certs\") pod \"maas-api-6d669c7fcd-g6klw\" (UID: \"1c451c5c-f091-446f-944a-f76d0749b6dc\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-g6klw"
Oct 31 19:25:40.904679 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:40.904607    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c451c5c-f091-446f-944a-f76d0749b6dc-maas-api-tls\") pod \"maas-api-6d669c7fcd-g6klw\" (UID: \"1c451c5c-f091-446f-944a-f76d0749b6dc\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-g6klw"
Oct 31 19:25:40.915579 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:40.915554    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v259r\" (UniqueName: \"kubernetes.io/projected/1c451c5c-f091-446f-944a-f76d0749b6dc-kube-api-access-v259r\") pod \"maas-api-6d669c7fcd-g6klw\" (UID: \"1c451c5c-f091-446f-944a-f76d0749b6dc\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-g6klw"
Oct 31 19:25:41.058223 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:41.058185    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-g6klw"
Oct 31 19:25:41.330955 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:41.330906    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-g6klw"]
Oct 31 19:25:41.333624 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:25:41.333593    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c451c5c_f091_446f_944a_f76d0749b6dc.slice/crio-435d423050bb4e92ba3e487db2755d21a25ddcb733082461b7c8d73e8bdb4ab2 WatchSource:0}: Error finding container 435d423050bb4e92ba3e487db2755d21a25ddcb733082461b7c8d73e8bdb4ab2: Status 404 returned error can't find the container with id 435d423050bb4e92ba3e487db2755d21a25ddcb733082461b7c8d73e8bdb4ab2
Oct 31 19:25:42.134364 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:42.134310    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-g6klw" event={"ID":"1c451c5c-f091-446f-944a-f76d0749b6dc","Type":"ContainerStarted","Data":"435d423050bb4e92ba3e487db2755d21a25ddcb733082461b7c8d73e8bdb4ab2"}
Oct 31 19:25:44.144690 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:44.144595    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-g6klw" event={"ID":"1c451c5c-f091-446f-944a-f76d0749b6dc","Type":"ContainerStarted","Data":"d732e4a88b04f0d270bd83f217e47f394b8183693402445b75810dac754d6a1f"}
Oct 31 19:25:44.145115 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:44.144729    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-g6klw"
Oct 31 19:25:44.165893 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:44.165837    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-g6klw" podStartSLOduration=1.650113226 podStartE2EDuration="4.165821298s" totalImagesPullingTime="2.515708072s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:25:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:25:44.164910376 +0000 UTC m=+374.738144505" watchObservedRunningTime="2026-10-31 19:25:44.165821298 +0000 UTC m=+374.739055428"
Oct 31 19:25:50.153971 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:50.153941    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-g6klw"
Oct 31 19:25:50.578317 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:50.578288    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:25:50.685113 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:50.685088    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:25:50.869488 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:50.869431    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6dd76875b5-q5cfx"]
Oct 31 19:25:50.873911 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:50.873892    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6dd76875b5-q5cfx"
Oct 31 19:25:50.885945 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:50.885927    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Oct 31 19:25:50.885945 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:50.885940    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-42cfq\""
Oct 31 19:25:50.890076 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:50.890059    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 31 19:25:50.890314 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:50.890290    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 31 19:25:50.891612 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:50.891592    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6dd76875b5-q5cfx"]
Oct 31 19:25:50.919040 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:50.919016    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6dd76875b5-q5cfx"]
Oct 31 19:25:50.919177 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:25:50.919160    4645 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-j58w7 tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6dd76875b5-q5cfx" podUID="e666ed5b-5362-4d75-a71b-c1eaee3a23aa"
Oct 31 19:25:50.954462 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:50.954439    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-5bfkr"]
Oct 31 19:25:50.957456 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:50.957441    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-5bfkr"
Oct 31 19:25:50.969571 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:50.969549    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-5bfkr"]
Oct 31 19:25:50.979486 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:50.979463    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e666ed5b-5362-4d75-a71b-c1eaee3a23aa-tls-cert\") pod \"authorino-6dd76875b5-q5cfx\" (UID: \"e666ed5b-5362-4d75-a71b-c1eaee3a23aa\") " pod="kuadrant-system/authorino-6dd76875b5-q5cfx"
Oct 31 19:25:50.979573 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:50.979503    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j58w7\" (UniqueName: \"kubernetes.io/projected/e666ed5b-5362-4d75-a71b-c1eaee3a23aa-kube-api-access-j58w7\") pod \"authorino-6dd76875b5-q5cfx\" (UID: \"e666ed5b-5362-4d75-a71b-c1eaee3a23aa\") " pod="kuadrant-system/authorino-6dd76875b5-q5cfx"
Oct 31 19:25:51.075191 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:51.075135    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7b784f56d7-phm8m"]
Oct 31 19:25:51.078295 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:51.078279    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7b784f56d7-phm8m"
Oct 31 19:25:51.079798 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:51.079775    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxbvg\" (UniqueName: \"kubernetes.io/projected/ca5aab9a-7f55-4ad9-b120-72470ea75774-kube-api-access-xxbvg\") pod \"authorino-6889cbd8c4-5bfkr\" (UID: \"ca5aab9a-7f55-4ad9-b120-72470ea75774\") " pod="kuadrant-system/authorino-6889cbd8c4-5bfkr"
Oct 31 19:25:51.079914 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:51.079818    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e666ed5b-5362-4d75-a71b-c1eaee3a23aa-tls-cert\") pod \"authorino-6dd76875b5-q5cfx\" (UID: \"e666ed5b-5362-4d75-a71b-c1eaee3a23aa\") " pod="kuadrant-system/authorino-6dd76875b5-q5cfx"
Oct 31 19:25:51.079914 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:51.079848    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j58w7\" (UniqueName: \"kubernetes.io/projected/e666ed5b-5362-4d75-a71b-c1eaee3a23aa-kube-api-access-j58w7\") pod \"authorino-6dd76875b5-q5cfx\" (UID: \"e666ed5b-5362-4d75-a71b-c1eaee3a23aa\") " pod="kuadrant-system/authorino-6dd76875b5-q5cfx"
Oct 31 19:25:51.079991 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:51.079926    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ca5aab9a-7f55-4ad9-b120-72470ea75774-tls-cert\") pod \"authorino-6889cbd8c4-5bfkr\" (UID: \"ca5aab9a-7f55-4ad9-b120-72470ea75774\") " pod="kuadrant-system/authorino-6889cbd8c4-5bfkr"
Oct 31 19:25:51.082357 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:51.082322    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e666ed5b-5362-4d75-a71b-c1eaee3a23aa-tls-cert\") pod \"authorino-6dd76875b5-q5cfx\" (UID: \"e666ed5b-5362-4d75-a71b-c1eaee3a23aa\") " pod="kuadrant-system/authorino-6dd76875b5-q5cfx"
Oct 31 19:25:51.091451 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:51.090597    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b784f56d7-phm8m"]
Oct 31 19:25:51.091732 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:51.091715    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j58w7\" (UniqueName: \"kubernetes.io/projected/e666ed5b-5362-4d75-a71b-c1eaee3a23aa-kube-api-access-j58w7\") pod \"authorino-6dd76875b5-q5cfx\" (UID: \"e666ed5b-5362-4d75-a71b-c1eaee3a23aa\") " pod="kuadrant-system/authorino-6dd76875b5-q5cfx"
Oct 31 19:25:51.170203 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:51.170180    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6dd76875b5-q5cfx"
Oct 31 19:25:51.175974 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:51.175956    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6dd76875b5-q5cfx"
Oct 31 19:25:51.180230 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:51.180212    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/835bc094-57b0-4d4e-adf9-df4f189ca317-openshift-service-ca\") pod \"maas-api-7b784f56d7-phm8m\" (UID: \"835bc094-57b0-4d4e-adf9-df4f189ca317\") " pod="odh-ai-gateway-infra/maas-api-7b784f56d7-phm8m"
Oct 31 19:25:51.180287 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:51.180242    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/835bc094-57b0-4d4e-adf9-df4f189ca317-metrics-certs\") pod \"maas-api-7b784f56d7-phm8m\" (UID: \"835bc094-57b0-4d4e-adf9-df4f189ca317\") " pod="odh-ai-gateway-infra/maas-api-7b784f56d7-phm8m"
Oct 31 19:25:51.180287 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:51.180265    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ca5aab9a-7f55-4ad9-b120-72470ea75774-tls-cert\") pod \"authorino-6889cbd8c4-5bfkr\" (UID: \"ca5aab9a-7f55-4ad9-b120-72470ea75774\") " pod="kuadrant-system/authorino-6889cbd8c4-5bfkr"
Oct 31 19:25:51.180406 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:51.180389    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxbvg\" (UniqueName: \"kubernetes.io/projected/ca5aab9a-7f55-4ad9-b120-72470ea75774-kube-api-access-xxbvg\") pod \"authorino-6889cbd8c4-5bfkr\" (UID: \"ca5aab9a-7f55-4ad9-b120-72470ea75774\") " pod="kuadrant-system/authorino-6889cbd8c4-5bfkr"
Oct 31 19:25:51.180449 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:51.180436    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/835bc094-57b0-4d4e-adf9-df4f189ca317-maas-api-tls\") pod \"maas-api-7b784f56d7-phm8m\" (UID: \"835bc094-57b0-4d4e-adf9-df4f189ca317\") " pod="odh-ai-gateway-infra/maas-api-7b784f56d7-phm8m"
Oct 31 19:25:51.180483 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:51.180458    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncrz8\" (UniqueName: \"kubernetes.io/projected/835bc094-57b0-4d4e-adf9-df4f189ca317-kube-api-access-ncrz8\") pod \"maas-api-7b784f56d7-phm8m\" (UID: \"835bc094-57b0-4d4e-adf9-df4f189ca317\") " pod="odh-ai-gateway-infra/maas-api-7b784f56d7-phm8m"
Oct 31 19:25:51.182395 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:51.182377    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ca5aab9a-7f55-4ad9-b120-72470ea75774-tls-cert\") pod \"authorino-6889cbd8c4-5bfkr\" (UID: \"ca5aab9a-7f55-4ad9-b120-72470ea75774\") " pod="kuadrant-system/authorino-6889cbd8c4-5bfkr"
Oct 31 19:25:51.193280 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:51.193261    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxbvg\" (UniqueName: \"kubernetes.io/projected/ca5aab9a-7f55-4ad9-b120-72470ea75774-kube-api-access-xxbvg\") pod \"authorino-6889cbd8c4-5bfkr\" (UID: \"ca5aab9a-7f55-4ad9-b120-72470ea75774\") " pod="kuadrant-system/authorino-6889cbd8c4-5bfkr"
Oct 31 19:25:51.251993 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:51.251890    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-5bfkr"]
Oct 31 19:25:51.252093 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:51.252052    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-5bfkr"
Oct 31 19:25:51.281073 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:51.281050    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e666ed5b-5362-4d75-a71b-c1eaee3a23aa-tls-cert\" (UniqueName: \"kubernetes.io/secret/e666ed5b-5362-4d75-a71b-c1eaee3a23aa-tls-cert\") pod \"e666ed5b-5362-4d75-a71b-c1eaee3a23aa\" (UID: \"e666ed5b-5362-4d75-a71b-c1eaee3a23aa\") "
Oct 31 19:25:51.281164 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:51.281093    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e666ed5b-5362-4d75-a71b-c1eaee3a23aa-kube-api-access-j58w7\" (UniqueName: \"kubernetes.io/projected/e666ed5b-5362-4d75-a71b-c1eaee3a23aa-kube-api-access-j58w7\") pod \"e666ed5b-5362-4d75-a71b-c1eaee3a23aa\" (UID: \"e666ed5b-5362-4d75-a71b-c1eaee3a23aa\") "
Oct 31 19:25:51.281232 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:51.281219    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/835bc094-57b0-4d4e-adf9-df4f189ca317-maas-api-tls\") pod \"maas-api-7b784f56d7-phm8m\" (UID: \"835bc094-57b0-4d4e-adf9-df4f189ca317\") " pod="odh-ai-gateway-infra/maas-api-7b784f56d7-phm8m"
Oct 31 19:25:51.281294 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:51.281243    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncrz8\" (UniqueName: \"kubernetes.io/projected/835bc094-57b0-4d4e-adf9-df4f189ca317-kube-api-access-ncrz8\") pod \"maas-api-7b784f56d7-phm8m\" (UID: \"835bc094-57b0-4d4e-adf9-df4f189ca317\") " pod="odh-ai-gateway-infra/maas-api-7b784f56d7-phm8m"
Oct 31 19:25:51.281294 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:51.281279    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/835bc094-57b0-4d4e-adf9-df4f189ca317-openshift-service-ca\") pod \"maas-api-7b784f56d7-phm8m\" (UID: \"835bc094-57b0-4d4e-adf9-df4f189ca317\") " pod="odh-ai-gateway-infra/maas-api-7b784f56d7-phm8m"
Oct 31 19:25:51.281426 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:51.281310    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/835bc094-57b0-4d4e-adf9-df4f189ca317-metrics-certs\") pod \"maas-api-7b784f56d7-phm8m\" (UID: \"835bc094-57b0-4d4e-adf9-df4f189ca317\") " pod="odh-ai-gateway-infra/maas-api-7b784f56d7-phm8m"
Oct 31 19:25:51.282170 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:51.282137    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/835bc094-57b0-4d4e-adf9-df4f189ca317-openshift-service-ca\") pod \"maas-api-7b784f56d7-phm8m\" (UID: \"835bc094-57b0-4d4e-adf9-df4f189ca317\") " pod="odh-ai-gateway-infra/maas-api-7b784f56d7-phm8m"
Oct 31 19:25:51.283410 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:51.283381    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e666ed5b-5362-4d75-a71b-c1eaee3a23aa-tls-cert" pod "e666ed5b-5362-4d75-a71b-c1eaee3a23aa" (UID: "e666ed5b-5362-4d75-a71b-c1eaee3a23aa"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:25:51.283682 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:51.283656    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e666ed5b-5362-4d75-a71b-c1eaee3a23aa-kube-api-access-j58w7" pod "e666ed5b-5362-4d75-a71b-c1eaee3a23aa" (UID: "e666ed5b-5362-4d75-a71b-c1eaee3a23aa"). InnerVolumeSpecName "kube-api-access-j58w7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:25:51.283779 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:51.283715    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/835bc094-57b0-4d4e-adf9-df4f189ca317-metrics-certs\") pod \"maas-api-7b784f56d7-phm8m\" (UID: \"835bc094-57b0-4d4e-adf9-df4f189ca317\") " pod="odh-ai-gateway-infra/maas-api-7b784f56d7-phm8m"
Oct 31 19:25:51.283943 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:51.283927    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/835bc094-57b0-4d4e-adf9-df4f189ca317-maas-api-tls\") pod \"maas-api-7b784f56d7-phm8m\" (UID: \"835bc094-57b0-4d4e-adf9-df4f189ca317\") " pod="odh-ai-gateway-infra/maas-api-7b784f56d7-phm8m"
Oct 31 19:25:51.286424 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:51.286401    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7dd6b9667f-kwrcm"]
Oct 31 19:25:51.291256 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:51.291237    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7dd6b9667f-kwrcm"
Oct 31 19:25:51.293633 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:51.293612    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncrz8\" (UniqueName: \"kubernetes.io/projected/835bc094-57b0-4d4e-adf9-df4f189ca317-kube-api-access-ncrz8\") pod \"maas-api-7b784f56d7-phm8m\" (UID: \"835bc094-57b0-4d4e-adf9-df4f189ca317\") " pod="odh-ai-gateway-infra/maas-api-7b784f56d7-phm8m"
Oct 31 19:25:51.299154 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:51.299131    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7dd6b9667f-kwrcm"]
Oct 31 19:25:51.381972 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:51.381943    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/83610c38-d76f-4ffa-8bcd-2c8230b4988a-tls-cert\") pod \"authorino-7dd6b9667f-kwrcm\" (UID: \"83610c38-d76f-4ffa-8bcd-2c8230b4988a\") " pod="kuadrant-system/authorino-7dd6b9667f-kwrcm"
Oct 31 19:25:51.382100 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:51.381983    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrmzh\" (UniqueName: \"kubernetes.io/projected/83610c38-d76f-4ffa-8bcd-2c8230b4988a-kube-api-access-xrmzh\") pod \"authorino-7dd6b9667f-kwrcm\" (UID: \"83610c38-d76f-4ffa-8bcd-2c8230b4988a\") " pod="kuadrant-system/authorino-7dd6b9667f-kwrcm"
Oct 31 19:25:51.382100 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:51.382054    4645 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e666ed5b-5362-4d75-a71b-c1eaee3a23aa-tls-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:25:51.382100 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:51.382066    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j58w7\" (UniqueName: \"kubernetes.io/projected/e666ed5b-5362-4d75-a71b-c1eaee3a23aa-kube-api-access-j58w7\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:25:51.411502 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:51.411473    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7b784f56d7-phm8m"
Oct 31 19:25:51.483065 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:51.483036    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/83610c38-d76f-4ffa-8bcd-2c8230b4988a-tls-cert\") pod \"authorino-7dd6b9667f-kwrcm\" (UID: \"83610c38-d76f-4ffa-8bcd-2c8230b4988a\") " pod="kuadrant-system/authorino-7dd6b9667f-kwrcm"
Oct 31 19:25:51.483195 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:51.483091    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrmzh\" (UniqueName: \"kubernetes.io/projected/83610c38-d76f-4ffa-8bcd-2c8230b4988a-kube-api-access-xrmzh\") pod \"authorino-7dd6b9667f-kwrcm\" (UID: \"83610c38-d76f-4ffa-8bcd-2c8230b4988a\") " pod="kuadrant-system/authorino-7dd6b9667f-kwrcm"
Oct 31 19:25:51.485800 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:51.485773    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/83610c38-d76f-4ffa-8bcd-2c8230b4988a-tls-cert\") pod \"authorino-7dd6b9667f-kwrcm\" (UID: \"83610c38-d76f-4ffa-8bcd-2c8230b4988a\") " pod="kuadrant-system/authorino-7dd6b9667f-kwrcm"
Oct 31 19:25:51.495116 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:51.495088    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrmzh\" (UniqueName: \"kubernetes.io/projected/83610c38-d76f-4ffa-8bcd-2c8230b4988a-kube-api-access-xrmzh\") pod \"authorino-7dd6b9667f-kwrcm\" (UID: \"83610c38-d76f-4ffa-8bcd-2c8230b4988a\") " pod="kuadrant-system/authorino-7dd6b9667f-kwrcm"
Oct 31 19:25:51.523244 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:51.523220    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-5bfkr"]
Oct 31 19:25:51.526192 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:25:51.526167    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podca5aab9a_7f55_4ad9_b120_72470ea75774.slice/crio-1cd4c3c121902e164f2f7a734a081b1fcd79535a949e391cda4d52d1af7dd3a6 WatchSource:0}: Error finding container 1cd4c3c121902e164f2f7a734a081b1fcd79535a949e391cda4d52d1af7dd3a6: Status 404 returned error can't find the container with id 1cd4c3c121902e164f2f7a734a081b1fcd79535a949e391cda4d52d1af7dd3a6
Oct 31 19:25:51.611919 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:51.611897    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7dd6b9667f-kwrcm"
Oct 31 19:25:51.696775 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:51.696752    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b784f56d7-phm8m"]
Oct 31 19:25:51.698772 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:25:51.698744    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod835bc094_57b0_4d4e_adf9_df4f189ca317.slice/crio-427b9a575c78470484fc503dd60547533c2b7349a4db61702fcda5f72e2e1ca0 WatchSource:0}: Error finding container 427b9a575c78470484fc503dd60547533c2b7349a4db61702fcda5f72e2e1ca0: Status 404 returned error can't find the container with id 427b9a575c78470484fc503dd60547533c2b7349a4db61702fcda5f72e2e1ca0
Oct 31 19:25:51.885008 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:51.884984    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7dd6b9667f-kwrcm"]
Oct 31 19:25:51.887499 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:25:51.887473    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod83610c38_d76f_4ffa_8bcd_2c8230b4988a.slice/crio-e7bc2849a4313f3035cce380c17e54208ba3a23e0ddf3d2bebb0862caf1a2cc6 WatchSource:0}: Error finding container e7bc2849a4313f3035cce380c17e54208ba3a23e0ddf3d2bebb0862caf1a2cc6: Status 404 returned error can't find the container with id e7bc2849a4313f3035cce380c17e54208ba3a23e0ddf3d2bebb0862caf1a2cc6
Oct 31 19:25:52.179883 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:52.179853    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-5bfkr" event={"ID":"ca5aab9a-7f55-4ad9-b120-72470ea75774","Type":"ContainerStarted","Data":"1cd4c3c121902e164f2f7a734a081b1fcd79535a949e391cda4d52d1af7dd3a6"}
Oct 31 19:25:52.181825 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:52.181801    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7b784f56d7-phm8m" event={"ID":"835bc094-57b0-4d4e-adf9-df4f189ca317","Type":"ContainerStarted","Data":"6f677490caaaf6fd588a1cbaad6bfe170c2bf86eb22db68931ab850b874a5b13"}
Oct 31 19:25:52.181963 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:52.181849    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7b784f56d7-phm8m" event={"ID":"835bc094-57b0-4d4e-adf9-df4f189ca317","Type":"ContainerStarted","Data":"427b9a575c78470484fc503dd60547533c2b7349a4db61702fcda5f72e2e1ca0"}
Oct 31 19:25:52.181963 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:52.181921    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7b784f56d7-phm8m"
Oct 31 19:25:52.183190 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:52.183166    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7dd6b9667f-kwrcm" event={"ID":"83610c38-d76f-4ffa-8bcd-2c8230b4988a","Type":"ContainerStarted","Data":"e7bc2849a4313f3035cce380c17e54208ba3a23e0ddf3d2bebb0862caf1a2cc6"}
Oct 31 19:25:52.183313 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:52.183198    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6dd76875b5-q5cfx"
Oct 31 19:25:52.201408 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:52.201364    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7b784f56d7-phm8m" podStartSLOduration=0.810759336 podStartE2EDuration="1.201352814s" totalImagesPullingTime="390.593478ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:25:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:25:52.200566407 +0000 UTC m=+382.773800536" watchObservedRunningTime="2026-10-31 19:25:52.201352814 +0000 UTC m=+382.774586937"
Oct 31 19:25:52.226496 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:52.226469    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6dd76875b5-q5cfx"]
Oct 31 19:25:52.228649 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:52.228625    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6dd76875b5-q5cfx"]
Oct 31 19:25:53.194006 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:53.193970    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-5bfkr" event={"ID":"ca5aab9a-7f55-4ad9-b120-72470ea75774","Type":"ContainerStarted","Data":"4606d162bedd853b794a907692b90010cf16973e668bfb9f21e65c6573893522"}
Oct 31 19:25:53.194536 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:53.194063    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6889cbd8c4-5bfkr" podUID="ca5aab9a-7f55-4ad9-b120-72470ea75774" containerName="authorino" containerID="cri-o://4606d162bedd853b794a907692b90010cf16973e668bfb9f21e65c6573893522" gracePeriod=30
Oct 31 19:25:53.196071 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:53.196000    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7dd6b9667f-kwrcm" event={"ID":"83610c38-d76f-4ffa-8bcd-2c8230b4988a","Type":"ContainerStarted","Data":"cf019030856932613fdd6f65ce414ef1fc5b0129201b311f40d231127393a668"}
Oct 31 19:25:53.210915 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:53.210865    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6889cbd8c4-5bfkr" podStartSLOduration=2.218801555 podStartE2EDuration="3.210848406s" totalImagesPullingTime="992.046851ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:25:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:25:53.209793079 +0000 UTC m=+383.783027220" watchObservedRunningTime="2026-10-31 19:25:53.210848406 +0000 UTC m=+383.784082535"
Oct 31 19:25:53.427637 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:53.427611    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-5bfkr"
Oct 31 19:25:53.444528 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:53.444428    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-7dd6b9667f-kwrcm" podStartSLOduration=1.809653736 podStartE2EDuration="2.444415002s" totalImagesPullingTime="634.761266ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:25:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:25:53.22580883 +0000 UTC m=+383.799042964" watchObservedRunningTime="2026-10-31 19:25:53.444415002 +0000 UTC m=+384.017649138"
Oct 31 19:25:53.504543 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:53.504517    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca5aab9a-7f55-4ad9-b120-72470ea75774-tls-cert\" (UniqueName: \"kubernetes.io/secret/ca5aab9a-7f55-4ad9-b120-72470ea75774-tls-cert\") pod \"ca5aab9a-7f55-4ad9-b120-72470ea75774\" (UID: \"ca5aab9a-7f55-4ad9-b120-72470ea75774\") "
Oct 31 19:25:53.504628 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:53.504557    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca5aab9a-7f55-4ad9-b120-72470ea75774-kube-api-access-xxbvg\" (UniqueName: \"kubernetes.io/projected/ca5aab9a-7f55-4ad9-b120-72470ea75774-kube-api-access-xxbvg\") pod \"ca5aab9a-7f55-4ad9-b120-72470ea75774\" (UID: \"ca5aab9a-7f55-4ad9-b120-72470ea75774\") "
Oct 31 19:25:53.506537 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:53.506499    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca5aab9a-7f55-4ad9-b120-72470ea75774-kube-api-access-xxbvg" pod "ca5aab9a-7f55-4ad9-b120-72470ea75774" (UID: "ca5aab9a-7f55-4ad9-b120-72470ea75774"). InnerVolumeSpecName "kube-api-access-xxbvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:25:53.514048 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:53.514020    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca5aab9a-7f55-4ad9-b120-72470ea75774-tls-cert" pod "ca5aab9a-7f55-4ad9-b120-72470ea75774" (UID: "ca5aab9a-7f55-4ad9-b120-72470ea75774"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:25:53.605665 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:53.605642    4645 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ca5aab9a-7f55-4ad9-b120-72470ea75774-tls-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:25:53.605665 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:53.605663    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxbvg\" (UniqueName: \"kubernetes.io/projected/ca5aab9a-7f55-4ad9-b120-72470ea75774-kube-api-access-xxbvg\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:25:53.616906 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:53.616866    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e666ed5b-5362-4d75-a71b-c1eaee3a23aa" path="/var/lib/kubelet/pods/e666ed5b-5362-4d75-a71b-c1eaee3a23aa/volumes"
Oct 31 19:25:54.201603 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:54.201569    4645 generic.go:356] "Generic (PLEG): container finished" podID="ca5aab9a-7f55-4ad9-b120-72470ea75774" containerID="4606d162bedd853b794a907692b90010cf16973e668bfb9f21e65c6573893522" exitCode=0
Oct 31 19:25:54.202019 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:54.201620    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-5bfkr"
Oct 31 19:25:54.202019 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:54.201659    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-5bfkr" event={"ID":"ca5aab9a-7f55-4ad9-b120-72470ea75774","Type":"ContainerDied","Data":"4606d162bedd853b794a907692b90010cf16973e668bfb9f21e65c6573893522"}
Oct 31 19:25:54.202019 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:54.201714    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-5bfkr" event={"ID":"ca5aab9a-7f55-4ad9-b120-72470ea75774","Type":"ContainerDied","Data":"1cd4c3c121902e164f2f7a734a081b1fcd79535a949e391cda4d52d1af7dd3a6"}
Oct 31 19:25:54.202019 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:54.201744    4645 scope.go:122] "RemoveContainer" containerID="4606d162bedd853b794a907692b90010cf16973e668bfb9f21e65c6573893522"
Oct 31 19:25:54.210807 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:54.210788    4645 scope.go:122] "RemoveContainer" containerID="4606d162bedd853b794a907692b90010cf16973e668bfb9f21e65c6573893522"
Oct 31 19:25:54.211114 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:25:54.211095    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4606d162bedd853b794a907692b90010cf16973e668bfb9f21e65c6573893522\": container with ID starting with 4606d162bedd853b794a907692b90010cf16973e668bfb9f21e65c6573893522 not found: ID does not exist" containerID="4606d162bedd853b794a907692b90010cf16973e668bfb9f21e65c6573893522"
Oct 31 19:25:54.211163 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:54.211122    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4606d162bedd853b794a907692b90010cf16973e668bfb9f21e65c6573893522"} err="failed to get container status \"4606d162bedd853b794a907692b90010cf16973e668bfb9f21e65c6573893522\": rpc error: code = NotFound desc = could not find container \"4606d162bedd853b794a907692b90010cf16973e668bfb9f21e65c6573893522\": container with ID starting with 4606d162bedd853b794a907692b90010cf16973e668bfb9f21e65c6573893522 not found: ID does not exist"
Oct 31 19:25:54.218645 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:54.218620    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-5bfkr"]
Oct 31 19:25:54.222888 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:54.222864    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-5bfkr"]
Oct 31 19:25:55.614810 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:55.614778    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca5aab9a-7f55-4ad9-b120-72470ea75774" path="/var/lib/kubelet/pods/ca5aab9a-7f55-4ad9-b120-72470ea75774/volumes"
Oct 31 19:25:58.201580 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:58.201542    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7b784f56d7-phm8m"
Oct 31 19:25:58.253244 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:58.253211    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-g6klw"]
Oct 31 19:25:58.253467 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:58.253437    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-g6klw" podUID="1c451c5c-f091-446f-944a-f76d0749b6dc" containerName="maas-api" containerID="cri-o://d732e4a88b04f0d270bd83f217e47f394b8183693402445b75810dac754d6a1f" gracePeriod=30
Oct 31 19:25:58.486224 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:58.486203    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-g6klw"
Oct 31 19:25:58.641549 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:58.641515    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c451c5c-f091-446f-944a-f76d0749b6dc-kube-api-access-v259r\" (UniqueName: \"kubernetes.io/projected/1c451c5c-f091-446f-944a-f76d0749b6dc-kube-api-access-v259r\") pod \"1c451c5c-f091-446f-944a-f76d0749b6dc\" (UID: \"1c451c5c-f091-446f-944a-f76d0749b6dc\") "
Oct 31 19:25:58.641674 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:58.641561    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c451c5c-f091-446f-944a-f76d0749b6dc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c451c5c-f091-446f-944a-f76d0749b6dc-openshift-service-ca\") pod \"1c451c5c-f091-446f-944a-f76d0749b6dc\" (UID: \"1c451c5c-f091-446f-944a-f76d0749b6dc\") "
Oct 31 19:25:58.641674 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:58.641595    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c451c5c-f091-446f-944a-f76d0749b6dc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c451c5c-f091-446f-944a-f76d0749b6dc-metrics-certs\") pod \"1c451c5c-f091-446f-944a-f76d0749b6dc\" (UID: \"1c451c5c-f091-446f-944a-f76d0749b6dc\") "
Oct 31 19:25:58.641751 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:58.641698    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c451c5c-f091-446f-944a-f76d0749b6dc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c451c5c-f091-446f-944a-f76d0749b6dc-maas-api-tls\") pod \"1c451c5c-f091-446f-944a-f76d0749b6dc\" (UID: \"1c451c5c-f091-446f-944a-f76d0749b6dc\") "
Oct 31 19:25:58.642005 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:58.641975    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c451c5c-f091-446f-944a-f76d0749b6dc-openshift-service-ca" pod "1c451c5c-f091-446f-944a-f76d0749b6dc" (UID: "1c451c5c-f091-446f-944a-f76d0749b6dc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:25:58.643766 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:58.643736    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c451c5c-f091-446f-944a-f76d0749b6dc-metrics-certs" pod "1c451c5c-f091-446f-944a-f76d0749b6dc" (UID: "1c451c5c-f091-446f-944a-f76d0749b6dc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:25:58.643874 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:58.643793    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c451c5c-f091-446f-944a-f76d0749b6dc-maas-api-tls" pod "1c451c5c-f091-446f-944a-f76d0749b6dc" (UID: "1c451c5c-f091-446f-944a-f76d0749b6dc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:25:58.643874 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:58.643864    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c451c5c-f091-446f-944a-f76d0749b6dc-kube-api-access-v259r" pod "1c451c5c-f091-446f-944a-f76d0749b6dc" (UID: "1c451c5c-f091-446f-944a-f76d0749b6dc"). InnerVolumeSpecName "kube-api-access-v259r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:25:58.742369 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:58.742306    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v259r\" (UniqueName: \"kubernetes.io/projected/1c451c5c-f091-446f-944a-f76d0749b6dc-kube-api-access-v259r\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:25:58.742369 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:58.742341    4645 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c451c5c-f091-446f-944a-f76d0749b6dc-openshift-service-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:25:58.742369 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:58.742359    4645 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c451c5c-f091-446f-944a-f76d0749b6dc-metrics-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:25:58.742518 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:58.742372    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1c451c5c-f091-446f-944a-f76d0749b6dc-maas-api-tls\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:25:59.223529 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:59.223500    4645 generic.go:356] "Generic (PLEG): container finished" podID="1c451c5c-f091-446f-944a-f76d0749b6dc" containerID="d732e4a88b04f0d270bd83f217e47f394b8183693402445b75810dac754d6a1f" exitCode=0
Oct 31 19:25:59.223885 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:59.223554    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-g6klw"
Oct 31 19:25:59.223885 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:59.223572    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-g6klw" event={"ID":"1c451c5c-f091-446f-944a-f76d0749b6dc","Type":"ContainerDied","Data":"d732e4a88b04f0d270bd83f217e47f394b8183693402445b75810dac754d6a1f"}
Oct 31 19:25:59.223885 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:59.223625    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-g6klw" event={"ID":"1c451c5c-f091-446f-944a-f76d0749b6dc","Type":"ContainerDied","Data":"435d423050bb4e92ba3e487db2755d21a25ddcb733082461b7c8d73e8bdb4ab2"}
Oct 31 19:25:59.223885 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:59.223656    4645 scope.go:122] "RemoveContainer" containerID="d732e4a88b04f0d270bd83f217e47f394b8183693402445b75810dac754d6a1f"
Oct 31 19:25:59.232811 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:59.232789    4645 scope.go:122] "RemoveContainer" containerID="d732e4a88b04f0d270bd83f217e47f394b8183693402445b75810dac754d6a1f"
Oct 31 19:25:59.233084 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:25:59.233063    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d732e4a88b04f0d270bd83f217e47f394b8183693402445b75810dac754d6a1f\": container with ID starting with d732e4a88b04f0d270bd83f217e47f394b8183693402445b75810dac754d6a1f not found: ID does not exist" containerID="d732e4a88b04f0d270bd83f217e47f394b8183693402445b75810dac754d6a1f"
Oct 31 19:25:59.233149 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:59.233090    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d732e4a88b04f0d270bd83f217e47f394b8183693402445b75810dac754d6a1f"} err="failed to get container status \"d732e4a88b04f0d270bd83f217e47f394b8183693402445b75810dac754d6a1f\": rpc error: code = NotFound desc = could not find container \"d732e4a88b04f0d270bd83f217e47f394b8183693402445b75810dac754d6a1f\": container with ID starting with d732e4a88b04f0d270bd83f217e47f394b8183693402445b75810dac754d6a1f not found: ID does not exist"
Oct 31 19:25:59.243759 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:59.243736    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-g6klw"]
Oct 31 19:25:59.253878 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:59.253855    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-g6klw"]
Oct 31 19:25:59.616015 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:25:59.615953    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c451c5c-f091-446f-944a-f76d0749b6dc" path="/var/lib/kubelet/pods/1c451c5c-f091-446f-944a-f76d0749b6dc/volumes"
Oct 31 19:26:06.247219 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:26:06.247166    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 19:26:14.725426 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:26:14.725313    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:26:25.248555 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:26:25.248512    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:26:36.247710 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:26:36.247665    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 19:27:00.406950 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:00.406907    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:27:06.246206 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:27:06.246165    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 19:27:07.991364 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:07.991313    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:27:08.098385 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:08.098352    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:27:14.776562 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:14.776523    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:27:18.124087 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:18.124045    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:27:18.226386 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:18.226348    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:27:24.320364 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:24.320311    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw"]
Oct 31 19:27:24.320775 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:24.320758    4645 cpu_manager.go:414] "containerMap: removing container" podUID="ca5aab9a-7f55-4ad9-b120-72470ea75774" containerName="authorino"
Oct 31 19:27:24.320829 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:24.320777    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca5aab9a-7f55-4ad9-b120-72470ea75774" containerName="authorino"
Oct 31 19:27:24.320829 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:24.320817    4645 cpu_manager.go:414] "containerMap: removing container" podUID="1c451c5c-f091-446f-944a-f76d0749b6dc" containerName="maas-api"
Oct 31 19:27:24.320829 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:24.320826    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c451c5c-f091-446f-944a-f76d0749b6dc" containerName="maas-api"
Oct 31 19:27:24.320948 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:24.320926    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c451c5c-f091-446f-944a-f76d0749b6dc" containerName="maas-api"
Oct 31 19:27:24.320948 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:24.320945    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca5aab9a-7f55-4ad9-b120-72470ea75774" containerName="authorino"
Oct 31 19:27:24.324092 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:24.324066    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw"
Oct 31 19:27:24.333263 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:24.333229    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 31 19:27:24.333425 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:24.333304    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 31 19:27:24.334079 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:24.334054    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw"]
Oct 31 19:27:24.334429 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:24.334405    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Oct 31 19:27:24.334558 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:24.334468    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-5hpmb\""
Oct 31 19:27:24.385682 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:24.385646    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:27:24.469910 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:24.469868    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1287db54-a587-4c11-9eb6-1606de555894-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw\" (UID: \"1287db54-a587-4c11-9eb6-1606de555894\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw"
Oct 31 19:27:24.470033 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:24.469922    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1287db54-a587-4c11-9eb6-1606de555894-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw\" (UID: \"1287db54-a587-4c11-9eb6-1606de555894\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw"
Oct 31 19:27:24.470033 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:24.469993    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hdjc\" (UniqueName: \"kubernetes.io/projected/1287db54-a587-4c11-9eb6-1606de555894-kube-api-access-2hdjc\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw\" (UID: \"1287db54-a587-4c11-9eb6-1606de555894\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw"
Oct 31 19:27:24.470171 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:24.470100    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1287db54-a587-4c11-9eb6-1606de555894-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw\" (UID: \"1287db54-a587-4c11-9eb6-1606de555894\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw"
Oct 31 19:27:24.470171 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:24.470134    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1287db54-a587-4c11-9eb6-1606de555894-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw\" (UID: \"1287db54-a587-4c11-9eb6-1606de555894\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw"
Oct 31 19:27:24.470171 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:24.470162    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1287db54-a587-4c11-9eb6-1606de555894-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw\" (UID: \"1287db54-a587-4c11-9eb6-1606de555894\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw"
Oct 31 19:27:24.570546 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:24.570492    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1287db54-a587-4c11-9eb6-1606de555894-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw\" (UID: \"1287db54-a587-4c11-9eb6-1606de555894\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw"
Oct 31 19:27:24.570546 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:24.570525    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1287db54-a587-4c11-9eb6-1606de555894-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw\" (UID: \"1287db54-a587-4c11-9eb6-1606de555894\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw"
Oct 31 19:27:24.570716 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:24.570551    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hdjc\" (UniqueName: \"kubernetes.io/projected/1287db54-a587-4c11-9eb6-1606de555894-kube-api-access-2hdjc\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw\" (UID: \"1287db54-a587-4c11-9eb6-1606de555894\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw"
Oct 31 19:27:24.570716 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:24.570624    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1287db54-a587-4c11-9eb6-1606de555894-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw\" (UID: \"1287db54-a587-4c11-9eb6-1606de555894\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw"
Oct 31 19:27:24.570716 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:24.570655    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1287db54-a587-4c11-9eb6-1606de555894-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw\" (UID: \"1287db54-a587-4c11-9eb6-1606de555894\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw"
Oct 31 19:27:24.570716 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:24.570685    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1287db54-a587-4c11-9eb6-1606de555894-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw\" (UID: \"1287db54-a587-4c11-9eb6-1606de555894\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw"
Oct 31 19:27:24.570979 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:24.570958    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1287db54-a587-4c11-9eb6-1606de555894-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw\" (UID: \"1287db54-a587-4c11-9eb6-1606de555894\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw"
Oct 31 19:27:24.571044 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:24.571010    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1287db54-a587-4c11-9eb6-1606de555894-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw\" (UID: \"1287db54-a587-4c11-9eb6-1606de555894\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw"
Oct 31 19:27:24.571044 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:24.571024    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1287db54-a587-4c11-9eb6-1606de555894-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw\" (UID: \"1287db54-a587-4c11-9eb6-1606de555894\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw"
Oct 31 19:27:24.573165 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:24.573136    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1287db54-a587-4c11-9eb6-1606de555894-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw\" (UID: \"1287db54-a587-4c11-9eb6-1606de555894\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw"
Oct 31 19:27:24.573279 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:24.573204    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1287db54-a587-4c11-9eb6-1606de555894-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw\" (UID: \"1287db54-a587-4c11-9eb6-1606de555894\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw"
Oct 31 19:27:24.581832 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:24.581809    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hdjc\" (UniqueName: \"kubernetes.io/projected/1287db54-a587-4c11-9eb6-1606de555894-kube-api-access-2hdjc\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw\" (UID: \"1287db54-a587-4c11-9eb6-1606de555894\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw"
Oct 31 19:27:24.635139 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:24.635108    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw"
Oct 31 19:27:24.918892 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:24.918861    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw"]
Oct 31 19:27:24.921401 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:27:24.921370    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1287db54_a587_4c11_9eb6_1606de555894.slice/crio-6c7e9eb2b5421981078d8edd9a18091ff295bfaa950ac6981a5289bad2bf50ad WatchSource:0}: Error finding container 6c7e9eb2b5421981078d8edd9a18091ff295bfaa950ac6981a5289bad2bf50ad: Status 404 returned error can't find the container with id 6c7e9eb2b5421981078d8edd9a18091ff295bfaa950ac6981a5289bad2bf50ad
Oct 31 19:27:25.489205 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:25.489175    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-88sql"]
Oct 31 19:27:25.492858 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:25.492831    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-88sql"
Oct 31 19:27:25.497377 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:25.497352    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Oct 31 19:27:25.506189 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:25.506158    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-88sql"]
Oct 31 19:27:25.557219 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:25.557182    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:27:25.578875 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:25.578846    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53abb174-56be-4e64-aadc-e871ec9c40f3-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-88sql\" (UID: \"53abb174-56be-4e64-aadc-e871ec9c40f3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-88sql"
Oct 31 19:27:25.579029 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:25.578893    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53abb174-56be-4e64-aadc-e871ec9c40f3-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-88sql\" (UID: \"53abb174-56be-4e64-aadc-e871ec9c40f3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-88sql"
Oct 31 19:27:25.579029 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:25.578918    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53abb174-56be-4e64-aadc-e871ec9c40f3-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-88sql\" (UID: \"53abb174-56be-4e64-aadc-e871ec9c40f3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-88sql"
Oct 31 19:27:25.579029 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:25.579009    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53abb174-56be-4e64-aadc-e871ec9c40f3-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-88sql\" (UID: \"53abb174-56be-4e64-aadc-e871ec9c40f3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-88sql"
Oct 31 19:27:25.579197 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:25.579035    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53abb174-56be-4e64-aadc-e871ec9c40f3-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-88sql\" (UID: \"53abb174-56be-4e64-aadc-e871ec9c40f3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-88sql"
Oct 31 19:27:25.579197 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:25.579056    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dp27\" (UniqueName: \"kubernetes.io/projected/53abb174-56be-4e64-aadc-e871ec9c40f3-kube-api-access-7dp27\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-88sql\" (UID: \"53abb174-56be-4e64-aadc-e871ec9c40f3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-88sql"
Oct 31 19:27:25.583655 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:25.583624    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw" event={"ID":"1287db54-a587-4c11-9eb6-1606de555894","Type":"ContainerStarted","Data":"6c7e9eb2b5421981078d8edd9a18091ff295bfaa950ac6981a5289bad2bf50ad"}
Oct 31 19:27:25.679723 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:25.679686    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53abb174-56be-4e64-aadc-e871ec9c40f3-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-88sql\" (UID: \"53abb174-56be-4e64-aadc-e871ec9c40f3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-88sql"
Oct 31 19:27:25.679871 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:25.679735    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53abb174-56be-4e64-aadc-e871ec9c40f3-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-88sql\" (UID: \"53abb174-56be-4e64-aadc-e871ec9c40f3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-88sql"
Oct 31 19:27:25.679871 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:25.679766    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dp27\" (UniqueName: \"kubernetes.io/projected/53abb174-56be-4e64-aadc-e871ec9c40f3-kube-api-access-7dp27\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-88sql\" (UID: \"53abb174-56be-4e64-aadc-e871ec9c40f3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-88sql"
Oct 31 19:27:25.679871 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:25.679822    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53abb174-56be-4e64-aadc-e871ec9c40f3-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-88sql\" (UID: \"53abb174-56be-4e64-aadc-e871ec9c40f3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-88sql"
Oct 31 19:27:25.679871 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:25.679843    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53abb174-56be-4e64-aadc-e871ec9c40f3-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-88sql\" (UID: \"53abb174-56be-4e64-aadc-e871ec9c40f3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-88sql"
Oct 31 19:27:25.679871 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:25.679856    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53abb174-56be-4e64-aadc-e871ec9c40f3-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-88sql\" (UID: \"53abb174-56be-4e64-aadc-e871ec9c40f3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-88sql"
Oct 31 19:27:25.680893 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:25.680852    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53abb174-56be-4e64-aadc-e871ec9c40f3-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-88sql\" (UID: \"53abb174-56be-4e64-aadc-e871ec9c40f3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-88sql"
Oct 31 19:27:25.681101 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:25.681081    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53abb174-56be-4e64-aadc-e871ec9c40f3-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-88sql\" (UID: \"53abb174-56be-4e64-aadc-e871ec9c40f3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-88sql"
Oct 31 19:27:25.681697 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:25.681643    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53abb174-56be-4e64-aadc-e871ec9c40f3-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-88sql\" (UID: \"53abb174-56be-4e64-aadc-e871ec9c40f3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-88sql"
Oct 31 19:27:25.682701 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:25.682644    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53abb174-56be-4e64-aadc-e871ec9c40f3-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-88sql\" (UID: \"53abb174-56be-4e64-aadc-e871ec9c40f3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-88sql"
Oct 31 19:27:25.687102 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:25.687080    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53abb174-56be-4e64-aadc-e871ec9c40f3-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-88sql\" (UID: \"53abb174-56be-4e64-aadc-e871ec9c40f3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-88sql"
Oct 31 19:27:25.693611 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:25.693570    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dp27\" (UniqueName: \"kubernetes.io/projected/53abb174-56be-4e64-aadc-e871ec9c40f3-kube-api-access-7dp27\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-88sql\" (UID: \"53abb174-56be-4e64-aadc-e871ec9c40f3\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-88sql"
Oct 31 19:27:25.808972 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:25.808908    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-88sql"
Oct 31 19:27:26.098031 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:26.097948    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-88sql"]
Oct 31 19:27:26.102936 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:27:26.102858    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53abb174_56be_4e64_aadc_e871ec9c40f3.slice/crio-f5d9c91cf9e907ce9f6fa336d883d6e46f6444eae3ad6051ce5343bebff3a86b WatchSource:0}: Error finding container f5d9c91cf9e907ce9f6fa336d883d6e46f6444eae3ad6051ce5343bebff3a86b: Status 404 returned error can't find the container with id f5d9c91cf9e907ce9f6fa336d883d6e46f6444eae3ad6051ce5343bebff3a86b
Oct 31 19:27:26.426778 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:26.426752    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-gg8cz"]
Oct 31 19:27:26.431541 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:26.431520    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gg8cz"
Oct 31 19:27:26.435494 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:26.435474    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Oct 31 19:27:26.441863 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:26.441840    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-gg8cz"]
Oct 31 19:27:26.488885 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:26.488858    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:27:26.587475 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:26.587448    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzkf8\" (UniqueName: \"kubernetes.io/projected/7e0baa6d-56cc-4248-853b-540777aa7b9b-kube-api-access-pzkf8\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-gg8cz\" (UID: \"7e0baa6d-56cc-4248-853b-540777aa7b9b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gg8cz"
Oct 31 19:27:26.587475 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:26.587477    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e0baa6d-56cc-4248-853b-540777aa7b9b-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-gg8cz\" (UID: \"7e0baa6d-56cc-4248-853b-540777aa7b9b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gg8cz"
Oct 31 19:27:26.587942 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:26.587513    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e0baa6d-56cc-4248-853b-540777aa7b9b-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-gg8cz\" (UID: \"7e0baa6d-56cc-4248-853b-540777aa7b9b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gg8cz"
Oct 31 19:27:26.587942 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:26.587635    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e0baa6d-56cc-4248-853b-540777aa7b9b-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-gg8cz\" (UID: \"7e0baa6d-56cc-4248-853b-540777aa7b9b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gg8cz"
Oct 31 19:27:26.587942 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:26.587658    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e0baa6d-56cc-4248-853b-540777aa7b9b-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-gg8cz\" (UID: \"7e0baa6d-56cc-4248-853b-540777aa7b9b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gg8cz"
Oct 31 19:27:26.587942 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:26.587687    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e0baa6d-56cc-4248-853b-540777aa7b9b-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-gg8cz\" (UID: \"7e0baa6d-56cc-4248-853b-540777aa7b9b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gg8cz"
Oct 31 19:27:26.589051 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:26.589018    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-88sql" event={"ID":"53abb174-56be-4e64-aadc-e871ec9c40f3","Type":"ContainerStarted","Data":"320727bf7f9b3bc9f4ee42aabd3c8dc5339d021eff3940abcd03c55dbad1481d"}
Oct 31 19:27:26.589165 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:26.589077    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-88sql" event={"ID":"53abb174-56be-4e64-aadc-e871ec9c40f3","Type":"ContainerStarted","Data":"f5d9c91cf9e907ce9f6fa336d883d6e46f6444eae3ad6051ce5343bebff3a86b"}
Oct 31 19:27:26.590723 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:26.590700    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw" event={"ID":"1287db54-a587-4c11-9eb6-1606de555894","Type":"ContainerStarted","Data":"bb66c8d55d12521df09dd441f711b5484b9b6967e38587458c22ea2c3fbfee7d"}
Oct 31 19:27:26.608935 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:26.608875    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-88sql" podStartSLOduration=1.300601408 podStartE2EDuration="1.608856875s" totalImagesPullingTime="308.255467ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:27:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:27:26.608079266 +0000 UTC m=+477.181313386" watchObservedRunningTime="2026-10-31 19:27:26.608856875 +0000 UTC m=+477.182091005"
Oct 31 19:27:26.629446 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:26.629402    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw" podStartSLOduration=1.373544836 podStartE2EDuration="2.629388848s" totalImagesPullingTime="1.255844012s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:27:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:27:26.628396637 +0000 UTC m=+477.201630763" watchObservedRunningTime="2026-10-31 19:27:26.629388848 +0000 UTC m=+477.202622977"
Oct 31 19:27:26.635629 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:26.635601    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw"
Oct 31 19:27:26.688766 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:26.688695    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzkf8\" (UniqueName: \"kubernetes.io/projected/7e0baa6d-56cc-4248-853b-540777aa7b9b-kube-api-access-pzkf8\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-gg8cz\" (UID: \"7e0baa6d-56cc-4248-853b-540777aa7b9b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gg8cz"
Oct 31 19:27:26.688766 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:26.688726    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e0baa6d-56cc-4248-853b-540777aa7b9b-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-gg8cz\" (UID: \"7e0baa6d-56cc-4248-853b-540777aa7b9b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gg8cz"
Oct 31 19:27:26.688982 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:26.688796    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e0baa6d-56cc-4248-853b-540777aa7b9b-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-gg8cz\" (UID: \"7e0baa6d-56cc-4248-853b-540777aa7b9b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gg8cz"
Oct 31 19:27:26.688982 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:26.688862    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e0baa6d-56cc-4248-853b-540777aa7b9b-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-gg8cz\" (UID: \"7e0baa6d-56cc-4248-853b-540777aa7b9b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gg8cz"
Oct 31 19:27:26.688982 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:26.688882    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e0baa6d-56cc-4248-853b-540777aa7b9b-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-gg8cz\" (UID: \"7e0baa6d-56cc-4248-853b-540777aa7b9b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gg8cz"
Oct 31 19:27:26.688982 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:26.688913    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e0baa6d-56cc-4248-853b-540777aa7b9b-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-gg8cz\" (UID: \"7e0baa6d-56cc-4248-853b-540777aa7b9b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gg8cz"
Oct 31 19:27:26.689190 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:26.689167    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e0baa6d-56cc-4248-853b-540777aa7b9b-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-gg8cz\" (UID: \"7e0baa6d-56cc-4248-853b-540777aa7b9b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gg8cz"
Oct 31 19:27:26.689494 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:26.689468    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e0baa6d-56cc-4248-853b-540777aa7b9b-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-gg8cz\" (UID: \"7e0baa6d-56cc-4248-853b-540777aa7b9b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gg8cz"
Oct 31 19:27:26.689639 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:26.689614    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e0baa6d-56cc-4248-853b-540777aa7b9b-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-gg8cz\" (UID: \"7e0baa6d-56cc-4248-853b-540777aa7b9b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gg8cz"
Oct 31 19:27:26.691268 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:26.691239    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e0baa6d-56cc-4248-853b-540777aa7b9b-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-gg8cz\" (UID: \"7e0baa6d-56cc-4248-853b-540777aa7b9b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gg8cz"
Oct 31 19:27:26.691493 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:26.691471    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e0baa6d-56cc-4248-853b-540777aa7b9b-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-gg8cz\" (UID: \"7e0baa6d-56cc-4248-853b-540777aa7b9b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gg8cz"
Oct 31 19:27:26.700322 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:26.700288    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzkf8\" (UniqueName: \"kubernetes.io/projected/7e0baa6d-56cc-4248-853b-540777aa7b9b-kube-api-access-pzkf8\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-gg8cz\" (UID: \"7e0baa6d-56cc-4248-853b-540777aa7b9b\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gg8cz"
Oct 31 19:27:26.775760 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:26.775487    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gg8cz"
Oct 31 19:27:26.809631 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:26.809574    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-88sql"
Oct 31 19:27:27.075350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:27.074885    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-gg8cz"]
Oct 31 19:27:27.077474 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:27:27.077446    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e0baa6d_56cc_4248_853b_540777aa7b9b.slice/crio-d279a20c2fa765d372c2a3e6887d86716c23662267ba2cee9a1bba299ceace3e WatchSource:0}: Error finding container d279a20c2fa765d372c2a3e6887d86716c23662267ba2cee9a1bba299ceace3e: Status 404 returned error can't find the container with id d279a20c2fa765d372c2a3e6887d86716c23662267ba2cee9a1bba299ceace3e
Oct 31 19:27:27.596817 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:27.596777    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gg8cz" event={"ID":"7e0baa6d-56cc-4248-853b-540777aa7b9b","Type":"ContainerStarted","Data":"3f4b69a0bc64218d2094ab94ef5e0ddaf573e252e087f0a932a67ff3122af56c"}
Oct 31 19:27:27.597217 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:27.596841    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gg8cz" event={"ID":"7e0baa6d-56cc-4248-853b-540777aa7b9b","Type":"ContainerStarted","Data":"d279a20c2fa765d372c2a3e6887d86716c23662267ba2cee9a1bba299ceace3e"}
Oct 31 19:27:27.616432 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:27.616387    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gg8cz" podStartSLOduration=1.2842499840000001 podStartE2EDuration="1.616372544s" totalImagesPullingTime="332.12256ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:27:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:27:27.615340517 +0000 UTC m=+478.188574637" watchObservedRunningTime="2026-10-31 19:27:27.616372544 +0000 UTC m=+478.189606672"
Oct 31 19:27:27.776844 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:27.776772    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gg8cz"
Oct 31 19:27:27.806378 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:27.806339    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-k8zgg"]
Oct 31 19:27:27.809543 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:27.809525    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-k8zgg"
Oct 31 19:27:27.815496 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:27.815476    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Oct 31 19:27:27.820470 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:27.820446    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-k8zgg"]
Oct 31 19:27:27.866601 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:27.866578    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:27:27.899598 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:27.899576    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ac0d131-8660-4018-88c9-1e2d06f3c038-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-k8zgg\" (UID: \"7ac0d131-8660-4018-88c9-1e2d06f3c038\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-k8zgg"
Oct 31 19:27:27.899704 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:27.899608    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ac0d131-8660-4018-88c9-1e2d06f3c038-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-k8zgg\" (UID: \"7ac0d131-8660-4018-88c9-1e2d06f3c038\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-k8zgg"
Oct 31 19:27:27.899704 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:27.899626    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ac0d131-8660-4018-88c9-1e2d06f3c038-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-k8zgg\" (UID: \"7ac0d131-8660-4018-88c9-1e2d06f3c038\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-k8zgg"
Oct 31 19:27:27.899704 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:27.899644    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ac0d131-8660-4018-88c9-1e2d06f3c038-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-k8zgg\" (UID: \"7ac0d131-8660-4018-88c9-1e2d06f3c038\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-k8zgg"
Oct 31 19:27:27.899807 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:27.899746    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ac0d131-8660-4018-88c9-1e2d06f3c038-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-k8zgg\" (UID: \"7ac0d131-8660-4018-88c9-1e2d06f3c038\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-k8zgg"
Oct 31 19:27:27.899840 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:27.899803    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbbc5\" (UniqueName: \"kubernetes.io/projected/7ac0d131-8660-4018-88c9-1e2d06f3c038-kube-api-access-dbbc5\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-k8zgg\" (UID: \"7ac0d131-8660-4018-88c9-1e2d06f3c038\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-k8zgg"
Oct 31 19:27:28.001052 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:28.001026    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ac0d131-8660-4018-88c9-1e2d06f3c038-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-k8zgg\" (UID: \"7ac0d131-8660-4018-88c9-1e2d06f3c038\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-k8zgg"
Oct 31 19:27:28.001151 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:28.001060    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ac0d131-8660-4018-88c9-1e2d06f3c038-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-k8zgg\" (UID: \"7ac0d131-8660-4018-88c9-1e2d06f3c038\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-k8zgg"
Oct 31 19:27:28.001151 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:28.001079    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ac0d131-8660-4018-88c9-1e2d06f3c038-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-k8zgg\" (UID: \"7ac0d131-8660-4018-88c9-1e2d06f3c038\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-k8zgg"
Oct 31 19:27:28.001151 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:28.001114    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ac0d131-8660-4018-88c9-1e2d06f3c038-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-k8zgg\" (UID: \"7ac0d131-8660-4018-88c9-1e2d06f3c038\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-k8zgg"
Oct 31 19:27:28.001151 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:28.001144    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbbc5\" (UniqueName: \"kubernetes.io/projected/7ac0d131-8660-4018-88c9-1e2d06f3c038-kube-api-access-dbbc5\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-k8zgg\" (UID: \"7ac0d131-8660-4018-88c9-1e2d06f3c038\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-k8zgg"
Oct 31 19:27:28.001388 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:28.001233    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ac0d131-8660-4018-88c9-1e2d06f3c038-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-k8zgg\" (UID: \"7ac0d131-8660-4018-88c9-1e2d06f3c038\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-k8zgg"
Oct 31 19:27:28.001504 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:28.001479    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ac0d131-8660-4018-88c9-1e2d06f3c038-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-k8zgg\" (UID: \"7ac0d131-8660-4018-88c9-1e2d06f3c038\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-k8zgg"
Oct 31 19:27:28.001630 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:28.001530    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ac0d131-8660-4018-88c9-1e2d06f3c038-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-k8zgg\" (UID: \"7ac0d131-8660-4018-88c9-1e2d06f3c038\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-k8zgg"
Oct 31 19:27:28.001630 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:28.001593    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ac0d131-8660-4018-88c9-1e2d06f3c038-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-k8zgg\" (UID: \"7ac0d131-8660-4018-88c9-1e2d06f3c038\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-k8zgg"
Oct 31 19:27:28.003303 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:28.003286    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ac0d131-8660-4018-88c9-1e2d06f3c038-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-k8zgg\" (UID: \"7ac0d131-8660-4018-88c9-1e2d06f3c038\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-k8zgg"
Oct 31 19:27:28.003581 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:28.003562    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ac0d131-8660-4018-88c9-1e2d06f3c038-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-k8zgg\" (UID: \"7ac0d131-8660-4018-88c9-1e2d06f3c038\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-k8zgg"
Oct 31 19:27:28.012720 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:28.012695    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbbc5\" (UniqueName: \"kubernetes.io/projected/7ac0d131-8660-4018-88c9-1e2d06f3c038-kube-api-access-dbbc5\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-k8zgg\" (UID: \"7ac0d131-8660-4018-88c9-1e2d06f3c038\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-k8zgg"
Oct 31 19:27:28.125035 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:28.124984    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-k8zgg"
Oct 31 19:27:28.400144 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:28.400055    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-k8zgg"]
Oct 31 19:27:28.402929 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:27:28.402889    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ac0d131_8660_4018_88c9_1e2d06f3c038.slice/crio-840411e22d28d21142d4fb72db37a831db9f5338a9c4a1c7e7de10c172d759a4 WatchSource:0}: Error finding container 840411e22d28d21142d4fb72db37a831db9f5338a9c4a1c7e7de10c172d759a4: Status 404 returned error can't find the container with id 840411e22d28d21142d4fb72db37a831db9f5338a9c4a1c7e7de10c172d759a4
Oct 31 19:27:28.615919 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:28.615884    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-k8zgg" event={"ID":"7ac0d131-8660-4018-88c9-1e2d06f3c038","Type":"ContainerStarted","Data":"840411e22d28d21142d4fb72db37a831db9f5338a9c4a1c7e7de10c172d759a4"}
Oct 31 19:27:29.622144 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:29.622096    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-k8zgg" event={"ID":"7ac0d131-8660-4018-88c9-1e2d06f3c038","Type":"ContainerStarted","Data":"780d42349d2f8397dbf10fac64afe7f5c0e44797e6db169b63eea626b3a95990"}
Oct 31 19:27:29.641450 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:29.641401    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-k8zgg" podStartSLOduration=2.3146624190000002 podStartE2EDuration="2.641387026s" totalImagesPullingTime="326.724607ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:27:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:27:29.64089132 +0000 UTC m=+480.214125448" watchObservedRunningTime="2026-10-31 19:27:29.641387026 +0000 UTC m=+480.214621154"
Oct 31 19:27:30.125940 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:30.125909    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-k8zgg"
Oct 31 19:27:30.166911 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:30.166885    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:27:30.626233 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:30.626192    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-wntcw"]
Oct 31 19:27:30.629920 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:30.629896    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-wntcw"
Oct 31 19:27:30.636536 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:30.636513    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Oct 31 19:27:30.642322 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:30.642287    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-wntcw"]
Oct 31 19:27:30.685436 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:30.685406    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:27:30.723161 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:30.723137    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6b653ca-0bd2-488f-925b-02a9a8dc509b-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-wntcw\" (UID: \"d6b653ca-0bd2-488f-925b-02a9a8dc509b\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-wntcw"
Oct 31 19:27:30.723284 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:30.723180    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6b653ca-0bd2-488f-925b-02a9a8dc509b-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-wntcw\" (UID: \"d6b653ca-0bd2-488f-925b-02a9a8dc509b\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-wntcw"
Oct 31 19:27:30.723284 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:30.723205    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6b653ca-0bd2-488f-925b-02a9a8dc509b-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-wntcw\" (UID: \"d6b653ca-0bd2-488f-925b-02a9a8dc509b\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-wntcw"
Oct 31 19:27:30.723397 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:30.723353    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6b653ca-0bd2-488f-925b-02a9a8dc509b-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-wntcw\" (UID: \"d6b653ca-0bd2-488f-925b-02a9a8dc509b\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-wntcw"
Oct 31 19:27:30.723436 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:30.723411    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6b653ca-0bd2-488f-925b-02a9a8dc509b-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-wntcw\" (UID: \"d6b653ca-0bd2-488f-925b-02a9a8dc509b\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-wntcw"
Oct 31 19:27:30.723493 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:30.723473    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnrpc\" (UniqueName: \"kubernetes.io/projected/d6b653ca-0bd2-488f-925b-02a9a8dc509b-kube-api-access-hnrpc\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-wntcw\" (UID: \"d6b653ca-0bd2-488f-925b-02a9a8dc509b\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-wntcw"
Oct 31 19:27:30.824621 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:30.824592    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6b653ca-0bd2-488f-925b-02a9a8dc509b-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-wntcw\" (UID: \"d6b653ca-0bd2-488f-925b-02a9a8dc509b\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-wntcw"
Oct 31 19:27:30.824621 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:30.824624    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6b653ca-0bd2-488f-925b-02a9a8dc509b-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-wntcw\" (UID: \"d6b653ca-0bd2-488f-925b-02a9a8dc509b\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-wntcw"
Oct 31 19:27:30.824771 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:30.824642    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6b653ca-0bd2-488f-925b-02a9a8dc509b-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-wntcw\" (UID: \"d6b653ca-0bd2-488f-925b-02a9a8dc509b\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-wntcw"
Oct 31 19:27:30.824771 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:30.824657    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6b653ca-0bd2-488f-925b-02a9a8dc509b-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-wntcw\" (UID: \"d6b653ca-0bd2-488f-925b-02a9a8dc509b\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-wntcw"
Oct 31 19:27:30.824771 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:30.824676    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6b653ca-0bd2-488f-925b-02a9a8dc509b-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-wntcw\" (UID: \"d6b653ca-0bd2-488f-925b-02a9a8dc509b\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-wntcw"
Oct 31 19:27:30.824913 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:30.824893    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnrpc\" (UniqueName: \"kubernetes.io/projected/d6b653ca-0bd2-488f-925b-02a9a8dc509b-kube-api-access-hnrpc\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-wntcw\" (UID: \"d6b653ca-0bd2-488f-925b-02a9a8dc509b\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-wntcw"
Oct 31 19:27:30.825014 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:30.824995    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6b653ca-0bd2-488f-925b-02a9a8dc509b-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-wntcw\" (UID: \"d6b653ca-0bd2-488f-925b-02a9a8dc509b\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-wntcw"
Oct 31 19:27:30.825063 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:30.825027    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6b653ca-0bd2-488f-925b-02a9a8dc509b-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-wntcw\" (UID: \"d6b653ca-0bd2-488f-925b-02a9a8dc509b\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-wntcw"
Oct 31 19:27:30.825094 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:30.825072    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6b653ca-0bd2-488f-925b-02a9a8dc509b-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-wntcw\" (UID: \"d6b653ca-0bd2-488f-925b-02a9a8dc509b\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-wntcw"
Oct 31 19:27:30.826923 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:30.826898    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6b653ca-0bd2-488f-925b-02a9a8dc509b-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-wntcw\" (UID: \"d6b653ca-0bd2-488f-925b-02a9a8dc509b\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-wntcw"
Oct 31 19:27:30.827191 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:30.827169    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6b653ca-0bd2-488f-925b-02a9a8dc509b-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-wntcw\" (UID: \"d6b653ca-0bd2-488f-925b-02a9a8dc509b\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-wntcw"
Oct 31 19:27:30.836615 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:30.836592    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnrpc\" (UniqueName: \"kubernetes.io/projected/d6b653ca-0bd2-488f-925b-02a9a8dc509b-kube-api-access-hnrpc\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-wntcw\" (UID: \"d6b653ca-0bd2-488f-925b-02a9a8dc509b\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-wntcw"
Oct 31 19:27:30.943152 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:30.943130    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-wntcw"
Oct 31 19:27:31.085439 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:31.085402    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbsgn"]
Oct 31 19:27:31.089713 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:31.089692    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbsgn"
Oct 31 19:27:31.092920 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:31.092902    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Oct 31 19:27:31.101511 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:31.101492    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbsgn"]
Oct 31 19:27:31.142089 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:31.142064    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:27:31.213165 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:31.213067    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-wntcw"]
Oct 31 19:27:31.215605 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:27:31.215584    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6b653ca_0bd2_488f_925b_02a9a8dc509b.slice/crio-271903d472c636523db4895cd8ce2282f39de9802f485acfa767454d87465ff6 WatchSource:0}: Error finding container 271903d472c636523db4895cd8ce2282f39de9802f485acfa767454d87465ff6: Status 404 returned error can't find the container with id 271903d472c636523db4895cd8ce2282f39de9802f485acfa767454d87465ff6
Oct 31 19:27:31.228934 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:31.228912    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/280abee2-f10e-4a96-9735-a3aadde28b69-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbsgn\" (UID: \"280abee2-f10e-4a96-9735-a3aadde28b69\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbsgn"
Oct 31 19:27:31.229019 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:31.228941    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/280abee2-f10e-4a96-9735-a3aadde28b69-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbsgn\" (UID: \"280abee2-f10e-4a96-9735-a3aadde28b69\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbsgn"
Oct 31 19:27:31.229019 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:31.228964    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/280abee2-f10e-4a96-9735-a3aadde28b69-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbsgn\" (UID: \"280abee2-f10e-4a96-9735-a3aadde28b69\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbsgn"
Oct 31 19:27:31.229123 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:31.229102    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/280abee2-f10e-4a96-9735-a3aadde28b69-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbsgn\" (UID: \"280abee2-f10e-4a96-9735-a3aadde28b69\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbsgn"
Oct 31 19:27:31.229162 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:31.229149    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/280abee2-f10e-4a96-9735-a3aadde28b69-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbsgn\" (UID: \"280abee2-f10e-4a96-9735-a3aadde28b69\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbsgn"
Oct 31 19:27:31.229221 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:31.229208    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2p5r\" (UniqueName: \"kubernetes.io/projected/280abee2-f10e-4a96-9735-a3aadde28b69-kube-api-access-d2p5r\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbsgn\" (UID: \"280abee2-f10e-4a96-9735-a3aadde28b69\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbsgn"
Oct 31 19:27:31.329824 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:31.329802    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/280abee2-f10e-4a96-9735-a3aadde28b69-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbsgn\" (UID: \"280abee2-f10e-4a96-9735-a3aadde28b69\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbsgn"
Oct 31 19:27:31.330002 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:31.329831    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2p5r\" (UniqueName: \"kubernetes.io/projected/280abee2-f10e-4a96-9735-a3aadde28b69-kube-api-access-d2p5r\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbsgn\" (UID: \"280abee2-f10e-4a96-9735-a3aadde28b69\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbsgn"
Oct 31 19:27:31.330002 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:31.329853    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/280abee2-f10e-4a96-9735-a3aadde28b69-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbsgn\" (UID: \"280abee2-f10e-4a96-9735-a3aadde28b69\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbsgn"
Oct 31 19:27:31.330002 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:31.329879    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/280abee2-f10e-4a96-9735-a3aadde28b69-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbsgn\" (UID: \"280abee2-f10e-4a96-9735-a3aadde28b69\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbsgn"
Oct 31 19:27:31.330002 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:31.329917    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/280abee2-f10e-4a96-9735-a3aadde28b69-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbsgn\" (UID: \"280abee2-f10e-4a96-9735-a3aadde28b69\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbsgn"
Oct 31 19:27:31.330218 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:31.330069    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/280abee2-f10e-4a96-9735-a3aadde28b69-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbsgn\" (UID: \"280abee2-f10e-4a96-9735-a3aadde28b69\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbsgn"
Oct 31 19:27:31.330218 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:31.330180    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/280abee2-f10e-4a96-9735-a3aadde28b69-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbsgn\" (UID: \"280abee2-f10e-4a96-9735-a3aadde28b69\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbsgn"
Oct 31 19:27:31.330218 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:31.330213    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/280abee2-f10e-4a96-9735-a3aadde28b69-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbsgn\" (UID: \"280abee2-f10e-4a96-9735-a3aadde28b69\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbsgn"
Oct 31 19:27:31.330409 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:31.330276    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/280abee2-f10e-4a96-9735-a3aadde28b69-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbsgn\" (UID: \"280abee2-f10e-4a96-9735-a3aadde28b69\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbsgn"
Oct 31 19:27:31.331969 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:31.331951    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/280abee2-f10e-4a96-9735-a3aadde28b69-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbsgn\" (UID: \"280abee2-f10e-4a96-9735-a3aadde28b69\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbsgn"
Oct 31 19:27:31.332310 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:31.332294    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/280abee2-f10e-4a96-9735-a3aadde28b69-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbsgn\" (UID: \"280abee2-f10e-4a96-9735-a3aadde28b69\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbsgn"
Oct 31 19:27:31.341339 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:31.341303    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2p5r\" (UniqueName: \"kubernetes.io/projected/280abee2-f10e-4a96-9735-a3aadde28b69-kube-api-access-d2p5r\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbsgn\" (UID: \"280abee2-f10e-4a96-9735-a3aadde28b69\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbsgn"
Oct 31 19:27:31.402867 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:31.402843    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbsgn"
Oct 31 19:27:31.631794 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:31.631758    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-wntcw" event={"ID":"d6b653ca-0bd2-488f-925b-02a9a8dc509b","Type":"ContainerStarted","Data":"06817b97963b6c5da8707824674e42e7d309f5f13159a2382299fdfafe81787b"}
Oct 31 19:27:31.632225 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:31.631816    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-wntcw" event={"ID":"d6b653ca-0bd2-488f-925b-02a9a8dc509b","Type":"ContainerStarted","Data":"271903d472c636523db4895cd8ce2282f39de9802f485acfa767454d87465ff6"}
Oct 31 19:27:31.654505 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:31.654457    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-wntcw" podStartSLOduration=1.38427951 podStartE2EDuration="1.654442741s" totalImagesPullingTime="270.163231ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:27:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:27:31.653568562 +0000 UTC m=+482.226802693" watchObservedRunningTime="2026-10-31 19:27:31.654442741 +0000 UTC m=+482.227676863"
Oct 31 19:27:31.679399 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:31.679298    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbsgn"]
Oct 31 19:27:31.682178 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:27:31.682143    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod280abee2_f10e_4a96_9735_a3aadde28b69.slice/crio-c931e94fe18943ef859ed40653da316bfcc8512b823246cb72c90933fb5d4ae6 WatchSource:0}: Error finding container c931e94fe18943ef859ed40653da316bfcc8512b823246cb72c90933fb5d4ae6: Status 404 returned error can't find the container with id c931e94fe18943ef859ed40653da316bfcc8512b823246cb72c90933fb5d4ae6
Oct 31 19:27:31.943425 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:31.943391    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-wntcw"
Oct 31 19:27:32.637556 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:32.637514    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbsgn" event={"ID":"280abee2-f10e-4a96-9735-a3aadde28b69","Type":"ContainerStarted","Data":"e125a142761b19f462e122bb66e679e30d6ebb48f5c40cee8b7460ddc737818d"}
Oct 31 19:27:32.637991 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:32.637613    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbsgn" event={"ID":"280abee2-f10e-4a96-9735-a3aadde28b69","Type":"ContainerStarted","Data":"c931e94fe18943ef859ed40653da316bfcc8512b823246cb72c90933fb5d4ae6"}
Oct 31 19:27:32.673664 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:32.673602    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbsgn" podStartSLOduration=1.392690594 podStartE2EDuration="1.673585227s" totalImagesPullingTime="280.894633ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:27:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:27:32.672624354 +0000 UTC m=+483.245858508" watchObservedRunningTime="2026-10-31 19:27:32.673585227 +0000 UTC m=+483.246819353"
Oct 31 19:27:32.835515 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:32.835480    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:27:33.403614 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:33.403583    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbsgn"
Oct 31 19:27:33.989703 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:33.989669    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-msrfk"]
Oct 31 19:27:33.993233 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:33.993210    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-msrfk"
Oct 31 19:27:34.009141 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:34.009109    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-msrfk"]
Oct 31 19:27:34.036122 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:34.036098    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 31 19:27:34.083537 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:34.083517    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-jkgv9\""
Oct 31 19:27:34.155397 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:34.155374    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48b0949c-0c89-4c80-a1dc-64d14839f620-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-msrfk\" (UID: \"48b0949c-0c89-4c80-a1dc-64d14839f620\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-msrfk"
Oct 31 19:27:34.155485 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:34.155437    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48b0949c-0c89-4c80-a1dc-64d14839f620-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-msrfk\" (UID: \"48b0949c-0c89-4c80-a1dc-64d14839f620\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-msrfk"
Oct 31 19:27:34.155485 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:34.155480    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48b0949c-0c89-4c80-a1dc-64d14839f620-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-msrfk\" (UID: \"48b0949c-0c89-4c80-a1dc-64d14839f620\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-msrfk"
Oct 31 19:27:34.155565 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:34.155507    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48b0949c-0c89-4c80-a1dc-64d14839f620-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-msrfk\" (UID: \"48b0949c-0c89-4c80-a1dc-64d14839f620\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-msrfk"
Oct 31 19:27:34.155616 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:34.155596    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48b0949c-0c89-4c80-a1dc-64d14839f620-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-msrfk\" (UID: \"48b0949c-0c89-4c80-a1dc-64d14839f620\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-msrfk"
Oct 31 19:27:34.155652 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:34.155623    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kldvq\" (UniqueName: \"kubernetes.io/projected/48b0949c-0c89-4c80-a1dc-64d14839f620-kube-api-access-kldvq\") pod \"maas-default-gateway-openshift-default-75d7c785f-msrfk\" (UID: \"48b0949c-0c89-4c80-a1dc-64d14839f620\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-msrfk"
Oct 31 19:27:34.155704 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:34.155689    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48b0949c-0c89-4c80-a1dc-64d14839f620-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-msrfk\" (UID: \"48b0949c-0c89-4c80-a1dc-64d14839f620\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-msrfk"
Oct 31 19:27:34.155754 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:34.155742    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48b0949c-0c89-4c80-a1dc-64d14839f620-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-msrfk\" (UID: \"48b0949c-0c89-4c80-a1dc-64d14839f620\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-msrfk"
Oct 31 19:27:34.155791 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:34.155764    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48b0949c-0c89-4c80-a1dc-64d14839f620-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-msrfk\" (UID: \"48b0949c-0c89-4c80-a1dc-64d14839f620\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-msrfk"
Oct 31 19:27:34.256248 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:34.256188    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48b0949c-0c89-4c80-a1dc-64d14839f620-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-msrfk\" (UID: \"48b0949c-0c89-4c80-a1dc-64d14839f620\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-msrfk"
Oct 31 19:27:34.256248 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:34.256219    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kldvq\" (UniqueName: \"kubernetes.io/projected/48b0949c-0c89-4c80-a1dc-64d14839f620-kube-api-access-kldvq\") pod \"maas-default-gateway-openshift-default-75d7c785f-msrfk\" (UID: \"48b0949c-0c89-4c80-a1dc-64d14839f620\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-msrfk"
Oct 31 19:27:34.256400 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:34.256259    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48b0949c-0c89-4c80-a1dc-64d14839f620-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-msrfk\" (UID: \"48b0949c-0c89-4c80-a1dc-64d14839f620\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-msrfk"
Oct 31 19:27:34.256478 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:34.256451    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48b0949c-0c89-4c80-a1dc-64d14839f620-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-msrfk\" (UID: \"48b0949c-0c89-4c80-a1dc-64d14839f620\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-msrfk"
Oct 31 19:27:34.256525 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:34.256503    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48b0949c-0c89-4c80-a1dc-64d14839f620-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-msrfk\" (UID: \"48b0949c-0c89-4c80-a1dc-64d14839f620\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-msrfk"
Oct 31 19:27:34.256572 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:34.256539    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48b0949c-0c89-4c80-a1dc-64d14839f620-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-msrfk\" (UID: \"48b0949c-0c89-4c80-a1dc-64d14839f620\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-msrfk"
Oct 31 19:27:34.256678 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:34.256615    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48b0949c-0c89-4c80-a1dc-64d14839f620-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-msrfk\" (UID: \"48b0949c-0c89-4c80-a1dc-64d14839f620\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-msrfk"
Oct 31 19:27:34.256678 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:34.256641    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48b0949c-0c89-4c80-a1dc-64d14839f620-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-msrfk\" (UID: \"48b0949c-0c89-4c80-a1dc-64d14839f620\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-msrfk"
Oct 31 19:27:34.256678 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:34.256670    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48b0949c-0c89-4c80-a1dc-64d14839f620-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-msrfk\" (UID: \"48b0949c-0c89-4c80-a1dc-64d14839f620\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-msrfk"
Oct 31 19:27:34.256886 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:34.256859    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48b0949c-0c89-4c80-a1dc-64d14839f620-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-msrfk\" (UID: \"48b0949c-0c89-4c80-a1dc-64d14839f620\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-msrfk"
Oct 31 19:27:34.257081 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:34.257051    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48b0949c-0c89-4c80-a1dc-64d14839f620-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-msrfk\" (UID: \"48b0949c-0c89-4c80-a1dc-64d14839f620\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-msrfk"
Oct 31 19:27:34.257193 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:34.256927    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48b0949c-0c89-4c80-a1dc-64d14839f620-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-msrfk\" (UID: \"48b0949c-0c89-4c80-a1dc-64d14839f620\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-msrfk"
Oct 31 19:27:34.257193 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:34.257120    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48b0949c-0c89-4c80-a1dc-64d14839f620-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-msrfk\" (UID: \"48b0949c-0c89-4c80-a1dc-64d14839f620\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-msrfk"
Oct 31 19:27:34.257267 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:34.257234    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48b0949c-0c89-4c80-a1dc-64d14839f620-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-msrfk\" (UID: \"48b0949c-0c89-4c80-a1dc-64d14839f620\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-msrfk"
Oct 31 19:27:34.259111 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:34.259086    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48b0949c-0c89-4c80-a1dc-64d14839f620-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-msrfk\" (UID: \"48b0949c-0c89-4c80-a1dc-64d14839f620\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-msrfk"
Oct 31 19:27:34.259213 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:34.259119    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48b0949c-0c89-4c80-a1dc-64d14839f620-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-msrfk\" (UID: \"48b0949c-0c89-4c80-a1dc-64d14839f620\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-msrfk"
Oct 31 19:27:34.268666 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:34.268639    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kldvq\" (UniqueName: \"kubernetes.io/projected/48b0949c-0c89-4c80-a1dc-64d14839f620-kube-api-access-kldvq\") pod \"maas-default-gateway-openshift-default-75d7c785f-msrfk\" (UID: \"48b0949c-0c89-4c80-a1dc-64d14839f620\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-msrfk"
Oct 31 19:27:34.269510 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:34.269486    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48b0949c-0c89-4c80-a1dc-64d14839f620-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-msrfk\" (UID: \"48b0949c-0c89-4c80-a1dc-64d14839f620\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-msrfk"
Oct 31 19:27:34.306252 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:34.306228    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-msrfk"
Oct 31 19:27:34.596002 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:34.595977    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-msrfk"]
Oct 31 19:27:34.599176 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:27:34.599137    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48b0949c_0c89_4c80_a1dc_64d14839f620.slice/crio-4776e22c2b14ec115176f74ad372e5f5b3c1a8e12e852e1ab9e1cc58298eb863 WatchSource:0}: Error finding container 4776e22c2b14ec115176f74ad372e5f5b3c1a8e12e852e1ab9e1cc58298eb863: Status 404 returned error can't find the container with id 4776e22c2b14ec115176f74ad372e5f5b3c1a8e12e852e1ab9e1cc58298eb863
Oct 31 19:27:34.635977 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:34.635944    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw"
Oct 31 19:27:34.647120 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:34.647093    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-msrfk" event={"ID":"48b0949c-0c89-4c80-a1dc-64d14839f620","Type":"ContainerStarted","Data":"4776e22c2b14ec115176f74ad372e5f5b3c1a8e12e852e1ab9e1cc58298eb863"}
Oct 31 19:27:34.650354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:34.650320    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw"
Oct 31 19:27:34.661605 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:34.661583    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw"
Oct 31 19:27:34.825302 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:34.825267    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:27:35.809722 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:35.809688    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-88sql"
Oct 31 19:27:35.824749 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:35.824720    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-88sql"
Oct 31 19:27:36.250748 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:27:36.250710    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 19:27:36.688832 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:36.688799    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-88sql"
Oct 31 19:27:36.776350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:36.776271    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gg8cz"
Oct 31 19:27:36.792506 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:36.792471    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gg8cz"
Oct 31 19:27:37.694675 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:37.694636    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-gg8cz"
Oct 31 19:27:37.808769 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:37.808737    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:27:37.808849 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:37.808808    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:27:37.808849 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:37.808836    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:27:38.125286 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:38.125195    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-k8zgg"
Oct 31 19:27:38.137923 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:38.137898    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-k8zgg"
Oct 31 19:27:38.691671 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:38.691634    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-msrfk" event={"ID":"48b0949c-0c89-4c80-a1dc-64d14839f620","Type":"ContainerStarted","Data":"ae2f8d98f60d251c600ecf1c895d943af254d216228ec8e3d6119638df3befcb"}
Oct 31 19:27:38.703978 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:38.703955    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-k8zgg"
Oct 31 19:27:38.715015 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:38.714970    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-msrfk" podStartSLOduration=2.508461027 podStartE2EDuration="5.714957563s" totalImagesPullingTime="3.206496536s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:27:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:27:38.712689647 +0000 UTC m=+489.285923776" watchObservedRunningTime="2026-10-31 19:27:38.714957563 +0000 UTC m=+489.288191692"
Oct 31 19:27:39.306952 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:39.306921    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-msrfk"
Oct 31 19:27:39.308583 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:39.308549    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-msrfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 31 19:27:39.308709 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:39.308607    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-msrfk" podUID="48b0949c-0c89-4c80-a1dc-64d14839f620" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 31 19:27:40.306906 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:40.306868    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-msrfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 31 19:27:40.307394 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:40.306930    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-msrfk" podUID="48b0949c-0c89-4c80-a1dc-64d14839f620" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 31 19:27:40.943800 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:40.943764    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-wntcw"
Oct 31 19:27:40.980364 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:40.978705    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-wntcw"
Oct 31 19:27:41.306915 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:41.306830    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-msrfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 31 19:27:41.306915 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:41.306881    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-msrfk" podUID="48b0949c-0c89-4c80-a1dc-64d14839f620" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 31 19:27:41.403384 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:41.403351    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbsgn"
Oct 31 19:27:41.415909 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:41.415885    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbsgn"
Oct 31 19:27:41.716729 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:41.716702    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbsgn"
Oct 31 19:27:41.717498 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:41.717477    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-wntcw"
Oct 31 19:27:42.306925 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:42.306887    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-msrfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 31 19:27:42.307112 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:42.306952    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-msrfk" podUID="48b0949c-0c89-4c80-a1dc-64d14839f620" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 31 19:27:43.306911 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:43.306873    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-msrfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 31 19:27:43.307118 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:43.306933    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-msrfk" podUID="48b0949c-0c89-4c80-a1dc-64d14839f620" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 31 19:27:44.307115 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:44.307086    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-msrfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 31 19:27:44.307612 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:44.307132    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-msrfk" podUID="48b0949c-0c89-4c80-a1dc-64d14839f620" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 31 19:27:45.307380 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:45.307310    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-msrfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 31 19:27:45.307766 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:45.307416    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-msrfk" podUID="48b0949c-0c89-4c80-a1dc-64d14839f620" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 31 19:27:46.311033 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:46.311003    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-msrfk"
Oct 31 19:27:46.311474 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:46.311426    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-msrfk"
Oct 31 19:27:46.312008 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:46.311988    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-msrfk"
Oct 31 19:27:49.021273 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:49.021241    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-l6wqz"]
Oct 31 19:27:49.024945 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:49.024924    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-l6wqz"
Oct 31 19:27:49.040479 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:49.040456    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-l6wqz"]
Oct 31 19:27:49.089164 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:49.089142    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cc617a4-ee55-4218-9907-1f43b5d0a18a-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-l6wqz\" (UID: \"5cc617a4-ee55-4218-9907-1f43b5d0a18a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-l6wqz"
Oct 31 19:27:49.089282 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:49.089176    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cc617a4-ee55-4218-9907-1f43b5d0a18a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-l6wqz\" (UID: \"5cc617a4-ee55-4218-9907-1f43b5d0a18a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-l6wqz"
Oct 31 19:27:49.089282 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:49.089263    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cc617a4-ee55-4218-9907-1f43b5d0a18a-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-l6wqz\" (UID: \"5cc617a4-ee55-4218-9907-1f43b5d0a18a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-l6wqz"
Oct 31 19:27:49.089390 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:49.089309    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kch7w\" (UniqueName: \"kubernetes.io/projected/5cc617a4-ee55-4218-9907-1f43b5d0a18a-kube-api-access-kch7w\") pod \"maas-default-gateway-openshift-default-75d7c785f-l6wqz\" (UID: \"5cc617a4-ee55-4218-9907-1f43b5d0a18a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-l6wqz"
Oct 31 19:27:49.089469 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:49.089453    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cc617a4-ee55-4218-9907-1f43b5d0a18a-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-l6wqz\" (UID: \"5cc617a4-ee55-4218-9907-1f43b5d0a18a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-l6wqz"
Oct 31 19:27:49.089527 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:49.089493    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cc617a4-ee55-4218-9907-1f43b5d0a18a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-l6wqz\" (UID: \"5cc617a4-ee55-4218-9907-1f43b5d0a18a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-l6wqz"
Oct 31 19:27:49.089572 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:49.089535    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cc617a4-ee55-4218-9907-1f43b5d0a18a-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-l6wqz\" (UID: \"5cc617a4-ee55-4218-9907-1f43b5d0a18a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-l6wqz"
Oct 31 19:27:49.089633 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:49.089615    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cc617a4-ee55-4218-9907-1f43b5d0a18a-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-l6wqz\" (UID: \"5cc617a4-ee55-4218-9907-1f43b5d0a18a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-l6wqz"
Oct 31 19:27:49.089672 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:49.089642    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cc617a4-ee55-4218-9907-1f43b5d0a18a-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-l6wqz\" (UID: \"5cc617a4-ee55-4218-9907-1f43b5d0a18a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-l6wqz"
Oct 31 19:27:49.190851 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:49.190831    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cc617a4-ee55-4218-9907-1f43b5d0a18a-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-l6wqz\" (UID: \"5cc617a4-ee55-4218-9907-1f43b5d0a18a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-l6wqz"
Oct 31 19:27:49.190962 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:49.190858    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cc617a4-ee55-4218-9907-1f43b5d0a18a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-l6wqz\" (UID: \"5cc617a4-ee55-4218-9907-1f43b5d0a18a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-l6wqz"
Oct 31 19:27:49.190962 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:49.190883    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cc617a4-ee55-4218-9907-1f43b5d0a18a-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-l6wqz\" (UID: \"5cc617a4-ee55-4218-9907-1f43b5d0a18a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-l6wqz"
Oct 31 19:27:49.191075 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:49.191019    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kch7w\" (UniqueName: \"kubernetes.io/projected/5cc617a4-ee55-4218-9907-1f43b5d0a18a-kube-api-access-kch7w\") pod \"maas-default-gateway-openshift-default-75d7c785f-l6wqz\" (UID: \"5cc617a4-ee55-4218-9907-1f43b5d0a18a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-l6wqz"
Oct 31 19:27:49.191132 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:49.191092    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cc617a4-ee55-4218-9907-1f43b5d0a18a-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-l6wqz\" (UID: \"5cc617a4-ee55-4218-9907-1f43b5d0a18a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-l6wqz"
Oct 31 19:27:49.191191 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:49.191129    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cc617a4-ee55-4218-9907-1f43b5d0a18a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-l6wqz\" (UID: \"5cc617a4-ee55-4218-9907-1f43b5d0a18a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-l6wqz"
Oct 31 19:27:49.191191 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:49.191170    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cc617a4-ee55-4218-9907-1f43b5d0a18a-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-l6wqz\" (UID: \"5cc617a4-ee55-4218-9907-1f43b5d0a18a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-l6wqz"
Oct 31 19:27:49.191293 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:49.191199    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cc617a4-ee55-4218-9907-1f43b5d0a18a-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-l6wqz\" (UID: \"5cc617a4-ee55-4218-9907-1f43b5d0a18a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-l6wqz"
Oct 31 19:27:49.191293 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:49.191223    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cc617a4-ee55-4218-9907-1f43b5d0a18a-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-l6wqz\" (UID: \"5cc617a4-ee55-4218-9907-1f43b5d0a18a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-l6wqz"
Oct 31 19:27:49.191424 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:49.191300    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cc617a4-ee55-4218-9907-1f43b5d0a18a-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-l6wqz\" (UID: \"5cc617a4-ee55-4218-9907-1f43b5d0a18a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-l6wqz"
Oct 31 19:27:49.191593 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:49.191568    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cc617a4-ee55-4218-9907-1f43b5d0a18a-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-l6wqz\" (UID: \"5cc617a4-ee55-4218-9907-1f43b5d0a18a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-l6wqz"
Oct 31 19:27:49.191642 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:49.191606    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cc617a4-ee55-4218-9907-1f43b5d0a18a-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-l6wqz\" (UID: \"5cc617a4-ee55-4218-9907-1f43b5d0a18a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-l6wqz"
Oct 31 19:27:49.191676 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:49.191659    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cc617a4-ee55-4218-9907-1f43b5d0a18a-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-l6wqz\" (UID: \"5cc617a4-ee55-4218-9907-1f43b5d0a18a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-l6wqz"
Oct 31 19:27:49.191749 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:49.191726    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cc617a4-ee55-4218-9907-1f43b5d0a18a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-l6wqz\" (UID: \"5cc617a4-ee55-4218-9907-1f43b5d0a18a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-l6wqz"
Oct 31 19:27:49.193188 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:49.193165    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cc617a4-ee55-4218-9907-1f43b5d0a18a-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-l6wqz\" (UID: \"5cc617a4-ee55-4218-9907-1f43b5d0a18a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-l6wqz"
Oct 31 19:27:49.193856 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:49.193832    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cc617a4-ee55-4218-9907-1f43b5d0a18a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-l6wqz\" (UID: \"5cc617a4-ee55-4218-9907-1f43b5d0a18a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-l6wqz"
Oct 31 19:27:49.202642 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:49.202619    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cc617a4-ee55-4218-9907-1f43b5d0a18a-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-l6wqz\" (UID: \"5cc617a4-ee55-4218-9907-1f43b5d0a18a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-l6wqz"
Oct 31 19:27:49.202937 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:49.202915    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kch7w\" (UniqueName: \"kubernetes.io/projected/5cc617a4-ee55-4218-9907-1f43b5d0a18a-kube-api-access-kch7w\") pod \"maas-default-gateway-openshift-default-75d7c785f-l6wqz\" (UID: \"5cc617a4-ee55-4218-9907-1f43b5d0a18a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-l6wqz"
Oct 31 19:27:49.339534 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:49.339476    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-l6wqz"
Oct 31 19:27:49.618379 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:49.618357    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-l6wqz"]
Oct 31 19:27:49.620175 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:27:49.620147    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cc617a4_ee55_4218_9907_1f43b5d0a18a.slice/crio-1b01726061b9fa14bd5dc8abd9590b2b8c6f83d357c826bb033ce161a9a189b7 WatchSource:0}: Error finding container 1b01726061b9fa14bd5dc8abd9590b2b8c6f83d357c826bb033ce161a9a189b7: Status 404 returned error can't find the container with id 1b01726061b9fa14bd5dc8abd9590b2b8c6f83d357c826bb033ce161a9a189b7
Oct 31 19:27:49.623189 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:49.623160    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:27:49.623257 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:49.623221    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:27:49.623257 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:49.623249    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:27:49.742499 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:49.742468    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-l6wqz" event={"ID":"5cc617a4-ee55-4218-9907-1f43b5d0a18a","Type":"ContainerStarted","Data":"1ee327e16fe4e3f2196d2d2d2d355098434730dae143457b2c46e503404fa44a"}
Oct 31 19:27:49.742641 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:49.742520    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-l6wqz" event={"ID":"5cc617a4-ee55-4218-9907-1f43b5d0a18a","Type":"ContainerStarted","Data":"1b01726061b9fa14bd5dc8abd9590b2b8c6f83d357c826bb033ce161a9a189b7"}
Oct 31 19:27:49.765355 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:49.765297    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-l6wqz" podStartSLOduration=1.7652805539999998 podStartE2EDuration="1.765280554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:27:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:27:49.764249076 +0000 UTC m=+500.337483206" watchObservedRunningTime="2026-10-31 19:27:49.765280554 +0000 UTC m=+500.338514682"
Oct 31 19:27:50.340006 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:50.339970    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-l6wqz"
Oct 31 19:27:50.341668 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:50.341644    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-l6wqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Oct 31 19:27:50.341789 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:50.341686    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-l6wqz" podUID="5cc617a4-ee55-4218-9907-1f43b5d0a18a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Oct 31 19:27:51.340844 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:51.340806    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-l6wqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Oct 31 19:27:51.341223 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:51.340859    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-l6wqz" podUID="5cc617a4-ee55-4218-9907-1f43b5d0a18a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Oct 31 19:27:52.340581 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:52.340548    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-l6wqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Oct 31 19:27:52.340742 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:52.340600    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-l6wqz" podUID="5cc617a4-ee55-4218-9907-1f43b5d0a18a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Oct 31 19:27:53.340342 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:53.340287    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-l6wqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Oct 31 19:27:53.340764 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:53.340369    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-l6wqz" podUID="5cc617a4-ee55-4218-9907-1f43b5d0a18a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Oct 31 19:27:54.340130 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:54.340094    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-l6wqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Oct 31 19:27:54.340313 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:54.340142    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-l6wqz" podUID="5cc617a4-ee55-4218-9907-1f43b5d0a18a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Oct 31 19:27:55.339770 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:55.339736    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-l6wqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Oct 31 19:27:55.340149 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:55.339786    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-l6wqz" podUID="5cc617a4-ee55-4218-9907-1f43b5d0a18a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Oct 31 19:27:56.340778 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:56.340741    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-l6wqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Oct 31 19:27:56.341147 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:56.340824    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-l6wqz" podUID="5cc617a4-ee55-4218-9907-1f43b5d0a18a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Oct 31 19:27:57.864294 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:57.864260    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-l6wqz"
Oct 31 19:27:57.864756 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:57.864693    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-l6wqz"
Oct 31 19:27:57.865379 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:57.865359    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-l6wqz"
Oct 31 19:27:58.832259 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:58.832206    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-559d65ddd-gx66l"]
Oct 31 19:27:58.835974 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:58.835954    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-559d65ddd-gx66l"
Oct 31 19:27:58.846292 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:58.846267    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-559d65ddd-gx66l"]
Oct 31 19:27:58.873785 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:58.873756    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c8726e82-9df2-47ab-bfbc-0babdddf5ee0-tls-cert\") pod \"authorino-559d65ddd-gx66l\" (UID: \"c8726e82-9df2-47ab-bfbc-0babdddf5ee0\") " pod="kuadrant-system/authorino-559d65ddd-gx66l"
Oct 31 19:27:58.874119 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:58.873787    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb779\" (UniqueName: \"kubernetes.io/projected/c8726e82-9df2-47ab-bfbc-0babdddf5ee0-kube-api-access-nb779\") pod \"authorino-559d65ddd-gx66l\" (UID: \"c8726e82-9df2-47ab-bfbc-0babdddf5ee0\") " pod="kuadrant-system/authorino-559d65ddd-gx66l"
Oct 31 19:27:58.974765 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:58.974735    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c8726e82-9df2-47ab-bfbc-0babdddf5ee0-tls-cert\") pod \"authorino-559d65ddd-gx66l\" (UID: \"c8726e82-9df2-47ab-bfbc-0babdddf5ee0\") " pod="kuadrant-system/authorino-559d65ddd-gx66l"
Oct 31 19:27:58.974854 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:58.974770    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb779\" (UniqueName: \"kubernetes.io/projected/c8726e82-9df2-47ab-bfbc-0babdddf5ee0-kube-api-access-nb779\") pod \"authorino-559d65ddd-gx66l\" (UID: \"c8726e82-9df2-47ab-bfbc-0babdddf5ee0\") " pod="kuadrant-system/authorino-559d65ddd-gx66l"
Oct 31 19:27:58.977260 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:58.977241    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c8726e82-9df2-47ab-bfbc-0babdddf5ee0-tls-cert\") pod \"authorino-559d65ddd-gx66l\" (UID: \"c8726e82-9df2-47ab-bfbc-0babdddf5ee0\") " pod="kuadrant-system/authorino-559d65ddd-gx66l"
Oct 31 19:27:58.986486 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:58.986463    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb779\" (UniqueName: \"kubernetes.io/projected/c8726e82-9df2-47ab-bfbc-0babdddf5ee0-kube-api-access-nb779\") pod \"authorino-559d65ddd-gx66l\" (UID: \"c8726e82-9df2-47ab-bfbc-0babdddf5ee0\") " pod="kuadrant-system/authorino-559d65ddd-gx66l"
Oct 31 19:27:59.147771 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:59.147720    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-559d65ddd-gx66l"
Oct 31 19:27:59.198210 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:59.197233    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:27:59.302692 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:59.302660    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:27:59.425364 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:59.425316    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-559d65ddd-gx66l"]
Oct 31 19:27:59.427003 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:27:59.426975    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc8726e82_9df2_47ab_bfbc_0babdddf5ee0.slice/crio-f15a88ef487b3aa61ca9ddbc93ff2c5d7d535a517a5f4c9eac9016b72d374b9f WatchSource:0}: Error finding container f15a88ef487b3aa61ca9ddbc93ff2c5d7d535a517a5f4c9eac9016b72d374b9f: Status 404 returned error can't find the container with id f15a88ef487b3aa61ca9ddbc93ff2c5d7d535a517a5f4c9eac9016b72d374b9f
Oct 31 19:27:59.809006 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:27:59.808928    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-559d65ddd-gx66l" event={"ID":"c8726e82-9df2-47ab-bfbc-0babdddf5ee0","Type":"ContainerStarted","Data":"f15a88ef487b3aa61ca9ddbc93ff2c5d7d535a517a5f4c9eac9016b72d374b9f"}
Oct 31 19:28:00.816769 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:00.816687    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-559d65ddd-gx66l" event={"ID":"c8726e82-9df2-47ab-bfbc-0babdddf5ee0","Type":"ContainerStarted","Data":"3423a4af46bde566cf9a994b76786c0d696e920e66c7c05cc68fc9c53b79bbfc"}
Oct 31 19:28:00.845497 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:00.845445    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-559d65ddd-gx66l" podStartSLOduration=1.800412953 podStartE2EDuration="2.845431251s" totalImagesPullingTime="1.045018298s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:27:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:28:00.843756019 +0000 UTC m=+511.416990150" watchObservedRunningTime="2026-10-31 19:28:00.845431251 +0000 UTC m=+511.418665379"
Oct 31 19:28:00.920577 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:00.916195    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7dd6b9667f-kwrcm"]
Oct 31 19:28:00.920577 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:00.916435    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-7dd6b9667f-kwrcm" podUID="83610c38-d76f-4ffa-8bcd-2c8230b4988a" containerName="authorino" containerID="cri-o://cf019030856932613fdd6f65ce414ef1fc5b0129201b311f40d231127393a668" gracePeriod=30
Oct 31 19:28:01.245871 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:01.245841    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7dd6b9667f-kwrcm"
Oct 31 19:28:01.331072 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:01.330640    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83610c38-d76f-4ffa-8bcd-2c8230b4988a-kube-api-access-xrmzh\" (UniqueName: \"kubernetes.io/projected/83610c38-d76f-4ffa-8bcd-2c8230b4988a-kube-api-access-xrmzh\") pod \"83610c38-d76f-4ffa-8bcd-2c8230b4988a\" (UID: \"83610c38-d76f-4ffa-8bcd-2c8230b4988a\") "
Oct 31 19:28:01.331072 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:01.330700    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83610c38-d76f-4ffa-8bcd-2c8230b4988a-tls-cert\" (UniqueName: \"kubernetes.io/secret/83610c38-d76f-4ffa-8bcd-2c8230b4988a-tls-cert\") pod \"83610c38-d76f-4ffa-8bcd-2c8230b4988a\" (UID: \"83610c38-d76f-4ffa-8bcd-2c8230b4988a\") "
Oct 31 19:28:01.333087 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:01.333051    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83610c38-d76f-4ffa-8bcd-2c8230b4988a-kube-api-access-xrmzh" pod "83610c38-d76f-4ffa-8bcd-2c8230b4988a" (UID: "83610c38-d76f-4ffa-8bcd-2c8230b4988a"). InnerVolumeSpecName "kube-api-access-xrmzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:28:01.340852 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:01.340826    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83610c38-d76f-4ffa-8bcd-2c8230b4988a-tls-cert" pod "83610c38-d76f-4ffa-8bcd-2c8230b4988a" (UID: "83610c38-d76f-4ffa-8bcd-2c8230b4988a"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:28:01.432438 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:01.432385    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrmzh\" (UniqueName: \"kubernetes.io/projected/83610c38-d76f-4ffa-8bcd-2c8230b4988a-kube-api-access-xrmzh\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:28:01.432438 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:01.432416    4645 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/83610c38-d76f-4ffa-8bcd-2c8230b4988a-tls-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:28:01.450985 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:01.450949    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:28:01.464672 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:01.464610    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-5f6dd4f69f-v2znx"]
Oct 31 19:28:01.466842 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:01.465572    4645 cpu_manager.go:414] "containerMap: removing container" podUID="83610c38-d76f-4ffa-8bcd-2c8230b4988a" containerName="authorino"
Oct 31 19:28:01.466842 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:01.465597    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83610c38-d76f-4ffa-8bcd-2c8230b4988a" containerName="authorino"
Oct 31 19:28:01.466842 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:01.465696    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="83610c38-d76f-4ffa-8bcd-2c8230b4988a" containerName="authorino"
Oct 31 19:28:01.490756 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:01.489698    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-v2znx"
Oct 31 19:28:01.494819 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:01.494215    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5f6dd4f69f-v2znx"]
Oct 31 19:28:01.509640 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:01.505576    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Oct 31 19:28:01.509640 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:01.505639    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Oct 31 19:28:01.509640 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:01.505746    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-nmd82\""
Oct 31 19:28:01.629836 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:01.629801    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:28:01.634202 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:01.634174    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd28x\" (UniqueName: \"kubernetes.io/projected/c8470944-e312-44f9-8d73-0e37062db79e-kube-api-access-hd28x\") pod \"ai-gateway-controller-5f6dd4f69f-v2znx\" (UID: \"c8470944-e312-44f9-8d73-0e37062db79e\") " pod="opendatahub/ai-gateway-controller-5f6dd4f69f-v2znx"
Oct 31 19:28:01.634307 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:01.634224    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8470944-e312-44f9-8d73-0e37062db79e-metrics-certs\") pod \"ai-gateway-controller-5f6dd4f69f-v2znx\" (UID: \"c8470944-e312-44f9-8d73-0e37062db79e\") " pod="opendatahub/ai-gateway-controller-5f6dd4f69f-v2znx"
Oct 31 19:28:01.735181 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:01.735088    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd28x\" (UniqueName: \"kubernetes.io/projected/c8470944-e312-44f9-8d73-0e37062db79e-kube-api-access-hd28x\") pod \"ai-gateway-controller-5f6dd4f69f-v2znx\" (UID: \"c8470944-e312-44f9-8d73-0e37062db79e\") " pod="opendatahub/ai-gateway-controller-5f6dd4f69f-v2znx"
Oct 31 19:28:01.735367 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:01.735235    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8470944-e312-44f9-8d73-0e37062db79e-metrics-certs\") pod \"ai-gateway-controller-5f6dd4f69f-v2znx\" (UID: \"c8470944-e312-44f9-8d73-0e37062db79e\") " pod="opendatahub/ai-gateway-controller-5f6dd4f69f-v2znx"
Oct 31 19:28:01.738014 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:01.737988    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8470944-e312-44f9-8d73-0e37062db79e-metrics-certs\") pod \"ai-gateway-controller-5f6dd4f69f-v2znx\" (UID: \"c8470944-e312-44f9-8d73-0e37062db79e\") " pod="opendatahub/ai-gateway-controller-5f6dd4f69f-v2znx"
Oct 31 19:28:01.748007 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:01.747981    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd28x\" (UniqueName: \"kubernetes.io/projected/c8470944-e312-44f9-8d73-0e37062db79e-kube-api-access-hd28x\") pod \"ai-gateway-controller-5f6dd4f69f-v2znx\" (UID: \"c8470944-e312-44f9-8d73-0e37062db79e\") " pod="opendatahub/ai-gateway-controller-5f6dd4f69f-v2znx"
Oct 31 19:28:01.823558 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:01.823251    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-v2znx"
Oct 31 19:28:01.825460 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:01.825429    4645 generic.go:356] "Generic (PLEG): container finished" podID="83610c38-d76f-4ffa-8bcd-2c8230b4988a" containerID="cf019030856932613fdd6f65ce414ef1fc5b0129201b311f40d231127393a668" exitCode=0
Oct 31 19:28:01.825596 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:01.825486    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7dd6b9667f-kwrcm"
Oct 31 19:28:01.825596 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:01.825505    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7dd6b9667f-kwrcm" event={"ID":"83610c38-d76f-4ffa-8bcd-2c8230b4988a","Type":"ContainerDied","Data":"cf019030856932613fdd6f65ce414ef1fc5b0129201b311f40d231127393a668"}
Oct 31 19:28:01.825596 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:01.825549    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7dd6b9667f-kwrcm" event={"ID":"83610c38-d76f-4ffa-8bcd-2c8230b4988a","Type":"ContainerDied","Data":"e7bc2849a4313f3035cce380c17e54208ba3a23e0ddf3d2bebb0862caf1a2cc6"}
Oct 31 19:28:01.825596 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:01.825572    4645 scope.go:122] "RemoveContainer" containerID="cf019030856932613fdd6f65ce414ef1fc5b0129201b311f40d231127393a668"
Oct 31 19:28:01.837938 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:01.837907    4645 scope.go:122] "RemoveContainer" containerID="cf019030856932613fdd6f65ce414ef1fc5b0129201b311f40d231127393a668"
Oct 31 19:28:01.838217 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:28:01.838189    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf019030856932613fdd6f65ce414ef1fc5b0129201b311f40d231127393a668\": container with ID starting with cf019030856932613fdd6f65ce414ef1fc5b0129201b311f40d231127393a668 not found: ID does not exist" containerID="cf019030856932613fdd6f65ce414ef1fc5b0129201b311f40d231127393a668"
Oct 31 19:28:01.838300 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:01.838216    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf019030856932613fdd6f65ce414ef1fc5b0129201b311f40d231127393a668"} err="failed to get container status \"cf019030856932613fdd6f65ce414ef1fc5b0129201b311f40d231127393a668\": rpc error: code = NotFound desc = could not find container \"cf019030856932613fdd6f65ce414ef1fc5b0129201b311f40d231127393a668\": container with ID starting with cf019030856932613fdd6f65ce414ef1fc5b0129201b311f40d231127393a668 not found: ID does not exist"
Oct 31 19:28:01.845393 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:01.845355    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7dd6b9667f-kwrcm"]
Oct 31 19:28:01.849609 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:01.849581    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7dd6b9667f-kwrcm"]
Oct 31 19:28:02.155257 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:02.155233    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5f6dd4f69f-v2znx"]
Oct 31 19:28:02.162395 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:28:02.161718    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8470944_e312_44f9_8d73_0e37062db79e.slice/crio-9a4512aaf7dc560617052f04e48671e8776753a4c78d9684c85c9a6f2a989012 WatchSource:0}: Error finding container 9a4512aaf7dc560617052f04e48671e8776753a4c78d9684c85c9a6f2a989012: Status 404 returned error can't find the container with id 9a4512aaf7dc560617052f04e48671e8776753a4c78d9684c85c9a6f2a989012
Oct 31 19:28:02.358542 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:02.358482    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-msrfk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:28:02.358741 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:02.358560    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-msrfk" podUID="48b0949c-0c89-4c80-a1dc-64d14839f620" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:28:02.831396 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:02.831361    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-v2znx" event={"ID":"c8470944-e312-44f9-8d73-0e37062db79e","Type":"ContainerStarted","Data":"9a4512aaf7dc560617052f04e48671e8776753a4c78d9684c85c9a6f2a989012"}
Oct 31 19:28:03.616625 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:03.616590    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83610c38-d76f-4ffa-8bcd-2c8230b4988a" path="/var/lib/kubelet/pods/83610c38-d76f-4ffa-8bcd-2c8230b4988a/volumes"
Oct 31 19:28:04.043311 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:04.043275    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9th7f"]
Oct 31 19:28:04.046985 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:04.046963    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9th7f"
Oct 31 19:28:04.062801 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:04.062739    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9th7f"]
Oct 31 19:28:04.077033 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:04.076662    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6k6cd"]
Oct 31 19:28:04.081824 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:04.080489    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6k6cd"
Oct 31 19:28:04.099037 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:04.098970    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6k6cd"]
Oct 31 19:28:04.163807 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:04.163774    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76bc1f3a-74b3-49b3-bde1-cc3b305260d6-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-9th7f\" (UID: \"76bc1f3a-74b3-49b3-bde1-cc3b305260d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9th7f"
Oct 31 19:28:04.163971 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:04.163813    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8fca631-527d-46c0-9197-5edd7762adb9-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-6k6cd\" (UID: \"a8fca631-527d-46c0-9197-5edd7762adb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6k6cd"
Oct 31 19:28:04.163971 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:04.163843    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76bc1f3a-74b3-49b3-bde1-cc3b305260d6-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-9th7f\" (UID: \"76bc1f3a-74b3-49b3-bde1-cc3b305260d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9th7f"
Oct 31 19:28:04.164057 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:04.163969    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v574w\" (UniqueName: \"kubernetes.io/projected/76bc1f3a-74b3-49b3-bde1-cc3b305260d6-kube-api-access-v574w\") pod \"maas-default-gateway-openshift-default-75d7c785f-9th7f\" (UID: \"76bc1f3a-74b3-49b3-bde1-cc3b305260d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9th7f"
Oct 31 19:28:04.164057 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:04.164000    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8fca631-527d-46c0-9197-5edd7762adb9-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-6k6cd\" (UID: \"a8fca631-527d-46c0-9197-5edd7762adb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6k6cd"
Oct 31 19:28:04.164140 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:04.164064    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8fca631-527d-46c0-9197-5edd7762adb9-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-6k6cd\" (UID: \"a8fca631-527d-46c0-9197-5edd7762adb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6k6cd"
Oct 31 19:28:04.164140 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:04.164098    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76bc1f3a-74b3-49b3-bde1-cc3b305260d6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-9th7f\" (UID: \"76bc1f3a-74b3-49b3-bde1-cc3b305260d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9th7f"
Oct 31 19:28:04.164140 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:04.164121    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76bc1f3a-74b3-49b3-bde1-cc3b305260d6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-9th7f\" (UID: \"76bc1f3a-74b3-49b3-bde1-cc3b305260d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9th7f"
Oct 31 19:28:04.164244 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:04.164164    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8fca631-527d-46c0-9197-5edd7762adb9-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-6k6cd\" (UID: \"a8fca631-527d-46c0-9197-5edd7762adb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6k6cd"
Oct 31 19:28:04.164244 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:04.164205    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76bc1f3a-74b3-49b3-bde1-cc3b305260d6-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-9th7f\" (UID: \"76bc1f3a-74b3-49b3-bde1-cc3b305260d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9th7f"
Oct 31 19:28:04.164313 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:04.164251    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8fca631-527d-46c0-9197-5edd7762adb9-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-6k6cd\" (UID: \"a8fca631-527d-46c0-9197-5edd7762adb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6k6cd"
Oct 31 19:28:04.164313 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:04.164281    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76bc1f3a-74b3-49b3-bde1-cc3b305260d6-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-9th7f\" (UID: \"76bc1f3a-74b3-49b3-bde1-cc3b305260d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9th7f"
Oct 31 19:28:04.164404 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:04.164321    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76bc1f3a-74b3-49b3-bde1-cc3b305260d6-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-9th7f\" (UID: \"76bc1f3a-74b3-49b3-bde1-cc3b305260d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9th7f"
Oct 31 19:28:04.164404 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:04.164359    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8fca631-527d-46c0-9197-5edd7762adb9-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-6k6cd\" (UID: \"a8fca631-527d-46c0-9197-5edd7762adb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6k6cd"
Oct 31 19:28:04.164473 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:04.164404    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8fca631-527d-46c0-9197-5edd7762adb9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-6k6cd\" (UID: \"a8fca631-527d-46c0-9197-5edd7762adb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6k6cd"
Oct 31 19:28:04.164473 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:04.164430    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4m458\" (UniqueName: \"kubernetes.io/projected/a8fca631-527d-46c0-9197-5edd7762adb9-kube-api-access-4m458\") pod \"maas-default-gateway-openshift-default-75d7c785f-6k6cd\" (UID: \"a8fca631-527d-46c0-9197-5edd7762adb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6k6cd"
Oct 31 19:28:04.164473 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:04.164446    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76bc1f3a-74b3-49b3-bde1-cc3b305260d6-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-9th7f\" (UID: \"76bc1f3a-74b3-49b3-bde1-cc3b305260d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9th7f"
Oct 31 19:28:04.164473 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:04.164461    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8fca631-527d-46c0-9197-5edd7762adb9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-6k6cd\" (UID: \"a8fca631-527d-46c0-9197-5edd7762adb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6k6cd"
Oct 31 19:28:04.265411 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:04.265375    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m458\" (UniqueName: \"kubernetes.io/projected/a8fca631-527d-46c0-9197-5edd7762adb9-kube-api-access-4m458\") pod \"maas-default-gateway-openshift-default-75d7c785f-6k6cd\" (UID: \"a8fca631-527d-46c0-9197-5edd7762adb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6k6cd"
Oct 31 19:28:04.265411 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:04.265414    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76bc1f3a-74b3-49b3-bde1-cc3b305260d6-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-9th7f\" (UID: \"76bc1f3a-74b3-49b3-bde1-cc3b305260d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9th7f"
Oct 31 19:28:04.265676 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:04.265435    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8fca631-527d-46c0-9197-5edd7762adb9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-6k6cd\" (UID: \"a8fca631-527d-46c0-9197-5edd7762adb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6k6cd"
Oct 31 19:28:04.265676 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:04.265462    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76bc1f3a-74b3-49b3-bde1-cc3b305260d6-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-9th7f\" (UID: \"76bc1f3a-74b3-49b3-bde1-cc3b305260d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9th7f"
Oct 31 19:28:04.265676 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:04.265493    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8fca631-527d-46c0-9197-5edd7762adb9-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-6k6cd\" (UID: \"a8fca631-527d-46c0-9197-5edd7762adb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6k6cd"
Oct 31 19:28:04.265676 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:04.265523    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76bc1f3a-74b3-49b3-bde1-cc3b305260d6-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-9th7f\" (UID: \"76bc1f3a-74b3-49b3-bde1-cc3b305260d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9th7f"
Oct 31 19:28:04.265676 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:04.265557    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v574w\" (UniqueName: \"kubernetes.io/projected/76bc1f3a-74b3-49b3-bde1-cc3b305260d6-kube-api-access-v574w\") pod \"maas-default-gateway-openshift-default-75d7c785f-9th7f\" (UID: \"76bc1f3a-74b3-49b3-bde1-cc3b305260d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9th7f"
Oct 31 19:28:04.265676 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:04.265583    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8fca631-527d-46c0-9197-5edd7762adb9-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-6k6cd\" (UID: \"a8fca631-527d-46c0-9197-5edd7762adb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6k6cd"
Oct 31 19:28:04.265676 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:04.265621    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8fca631-527d-46c0-9197-5edd7762adb9-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-6k6cd\" (UID: \"a8fca631-527d-46c0-9197-5edd7762adb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6k6cd"
Oct 31 19:28:04.265676 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:04.265657    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76bc1f3a-74b3-49b3-bde1-cc3b305260d6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-9th7f\" (UID: \"76bc1f3a-74b3-49b3-bde1-cc3b305260d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9th7f"
Oct 31 19:28:04.266078 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:04.265694    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76bc1f3a-74b3-49b3-bde1-cc3b305260d6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-9th7f\" (UID: \"76bc1f3a-74b3-49b3-bde1-cc3b305260d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9th7f"
Oct 31 19:28:04.266078 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:04.265727    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8fca631-527d-46c0-9197-5edd7762adb9-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-6k6cd\" (UID: \"a8fca631-527d-46c0-9197-5edd7762adb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6k6cd"
Oct 31 19:28:04.266078 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:04.265774    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76bc1f3a-74b3-49b3-bde1-cc3b305260d6-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-9th7f\" (UID: \"76bc1f3a-74b3-49b3-bde1-cc3b305260d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9th7f"
Oct 31 19:28:04.266078 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:04.265813    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8fca631-527d-46c0-9197-5edd7762adb9-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-6k6cd\" (UID: \"a8fca631-527d-46c0-9197-5edd7762adb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6k6cd"
Oct 31 19:28:04.266078 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:04.265844    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76bc1f3a-74b3-49b3-bde1-cc3b305260d6-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-9th7f\" (UID: \"76bc1f3a-74b3-49b3-bde1-cc3b305260d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9th7f"
Oct 31 19:28:04.266078 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:04.265904    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76bc1f3a-74b3-49b3-bde1-cc3b305260d6-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-9th7f\" (UID: \"76bc1f3a-74b3-49b3-bde1-cc3b305260d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9th7f"
Oct 31 19:28:04.266078 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:04.266054    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8fca631-527d-46c0-9197-5edd7762adb9-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-6k6cd\" (UID: \"a8fca631-527d-46c0-9197-5edd7762adb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6k6cd"
Oct 31 19:28:04.266078 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:04.265913    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76bc1f3a-74b3-49b3-bde1-cc3b305260d6-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-9th7f\" (UID: \"76bc1f3a-74b3-49b3-bde1-cc3b305260d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9th7f"
Oct 31 19:28:04.266483 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:04.266094    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8fca631-527d-46c0-9197-5edd7762adb9-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-6k6cd\" (UID: \"a8fca631-527d-46c0-9197-5edd7762adb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6k6cd"
Oct 31 19:28:04.266483 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:04.266128    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8fca631-527d-46c0-9197-5edd7762adb9-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-6k6cd\" (UID: \"a8fca631-527d-46c0-9197-5edd7762adb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6k6cd"
Oct 31 19:28:04.266483 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:04.266168    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8fca631-527d-46c0-9197-5edd7762adb9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-6k6cd\" (UID: \"a8fca631-527d-46c0-9197-5edd7762adb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6k6cd"
Oct 31 19:28:04.266483 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:04.266321    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8fca631-527d-46c0-9197-5edd7762adb9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-6k6cd\" (UID: \"a8fca631-527d-46c0-9197-5edd7762adb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6k6cd"
Oct 31 19:28:04.266684 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:04.266587    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76bc1f3a-74b3-49b3-bde1-cc3b305260d6-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-9th7f\" (UID: \"76bc1f3a-74b3-49b3-bde1-cc3b305260d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9th7f"
Oct 31 19:28:04.266684 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:04.266590    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76bc1f3a-74b3-49b3-bde1-cc3b305260d6-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-9th7f\" (UID: \"76bc1f3a-74b3-49b3-bde1-cc3b305260d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9th7f"
Oct 31 19:28:04.266684 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:04.266639    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76bc1f3a-74b3-49b3-bde1-cc3b305260d6-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-9th7f\" (UID: \"76bc1f3a-74b3-49b3-bde1-cc3b305260d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9th7f"
Oct 31 19:28:04.266684 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:04.266674    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8fca631-527d-46c0-9197-5edd7762adb9-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-6k6cd\" (UID: \"a8fca631-527d-46c0-9197-5edd7762adb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6k6cd"
Oct 31 19:28:04.266897 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:04.266757    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8fca631-527d-46c0-9197-5edd7762adb9-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-6k6cd\" (UID: \"a8fca631-527d-46c0-9197-5edd7762adb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6k6cd"
Oct 31 19:28:04.267108 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:04.267084    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76bc1f3a-74b3-49b3-bde1-cc3b305260d6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-9th7f\" (UID: \"76bc1f3a-74b3-49b3-bde1-cc3b305260d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9th7f"
Oct 31 19:28:04.268683 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:04.268656    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76bc1f3a-74b3-49b3-bde1-cc3b305260d6-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-9th7f\" (UID: \"76bc1f3a-74b3-49b3-bde1-cc3b305260d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9th7f"
Oct 31 19:28:04.269059 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:04.269033    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8fca631-527d-46c0-9197-5edd7762adb9-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-6k6cd\" (UID: \"a8fca631-527d-46c0-9197-5edd7762adb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6k6cd"
Oct 31 19:28:04.269137 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:04.269108    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8fca631-527d-46c0-9197-5edd7762adb9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-6k6cd\" (UID: \"a8fca631-527d-46c0-9197-5edd7762adb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6k6cd"
Oct 31 19:28:04.269197 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:04.269181    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76bc1f3a-74b3-49b3-bde1-cc3b305260d6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-9th7f\" (UID: \"76bc1f3a-74b3-49b3-bde1-cc3b305260d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9th7f"
Oct 31 19:28:04.281515 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:04.281486    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76bc1f3a-74b3-49b3-bde1-cc3b305260d6-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-9th7f\" (UID: \"76bc1f3a-74b3-49b3-bde1-cc3b305260d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9th7f"
Oct 31 19:28:04.281642 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:04.281620    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m458\" (UniqueName: \"kubernetes.io/projected/a8fca631-527d-46c0-9197-5edd7762adb9-kube-api-access-4m458\") pod \"maas-default-gateway-openshift-default-75d7c785f-6k6cd\" (UID: \"a8fca631-527d-46c0-9197-5edd7762adb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6k6cd"
Oct 31 19:28:04.282184 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:04.282166    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8fca631-527d-46c0-9197-5edd7762adb9-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-6k6cd\" (UID: \"a8fca631-527d-46c0-9197-5edd7762adb9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6k6cd"
Oct 31 19:28:04.286549 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:04.286523    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v574w\" (UniqueName: \"kubernetes.io/projected/76bc1f3a-74b3-49b3-bde1-cc3b305260d6-kube-api-access-v574w\") pod \"maas-default-gateway-openshift-default-75d7c785f-9th7f\" (UID: \"76bc1f3a-74b3-49b3-bde1-cc3b305260d6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9th7f"
Oct 31 19:28:04.360151 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:04.360065    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9th7f"
Oct 31 19:28:04.394424 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:04.394391    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6k6cd"
Oct 31 19:28:04.652673 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:04.652538    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9th7f"]
Oct 31 19:28:04.658477 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:28:04.658446    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76bc1f3a_74b3_49b3_bde1_cc3b305260d6.slice/crio-74f01100ae9d6e4cd9d6ca79a35d7def9de1fc822c2afcaa12e4d3f15865572a WatchSource:0}: Error finding container 74f01100ae9d6e4cd9d6ca79a35d7def9de1fc822c2afcaa12e4d3f15865572a: Status 404 returned error can't find the container with id 74f01100ae9d6e4cd9d6ca79a35d7def9de1fc822c2afcaa12e4d3f15865572a
Oct 31 19:28:04.660588 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:04.660542    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:28:04.660701 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:04.660633    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:28:04.660701 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:04.660681    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:28:04.705247 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:04.702884    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6k6cd"]
Oct 31 19:28:04.717729 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:04.717283    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:28:04.717729 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:04.717365    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:28:04.717729 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:04.717396    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:28:04.845351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:04.844710    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9th7f" event={"ID":"76bc1f3a-74b3-49b3-bde1-cc3b305260d6","Type":"ContainerStarted","Data":"2f0c74238d8b1117dda0cd0bb0aada4ee7f7ac6057576ed3a76b0fc2162f754e"}
Oct 31 19:28:04.845351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:04.844761    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9th7f" event={"ID":"76bc1f3a-74b3-49b3-bde1-cc3b305260d6","Type":"ContainerStarted","Data":"74f01100ae9d6e4cd9d6ca79a35d7def9de1fc822c2afcaa12e4d3f15865572a"}
Oct 31 19:28:04.849710 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:04.848399    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6k6cd" event={"ID":"a8fca631-527d-46c0-9197-5edd7762adb9","Type":"ContainerStarted","Data":"c32d0cfd6f6abb7c3db39cdc98f6040183989685f651797cfccbc4225e3e4835"}
Oct 31 19:28:04.849710 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:04.848456    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6k6cd" event={"ID":"a8fca631-527d-46c0-9197-5edd7762adb9","Type":"ContainerStarted","Data":"dbc30710aab6cec5232bf550442aa7007e18d311211c26f203deea5efa6c6a60"}
Oct 31 19:28:04.871950 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:04.871253    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9th7f" podStartSLOduration=0.871235099 podStartE2EDuration="871.235099ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:28:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:28:04.868374908 +0000 UTC m=+515.441609041" watchObservedRunningTime="2026-10-31 19:28:04.871235099 +0000 UTC m=+515.444469229"
Oct 31 19:28:04.892189 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:04.891441    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6k6cd" podStartSLOduration=0.891423461 podStartE2EDuration="891.423461ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:28:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:28:04.889225702 +0000 UTC m=+515.462459831" watchObservedRunningTime="2026-10-31 19:28:04.891423461 +0000 UTC m=+515.464657590"
Oct 31 19:28:05.367350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:05.363679    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9th7f"
Oct 31 19:28:05.371348 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:05.369448    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-9th7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Oct 31 19:28:05.371348 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:05.369505    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9th7f" podUID="76bc1f3a-74b3-49b3-bde1-cc3b305260d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Oct 31 19:28:05.404034 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:05.403944    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6k6cd"
Oct 31 19:28:05.404247 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:05.404157    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-6k6cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Oct 31 19:28:05.404247 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:05.404204    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6k6cd" podUID="a8fca631-527d-46c0-9197-5edd7762adb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Oct 31 19:28:06.247107 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:28:06.247068    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 19:28:06.374353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:06.370683    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-9th7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Oct 31 19:28:06.374353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:06.370742    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9th7f" podUID="76bc1f3a-74b3-49b3-bde1-cc3b305260d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Oct 31 19:28:06.399358 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:06.397638    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-6k6cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Oct 31 19:28:06.399358 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:06.397688    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6k6cd" podUID="a8fca631-527d-46c0-9197-5edd7762adb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Oct 31 19:28:07.372352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:07.363707    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-9th7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Oct 31 19:28:07.372352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:07.363786    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9th7f" podUID="76bc1f3a-74b3-49b3-bde1-cc3b305260d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Oct 31 19:28:07.405406 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:07.396688    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-6k6cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Oct 31 19:28:07.405406 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:07.396749    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6k6cd" podUID="a8fca631-527d-46c0-9197-5edd7762adb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Oct 31 19:28:07.922576 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:07.922533    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-v2znx" event={"ID":"c8470944-e312-44f9-8d73-0e37062db79e","Type":"ContainerStarted","Data":"6b681d0120043237bc53a68767850b41256326cd2b5d58fae03ebafc2d8c8249"}
Oct 31 19:28:07.922769 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:07.922637    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-v2znx"
Oct 31 19:28:07.941478 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:07.941424    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-v2znx" podStartSLOduration=1.9866604369999998 podStartE2EDuration="6.941410152s" totalImagesPullingTime="4.954749715s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:28:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:28:07.939616472 +0000 UTC m=+518.512850601" watchObservedRunningTime="2026-10-31 19:28:07.941410152 +0000 UTC m=+518.514644281"
Oct 31 19:28:08.360926 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:08.360832    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-9th7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Oct 31 19:28:08.366468 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:08.361153    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9th7f" podUID="76bc1f3a-74b3-49b3-bde1-cc3b305260d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Oct 31 19:28:08.405757 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:08.405720    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-6k6cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Oct 31 19:28:08.406356 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:08.406278    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6k6cd" podUID="a8fca631-527d-46c0-9197-5edd7762adb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Oct 31 19:28:09.208965 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:09.208928    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:28:09.308423 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:09.308384    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-76hjw"]
Oct 31 19:28:09.312817 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:09.312796    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-76hjw"
Oct 31 19:28:09.315368 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:09.315347    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-76hjw"]
Oct 31 19:28:09.317130 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:09.317108    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-p774g\""
Oct 31 19:28:09.320057 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:09.320028    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:28:09.352937 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:09.352912    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Oct 31 19:28:09.369351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:09.363656    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-9th7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Oct 31 19:28:09.369351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:09.363711    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9th7f" podUID="76bc1f3a-74b3-49b3-bde1-cc3b305260d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Oct 31 19:28:09.414356 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:09.395646    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-6k6cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Oct 31 19:28:09.414356 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:09.395704    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6k6cd" podUID="a8fca631-527d-46c0-9197-5edd7762adb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Oct 31 19:28:09.483938 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:09.483842    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b14ee5b2-6bf8-4b41-a84d-d0a7dcc9b85a-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-76hjw\" (UID: \"b14ee5b2-6bf8-4b41-a84d-d0a7dcc9b85a\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-76hjw"
Oct 31 19:28:09.484265 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:09.484243    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7m9v7\" (UniqueName: \"kubernetes.io/projected/b14ee5b2-6bf8-4b41-a84d-d0a7dcc9b85a-kube-api-access-7m9v7\") pod \"payload-processing-55fbbbf9dc-76hjw\" (UID: \"b14ee5b2-6bf8-4b41-a84d-d0a7dcc9b85a\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-76hjw"
Oct 31 19:28:09.602064 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:09.602020    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m9v7\" (UniqueName: \"kubernetes.io/projected/b14ee5b2-6bf8-4b41-a84d-d0a7dcc9b85a-kube-api-access-7m9v7\") pod \"payload-processing-55fbbbf9dc-76hjw\" (UID: \"b14ee5b2-6bf8-4b41-a84d-d0a7dcc9b85a\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-76hjw"
Oct 31 19:28:09.602862 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:09.602834    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b14ee5b2-6bf8-4b41-a84d-d0a7dcc9b85a-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-76hjw\" (UID: \"b14ee5b2-6bf8-4b41-a84d-d0a7dcc9b85a\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-76hjw"
Oct 31 19:28:09.604024 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:09.603924    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b14ee5b2-6bf8-4b41-a84d-d0a7dcc9b85a-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-76hjw\" (UID: \"b14ee5b2-6bf8-4b41-a84d-d0a7dcc9b85a\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-76hjw"
Oct 31 19:28:09.614023 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:09.614001    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7m9v7\" (UniqueName: \"kubernetes.io/projected/b14ee5b2-6bf8-4b41-a84d-d0a7dcc9b85a-kube-api-access-7m9v7\") pod \"payload-processing-55fbbbf9dc-76hjw\" (UID: \"b14ee5b2-6bf8-4b41-a84d-d0a7dcc9b85a\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-76hjw"
Oct 31 19:28:09.631748 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:09.631730    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-76hjw"
Oct 31 19:28:09.974353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:09.967652    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-76hjw"]
Oct 31 19:28:10.362764 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:10.362674    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-9th7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Oct 31 19:28:10.363000 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:10.362970    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9th7f" podUID="76bc1f3a-74b3-49b3-bde1-cc3b305260d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Oct 31 19:28:10.401690 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:10.401654    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-6k6cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Oct 31 19:28:10.401932 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:10.401903    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6k6cd" podUID="a8fca631-527d-46c0-9197-5edd7762adb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Oct 31 19:28:10.940672 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:10.940558    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-76hjw" event={"ID":"b14ee5b2-6bf8-4b41-a84d-d0a7dcc9b85a","Type":"ContainerStarted","Data":"c594f6faeedd9cd68574c50828c29990f4482bbf0e979f372379f1fbaa895b33"}
Oct 31 19:28:11.361055 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:11.360961    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-9th7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Oct 31 19:28:11.361055 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:11.361025    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9th7f" podUID="76bc1f3a-74b3-49b3-bde1-cc3b305260d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Oct 31 19:28:11.394802 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:11.394767    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-6k6cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Oct 31 19:28:11.394995 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:11.394827    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6k6cd" podUID="a8fca631-527d-46c0-9197-5edd7762adb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Oct 31 19:28:12.361421 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:12.361381    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-9th7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body=
Oct 31 19:28:12.361887 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:12.361451    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9th7f" podUID="76bc1f3a-74b3-49b3-bde1-cc3b305260d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused"
Oct 31 19:28:12.395471 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:12.395441    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-6k6cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Oct 31 19:28:12.395659 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:12.395499    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6k6cd" podUID="a8fca631-527d-46c0-9197-5edd7762adb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Oct 31 19:28:12.955511 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:12.955472    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-76hjw" event={"ID":"b14ee5b2-6bf8-4b41-a84d-d0a7dcc9b85a","Type":"ContainerStarted","Data":"0b1a540b8db12a46b3256e90c1c16059cc4efc179d06803d2a46c8c461c21e1c"}
Oct 31 19:28:12.955805 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:12.955785    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-76hjw"
Oct 31 19:28:12.976144 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:12.976071    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-76hjw" podStartSLOduration=1.517599459 podStartE2EDuration="3.976054647s" totalImagesPullingTime="2.458455188s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:28:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:28:12.972251011 +0000 UTC m=+523.545485143" watchObservedRunningTime="2026-10-31 19:28:12.976054647 +0000 UTC m=+523.549288778"
Oct 31 19:28:14.361844 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:14.361809    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-9th7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:28:14.362385 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:14.361873    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9th7f" podUID="76bc1f3a-74b3-49b3-bde1-cc3b305260d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:28:14.436499 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:14.436453    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-6k6cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:28:14.436678 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:14.436511    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6k6cd" podUID="a8fca631-527d-46c0-9197-5edd7762adb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:28:14.835961 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:14.835925    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:28:15.402554 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:15.402495    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-9th7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:28:15.403304 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:15.403247    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9th7f" podUID="76bc1f3a-74b3-49b3-bde1-cc3b305260d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:28:15.478502 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:15.478460    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-6k6cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:28:15.478649 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:15.478522    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6k6cd" podUID="a8fca631-527d-46c0-9197-5edd7762adb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:28:15.840958 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:15.840927    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9th7f"
Oct 31 19:28:15.841257 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:15.841225    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9th7f"
Oct 31 19:28:15.841955 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:15.841929    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9th7f"
Oct 31 19:28:15.906568 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:15.906537    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6k6cd"
Oct 31 19:28:15.907318 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:15.907267    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6k6cd"
Oct 31 19:28:15.974087 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:15.974055    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6k6cd"
Oct 31 19:28:18.930656 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:18.930615    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-v2znx"
Oct 31 19:28:23.963813 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:23.963779    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-76hjw"
Oct 31 19:28:33.190151 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:33.190115    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5f6dd4f69f-v2znx"]
Oct 31 19:28:33.190668 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:33.190355    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-v2znx" podUID="c8470944-e312-44f9-8d73-0e37062db79e" containerName="manager" containerID="cri-o://6b681d0120043237bc53a68767850b41256326cd2b5d58fae03ebafc2d8c8249" gracePeriod=30
Oct 31 19:28:33.432587 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:33.432564    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-v2znx"
Oct 31 19:28:33.513955 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:33.513899    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8470944-e312-44f9-8d73-0e37062db79e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8470944-e312-44f9-8d73-0e37062db79e-metrics-certs\") pod \"c8470944-e312-44f9-8d73-0e37062db79e\" (UID: \"c8470944-e312-44f9-8d73-0e37062db79e\") "
Oct 31 19:28:33.514066 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:33.513962    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8470944-e312-44f9-8d73-0e37062db79e-kube-api-access-hd28x\" (UniqueName: \"kubernetes.io/projected/c8470944-e312-44f9-8d73-0e37062db79e-kube-api-access-hd28x\") pod \"c8470944-e312-44f9-8d73-0e37062db79e\" (UID: \"c8470944-e312-44f9-8d73-0e37062db79e\") "
Oct 31 19:28:33.516055 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:33.516023    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8470944-e312-44f9-8d73-0e37062db79e-kube-api-access-hd28x" pod "c8470944-e312-44f9-8d73-0e37062db79e" (UID: "c8470944-e312-44f9-8d73-0e37062db79e"). InnerVolumeSpecName "kube-api-access-hd28x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:28:33.516055 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:33.516044    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8470944-e312-44f9-8d73-0e37062db79e-metrics-certs" pod "c8470944-e312-44f9-8d73-0e37062db79e" (UID: "c8470944-e312-44f9-8d73-0e37062db79e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:28:33.614950 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:33.614929    4645 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8470944-e312-44f9-8d73-0e37062db79e-metrics-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:28:33.615041 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:33.614952    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hd28x\" (UniqueName: \"kubernetes.io/projected/c8470944-e312-44f9-8d73-0e37062db79e-kube-api-access-hd28x\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:28:34.058856 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:34.058824    4645 generic.go:356] "Generic (PLEG): container finished" podID="c8470944-e312-44f9-8d73-0e37062db79e" containerID="6b681d0120043237bc53a68767850b41256326cd2b5d58fae03ebafc2d8c8249" exitCode=0
Oct 31 19:28:34.058991 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:34.058876    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-v2znx"
Oct 31 19:28:34.058991 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:34.058883    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-v2znx" event={"ID":"c8470944-e312-44f9-8d73-0e37062db79e","Type":"ContainerDied","Data":"6b681d0120043237bc53a68767850b41256326cd2b5d58fae03ebafc2d8c8249"}
Oct 31 19:28:34.058991 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:34.058940    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-v2znx" event={"ID":"c8470944-e312-44f9-8d73-0e37062db79e","Type":"ContainerDied","Data":"9a4512aaf7dc560617052f04e48671e8776753a4c78d9684c85c9a6f2a989012"}
Oct 31 19:28:34.058991 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:34.058970    4645 scope.go:122] "RemoveContainer" containerID="6b681d0120043237bc53a68767850b41256326cd2b5d58fae03ebafc2d8c8249"
Oct 31 19:28:34.068059 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:34.068040    4645 scope.go:122] "RemoveContainer" containerID="6b681d0120043237bc53a68767850b41256326cd2b5d58fae03ebafc2d8c8249"
Oct 31 19:28:34.068359 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:28:34.068317    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b681d0120043237bc53a68767850b41256326cd2b5d58fae03ebafc2d8c8249\": container with ID starting with 6b681d0120043237bc53a68767850b41256326cd2b5d58fae03ebafc2d8c8249 not found: ID does not exist" containerID="6b681d0120043237bc53a68767850b41256326cd2b5d58fae03ebafc2d8c8249"
Oct 31 19:28:34.068440 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:34.068370    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b681d0120043237bc53a68767850b41256326cd2b5d58fae03ebafc2d8c8249"} err="failed to get container status \"6b681d0120043237bc53a68767850b41256326cd2b5d58fae03ebafc2d8c8249\": rpc error: code = NotFound desc = could not find container \"6b681d0120043237bc53a68767850b41256326cd2b5d58fae03ebafc2d8c8249\": container with ID starting with 6b681d0120043237bc53a68767850b41256326cd2b5d58fae03ebafc2d8c8249 not found: ID does not exist"
Oct 31 19:28:34.074568 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:34.074545    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5f6dd4f69f-v2znx"]
Oct 31 19:28:34.078995 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:34.078970    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-5f6dd4f69f-v2znx"]
Oct 31 19:28:34.929619 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:34.929587    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:28:35.516873 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:35.516841    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:28:35.616559 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:35.616527    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8470944-e312-44f9-8d73-0e37062db79e" path="/var/lib/kubelet/pods/c8470944-e312-44f9-8d73-0e37062db79e/volumes"
Oct 31 19:28:36.246438 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:28:36.246404    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 19:28:37.158727 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:37.158695    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:28:37.286008 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:37.285949    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:28:38.755706 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:38.755674    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:28:38.870427 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:28:38.870398    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:01.887544 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:01.887513    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:02.451477 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:02.451449    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s"]
Oct 31 19:29:02.452221 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:02.452194    4645 cpu_manager.go:414] "containerMap: removing container" podUID="c8470944-e312-44f9-8d73-0e37062db79e" containerName="manager"
Oct 31 19:29:02.452221 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:02.452220    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8470944-e312-44f9-8d73-0e37062db79e" containerName="manager"
Oct 31 19:29:02.452422 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:02.452358    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8470944-e312-44f9-8d73-0e37062db79e" containerName="manager"
Oct 31 19:29:02.455522 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:02.455500    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s"
Oct 31 19:29:02.466357 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:02.466322    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-5bbf75-openshift-default-dockercfg-qv9k6\""
Oct 31 19:29:02.480093 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:02.480063    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s"]
Oct 31 19:29:02.548797 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:02.548655    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/426c8fc3-1152-4a03-81e6-ada81e4d0007-workload-socket\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s\" (UID: \"426c8fc3-1152-4a03-81e6-ada81e4d0007\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s"
Oct 31 19:29:02.548797 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:02.548710    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26rf9\" (UniqueName: \"kubernetes.io/projected/426c8fc3-1152-4a03-81e6-ada81e4d0007-kube-api-access-26rf9\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s\" (UID: \"426c8fc3-1152-4a03-81e6-ada81e4d0007\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s"
Oct 31 19:29:02.548797 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:02.548740    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/426c8fc3-1152-4a03-81e6-ada81e4d0007-istiod-ca-cert\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s\" (UID: \"426c8fc3-1152-4a03-81e6-ada81e4d0007\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s"
Oct 31 19:29:02.548797 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:02.548770    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/426c8fc3-1152-4a03-81e6-ada81e4d0007-istio-podinfo\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s\" (UID: \"426c8fc3-1152-4a03-81e6-ada81e4d0007\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s"
Oct 31 19:29:02.549205 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:02.548832    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/426c8fc3-1152-4a03-81e6-ada81e4d0007-workload-certs\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s\" (UID: \"426c8fc3-1152-4a03-81e6-ada81e4d0007\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s"
Oct 31 19:29:02.549205 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:02.548861    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/426c8fc3-1152-4a03-81e6-ada81e4d0007-istio-envoy\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s\" (UID: \"426c8fc3-1152-4a03-81e6-ada81e4d0007\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s"
Oct 31 19:29:02.549205 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:02.548892    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/426c8fc3-1152-4a03-81e6-ada81e4d0007-credential-socket\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s\" (UID: \"426c8fc3-1152-4a03-81e6-ada81e4d0007\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s"
Oct 31 19:29:02.549205 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:02.548927    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/426c8fc3-1152-4a03-81e6-ada81e4d0007-istio-token\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s\" (UID: \"426c8fc3-1152-4a03-81e6-ada81e4d0007\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s"
Oct 31 19:29:02.549205 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:02.548964    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/426c8fc3-1152-4a03-81e6-ada81e4d0007-istio-data\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s\" (UID: \"426c8fc3-1152-4a03-81e6-ada81e4d0007\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s"
Oct 31 19:29:02.649900 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:02.649872    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/426c8fc3-1152-4a03-81e6-ada81e4d0007-istiod-ca-cert\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s\" (UID: \"426c8fc3-1152-4a03-81e6-ada81e4d0007\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s"
Oct 31 19:29:02.650035 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:02.649916    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/426c8fc3-1152-4a03-81e6-ada81e4d0007-istio-podinfo\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s\" (UID: \"426c8fc3-1152-4a03-81e6-ada81e4d0007\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s"
Oct 31 19:29:02.650035 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:02.649952    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/426c8fc3-1152-4a03-81e6-ada81e4d0007-workload-certs\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s\" (UID: \"426c8fc3-1152-4a03-81e6-ada81e4d0007\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s"
Oct 31 19:29:02.650035 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:02.649970    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/426c8fc3-1152-4a03-81e6-ada81e4d0007-istio-envoy\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s\" (UID: \"426c8fc3-1152-4a03-81e6-ada81e4d0007\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s"
Oct 31 19:29:02.650035 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:02.649994    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/426c8fc3-1152-4a03-81e6-ada81e4d0007-credential-socket\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s\" (UID: \"426c8fc3-1152-4a03-81e6-ada81e4d0007\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s"
Oct 31 19:29:02.650035 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:02.650014    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/426c8fc3-1152-4a03-81e6-ada81e4d0007-istio-token\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s\" (UID: \"426c8fc3-1152-4a03-81e6-ada81e4d0007\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s"
Oct 31 19:29:02.650305 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:02.650042    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/426c8fc3-1152-4a03-81e6-ada81e4d0007-istio-data\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s\" (UID: \"426c8fc3-1152-4a03-81e6-ada81e4d0007\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s"
Oct 31 19:29:02.650305 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:02.650083    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/426c8fc3-1152-4a03-81e6-ada81e4d0007-workload-socket\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s\" (UID: \"426c8fc3-1152-4a03-81e6-ada81e4d0007\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s"
Oct 31 19:29:02.650305 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:02.650121    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26rf9\" (UniqueName: \"kubernetes.io/projected/426c8fc3-1152-4a03-81e6-ada81e4d0007-kube-api-access-26rf9\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s\" (UID: \"426c8fc3-1152-4a03-81e6-ada81e4d0007\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s"
Oct 31 19:29:02.650715 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:02.650692    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/426c8fc3-1152-4a03-81e6-ada81e4d0007-istiod-ca-cert\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s\" (UID: \"426c8fc3-1152-4a03-81e6-ada81e4d0007\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s"
Oct 31 19:29:02.650818 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:02.650785    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/426c8fc3-1152-4a03-81e6-ada81e4d0007-workload-certs\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s\" (UID: \"426c8fc3-1152-4a03-81e6-ada81e4d0007\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s"
Oct 31 19:29:02.650907 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:02.650882    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/426c8fc3-1152-4a03-81e6-ada81e4d0007-istio-data\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s\" (UID: \"426c8fc3-1152-4a03-81e6-ada81e4d0007\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s"
Oct 31 19:29:02.650986 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:02.650916    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/426c8fc3-1152-4a03-81e6-ada81e4d0007-workload-socket\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s\" (UID: \"426c8fc3-1152-4a03-81e6-ada81e4d0007\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s"
Oct 31 19:29:02.651252 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:02.651232    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/426c8fc3-1152-4a03-81e6-ada81e4d0007-credential-socket\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s\" (UID: \"426c8fc3-1152-4a03-81e6-ada81e4d0007\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s"
Oct 31 19:29:02.652697 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:02.652677    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/426c8fc3-1152-4a03-81e6-ada81e4d0007-istio-envoy\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s\" (UID: \"426c8fc3-1152-4a03-81e6-ada81e4d0007\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s"
Oct 31 19:29:02.653114 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:02.653094    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/426c8fc3-1152-4a03-81e6-ada81e4d0007-istio-podinfo\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s\" (UID: \"426c8fc3-1152-4a03-81e6-ada81e4d0007\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s"
Oct 31 19:29:02.671177 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:02.671147    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26rf9\" (UniqueName: \"kubernetes.io/projected/426c8fc3-1152-4a03-81e6-ada81e4d0007-kube-api-access-26rf9\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s\" (UID: \"426c8fc3-1152-4a03-81e6-ada81e4d0007\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s"
Oct 31 19:29:02.671733 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:02.671707    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/426c8fc3-1152-4a03-81e6-ada81e4d0007-istio-token\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s\" (UID: \"426c8fc3-1152-4a03-81e6-ada81e4d0007\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s"
Oct 31 19:29:02.684012 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:02.683990    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln"]
Oct 31 19:29:02.689706 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:02.689686    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln"
Oct 31 19:29:02.693200 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:02.693164    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-aeb292-openshift-default-dockercfg-zq554\""
Oct 31 19:29:02.720530 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:02.720474    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln"]
Oct 31 19:29:02.769012 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:02.768991    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s"
Oct 31 19:29:02.852529 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:02.852495    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3b22b1e-9d49-4a54-b016-bc504f03390c-workload-socket\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln\" (UID: \"f3b22b1e-9d49-4a54-b016-bc504f03390c\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln"
Oct 31 19:29:02.852651 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:02.852542    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6t77\" (UniqueName: \"kubernetes.io/projected/f3b22b1e-9d49-4a54-b016-bc504f03390c-kube-api-access-b6t77\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln\" (UID: \"f3b22b1e-9d49-4a54-b016-bc504f03390c\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln"
Oct 31 19:29:02.852651 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:02.852579    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3b22b1e-9d49-4a54-b016-bc504f03390c-credential-socket\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln\" (UID: \"f3b22b1e-9d49-4a54-b016-bc504f03390c\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln"
Oct 31 19:29:02.852651 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:02.852611    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3b22b1e-9d49-4a54-b016-bc504f03390c-istiod-ca-cert\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln\" (UID: \"f3b22b1e-9d49-4a54-b016-bc504f03390c\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln"
Oct 31 19:29:02.852651 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:02.852639    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3b22b1e-9d49-4a54-b016-bc504f03390c-workload-certs\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln\" (UID: \"f3b22b1e-9d49-4a54-b016-bc504f03390c\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln"
Oct 31 19:29:02.852780 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:02.852686    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3b22b1e-9d49-4a54-b016-bc504f03390c-istio-podinfo\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln\" (UID: \"f3b22b1e-9d49-4a54-b016-bc504f03390c\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln"
Oct 31 19:29:02.852780 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:02.852748    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3b22b1e-9d49-4a54-b016-bc504f03390c-istio-token\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln\" (UID: \"f3b22b1e-9d49-4a54-b016-bc504f03390c\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln"
Oct 31 19:29:02.852843 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:02.852781    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3b22b1e-9d49-4a54-b016-bc504f03390c-istio-envoy\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln\" (UID: \"f3b22b1e-9d49-4a54-b016-bc504f03390c\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln"
Oct 31 19:29:02.852843 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:02.852832    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3b22b1e-9d49-4a54-b016-bc504f03390c-istio-data\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln\" (UID: \"f3b22b1e-9d49-4a54-b016-bc504f03390c\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln"
Oct 31 19:29:02.890810 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:02.890782    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:02.954123 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:02.954095    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3b22b1e-9d49-4a54-b016-bc504f03390c-credential-socket\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln\" (UID: \"f3b22b1e-9d49-4a54-b016-bc504f03390c\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln"
Oct 31 19:29:02.954246 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:02.954127    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3b22b1e-9d49-4a54-b016-bc504f03390c-istiod-ca-cert\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln\" (UID: \"f3b22b1e-9d49-4a54-b016-bc504f03390c\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln"
Oct 31 19:29:02.954246 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:02.954148    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3b22b1e-9d49-4a54-b016-bc504f03390c-workload-certs\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln\" (UID: \"f3b22b1e-9d49-4a54-b016-bc504f03390c\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln"
Oct 31 19:29:02.954246 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:02.954177    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3b22b1e-9d49-4a54-b016-bc504f03390c-istio-podinfo\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln\" (UID: \"f3b22b1e-9d49-4a54-b016-bc504f03390c\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln"
Oct 31 19:29:02.954246 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:02.954213    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3b22b1e-9d49-4a54-b016-bc504f03390c-istio-token\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln\" (UID: \"f3b22b1e-9d49-4a54-b016-bc504f03390c\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln"
Oct 31 19:29:02.954246 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:02.954233    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3b22b1e-9d49-4a54-b016-bc504f03390c-istio-envoy\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln\" (UID: \"f3b22b1e-9d49-4a54-b016-bc504f03390c\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln"
Oct 31 19:29:02.954519 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:02.954277    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3b22b1e-9d49-4a54-b016-bc504f03390c-istio-data\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln\" (UID: \"f3b22b1e-9d49-4a54-b016-bc504f03390c\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln"
Oct 31 19:29:02.954519 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:02.954352    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3b22b1e-9d49-4a54-b016-bc504f03390c-workload-socket\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln\" (UID: \"f3b22b1e-9d49-4a54-b016-bc504f03390c\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln"
Oct 31 19:29:02.954519 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:02.954457    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6t77\" (UniqueName: \"kubernetes.io/projected/f3b22b1e-9d49-4a54-b016-bc504f03390c-kube-api-access-b6t77\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln\" (UID: \"f3b22b1e-9d49-4a54-b016-bc504f03390c\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln"
Oct 31 19:29:02.954668 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:02.954597    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3b22b1e-9d49-4a54-b016-bc504f03390c-credential-socket\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln\" (UID: \"f3b22b1e-9d49-4a54-b016-bc504f03390c\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln"
Oct 31 19:29:02.954818 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:02.954796    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3b22b1e-9d49-4a54-b016-bc504f03390c-workload-certs\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln\" (UID: \"f3b22b1e-9d49-4a54-b016-bc504f03390c\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln"
Oct 31 19:29:02.954890 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:02.954881    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3b22b1e-9d49-4a54-b016-bc504f03390c-istio-data\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln\" (UID: \"f3b22b1e-9d49-4a54-b016-bc504f03390c\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln"
Oct 31 19:29:02.954947 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:02.954931    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3b22b1e-9d49-4a54-b016-bc504f03390c-workload-socket\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln\" (UID: \"f3b22b1e-9d49-4a54-b016-bc504f03390c\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln"
Oct 31 19:29:02.955029 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:02.955013    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3b22b1e-9d49-4a54-b016-bc504f03390c-istiod-ca-cert\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln\" (UID: \"f3b22b1e-9d49-4a54-b016-bc504f03390c\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln"
Oct 31 19:29:02.956451 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:02.956431    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3b22b1e-9d49-4a54-b016-bc504f03390c-istio-envoy\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln\" (UID: \"f3b22b1e-9d49-4a54-b016-bc504f03390c\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln"
Oct 31 19:29:02.956682 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:02.956663    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3b22b1e-9d49-4a54-b016-bc504f03390c-istio-podinfo\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln\" (UID: \"f3b22b1e-9d49-4a54-b016-bc504f03390c\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln"
Oct 31 19:29:02.969829 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:02.969805    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6t77\" (UniqueName: \"kubernetes.io/projected/f3b22b1e-9d49-4a54-b016-bc504f03390c-kube-api-access-b6t77\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln\" (UID: \"f3b22b1e-9d49-4a54-b016-bc504f03390c\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln"
Oct 31 19:29:02.978117 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:02.978071    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3b22b1e-9d49-4a54-b016-bc504f03390c-istio-token\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln\" (UID: \"f3b22b1e-9d49-4a54-b016-bc504f03390c\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln"
Oct 31 19:29:03.003835 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.003814    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln"
Oct 31 19:29:03.063816 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.063784    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:03.076246 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.076219    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s"]
Oct 31 19:29:03.080669 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:29:03.080630    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod426c8fc3_1152_4a03_81e6_ada81e4d0007.slice/crio-7c5692b7c4524597651ee600c9a5373c784a9b63775a53f991bdb169e453b2ae WatchSource:0}: Error finding container 7c5692b7c4524597651ee600c9a5373c784a9b63775a53f991bdb169e453b2ae: Status 404 returned error can't find the container with id 7c5692b7c4524597651ee600c9a5373c784a9b63775a53f991bdb169e453b2ae
Oct 31 19:29:03.082894 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.082859    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:29:03.082975 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.082943    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:29:03.083015 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.082993    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:29:03.163647 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.163616    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm"]
Oct 31 19:29:03.167200 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.167176    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm"
Oct 31 19:29:03.185019 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.184994    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-8730d5-openshift-default-dockercfg-hc6b7\""
Oct 31 19:29:03.190478 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.190075    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s" event={"ID":"426c8fc3-1152-4a03-81e6-ada81e4d0007","Type":"ContainerStarted","Data":"f87c1ec8f52beaaa4f9d63a1ec96e4d1889c7e71e71aa74a742ef3fa1f7ed58f"}
Oct 31 19:29:03.190478 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.190130    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s" event={"ID":"426c8fc3-1152-4a03-81e6-ada81e4d0007","Type":"ContainerStarted","Data":"7c5692b7c4524597651ee600c9a5373c784a9b63775a53f991bdb169e453b2ae"}
Oct 31 19:29:03.200838 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.200352    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm"]
Oct 31 19:29:03.257650 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.257588    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d78f81fc-d267-407f-8bd7-51ac2d787720-workload-socket\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm\" (UID: \"d78f81fc-d267-407f-8bd7-51ac2d787720\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm"
Oct 31 19:29:03.257650 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.257620    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d78f81fc-d267-407f-8bd7-51ac2d787720-istio-envoy\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm\" (UID: \"d78f81fc-d267-407f-8bd7-51ac2d787720\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm"
Oct 31 19:29:03.257809 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.257654    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d78f81fc-d267-407f-8bd7-51ac2d787720-istio-data\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm\" (UID: \"d78f81fc-d267-407f-8bd7-51ac2d787720\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm"
Oct 31 19:29:03.257809 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.257783    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d78f81fc-d267-407f-8bd7-51ac2d787720-credential-socket\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm\" (UID: \"d78f81fc-d267-407f-8bd7-51ac2d787720\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm"
Oct 31 19:29:03.257883 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.257814    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d78f81fc-d267-407f-8bd7-51ac2d787720-istiod-ca-cert\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm\" (UID: \"d78f81fc-d267-407f-8bd7-51ac2d787720\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm"
Oct 31 19:29:03.257945 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.257922    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d78f81fc-d267-407f-8bd7-51ac2d787720-workload-certs\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm\" (UID: \"d78f81fc-d267-407f-8bd7-51ac2d787720\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm"
Oct 31 19:29:03.258002 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.257988    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d78f81fc-d267-407f-8bd7-51ac2d787720-istio-podinfo\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm\" (UID: \"d78f81fc-d267-407f-8bd7-51ac2d787720\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm"
Oct 31 19:29:03.258038 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.258019    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d78f81fc-d267-407f-8bd7-51ac2d787720-istio-token\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm\" (UID: \"d78f81fc-d267-407f-8bd7-51ac2d787720\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm"
Oct 31 19:29:03.258110 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.258091    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpr6x\" (UniqueName: \"kubernetes.io/projected/d78f81fc-d267-407f-8bd7-51ac2d787720-kube-api-access-wpr6x\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm\" (UID: \"d78f81fc-d267-407f-8bd7-51ac2d787720\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm"
Oct 31 19:29:03.274105 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.274060    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:03.307405 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.307373    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln"]
Oct 31 19:29:03.308027 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.307991    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s" podStartSLOduration=1.30798042 podStartE2EDuration="1.30798042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:29:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:29:03.303542003 +0000 UTC m=+573.876776155" watchObservedRunningTime="2026-10-31 19:29:03.30798042 +0000 UTC m=+573.881214546"
Oct 31 19:29:03.308881 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:29:03.308856    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3b22b1e_9d49_4a54_b016_bc504f03390c.slice/crio-51aa454ee4f0e4362566b1f245e32ce6510814ba54e6f8551f9cf6d9e63c7195 WatchSource:0}: Error finding container 51aa454ee4f0e4362566b1f245e32ce6510814ba54e6f8551f9cf6d9e63c7195: Status 404 returned error can't find the container with id 51aa454ee4f0e4362566b1f245e32ce6510814ba54e6f8551f9cf6d9e63c7195
Oct 31 19:29:03.311363 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.311298    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:29:03.311422 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.311404    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:29:03.311464 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.311451    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:29:03.359052 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.359026    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt"]
Oct 31 19:29:03.359358 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.359312    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d78f81fc-d267-407f-8bd7-51ac2d787720-workload-certs\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm\" (UID: \"d78f81fc-d267-407f-8bd7-51ac2d787720\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm"
Oct 31 19:29:03.359482 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.359388    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d78f81fc-d267-407f-8bd7-51ac2d787720-istio-podinfo\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm\" (UID: \"d78f81fc-d267-407f-8bd7-51ac2d787720\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm"
Oct 31 19:29:03.359482 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.359434    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d78f81fc-d267-407f-8bd7-51ac2d787720-istio-token\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm\" (UID: \"d78f81fc-d267-407f-8bd7-51ac2d787720\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm"
Oct 31 19:29:03.359599 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.359481    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpr6x\" (UniqueName: \"kubernetes.io/projected/d78f81fc-d267-407f-8bd7-51ac2d787720-kube-api-access-wpr6x\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm\" (UID: \"d78f81fc-d267-407f-8bd7-51ac2d787720\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm"
Oct 31 19:29:03.359599 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.359542    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d78f81fc-d267-407f-8bd7-51ac2d787720-workload-socket\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm\" (UID: \"d78f81fc-d267-407f-8bd7-51ac2d787720\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm"
Oct 31 19:29:03.359599 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.359571    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d78f81fc-d267-407f-8bd7-51ac2d787720-istio-envoy\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm\" (UID: \"d78f81fc-d267-407f-8bd7-51ac2d787720\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm"
Oct 31 19:29:03.359858 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.359629    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d78f81fc-d267-407f-8bd7-51ac2d787720-istio-data\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm\" (UID: \"d78f81fc-d267-407f-8bd7-51ac2d787720\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm"
Oct 31 19:29:03.359858 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.359687    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d78f81fc-d267-407f-8bd7-51ac2d787720-credential-socket\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm\" (UID: \"d78f81fc-d267-407f-8bd7-51ac2d787720\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm"
Oct 31 19:29:03.359858 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.359717    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d78f81fc-d267-407f-8bd7-51ac2d787720-istiod-ca-cert\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm\" (UID: \"d78f81fc-d267-407f-8bd7-51ac2d787720\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm"
Oct 31 19:29:03.360720 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.360692    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d78f81fc-d267-407f-8bd7-51ac2d787720-workload-certs\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm\" (UID: \"d78f81fc-d267-407f-8bd7-51ac2d787720\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm"
Oct 31 19:29:03.360720 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.360700    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d78f81fc-d267-407f-8bd7-51ac2d787720-workload-socket\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm\" (UID: \"d78f81fc-d267-407f-8bd7-51ac2d787720\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm"
Oct 31 19:29:03.361414 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.361391    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d78f81fc-d267-407f-8bd7-51ac2d787720-istiod-ca-cert\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm\" (UID: \"d78f81fc-d267-407f-8bd7-51ac2d787720\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm"
Oct 31 19:29:03.361717 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.361691    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d78f81fc-d267-407f-8bd7-51ac2d787720-istio-data\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm\" (UID: \"d78f81fc-d267-407f-8bd7-51ac2d787720\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm"
Oct 31 19:29:03.361717 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.361708    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d78f81fc-d267-407f-8bd7-51ac2d787720-credential-socket\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm\" (UID: \"d78f81fc-d267-407f-8bd7-51ac2d787720\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm"
Oct 31 19:29:03.362746 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.362723    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d78f81fc-d267-407f-8bd7-51ac2d787720-istio-envoy\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm\" (UID: \"d78f81fc-d267-407f-8bd7-51ac2d787720\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm"
Oct 31 19:29:03.362931 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.362910    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d78f81fc-d267-407f-8bd7-51ac2d787720-istio-podinfo\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm\" (UID: \"d78f81fc-d267-407f-8bd7-51ac2d787720\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm"
Oct 31 19:29:03.364575 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.364554    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt"
Oct 31 19:29:03.375422 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.375406    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-23814d-openshift-default-dockercfg-m6pw6\""
Oct 31 19:29:03.382229 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.382207    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d78f81fc-d267-407f-8bd7-51ac2d787720-istio-token\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm\" (UID: \"d78f81fc-d267-407f-8bd7-51ac2d787720\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm"
Oct 31 19:29:03.382758 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.382737    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpr6x\" (UniqueName: \"kubernetes.io/projected/d78f81fc-d267-407f-8bd7-51ac2d787720-kube-api-access-wpr6x\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm\" (UID: \"d78f81fc-d267-407f-8bd7-51ac2d787720\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm"
Oct 31 19:29:03.384493 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.384473    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt"]
Oct 31 19:29:03.461412 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.461374    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fa97d00-d3af-4c53-bfce-83746cdd6ce3-istio-token\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt\" (UID: \"3fa97d00-d3af-4c53-bfce-83746cdd6ce3\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt"
Oct 31 19:29:03.461589 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.461432    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fa97d00-d3af-4c53-bfce-83746cdd6ce3-workload-certs\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt\" (UID: \"3fa97d00-d3af-4c53-bfce-83746cdd6ce3\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt"
Oct 31 19:29:03.461589 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.461530    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xx678\" (UniqueName: \"kubernetes.io/projected/3fa97d00-d3af-4c53-bfce-83746cdd6ce3-kube-api-access-xx678\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt\" (UID: \"3fa97d00-d3af-4c53-bfce-83746cdd6ce3\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt"
Oct 31 19:29:03.461589 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.461580    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa97d00-d3af-4c53-bfce-83746cdd6ce3-credential-socket\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt\" (UID: \"3fa97d00-d3af-4c53-bfce-83746cdd6ce3\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt"
Oct 31 19:29:03.461744 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.461607    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fa97d00-d3af-4c53-bfce-83746cdd6ce3-istio-podinfo\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt\" (UID: \"3fa97d00-d3af-4c53-bfce-83746cdd6ce3\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt"
Oct 31 19:29:03.461804 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.461734    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fa97d00-d3af-4c53-bfce-83746cdd6ce3-istio-envoy\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt\" (UID: \"3fa97d00-d3af-4c53-bfce-83746cdd6ce3\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt"
Oct 31 19:29:03.461858 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.461796    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fa97d00-d3af-4c53-bfce-83746cdd6ce3-istiod-ca-cert\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt\" (UID: \"3fa97d00-d3af-4c53-bfce-83746cdd6ce3\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt"
Oct 31 19:29:03.461858 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.461835    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa97d00-d3af-4c53-bfce-83746cdd6ce3-workload-socket\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt\" (UID: \"3fa97d00-d3af-4c53-bfce-83746cdd6ce3\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt"
Oct 31 19:29:03.461933 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.461914    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fa97d00-d3af-4c53-bfce-83746cdd6ce3-istio-data\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt\" (UID: \"3fa97d00-d3af-4c53-bfce-83746cdd6ce3\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt"
Oct 31 19:29:03.478978 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.478948    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:03.483944 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.483917    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm"
Oct 31 19:29:03.562896 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.562867    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fa97d00-d3af-4c53-bfce-83746cdd6ce3-istio-token\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt\" (UID: \"3fa97d00-d3af-4c53-bfce-83746cdd6ce3\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt"
Oct 31 19:29:03.562896 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.562904    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fa97d00-d3af-4c53-bfce-83746cdd6ce3-workload-certs\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt\" (UID: \"3fa97d00-d3af-4c53-bfce-83746cdd6ce3\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt"
Oct 31 19:29:03.563290 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.563266    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xx678\" (UniqueName: \"kubernetes.io/projected/3fa97d00-d3af-4c53-bfce-83746cdd6ce3-kube-api-access-xx678\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt\" (UID: \"3fa97d00-d3af-4c53-bfce-83746cdd6ce3\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt"
Oct 31 19:29:03.563422 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.563356    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa97d00-d3af-4c53-bfce-83746cdd6ce3-credential-socket\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt\" (UID: \"3fa97d00-d3af-4c53-bfce-83746cdd6ce3\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt"
Oct 31 19:29:03.563422 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.563384    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fa97d00-d3af-4c53-bfce-83746cdd6ce3-istio-podinfo\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt\" (UID: \"3fa97d00-d3af-4c53-bfce-83746cdd6ce3\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt"
Oct 31 19:29:03.563422 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.563416    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fa97d00-d3af-4c53-bfce-83746cdd6ce3-istio-envoy\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt\" (UID: \"3fa97d00-d3af-4c53-bfce-83746cdd6ce3\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt"
Oct 31 19:29:03.564284 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.563464    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fa97d00-d3af-4c53-bfce-83746cdd6ce3-istiod-ca-cert\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt\" (UID: \"3fa97d00-d3af-4c53-bfce-83746cdd6ce3\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt"
Oct 31 19:29:03.564284 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.563499    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa97d00-d3af-4c53-bfce-83746cdd6ce3-workload-socket\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt\" (UID: \"3fa97d00-d3af-4c53-bfce-83746cdd6ce3\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt"
Oct 31 19:29:03.564284 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.563535    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fa97d00-d3af-4c53-bfce-83746cdd6ce3-istio-data\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt\" (UID: \"3fa97d00-d3af-4c53-bfce-83746cdd6ce3\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt"
Oct 31 19:29:03.564284 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.563271    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fa97d00-d3af-4c53-bfce-83746cdd6ce3-workload-certs\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt\" (UID: \"3fa97d00-d3af-4c53-bfce-83746cdd6ce3\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt"
Oct 31 19:29:03.564284 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.563865    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fa97d00-d3af-4c53-bfce-83746cdd6ce3-istio-data\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt\" (UID: \"3fa97d00-d3af-4c53-bfce-83746cdd6ce3\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt"
Oct 31 19:29:03.564284 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.564104    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa97d00-d3af-4c53-bfce-83746cdd6ce3-workload-socket\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt\" (UID: \"3fa97d00-d3af-4c53-bfce-83746cdd6ce3\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt"
Oct 31 19:29:03.564621 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.564499    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fa97d00-d3af-4c53-bfce-83746cdd6ce3-istiod-ca-cert\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt\" (UID: \"3fa97d00-d3af-4c53-bfce-83746cdd6ce3\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt"
Oct 31 19:29:03.564725 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.564702    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa97d00-d3af-4c53-bfce-83746cdd6ce3-credential-socket\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt\" (UID: \"3fa97d00-d3af-4c53-bfce-83746cdd6ce3\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt"
Oct 31 19:29:03.567686 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.567662    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fa97d00-d3af-4c53-bfce-83746cdd6ce3-istio-envoy\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt\" (UID: \"3fa97d00-d3af-4c53-bfce-83746cdd6ce3\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt"
Oct 31 19:29:03.568038 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.568004    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fa97d00-d3af-4c53-bfce-83746cdd6ce3-istio-podinfo\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt\" (UID: \"3fa97d00-d3af-4c53-bfce-83746cdd6ce3\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt"
Oct 31 19:29:03.578441 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.578416    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fa97d00-d3af-4c53-bfce-83746cdd6ce3-istio-token\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt\" (UID: \"3fa97d00-d3af-4c53-bfce-83746cdd6ce3\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt"
Oct 31 19:29:03.579669 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.579647    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xx678\" (UniqueName: \"kubernetes.io/projected/3fa97d00-d3af-4c53-bfce-83746cdd6ce3-kube-api-access-xx678\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt\" (UID: \"3fa97d00-d3af-4c53-bfce-83746cdd6ce3\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt"
Oct 31 19:29:03.646476 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.646430    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:03.678592 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.678567    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt"
Oct 31 19:29:03.769714 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.769687    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s"
Oct 31 19:29:03.778703 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.778682    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm"]
Oct 31 19:29:03.779659 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:29:03.779633    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd78f81fc_d267_407f_8bd7_51ac2d787720.slice/crio-a270d828d57a88d096c122a801235f869a7a5a825044fd3081eef3ea34f1b946 WatchSource:0}: Error finding container a270d828d57a88d096c122a801235f869a7a5a825044fd3081eef3ea34f1b946: Status 404 returned error can't find the container with id a270d828d57a88d096c122a801235f869a7a5a825044fd3081eef3ea34f1b946
Oct 31 19:29:03.782136 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.782098    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:29:03.782220 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.782184    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:29:03.782257 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.782229    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:29:03.960153 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:29:03.960094    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fa97d00_d3af_4c53_bfce_83746cdd6ce3.slice/crio-5ab1e8360c6b574dd6a2def2a174ab5b28b5c3bd044801e8d35d2866e6fae286 WatchSource:0}: Error finding container 5ab1e8360c6b574dd6a2def2a174ab5b28b5c3bd044801e8d35d2866e6fae286: Status 404 returned error can't find the container with id 5ab1e8360c6b574dd6a2def2a174ab5b28b5c3bd044801e8d35d2866e6fae286
Oct 31 19:29:03.962282 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.962258    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt"]
Oct 31 19:29:03.962700 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.962673    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:29:03.962742 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.962731    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:29:03.962774 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:03.962763    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:29:04.197718 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:04.197680    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt" event={"ID":"3fa97d00-d3af-4c53-bfce-83746cdd6ce3","Type":"ContainerStarted","Data":"fee0e0f90b8dd9575f381dfe8d2748ea2ac697946824aa35184c529981f7d64b"}
Oct 31 19:29:04.197926 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:04.197745    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt" event={"ID":"3fa97d00-d3af-4c53-bfce-83746cdd6ce3","Type":"ContainerStarted","Data":"5ab1e8360c6b574dd6a2def2a174ab5b28b5c3bd044801e8d35d2866e6fae286"}
Oct 31 19:29:04.199413 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:04.199379    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm" event={"ID":"d78f81fc-d267-407f-8bd7-51ac2d787720","Type":"ContainerStarted","Data":"2471599d7086c3806a504ba57ca06e314c6c6f0d2a66edd6d38f44acd7e05b97"}
Oct 31 19:29:04.199540 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:04.199433    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm" event={"ID":"d78f81fc-d267-407f-8bd7-51ac2d787720","Type":"ContainerStarted","Data":"a270d828d57a88d096c122a801235f869a7a5a825044fd3081eef3ea34f1b946"}
Oct 31 19:29:04.201022 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:04.200993    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln" event={"ID":"f3b22b1e-9d49-4a54-b016-bc504f03390c","Type":"ContainerStarted","Data":"8429f3ed823bbf3a7a32a9203f1e0fd2704ca88b7be4e1ac071810c4fd8e5711"}
Oct 31 19:29:04.201151 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:04.201046    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln" event={"ID":"f3b22b1e-9d49-4a54-b016-bc504f03390c","Type":"ContainerStarted","Data":"51aa454ee4f0e4362566b1f245e32ce6510814ba54e6f8551f9cf6d9e63c7195"}
Oct 31 19:29:04.229795 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:04.229745    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt" podStartSLOduration=1.22973021 podStartE2EDuration="1.22973021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:29:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:29:04.227044318 +0000 UTC m=+574.800278445" watchObservedRunningTime="2026-10-31 19:29:04.22973021 +0000 UTC m=+574.802964339"
Oct 31 19:29:04.253381 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:04.253300    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln" podStartSLOduration=2.253287788 podStartE2EDuration="2.253287788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:29:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:29:04.250042181 +0000 UTC m=+574.823276307" watchObservedRunningTime="2026-10-31 19:29:04.253287788 +0000 UTC m=+574.826521917"
Oct 31 19:29:04.276922 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:04.276859    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm" podStartSLOduration=1.276844652 podStartE2EDuration="1.276844652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:29:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:29:04.274876888 +0000 UTC m=+574.848111015" watchObservedRunningTime="2026-10-31 19:29:04.276844652 +0000 UTC m=+574.850078782"
Oct 31 19:29:04.484583 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:04.484497    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm"
Oct 31 19:29:04.489869 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:04.489845    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm"
Oct 31 19:29:04.679066 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:04.679045    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt"
Oct 31 19:29:04.774633 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:04.774576    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s"
Oct 31 19:29:05.004912 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:05.004885    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln"
Oct 31 19:29:05.009805 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:05.009784    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln"
Oct 31 19:29:05.205357 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:05.205296    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s"
Oct 31 19:29:05.205505 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:05.205378    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm"
Oct 31 19:29:05.205815 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:05.205504    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln"
Oct 31 19:29:05.205962 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:05.205942    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm"
Oct 31 19:29:05.206019 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:05.205996    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s"
Oct 31 19:29:05.206054 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:05.206033    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln"
Oct 31 19:29:05.685070 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:05.685042    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt"
Oct 31 19:29:05.877538 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:05.877512    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:06.063017 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:06.062947    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:06.209937 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:06.209911    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt"
Oct 31 19:29:06.210494 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:06.210479    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt"
Oct 31 19:29:06.246961 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:29:06.246931    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 19:29:06.404196 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:06.404105    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:06.613146 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:06.613116    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:06.827501 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:06.827473    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:06.997284 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:06.997258    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:07.164084 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:07.164056    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:07.308511 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:07.308480    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:07.481857 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:07.481798    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:07.700319 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:07.700286    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:07.848041 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:07.847976    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:08.428806 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:08.428774    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:08.754080 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:08.754002    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:08.911923 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:08.911888    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:09.077721 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:09.077642    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:09.489964 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:09.489928    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:09.628458 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:09.628428    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:09.974627 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:09.974596    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-996c32-88bf7fc6f-twmpc"]
Oct 31 19:29:09.978445 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:09.978415    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-996c32-88bf7fc6f-twmpc"
Oct 31 19:29:09.992918 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:09.992898    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-996c32-88bf7fc6f-twmpc"]
Oct 31 19:29:10.019581 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:10.019564    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-996c32-serving-cert\""
Oct 31 19:29:10.036038 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:10.036016    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f67e9a88-e1fd-4151-8023-8d44ce90e441-maas-api-tls\") pod \"maas-api-e2e-worker-w4-996c32-88bf7fc6f-twmpc\" (UID: \"f67e9a88-e1fd-4151-8023-8d44ce90e441\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-996c32-88bf7fc6f-twmpc"
Oct 31 19:29:10.036172 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:10.036133    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f67e9a88-e1fd-4151-8023-8d44ce90e441-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-996c32-88bf7fc6f-twmpc\" (UID: \"f67e9a88-e1fd-4151-8023-8d44ce90e441\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-996c32-88bf7fc6f-twmpc"
Oct 31 19:29:10.036232 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:10.036216    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5gzd\" (UniqueName: \"kubernetes.io/projected/f67e9a88-e1fd-4151-8023-8d44ce90e441-kube-api-access-z5gzd\") pod \"maas-api-e2e-worker-w4-996c32-88bf7fc6f-twmpc\" (UID: \"f67e9a88-e1fd-4151-8023-8d44ce90e441\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-996c32-88bf7fc6f-twmpc"
Oct 31 19:29:10.036288 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:10.036277    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f67e9a88-e1fd-4151-8023-8d44ce90e441-metrics-certs\") pod \"maas-api-e2e-worker-w4-996c32-88bf7fc6f-twmpc\" (UID: \"f67e9a88-e1fd-4151-8023-8d44ce90e441\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-996c32-88bf7fc6f-twmpc"
Oct 31 19:29:10.044285 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:10.044261    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:10.138824 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:10.137642    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f67e9a88-e1fd-4151-8023-8d44ce90e441-metrics-certs\") pod \"maas-api-e2e-worker-w4-996c32-88bf7fc6f-twmpc\" (UID: \"f67e9a88-e1fd-4151-8023-8d44ce90e441\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-996c32-88bf7fc6f-twmpc"
Oct 31 19:29:10.138824 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:10.137743    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f67e9a88-e1fd-4151-8023-8d44ce90e441-maas-api-tls\") pod \"maas-api-e2e-worker-w4-996c32-88bf7fc6f-twmpc\" (UID: \"f67e9a88-e1fd-4151-8023-8d44ce90e441\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-996c32-88bf7fc6f-twmpc"
Oct 31 19:29:10.138824 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:10.137783    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f67e9a88-e1fd-4151-8023-8d44ce90e441-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-996c32-88bf7fc6f-twmpc\" (UID: \"f67e9a88-e1fd-4151-8023-8d44ce90e441\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-996c32-88bf7fc6f-twmpc"
Oct 31 19:29:10.138824 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:10.137857    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5gzd\" (UniqueName: \"kubernetes.io/projected/f67e9a88-e1fd-4151-8023-8d44ce90e441-kube-api-access-z5gzd\") pod \"maas-api-e2e-worker-w4-996c32-88bf7fc6f-twmpc\" (UID: \"f67e9a88-e1fd-4151-8023-8d44ce90e441\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-996c32-88bf7fc6f-twmpc"
Oct 31 19:29:10.139038 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:10.138862    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f67e9a88-e1fd-4151-8023-8d44ce90e441-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-996c32-88bf7fc6f-twmpc\" (UID: \"f67e9a88-e1fd-4151-8023-8d44ce90e441\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-996c32-88bf7fc6f-twmpc"
Oct 31 19:29:10.141360 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:10.140955    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f67e9a88-e1fd-4151-8023-8d44ce90e441-metrics-certs\") pod \"maas-api-e2e-worker-w4-996c32-88bf7fc6f-twmpc\" (UID: \"f67e9a88-e1fd-4151-8023-8d44ce90e441\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-996c32-88bf7fc6f-twmpc"
Oct 31 19:29:10.147364 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:10.146499    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f67e9a88-e1fd-4151-8023-8d44ce90e441-maas-api-tls\") pod \"maas-api-e2e-worker-w4-996c32-88bf7fc6f-twmpc\" (UID: \"f67e9a88-e1fd-4151-8023-8d44ce90e441\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-996c32-88bf7fc6f-twmpc"
Oct 31 19:29:10.159089 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:10.159060    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5gzd\" (UniqueName: \"kubernetes.io/projected/f67e9a88-e1fd-4151-8023-8d44ce90e441-kube-api-access-z5gzd\") pod \"maas-api-e2e-worker-w4-996c32-88bf7fc6f-twmpc\" (UID: \"f67e9a88-e1fd-4151-8023-8d44ce90e441\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-996c32-88bf7fc6f-twmpc"
Oct 31 19:29:10.162816 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:10.162792    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:10.300811 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:10.300747    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-996c32-88bf7fc6f-twmpc"
Oct 31 19:29:10.456090 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:10.456055    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:10.579100 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:10.579079    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-996c32-88bf7fc6f-twmpc"]
Oct 31 19:29:10.580837 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:29:10.580814    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf67e9a88_e1fd_4151_8023_8d44ce90e441.slice/crio-935dc961630466be4b65fe2054ce65babd4d380b6002d7c2e3c444ff5157b179 WatchSource:0}: Error finding container 935dc961630466be4b65fe2054ce65babd4d380b6002d7c2e3c444ff5157b179: Status 404 returned error can't find the container with id 935dc961630466be4b65fe2054ce65babd4d380b6002d7c2e3c444ff5157b179
Oct 31 19:29:10.635039 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:10.635013    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:11.206422 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:11.206366    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-23843f-6fc9d79f67-clgcx"]
Oct 31 19:29:11.212204 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:11.212180    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-23843f-6fc9d79f67-clgcx"
Oct 31 19:29:11.218073 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:11.217116    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-23843f-serving-cert\""
Oct 31 19:29:11.219834 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:11.219814    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-23843f-6fc9d79f67-clgcx"]
Oct 31 19:29:11.238195 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:11.238162    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-996c32-88bf7fc6f-twmpc" event={"ID":"f67e9a88-e1fd-4151-8023-8d44ce90e441","Type":"ContainerStarted","Data":"b29d029fe4bab602a01874f88ea7ac03b33b9c09fbe19dcf731fb5181b372450"}
Oct 31 19:29:11.238348 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:11.238235    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-996c32-88bf7fc6f-twmpc" event={"ID":"f67e9a88-e1fd-4151-8023-8d44ce90e441","Type":"ContainerStarted","Data":"935dc961630466be4b65fe2054ce65babd4d380b6002d7c2e3c444ff5157b179"}
Oct 31 19:29:11.238504 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:11.238468    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-996c32-88bf7fc6f-twmpc"
Oct 31 19:29:11.257554 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:11.257529    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56586a65-8824-40b4-9e94-b517f0307311-metrics-certs\") pod \"maas-api-e2e-worker-w0-23843f-6fc9d79f67-clgcx\" (UID: \"56586a65-8824-40b4-9e94-b517f0307311\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-23843f-6fc9d79f67-clgcx"
Oct 31 19:29:11.257689 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:11.257606    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56586a65-8824-40b4-9e94-b517f0307311-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-23843f-6fc9d79f67-clgcx\" (UID: \"56586a65-8824-40b4-9e94-b517f0307311\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-23843f-6fc9d79f67-clgcx"
Oct 31 19:29:11.257817 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:11.257791    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcv8c\" (UniqueName: \"kubernetes.io/projected/56586a65-8824-40b4-9e94-b517f0307311-kube-api-access-dcv8c\") pod \"maas-api-e2e-worker-w0-23843f-6fc9d79f67-clgcx\" (UID: \"56586a65-8824-40b4-9e94-b517f0307311\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-23843f-6fc9d79f67-clgcx"
Oct 31 19:29:11.257926 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:11.257842    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56586a65-8824-40b4-9e94-b517f0307311-maas-api-tls\") pod \"maas-api-e2e-worker-w0-23843f-6fc9d79f67-clgcx\" (UID: \"56586a65-8824-40b4-9e94-b517f0307311\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-23843f-6fc9d79f67-clgcx"
Oct 31 19:29:11.262147 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:11.261783    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-996c32-88bf7fc6f-twmpc" podStartSLOduration=1.732721176 podStartE2EDuration="2.261767614s" totalImagesPullingTime="529.046438ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:29:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:29:11.258790013 +0000 UTC m=+581.832024142" watchObservedRunningTime="2026-10-31 19:29:11.261767614 +0000 UTC m=+581.835001744"
Oct 31 19:29:11.289945 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:11.289844    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:11.373227 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:11.372889    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56586a65-8824-40b4-9e94-b517f0307311-maas-api-tls\") pod \"maas-api-e2e-worker-w0-23843f-6fc9d79f67-clgcx\" (UID: \"56586a65-8824-40b4-9e94-b517f0307311\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-23843f-6fc9d79f67-clgcx"
Oct 31 19:29:11.373665 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:11.373459    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56586a65-8824-40b4-9e94-b517f0307311-metrics-certs\") pod \"maas-api-e2e-worker-w0-23843f-6fc9d79f67-clgcx\" (UID: \"56586a65-8824-40b4-9e94-b517f0307311\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-23843f-6fc9d79f67-clgcx"
Oct 31 19:29:11.373665 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:11.373534    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56586a65-8824-40b4-9e94-b517f0307311-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-23843f-6fc9d79f67-clgcx\" (UID: \"56586a65-8824-40b4-9e94-b517f0307311\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-23843f-6fc9d79f67-clgcx"
Oct 31 19:29:11.373866 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:11.373683    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcv8c\" (UniqueName: \"kubernetes.io/projected/56586a65-8824-40b4-9e94-b517f0307311-kube-api-access-dcv8c\") pod \"maas-api-e2e-worker-w0-23843f-6fc9d79f67-clgcx\" (UID: \"56586a65-8824-40b4-9e94-b517f0307311\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-23843f-6fc9d79f67-clgcx"
Oct 31 19:29:11.375814 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:11.375283    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56586a65-8824-40b4-9e94-b517f0307311-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-23843f-6fc9d79f67-clgcx\" (UID: \"56586a65-8824-40b4-9e94-b517f0307311\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-23843f-6fc9d79f67-clgcx"
Oct 31 19:29:11.379362 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:11.376963    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56586a65-8824-40b4-9e94-b517f0307311-maas-api-tls\") pod \"maas-api-e2e-worker-w0-23843f-6fc9d79f67-clgcx\" (UID: \"56586a65-8824-40b4-9e94-b517f0307311\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-23843f-6fc9d79f67-clgcx"
Oct 31 19:29:11.379362 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:11.377858    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56586a65-8824-40b4-9e94-b517f0307311-metrics-certs\") pod \"maas-api-e2e-worker-w0-23843f-6fc9d79f67-clgcx\" (UID: \"56586a65-8824-40b4-9e94-b517f0307311\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-23843f-6fc9d79f67-clgcx"
Oct 31 19:29:11.388127 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:11.388101    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcv8c\" (UniqueName: \"kubernetes.io/projected/56586a65-8824-40b4-9e94-b517f0307311-kube-api-access-dcv8c\") pod \"maas-api-e2e-worker-w0-23843f-6fc9d79f67-clgcx\" (UID: \"56586a65-8824-40b4-9e94-b517f0307311\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-23843f-6fc9d79f67-clgcx"
Oct 31 19:29:11.403676 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:11.403648    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:11.529110 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:11.529078    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-23843f-6fc9d79f67-clgcx"
Oct 31 19:29:11.801460 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:11.801431    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:11.810022 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:11.809972    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-23843f-6fc9d79f67-clgcx"]
Oct 31 19:29:11.812288 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:29:11.812263    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56586a65_8824_40b4_9e94_b517f0307311.slice/crio-8f9ad2115d76936579f787d15893a81c850ae5b937256c1a3e0b67d59565ec0c WatchSource:0}: Error finding container 8f9ad2115d76936579f787d15893a81c850ae5b937256c1a3e0b67d59565ec0c: Status 404 returned error can't find the container with id 8f9ad2115d76936579f787d15893a81c850ae5b937256c1a3e0b67d59565ec0c
Oct 31 19:29:11.938893 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:11.938823    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:12.244670 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:12.244639    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-23843f-6fc9d79f67-clgcx" event={"ID":"56586a65-8824-40b4-9e94-b517f0307311","Type":"ContainerStarted","Data":"8f9ad2115d76936579f787d15893a81c850ae5b937256c1a3e0b67d59565ec0c"}
Oct 31 19:29:12.307131 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:12.307099    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-8730d5-549774fbc4-wrjg5"]
Oct 31 19:29:12.313478 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:12.313454    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-8730d5-549774fbc4-wrjg5"
Oct 31 19:29:12.325623 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:12.325594    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-8730d5-549774fbc4-wrjg5"]
Oct 31 19:29:12.325792 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:12.325774    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-8730d5-serving-cert\""
Oct 31 19:29:12.397027 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:12.396991    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a08b47e2-773b-4f08-9b57-957c735fb0f3-metrics-certs\") pod \"maas-api-e2e-worker-w6-8730d5-549774fbc4-wrjg5\" (UID: \"a08b47e2-773b-4f08-9b57-957c735fb0f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-8730d5-549774fbc4-wrjg5"
Oct 31 19:29:12.397211 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:12.397155    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a08b47e2-773b-4f08-9b57-957c735fb0f3-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-8730d5-549774fbc4-wrjg5\" (UID: \"a08b47e2-773b-4f08-9b57-957c735fb0f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-8730d5-549774fbc4-wrjg5"
Oct 31 19:29:12.397277 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:12.397214    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a08b47e2-773b-4f08-9b57-957c735fb0f3-maas-api-tls\") pod \"maas-api-e2e-worker-w6-8730d5-549774fbc4-wrjg5\" (UID: \"a08b47e2-773b-4f08-9b57-957c735fb0f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-8730d5-549774fbc4-wrjg5"
Oct 31 19:29:12.397365 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:12.397349    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7slt\" (UniqueName: \"kubernetes.io/projected/a08b47e2-773b-4f08-9b57-957c735fb0f3-kube-api-access-g7slt\") pod \"maas-api-e2e-worker-w6-8730d5-549774fbc4-wrjg5\" (UID: \"a08b47e2-773b-4f08-9b57-957c735fb0f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-8730d5-549774fbc4-wrjg5"
Oct 31 19:29:12.402892 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:12.402860    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:12.498724 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:12.498656    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a08b47e2-773b-4f08-9b57-957c735fb0f3-metrics-certs\") pod \"maas-api-e2e-worker-w6-8730d5-549774fbc4-wrjg5\" (UID: \"a08b47e2-773b-4f08-9b57-957c735fb0f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-8730d5-549774fbc4-wrjg5"
Oct 31 19:29:12.498853 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:12.498761    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a08b47e2-773b-4f08-9b57-957c735fb0f3-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-8730d5-549774fbc4-wrjg5\" (UID: \"a08b47e2-773b-4f08-9b57-957c735fb0f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-8730d5-549774fbc4-wrjg5"
Oct 31 19:29:12.498853 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:12.498811    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a08b47e2-773b-4f08-9b57-957c735fb0f3-maas-api-tls\") pod \"maas-api-e2e-worker-w6-8730d5-549774fbc4-wrjg5\" (UID: \"a08b47e2-773b-4f08-9b57-957c735fb0f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-8730d5-549774fbc4-wrjg5"
Oct 31 19:29:12.498984 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:12.498878    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7slt\" (UniqueName: \"kubernetes.io/projected/a08b47e2-773b-4f08-9b57-957c735fb0f3-kube-api-access-g7slt\") pod \"maas-api-e2e-worker-w6-8730d5-549774fbc4-wrjg5\" (UID: \"a08b47e2-773b-4f08-9b57-957c735fb0f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-8730d5-549774fbc4-wrjg5"
Oct 31 19:29:12.499500 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:12.499477    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a08b47e2-773b-4f08-9b57-957c735fb0f3-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-8730d5-549774fbc4-wrjg5\" (UID: \"a08b47e2-773b-4f08-9b57-957c735fb0f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-8730d5-549774fbc4-wrjg5"
Oct 31 19:29:12.501242 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:12.501217    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a08b47e2-773b-4f08-9b57-957c735fb0f3-metrics-certs\") pod \"maas-api-e2e-worker-w6-8730d5-549774fbc4-wrjg5\" (UID: \"a08b47e2-773b-4f08-9b57-957c735fb0f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-8730d5-549774fbc4-wrjg5"
Oct 31 19:29:12.501323 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:12.501214    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a08b47e2-773b-4f08-9b57-957c735fb0f3-maas-api-tls\") pod \"maas-api-e2e-worker-w6-8730d5-549774fbc4-wrjg5\" (UID: \"a08b47e2-773b-4f08-9b57-957c735fb0f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-8730d5-549774fbc4-wrjg5"
Oct 31 19:29:12.518846 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:12.518817    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7slt\" (UniqueName: \"kubernetes.io/projected/a08b47e2-773b-4f08-9b57-957c735fb0f3-kube-api-access-g7slt\") pod \"maas-api-e2e-worker-w6-8730d5-549774fbc4-wrjg5\" (UID: \"a08b47e2-773b-4f08-9b57-957c735fb0f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-8730d5-549774fbc4-wrjg5"
Oct 31 19:29:12.548631 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:12.548605    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:12.644723 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:12.644684    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-8730d5-549774fbc4-wrjg5"
Oct 31 19:29:12.647491 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:12.647460    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:12.942141 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:12.942118    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-8730d5-549774fbc4-wrjg5"]
Oct 31 19:29:12.943667 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:29:12.943643    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda08b47e2_773b_4f08_9b57_957c735fb0f3.slice/crio-6b001455e69cc5591b9c6c3d07c014af234dc4d728253a5812dcea49a49c2a6a WatchSource:0}: Error finding container 6b001455e69cc5591b9c6c3d07c014af234dc4d728253a5812dcea49a49c2a6a: Status 404 returned error can't find the container with id 6b001455e69cc5591b9c6c3d07c014af234dc4d728253a5812dcea49a49c2a6a
Oct 31 19:29:13.250618 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:13.250535    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-23843f-6fc9d79f67-clgcx" event={"ID":"56586a65-8824-40b4-9e94-b517f0307311","Type":"ContainerStarted","Data":"c75614b1ba6ef41d671864f39c92f108893048539cf56b32ac65d42b96425232"}
Oct 31 19:29:13.250618 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:13.250607    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-23843f-6fc9d79f67-clgcx"
Oct 31 19:29:13.251730 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:13.251708    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-8730d5-549774fbc4-wrjg5" event={"ID":"a08b47e2-773b-4f08-9b57-957c735fb0f3","Type":"ContainerStarted","Data":"6b001455e69cc5591b9c6c3d07c014af234dc4d728253a5812dcea49a49c2a6a"}
Oct 31 19:29:13.276861 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:13.276824    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-23843f-6fc9d79f67-clgcx" podStartSLOduration=1.815377485 podStartE2EDuration="2.276811956s" totalImagesPullingTime="461.434471ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:29:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:29:13.274689257 +0000 UTC m=+583.847923386" watchObservedRunningTime="2026-10-31 19:29:13.276811956 +0000 UTC m=+583.850046085"
Oct 31 19:29:13.539218 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:13.539144    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:13.661193 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:13.661160    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:14.258534 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:14.258501    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-8730d5-549774fbc4-wrjg5" event={"ID":"a08b47e2-773b-4f08-9b57-957c735fb0f3","Type":"ContainerStarted","Data":"4203006cdc9454ad518df90cb980fa826b20f148d5c8d25c17df5fba9298fb72"}
Oct 31 19:29:14.258953 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:14.258731    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-8730d5-549774fbc4-wrjg5"
Oct 31 19:29:14.282158 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:14.282108    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-8730d5-549774fbc4-wrjg5" podStartSLOduration=1.8742159630000002 podStartE2EDuration="2.282092936s" totalImagesPullingTime="407.876973ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:29:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:29:14.280793732 +0000 UTC m=+584.854027874" watchObservedRunningTime="2026-10-31 19:29:14.282092936 +0000 UTC m=+584.855327067"
Oct 31 19:29:14.537740 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:14.537665    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:14.895385 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:14.895302    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:15.462449 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:15.462419    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:15.578570 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:15.578543    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:16.382211 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:16.382179    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-5bbf75-6c987cff47-d4kgj"]
Oct 31 19:29:16.385688 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:16.385672    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-5bbf75-6c987cff47-d4kgj"
Oct 31 19:29:16.391492 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:16.391467    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-5bbf75-serving-cert\""
Oct 31 19:29:16.402784 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:16.402747    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-5bbf75-6c987cff47-d4kgj"]
Oct 31 19:29:16.452428 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:16.452394    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8274db35-2b32-4fae-9f7d-e3fbb51bf4ed-maas-api-tls\") pod \"maas-api-e2e-worker-w5-5bbf75-6c987cff47-d4kgj\" (UID: \"8274db35-2b32-4fae-9f7d-e3fbb51bf4ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-5bbf75-6c987cff47-d4kgj"
Oct 31 19:29:16.452560 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:16.452527    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8274db35-2b32-4fae-9f7d-e3fbb51bf4ed-metrics-certs\") pod \"maas-api-e2e-worker-w5-5bbf75-6c987cff47-d4kgj\" (UID: \"8274db35-2b32-4fae-9f7d-e3fbb51bf4ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-5bbf75-6c987cff47-d4kgj"
Oct 31 19:29:16.452611 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:16.452557    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8274db35-2b32-4fae-9f7d-e3fbb51bf4ed-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-5bbf75-6c987cff47-d4kgj\" (UID: \"8274db35-2b32-4fae-9f7d-e3fbb51bf4ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-5bbf75-6c987cff47-d4kgj"
Oct 31 19:29:16.452611 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:16.452599    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv6bv\" (UniqueName: \"kubernetes.io/projected/8274db35-2b32-4fae-9f7d-e3fbb51bf4ed-kube-api-access-zv6bv\") pod \"maas-api-e2e-worker-w5-5bbf75-6c987cff47-d4kgj\" (UID: \"8274db35-2b32-4fae-9f7d-e3fbb51bf4ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-5bbf75-6c987cff47-d4kgj"
Oct 31 19:29:16.467240 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:16.467211    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:16.553893 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:16.553854    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8274db35-2b32-4fae-9f7d-e3fbb51bf4ed-maas-api-tls\") pod \"maas-api-e2e-worker-w5-5bbf75-6c987cff47-d4kgj\" (UID: \"8274db35-2b32-4fae-9f7d-e3fbb51bf4ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-5bbf75-6c987cff47-d4kgj"
Oct 31 19:29:16.554084 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:16.553963    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8274db35-2b32-4fae-9f7d-e3fbb51bf4ed-metrics-certs\") pod \"maas-api-e2e-worker-w5-5bbf75-6c987cff47-d4kgj\" (UID: \"8274db35-2b32-4fae-9f7d-e3fbb51bf4ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-5bbf75-6c987cff47-d4kgj"
Oct 31 19:29:16.554084 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:16.553992    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8274db35-2b32-4fae-9f7d-e3fbb51bf4ed-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-5bbf75-6c987cff47-d4kgj\" (UID: \"8274db35-2b32-4fae-9f7d-e3fbb51bf4ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-5bbf75-6c987cff47-d4kgj"
Oct 31 19:29:16.554084 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:29:16.554022    4645 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w5-5bbf75-serving-cert: secret "maas-api-e2e-worker-w5-5bbf75-serving-cert" not found
Oct 31 19:29:16.554084 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:16.554045    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv6bv\" (UniqueName: \"kubernetes.io/projected/8274db35-2b32-4fae-9f7d-e3fbb51bf4ed-kube-api-access-zv6bv\") pod \"maas-api-e2e-worker-w5-5bbf75-6c987cff47-d4kgj\" (UID: \"8274db35-2b32-4fae-9f7d-e3fbb51bf4ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-5bbf75-6c987cff47-d4kgj"
Oct 31 19:29:16.554301 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:29:16.554101    4645 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8274db35-2b32-4fae-9f7d-e3fbb51bf4ed-maas-api-tls podName:8274db35-2b32-4fae-9f7d-e3fbb51bf4ed nodeName:}" failed. No retries permitted until 2026-10-31 19:29:17.054082778 +0000 UTC m=+587.627316905 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8274db35-2b32-4fae-9f7d-e3fbb51bf4ed-maas-api-tls") pod "maas-api-e2e-worker-w5-5bbf75-6c987cff47-d4kgj" (UID: "8274db35-2b32-4fae-9f7d-e3fbb51bf4ed") : secret "maas-api-e2e-worker-w5-5bbf75-serving-cert" not found
Oct 31 19:29:16.554888 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:16.554864    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8274db35-2b32-4fae-9f7d-e3fbb51bf4ed-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-5bbf75-6c987cff47-d4kgj\" (UID: \"8274db35-2b32-4fae-9f7d-e3fbb51bf4ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-5bbf75-6c987cff47-d4kgj"
Oct 31 19:29:16.556663 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:16.556632    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8274db35-2b32-4fae-9f7d-e3fbb51bf4ed-metrics-certs\") pod \"maas-api-e2e-worker-w5-5bbf75-6c987cff47-d4kgj\" (UID: \"8274db35-2b32-4fae-9f7d-e3fbb51bf4ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-5bbf75-6c987cff47-d4kgj"
Oct 31 19:29:16.568573 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:16.568550    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv6bv\" (UniqueName: \"kubernetes.io/projected/8274db35-2b32-4fae-9f7d-e3fbb51bf4ed-kube-api-access-zv6bv\") pod \"maas-api-e2e-worker-w5-5bbf75-6c987cff47-d4kgj\" (UID: \"8274db35-2b32-4fae-9f7d-e3fbb51bf4ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-5bbf75-6c987cff47-d4kgj"
Oct 31 19:29:16.583905 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:16.583877    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:16.701618 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:16.701591    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:17.058874 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:17.058788    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8274db35-2b32-4fae-9f7d-e3fbb51bf4ed-maas-api-tls\") pod \"maas-api-e2e-worker-w5-5bbf75-6c987cff47-d4kgj\" (UID: \"8274db35-2b32-4fae-9f7d-e3fbb51bf4ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-5bbf75-6c987cff47-d4kgj"
Oct 31 19:29:17.061190 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:17.061157    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8274db35-2b32-4fae-9f7d-e3fbb51bf4ed-maas-api-tls\") pod \"maas-api-e2e-worker-w5-5bbf75-6c987cff47-d4kgj\" (UID: \"8274db35-2b32-4fae-9f7d-e3fbb51bf4ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-5bbf75-6c987cff47-d4kgj"
Oct 31 19:29:17.251357 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:17.251317    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-996c32-88bf7fc6f-twmpc"
Oct 31 19:29:17.300047 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:17.300014    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-5bbf75-6c987cff47-d4kgj"
Oct 31 19:29:17.469070 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:17.469036    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:17.586546 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:29:17.586504    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8274db35_2b32_4fae_9f7d_e3fbb51bf4ed.slice/crio-979f319f84be73d13dc617b9278c1dd13494e14f69759d819fbd65b6228c0162 WatchSource:0}: Error finding container 979f319f84be73d13dc617b9278c1dd13494e14f69759d819fbd65b6228c0162: Status 404 returned error can't find the container with id 979f319f84be73d13dc617b9278c1dd13494e14f69759d819fbd65b6228c0162
Oct 31 19:29:17.588975 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:17.588945    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-5bbf75-6c987cff47-d4kgj"]
Oct 31 19:29:18.278815 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:18.278739    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-5bbf75-6c987cff47-d4kgj" event={"ID":"8274db35-2b32-4fae-9f7d-e3fbb51bf4ed","Type":"ContainerStarted","Data":"62687938032ac4db04b1747f9d42568acd00b1862feb0254780d2da59ce8f5e6"}
Oct 31 19:29:18.278815 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:18.278791    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-5bbf75-6c987cff47-d4kgj" event={"ID":"8274db35-2b32-4fae-9f7d-e3fbb51bf4ed","Type":"ContainerStarted","Data":"979f319f84be73d13dc617b9278c1dd13494e14f69759d819fbd65b6228c0162"}
Oct 31 19:29:18.279010 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:18.278864    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-5bbf75-6c987cff47-d4kgj"
Oct 31 19:29:18.302917 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:18.302869    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-5bbf75-6c987cff47-d4kgj" podStartSLOduration=1.900519961 podStartE2EDuration="2.302855402s" totalImagesPullingTime="402.335441ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:29:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:29:18.301184894 +0000 UTC m=+588.874419024" watchObservedRunningTime="2026-10-31 19:29:18.302855402 +0000 UTC m=+588.876089560"
Oct 31 19:29:18.400536 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:18.400502    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:19.265038 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:19.265008    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-23843f-6fc9d79f67-clgcx"
Oct 31 19:29:19.522377 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:19.522275    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:20.297353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:20.296524    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-8730d5-549774fbc4-wrjg5"
Oct 31 19:29:20.581206 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:20.581114    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:21.429353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:21.429300    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:21.544350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:21.544293    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:21.652134 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:21.652102    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:22.154162 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:22.154132    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:22.306441 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:22.306410    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:22.465378 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:22.462574    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:23.242004 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:23.241968    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:23.798357 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:23.796144    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:24.287353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:24.283897    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:24.327354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:24.327166    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-5bbf75-6c987cff47-d4kgj"
Oct 31 19:29:25.045451 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:25.045411    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:25.825661 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:25.825528    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:25.930947 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:25.930911    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:26.859036 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:26.858963    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:27.000655 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:27.000475    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:27.342745 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:27.342690    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:27.762686 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:27.762624    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:28.095549 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:28.095419    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:28.652422 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:28.652370    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:29.477753 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:29.477657    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:30.281901 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:30.281846    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:31.041452 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:31.041417    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:31.189960 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:31.189866    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:31.427845 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:31.427814    4645 generic.go:356] "Generic (PLEG): container finished" podID="d78f81fc-d267-407f-8bd7-51ac2d787720" containerID="2471599d7086c3806a504ba57ca06e314c6c6f0d2a66edd6d38f44acd7e05b97" exitCode=0
Oct 31 19:29:31.642044 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:31.427885    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm" event={"ID":"d78f81fc-d267-407f-8bd7-51ac2d787720","Type":"ContainerDied","Data":"2471599d7086c3806a504ba57ca06e314c6c6f0d2a66edd6d38f44acd7e05b97"}
Oct 31 19:29:31.642044 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:31.428317    4645 scope.go:122] "RemoveContainer" containerID="2471599d7086c3806a504ba57ca06e314c6c6f0d2a66edd6d38f44acd7e05b97"
Oct 31 19:29:31.642044 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:31.429564    4645 generic.go:356] "Generic (PLEG): container finished" podID="f3b22b1e-9d49-4a54-b016-bc504f03390c" containerID="8429f3ed823bbf3a7a32a9203f1e0fd2704ca88b7be4e1ac071810c4fd8e5711" exitCode=0
Oct 31 19:29:31.642044 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:31.429614    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln" event={"ID":"f3b22b1e-9d49-4a54-b016-bc504f03390c","Type":"ContainerDied","Data":"8429f3ed823bbf3a7a32a9203f1e0fd2704ca88b7be4e1ac071810c4fd8e5711"}
Oct 31 19:29:31.642044 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:31.429685    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:29:31.642044 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:31.429755    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:29:31.642044 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:31.429784    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:29:31.642044 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:31.429913    4645 scope.go:122] "RemoveContainer" containerID="8429f3ed823bbf3a7a32a9203f1e0fd2704ca88b7be4e1ac071810c4fd8e5711"
Oct 31 19:29:31.642044 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:31.431073    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:29:31.642044 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:31.431132    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:29:31.642044 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:31.431166    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:29:31.846210 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:31.846161    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:31.984350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:31.984246    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:32.105930 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:32.105896    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:32.437623 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:32.437584    4645 generic.go:356] "Generic (PLEG): container finished" podID="3fa97d00-d3af-4c53-bfce-83746cdd6ce3" containerID="fee0e0f90b8dd9575f381dfe8d2748ea2ac697946824aa35184c529981f7d64b" exitCode=0
Oct 31 19:29:32.437941 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:32.437658    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt" event={"ID":"3fa97d00-d3af-4c53-bfce-83746cdd6ce3","Type":"ContainerDied","Data":"fee0e0f90b8dd9575f381dfe8d2748ea2ac697946824aa35184c529981f7d64b"}
Oct 31 19:29:32.439210 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:32.439188    4645 scope.go:122] "RemoveContainer" containerID="fee0e0f90b8dd9575f381dfe8d2748ea2ac697946824aa35184c529981f7d64b"
Oct 31 19:29:32.439845 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:32.439770    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm" event={"ID":"d78f81fc-d267-407f-8bd7-51ac2d787720","Type":"ContainerStarted","Data":"2cf9ad422be8e2ec90162aa8aebe6302089535b2a06dd8da963a84db064ea742"}
Oct 31 19:29:32.440803 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:32.440774    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:29:32.440895 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:32.440845    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:29:32.440895 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:32.440878    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:29:32.441570 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:32.441554    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln" event={"ID":"f3b22b1e-9d49-4a54-b016-bc504f03390c","Type":"ContainerStarted","Data":"7948823885e27f58d3ed14f621d4c87fc5c250551f48ef8cc354a49116d4ef10"}
Oct 31 19:29:32.499501 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:32.499406    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm"
Oct 31 19:29:32.523353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:32.518642    4645 patch_prober.go:29] interesting pod/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Oct 31 19:29:32.523353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:32.518703    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm" podUID="d78f81fc-d267-407f-8bd7-51ac2d787720" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Oct 31 19:29:33.015397 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:33.008651    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln"
Oct 31 19:29:33.015397 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:33.008839    4645 patch_prober.go:29] interesting pod/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Oct 31 19:29:33.015397 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:33.008888    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln" podUID="f3b22b1e-9d49-4a54-b016-bc504f03390c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Oct 31 19:29:33.122904 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:33.122860    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:33.498401 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:33.497240    4645 patch_prober.go:29] interesting pod/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Oct 31 19:29:33.498401 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:33.497302    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm" podUID="d78f81fc-d267-407f-8bd7-51ac2d787720" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Oct 31 19:29:33.510355 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:33.506695    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt" event={"ID":"3fa97d00-d3af-4c53-bfce-83746cdd6ce3","Type":"ContainerStarted","Data":"8ed58bd86fce039d155cb3a7b6b9821ac00b1b0853fb3df59e5eee872377eae1"}
Oct 31 19:29:33.684357 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:33.683550    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt"
Oct 31 19:29:33.684357 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:33.683679    4645 patch_prober.go:29] interesting pod/e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Oct 31 19:29:33.684357 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:33.683721    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt" podUID="3fa97d00-d3af-4c53-bfce-83746cdd6ce3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Oct 31 19:29:33.972354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:33.969574    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:34.008271 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:34.006761    4645 patch_prober.go:29] interesting pod/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Oct 31 19:29:34.008271 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:34.006828    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln" podUID="f3b22b1e-9d49-4a54-b016-bc504f03390c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Oct 31 19:29:34.530356 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:34.512099    4645 patch_prober.go:29] interesting pod/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Oct 31 19:29:34.530356 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:34.512155    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm" podUID="d78f81fc-d267-407f-8bd7-51ac2d787720" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Oct 31 19:29:34.680353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:34.679467    4645 patch_prober.go:29] interesting pod/e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Oct 31 19:29:34.680353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:34.679528    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt" podUID="3fa97d00-d3af-4c53-bfce-83746cdd6ce3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Oct 31 19:29:34.845461 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:34.845344    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:35.009271 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:35.008806    4645 patch_prober.go:29] interesting pod/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Oct 31 19:29:35.009271 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:35.008867    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln" podUID="f3b22b1e-9d49-4a54-b016-bc504f03390c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Oct 31 19:29:35.208470 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:35.208431    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm"
Oct 31 19:29:35.211111 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:35.211085    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln"
Oct 31 19:29:35.487845 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:35.487753    4645 patch_prober.go:29] interesting pod/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Oct 31 19:29:35.601354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:35.600413    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm" podUID="d78f81fc-d267-407f-8bd7-51ac2d787720" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Oct 31 19:29:35.727381 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:35.715597    4645 patch_prober.go:29] interesting pod/e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Oct 31 19:29:35.727381 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:35.715670    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt" podUID="3fa97d00-d3af-4c53-bfce-83746cdd6ce3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Oct 31 19:29:35.752859 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:35.752765    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:36.007881 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:36.007783    4645 patch_prober.go:29] interesting pod/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Oct 31 19:29:36.008172 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:36.008137    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln" podUID="f3b22b1e-9d49-4a54-b016-bc504f03390c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Oct 31 19:29:36.228357 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:36.214432    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt"
Oct 31 19:29:36.254744 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:36.254480    4645 patch_prober.go:29] interesting pod/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:29:36.254744 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:36.254529    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s" podUID="426c8fc3-1152-4a03-81e6-ada81e4d0007" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:29:36.349024 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:29:36.348937    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 19:29:36.485352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:36.484362    4645 patch_prober.go:29] interesting pod/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Oct 31 19:29:36.485352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:36.484426    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm" podUID="d78f81fc-d267-407f-8bd7-51ac2d787720" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Oct 31 19:29:36.542661 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:36.542631    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:36.689354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:36.681425    4645 patch_prober.go:29] interesting pod/e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Oct 31 19:29:36.689354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:36.681486    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt" podUID="3fa97d00-d3af-4c53-bfce-83746cdd6ce3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Oct 31 19:29:37.018414 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:37.016793    4645 patch_prober.go:29] interesting pod/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body=
Oct 31 19:29:37.018414 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:37.016853    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln" podUID="f3b22b1e-9d49-4a54-b016-bc504f03390c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused"
Oct 31 19:29:37.226396 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:37.224236    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:37.481369 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:37.481303    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:37.486363 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:37.484694    4645 patch_prober.go:29] interesting pod/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body=
Oct 31 19:29:37.486363 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:37.484760    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm" podUID="d78f81fc-d267-407f-8bd7-51ac2d787720" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Oct 31 19:29:37.715525 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:37.715491    4645 patch_prober.go:29] interesting pod/e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Oct 31 19:29:37.716117 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:37.716073    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt" podUID="3fa97d00-d3af-4c53-bfce-83746cdd6ce3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Oct 31 19:29:37.827433 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:37.827317    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:37.999465 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:37.999353    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:38.112273 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:38.112052    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:38.175731 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:38.175700    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln"
Oct 31 19:29:38.184351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:38.179534    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln"
Oct 31 19:29:38.502350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:38.501542    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm"
Oct 31 19:29:38.509031 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:38.508012    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm"
Oct 31 19:29:38.532765 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:38.529237    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:38.686353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:38.685446    4645 patch_prober.go:29] interesting pod/e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Oct 31 19:29:38.686353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:38.685515    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt" podUID="3fa97d00-d3af-4c53-bfce-83746cdd6ce3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Oct 31 19:29:39.422447 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:39.422413    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw"]
Oct 31 19:29:39.427011 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:39.426984    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw"
Oct 31 19:29:39.428496 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:39.428465    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:39.435557 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:39.435520    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-aeb292\"/\"default-dockercfg-g2p8n\""
Oct 31 19:29:39.435992 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:39.435972    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-aeb292\"/\"openshift-service-ca.crt\""
Oct 31 19:29:39.436145 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:39.436126    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-aeb292\"/\"simulator-w2-aeb292-kserve-self-signed-certs\""
Oct 31 19:29:39.436264 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:39.436187    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-aeb292\"/\"kube-root-ca.crt\""
Oct 31 19:29:39.438454 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:39.438432    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw"]
Oct 31 19:29:39.517360 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:39.517316    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d504b2d3-c26d-4dcf-a7eb-efe20732ae4e-tmp-dir\") pod \"simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw\" (UID: \"d504b2d3-c26d-4dcf-a7eb-efe20732ae4e\") " pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw"
Oct 31 19:29:39.517516 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:39.517374    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d504b2d3-c26d-4dcf-a7eb-efe20732ae4e-dshm\") pod \"simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw\" (UID: \"d504b2d3-c26d-4dcf-a7eb-efe20732ae4e\") " pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw"
Oct 31 19:29:39.517516 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:39.517494    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d504b2d3-c26d-4dcf-a7eb-efe20732ae4e-model-cache\") pod \"simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw\" (UID: \"d504b2d3-c26d-4dcf-a7eb-efe20732ae4e\") " pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw"
Oct 31 19:29:39.517589 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:39.517550    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d504b2d3-c26d-4dcf-a7eb-efe20732ae4e-tls-certs\") pod \"simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw\" (UID: \"d504b2d3-c26d-4dcf-a7eb-efe20732ae4e\") " pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw"
Oct 31 19:29:39.517589 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:39.517580    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbz44\" (UniqueName: \"kubernetes.io/projected/d504b2d3-c26d-4dcf-a7eb-efe20732ae4e-kube-api-access-zbz44\") pod \"simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw\" (UID: \"d504b2d3-c26d-4dcf-a7eb-efe20732ae4e\") " pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw"
Oct 31 19:29:39.517656 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:39.517599    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d504b2d3-c26d-4dcf-a7eb-efe20732ae4e-home\") pod \"simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw\" (UID: \"d504b2d3-c26d-4dcf-a7eb-efe20732ae4e\") " pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw"
Oct 31 19:29:39.530813 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:39.530779    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:39.618275 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:39.618242    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d504b2d3-c26d-4dcf-a7eb-efe20732ae4e-model-cache\") pod \"simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw\" (UID: \"d504b2d3-c26d-4dcf-a7eb-efe20732ae4e\") " pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw"
Oct 31 19:29:39.618475 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:39.618316    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d504b2d3-c26d-4dcf-a7eb-efe20732ae4e-tls-certs\") pod \"simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw\" (UID: \"d504b2d3-c26d-4dcf-a7eb-efe20732ae4e\") " pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw"
Oct 31 19:29:39.618475 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:39.618380    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbz44\" (UniqueName: \"kubernetes.io/projected/d504b2d3-c26d-4dcf-a7eb-efe20732ae4e-kube-api-access-zbz44\") pod \"simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw\" (UID: \"d504b2d3-c26d-4dcf-a7eb-efe20732ae4e\") " pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw"
Oct 31 19:29:39.618475 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:39.618404    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d504b2d3-c26d-4dcf-a7eb-efe20732ae4e-home\") pod \"simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw\" (UID: \"d504b2d3-c26d-4dcf-a7eb-efe20732ae4e\") " pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw"
Oct 31 19:29:39.618632 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:39.618520    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d504b2d3-c26d-4dcf-a7eb-efe20732ae4e-tmp-dir\") pod \"simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw\" (UID: \"d504b2d3-c26d-4dcf-a7eb-efe20732ae4e\") " pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw"
Oct 31 19:29:39.618632 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:39.618565    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d504b2d3-c26d-4dcf-a7eb-efe20732ae4e-dshm\") pod \"simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw\" (UID: \"d504b2d3-c26d-4dcf-a7eb-efe20732ae4e\") " pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw"
Oct 31 19:29:39.618709 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:39.618681    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d504b2d3-c26d-4dcf-a7eb-efe20732ae4e-model-cache\") pod \"simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw\" (UID: \"d504b2d3-c26d-4dcf-a7eb-efe20732ae4e\") " pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw"
Oct 31 19:29:39.618948 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:39.618924    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d504b2d3-c26d-4dcf-a7eb-efe20732ae4e-tmp-dir\") pod \"simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw\" (UID: \"d504b2d3-c26d-4dcf-a7eb-efe20732ae4e\") " pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw"
Oct 31 19:29:39.619116 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:39.619091    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d504b2d3-c26d-4dcf-a7eb-efe20732ae4e-home\") pod \"simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw\" (UID: \"d504b2d3-c26d-4dcf-a7eb-efe20732ae4e\") " pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw"
Oct 31 19:29:39.621452 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:39.621256    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d504b2d3-c26d-4dcf-a7eb-efe20732ae4e-dshm\") pod \"simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw\" (UID: \"d504b2d3-c26d-4dcf-a7eb-efe20732ae4e\") " pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw"
Oct 31 19:29:39.621452 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:39.621276    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d504b2d3-c26d-4dcf-a7eb-efe20732ae4e-tls-certs\") pod \"simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw\" (UID: \"d504b2d3-c26d-4dcf-a7eb-efe20732ae4e\") " pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw"
Oct 31 19:29:39.635862 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:39.635838    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbz44\" (UniqueName: \"kubernetes.io/projected/d504b2d3-c26d-4dcf-a7eb-efe20732ae4e-kube-api-access-zbz44\") pod \"simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw\" (UID: \"d504b2d3-c26d-4dcf-a7eb-efe20732ae4e\") " pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw"
Oct 31 19:29:39.683379 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:39.683298    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt"
Oct 31 19:29:39.684532 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:39.684507    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt"
Oct 31 19:29:39.741578 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:39.741550    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw"
Oct 31 19:29:40.037176 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:40.037148    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw"]
Oct 31 19:29:40.039223 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:29:40.039197    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd504b2d3_c26d_4dcf_a7eb_efe20732ae4e.slice/crio-2a0a51a1702babeea1f099e644d328bc2ce72e19807a543de340ac2012fa7297 WatchSource:0}: Error finding container 2a0a51a1702babeea1f099e644d328bc2ce72e19807a543de340ac2012fa7297: Status 404 returned error can't find the container with id 2a0a51a1702babeea1f099e644d328bc2ce72e19807a543de340ac2012fa7297
Oct 31 19:29:40.254135 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:40.254096    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:40.717083 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:40.716989    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw" event={"ID":"d504b2d3-c26d-4dcf-a7eb-efe20732ae4e","Type":"ContainerStarted","Data":"a58c7aab57f48cf169ca6488f1bcb792f0a6180f3da1d7e066201d05a8684cf5"}
Oct 31 19:29:40.717083 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:40.717040    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw" event={"ID":"d504b2d3-c26d-4dcf-a7eb-efe20732ae4e","Type":"ContainerStarted","Data":"2a0a51a1702babeea1f099e644d328bc2ce72e19807a543de340ac2012fa7297"}
Oct 31 19:29:40.718622 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:40.718597    4645 generic.go:356] "Generic (PLEG): container finished" podID="426c8fc3-1152-4a03-81e6-ada81e4d0007" containerID="f87c1ec8f52beaaa4f9d63a1ec96e4d1889c7e71e71aa74a742ef3fa1f7ed58f" exitCode=0
Oct 31 19:29:40.718742 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:40.718663    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s" event={"ID":"426c8fc3-1152-4a03-81e6-ada81e4d0007","Type":"ContainerDied","Data":"f87c1ec8f52beaaa4f9d63a1ec96e4d1889c7e71e71aa74a742ef3fa1f7ed58f"}
Oct 31 19:29:40.719000 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:40.718985    4645 scope.go:122] "RemoveContainer" containerID="f87c1ec8f52beaaa4f9d63a1ec96e4d1889c7e71e71aa74a742ef3fa1f7ed58f"
Oct 31 19:29:40.720236 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:40.720206    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:29:40.720324 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:40.720267    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:29:40.720324 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:40.720299    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:29:40.736894 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:40.736848    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw" podStartSLOduration=1.7368321199999999 podStartE2EDuration="1.73683212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:29:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:29:40.734818336 +0000 UTC m=+611.308052480" watchObservedRunningTime="2026-10-31 19:29:40.73683212 +0000 UTC m=+611.310066251"
Oct 31 19:29:40.741799 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:40.741781    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw"
Oct 31 19:29:41.054627 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:41.054543    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:41.725482 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:41.725444    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s" event={"ID":"426c8fc3-1152-4a03-81e6-ada81e4d0007","Type":"ContainerStarted","Data":"9e97adecf885e756917ec05986dd7dbf6db69ab86c5cf75fcfea13ebdf5264ca"}
Oct 31 19:29:41.772777 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:41.772739    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s"
Oct 31 19:29:41.773104 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:41.773025    4645 patch_prober.go:29] interesting pod/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Oct 31 19:29:41.773266 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:41.773244    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s" podUID="426c8fc3-1152-4a03-81e6-ada81e4d0007" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Oct 31 19:29:41.782146 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:41.782111    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:42.530100 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:42.530051    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:42.778351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:42.777637    4645 patch_prober.go:29] interesting pod/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Oct 31 19:29:42.778351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:42.777693    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s" podUID="426c8fc3-1152-4a03-81e6-ada81e4d0007" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Oct 31 19:29:42.944348 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:42.944305    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:43.337234 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:43.337135    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:43.780354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:43.777708    4645 patch_prober.go:29] interesting pod/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Oct 31 19:29:43.780354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:43.777771    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s" podUID="426c8fc3-1152-4a03-81e6-ada81e4d0007" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Oct 31 19:29:44.141904 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:44.141810    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:44.780920 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:44.780886    4645 patch_prober.go:29] interesting pod/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Oct 31 19:29:44.781534 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:44.781496    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s" podUID="426c8fc3-1152-4a03-81e6-ada81e4d0007" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Oct 31 19:29:44.840109 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:44.840066    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:45.246115 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:45.246074    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x"]
Oct 31 19:29:45.249848 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:45.249782    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x"
Oct 31 19:29:45.286355 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:45.283504    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-5bbf75\"/\"kube-root-ca.crt\""
Oct 31 19:29:45.286355 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:45.283725    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-5bbf75\"/\"simulator-w5-5bbf75-kserve-self-signed-certs\""
Oct 31 19:29:45.286355 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:45.283805    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-5bbf75\"/\"default-dockercfg-8kc45\""
Oct 31 19:29:45.286355 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:45.284431    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-5bbf75\"/\"openshift-service-ca.crt\""
Oct 31 19:29:45.287492 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:45.287466    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x"]
Oct 31 19:29:45.309019 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:45.308985    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:45.417148 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:45.417109    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a774e286-dcb8-4a2f-8541-26bf231eb08d-home\") pod \"simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x\" (UID: \"a774e286-dcb8-4a2f-8541-26bf231eb08d\") " pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x"
Oct 31 19:29:45.417373 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:45.417198    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a774e286-dcb8-4a2f-8541-26bf231eb08d-tls-certs\") pod \"simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x\" (UID: \"a774e286-dcb8-4a2f-8541-26bf231eb08d\") " pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x"
Oct 31 19:29:45.417373 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:45.417264    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89x8m\" (UniqueName: \"kubernetes.io/projected/a774e286-dcb8-4a2f-8541-26bf231eb08d-kube-api-access-89x8m\") pod \"simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x\" (UID: \"a774e286-dcb8-4a2f-8541-26bf231eb08d\") " pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x"
Oct 31 19:29:45.417373 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:45.417286    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a774e286-dcb8-4a2f-8541-26bf231eb08d-tmp-dir\") pod \"simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x\" (UID: \"a774e286-dcb8-4a2f-8541-26bf231eb08d\") " pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x"
Oct 31 19:29:45.417373 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:45.417313    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a774e286-dcb8-4a2f-8541-26bf231eb08d-dshm\") pod \"simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x\" (UID: \"a774e286-dcb8-4a2f-8541-26bf231eb08d\") " pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x"
Oct 31 19:29:45.417373 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:45.417358    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a774e286-dcb8-4a2f-8541-26bf231eb08d-model-cache\") pod \"simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x\" (UID: \"a774e286-dcb8-4a2f-8541-26bf231eb08d\") " pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x"
Oct 31 19:29:45.518862 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:45.518775    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89x8m\" (UniqueName: \"kubernetes.io/projected/a774e286-dcb8-4a2f-8541-26bf231eb08d-kube-api-access-89x8m\") pod \"simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x\" (UID: \"a774e286-dcb8-4a2f-8541-26bf231eb08d\") " pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x"
Oct 31 19:29:45.518862 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:45.518820    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a774e286-dcb8-4a2f-8541-26bf231eb08d-tmp-dir\") pod \"simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x\" (UID: \"a774e286-dcb8-4a2f-8541-26bf231eb08d\") " pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x"
Oct 31 19:29:45.518862 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:45.518862    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a774e286-dcb8-4a2f-8541-26bf231eb08d-dshm\") pod \"simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x\" (UID: \"a774e286-dcb8-4a2f-8541-26bf231eb08d\") " pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x"
Oct 31 19:29:45.519145 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:45.518892    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a774e286-dcb8-4a2f-8541-26bf231eb08d-model-cache\") pod \"simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x\" (UID: \"a774e286-dcb8-4a2f-8541-26bf231eb08d\") " pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x"
Oct 31 19:29:45.519145 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:45.518947    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a774e286-dcb8-4a2f-8541-26bf231eb08d-home\") pod \"simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x\" (UID: \"a774e286-dcb8-4a2f-8541-26bf231eb08d\") " pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x"
Oct 31 19:29:45.519145 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:45.519023    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a774e286-dcb8-4a2f-8541-26bf231eb08d-tls-certs\") pod \"simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x\" (UID: \"a774e286-dcb8-4a2f-8541-26bf231eb08d\") " pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x"
Oct 31 19:29:45.519292 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:45.519258    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a774e286-dcb8-4a2f-8541-26bf231eb08d-tmp-dir\") pod \"simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x\" (UID: \"a774e286-dcb8-4a2f-8541-26bf231eb08d\") " pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x"
Oct 31 19:29:45.519383 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:45.519360    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a774e286-dcb8-4a2f-8541-26bf231eb08d-model-cache\") pod \"simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x\" (UID: \"a774e286-dcb8-4a2f-8541-26bf231eb08d\") " pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x"
Oct 31 19:29:45.519498 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:45.519482    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a774e286-dcb8-4a2f-8541-26bf231eb08d-home\") pod \"simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x\" (UID: \"a774e286-dcb8-4a2f-8541-26bf231eb08d\") " pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x"
Oct 31 19:29:45.521106 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:45.521087    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a774e286-dcb8-4a2f-8541-26bf231eb08d-dshm\") pod \"simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x\" (UID: \"a774e286-dcb8-4a2f-8541-26bf231eb08d\") " pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x"
Oct 31 19:29:45.521547 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:45.521529    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a774e286-dcb8-4a2f-8541-26bf231eb08d-tls-certs\") pod \"simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x\" (UID: \"a774e286-dcb8-4a2f-8541-26bf231eb08d\") " pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x"
Oct 31 19:29:45.530863 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:45.530836    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89x8m\" (UniqueName: \"kubernetes.io/projected/a774e286-dcb8-4a2f-8541-26bf231eb08d-kube-api-access-89x8m\") pod \"simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x\" (UID: \"a774e286-dcb8-4a2f-8541-26bf231eb08d\") " pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x"
Oct 31 19:29:45.589488 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:45.589450    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x"
Oct 31 19:29:45.650321 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:45.646553    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:45.780420 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:45.773759    4645 patch_prober.go:29] interesting pod/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Oct 31 19:29:45.780420 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:45.773837    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s" podUID="426c8fc3-1152-4a03-81e6-ada81e4d0007" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Oct 31 19:29:45.880738 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:45.869275    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x"]
Oct 31 19:29:45.885757 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:29:45.885725    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda774e286_dcb8_4a2f_8541_26bf231eb08d.slice/crio-1a1ddc08963e669aa5adfedffc62ae41eac6520e9080fc4069c6945dfed6e560 WatchSource:0}: Error finding container 1a1ddc08963e669aa5adfedffc62ae41eac6520e9080fc4069c6945dfed6e560: Status 404 returned error can't find the container with id 1a1ddc08963e669aa5adfedffc62ae41eac6520e9080fc4069c6945dfed6e560
Oct 31 19:29:46.478365 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:46.475185    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:46.754423 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:46.753757    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x" event={"ID":"a774e286-dcb8-4a2f-8541-26bf231eb08d","Type":"ContainerStarted","Data":"37915b42aa2185f229e626755de139318b29fc6f797c16185768157c18805912"}
Oct 31 19:29:46.754423 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:46.753810    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x" event={"ID":"a774e286-dcb8-4a2f-8541-26bf231eb08d","Type":"ContainerStarted","Data":"1a1ddc08963e669aa5adfedffc62ae41eac6520e9080fc4069c6945dfed6e560"}
Oct 31 19:29:46.786351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:46.785470    4645 patch_prober.go:29] interesting pod/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Oct 31 19:29:46.786351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:46.785547    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s" podUID="426c8fc3-1152-4a03-81e6-ada81e4d0007" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Oct 31 19:29:46.802352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:46.789548    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x" podStartSLOduration=1.7895310960000002 podStartE2EDuration="1.789531096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:29:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:29:46.786730172 +0000 UTC m=+617.359964303" watchObservedRunningTime="2026-10-31 19:29:46.789531096 +0000 UTC m=+617.362765226"
Oct 31 19:29:47.376345 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:47.376251    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:47.590440 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:47.590407    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x"
Oct 31 19:29:47.774844 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:47.774812    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s"
Oct 31 19:29:47.775110 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:47.775087    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s"
Oct 31 19:29:47.775770 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:47.775746    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s"
Oct 31 19:29:48.201198 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:48.201166    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:48.937050 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:48.937015    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:49.742820 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:49.742781    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw"
Oct 31 19:29:49.753191 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:49.753156    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:49.756258 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:49.756234    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw"
Oct 31 19:29:49.787442 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:49.787411    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw"
Oct 31 19:29:50.449305 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:50.449269    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:50.782727 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:50.782639    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:51.285316 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:51.285279    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:52.075011 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:52.074973    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:52.843417 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:52.843370    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:53.752279 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:53.752241    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:54.541678 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:54.541644    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:55.249385 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:55.249314    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n"]
Oct 31 19:29:55.299522 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:55.299491    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n"]
Oct 31 19:29:55.299821 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:55.299804    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n"
Oct 31 19:29:55.317852 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:55.317828    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-996c32\"/\"openshift-service-ca.crt\""
Oct 31 19:29:55.318458 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:55.318440    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-996c32\"/\"default-dockercfg-n5bxr\""
Oct 31 19:29:55.318789 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:55.318774    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-996c32\"/\"premium-w4-996c32-kserve-self-signed-certs\""
Oct 31 19:29:55.319102 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:55.319012    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-996c32\"/\"kube-root-ca.crt\""
Oct 31 19:29:55.319546 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:55.319519    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:55.356217 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:55.356180    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbb97168-db93-4757-9a1b-f9c5eb704fc5-model-cache\") pod \"premium-w4-996c32-kserve-864499dc8b-l9t9n\" (UID: \"dbb97168-db93-4757-9a1b-f9c5eb704fc5\") " pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n"
Oct 31 19:29:55.357429 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:55.356518    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbb97168-db93-4757-9a1b-f9c5eb704fc5-home\") pod \"premium-w4-996c32-kserve-864499dc8b-l9t9n\" (UID: \"dbb97168-db93-4757-9a1b-f9c5eb704fc5\") " pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n"
Oct 31 19:29:55.357665 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:55.357643    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbb97168-db93-4757-9a1b-f9c5eb704fc5-tmp-dir\") pod \"premium-w4-996c32-kserve-864499dc8b-l9t9n\" (UID: \"dbb97168-db93-4757-9a1b-f9c5eb704fc5\") " pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n"
Oct 31 19:29:55.357812 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:55.357797    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbb97168-db93-4757-9a1b-f9c5eb704fc5-tls-certs\") pod \"premium-w4-996c32-kserve-864499dc8b-l9t9n\" (UID: \"dbb97168-db93-4757-9a1b-f9c5eb704fc5\") " pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n"
Oct 31 19:29:55.357943 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:55.357928    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbb97168-db93-4757-9a1b-f9c5eb704fc5-dshm\") pod \"premium-w4-996c32-kserve-864499dc8b-l9t9n\" (UID: \"dbb97168-db93-4757-9a1b-f9c5eb704fc5\") " pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n"
Oct 31 19:29:55.358043 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:55.358029    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rnc9\" (UniqueName: \"kubernetes.io/projected/dbb97168-db93-4757-9a1b-f9c5eb704fc5-kube-api-access-7rnc9\") pod \"premium-w4-996c32-kserve-864499dc8b-l9t9n\" (UID: \"dbb97168-db93-4757-9a1b-f9c5eb704fc5\") " pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n"
Oct 31 19:29:55.456371 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:55.455982    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:55.458900 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:55.458866    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbb97168-db93-4757-9a1b-f9c5eb704fc5-dshm\") pod \"premium-w4-996c32-kserve-864499dc8b-l9t9n\" (UID: \"dbb97168-db93-4757-9a1b-f9c5eb704fc5\") " pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n"
Oct 31 19:29:55.459052 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:55.458916    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rnc9\" (UniqueName: \"kubernetes.io/projected/dbb97168-db93-4757-9a1b-f9c5eb704fc5-kube-api-access-7rnc9\") pod \"premium-w4-996c32-kserve-864499dc8b-l9t9n\" (UID: \"dbb97168-db93-4757-9a1b-f9c5eb704fc5\") " pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n"
Oct 31 19:29:55.459052 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:55.459036    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbb97168-db93-4757-9a1b-f9c5eb704fc5-model-cache\") pod \"premium-w4-996c32-kserve-864499dc8b-l9t9n\" (UID: \"dbb97168-db93-4757-9a1b-f9c5eb704fc5\") " pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n"
Oct 31 19:29:55.459149 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:55.459076    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbb97168-db93-4757-9a1b-f9c5eb704fc5-home\") pod \"premium-w4-996c32-kserve-864499dc8b-l9t9n\" (UID: \"dbb97168-db93-4757-9a1b-f9c5eb704fc5\") " pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n"
Oct 31 19:29:55.459209 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:55.459149    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbb97168-db93-4757-9a1b-f9c5eb704fc5-tmp-dir\") pod \"premium-w4-996c32-kserve-864499dc8b-l9t9n\" (UID: \"dbb97168-db93-4757-9a1b-f9c5eb704fc5\") " pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n"
Oct 31 19:29:55.459259 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:55.459203    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbb97168-db93-4757-9a1b-f9c5eb704fc5-tls-certs\") pod \"premium-w4-996c32-kserve-864499dc8b-l9t9n\" (UID: \"dbb97168-db93-4757-9a1b-f9c5eb704fc5\") " pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n"
Oct 31 19:29:55.459682 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:55.459649    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbb97168-db93-4757-9a1b-f9c5eb704fc5-home\") pod \"premium-w4-996c32-kserve-864499dc8b-l9t9n\" (UID: \"dbb97168-db93-4757-9a1b-f9c5eb704fc5\") " pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n"
Oct 31 19:29:55.459812 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:55.459683    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbb97168-db93-4757-9a1b-f9c5eb704fc5-model-cache\") pod \"premium-w4-996c32-kserve-864499dc8b-l9t9n\" (UID: \"dbb97168-db93-4757-9a1b-f9c5eb704fc5\") " pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n"
Oct 31 19:29:55.459917 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:55.459896    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbb97168-db93-4757-9a1b-f9c5eb704fc5-tmp-dir\") pod \"premium-w4-996c32-kserve-864499dc8b-l9t9n\" (UID: \"dbb97168-db93-4757-9a1b-f9c5eb704fc5\") " pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n"
Oct 31 19:29:55.462295 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:55.462265    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbb97168-db93-4757-9a1b-f9c5eb704fc5-dshm\") pod \"premium-w4-996c32-kserve-864499dc8b-l9t9n\" (UID: \"dbb97168-db93-4757-9a1b-f9c5eb704fc5\") " pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n"
Oct 31 19:29:55.463896 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:55.463871    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbb97168-db93-4757-9a1b-f9c5eb704fc5-tls-certs\") pod \"premium-w4-996c32-kserve-864499dc8b-l9t9n\" (UID: \"dbb97168-db93-4757-9a1b-f9c5eb704fc5\") " pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n"
Oct 31 19:29:55.470618 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:55.470596    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rnc9\" (UniqueName: \"kubernetes.io/projected/dbb97168-db93-4757-9a1b-f9c5eb704fc5-kube-api-access-7rnc9\") pod \"premium-w4-996c32-kserve-864499dc8b-l9t9n\" (UID: \"dbb97168-db93-4757-9a1b-f9c5eb704fc5\") " pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n"
Oct 31 19:29:55.590610 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:55.590514    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x"
Oct 31 19:29:55.604295 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:55.604266    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x"
Oct 31 19:29:55.697869 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:55.697822    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n"
Oct 31 19:29:55.823285 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:55.823027    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x"
Oct 31 19:29:55.905270 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:55.905168    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:55.983489 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:55.983457    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n"]
Oct 31 19:29:56.436798 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:56.436764    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:56.815046 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:56.814956    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n" event={"ID":"dbb97168-db93-4757-9a1b-f9c5eb704fc5","Type":"ContainerStarted","Data":"9672227baf80eaf23cdc05e00b5d9277b804295a45104e8114f256dda17f4a11"}
Oct 31 19:29:56.815046 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:56.815004    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n" event={"ID":"dbb97168-db93-4757-9a1b-f9c5eb704fc5","Type":"ContainerStarted","Data":"4d400e34f50a429f9efd96d72791662367965d6259aa9e299ab395bacb12533f"}
Oct 31 19:29:56.833927 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:56.833874    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n" podStartSLOduration=1.833859549 podStartE2EDuration="1.833859549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:29:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:29:56.831965128 +0000 UTC m=+627.405199257" watchObservedRunningTime="2026-10-31 19:29:56.833859549 +0000 UTC m=+627.407093679"
Oct 31 19:29:57.157651 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:57.157557    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:57.698061 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:57.698030    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n"
Oct 31 19:29:57.979481 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:57.979400    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:58.279920 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:58.279832    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:58.833526 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:58.833489    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:29:59.567605 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:29:59.567563    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:00.453616 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:00.453580    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:01.231664 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:01.231627    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:01.933393 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:01.933362    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:02.754925 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:02.754882    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:03.683212 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:03.683176    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:04.807530 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:04.807493    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:05.639972 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:05.639939    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:05.697962 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:05.697935    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n"
Oct 31 19:30:05.711534 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:05.711508    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n"
Oct 31 19:30:05.874562 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:05.874533    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n"
Oct 31 19:30:06.245789 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:30:06.245753    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 19:30:06.452318 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:06.452285    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:07.444028 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:07.443995    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:08.382197 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:08.382163    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:09.331699 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:09.331658    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:10.230180 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:10.230145    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:11.204139 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:11.204104    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:12.565569 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:12.565537    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:13.164788 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:13.164753    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4"]
Oct 31 19:30:13.169760 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:13.169738    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4"
Oct 31 19:30:13.182596 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:13.182572    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-5bbf75\"/\"premium-w5-5bbf75-kserve-self-signed-certs\""
Oct 31 19:30:13.183720 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:13.183700    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4"]
Oct 31 19:30:13.232601 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:13.232568    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:13.281161 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:13.281130    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nxdb\" (UniqueName: \"kubernetes.io/projected/6032a997-b795-4561-805c-110cf67a9d5d-kube-api-access-9nxdb\") pod \"premium-w5-5bbf75-kserve-79df6f5f68-8tgl4\" (UID: \"6032a997-b795-4561-805c-110cf67a9d5d\") " pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4"
Oct 31 19:30:13.281392 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:13.281176    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6032a997-b795-4561-805c-110cf67a9d5d-tls-certs\") pod \"premium-w5-5bbf75-kserve-79df6f5f68-8tgl4\" (UID: \"6032a997-b795-4561-805c-110cf67a9d5d\") " pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4"
Oct 31 19:30:13.281392 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:13.281246    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6032a997-b795-4561-805c-110cf67a9d5d-dshm\") pod \"premium-w5-5bbf75-kserve-79df6f5f68-8tgl4\" (UID: \"6032a997-b795-4561-805c-110cf67a9d5d\") " pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4"
Oct 31 19:30:13.281392 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:13.281298    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6032a997-b795-4561-805c-110cf67a9d5d-model-cache\") pod \"premium-w5-5bbf75-kserve-79df6f5f68-8tgl4\" (UID: \"6032a997-b795-4561-805c-110cf67a9d5d\") " pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4"
Oct 31 19:30:13.281392 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:13.281344    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6032a997-b795-4561-805c-110cf67a9d5d-home\") pod \"premium-w5-5bbf75-kserve-79df6f5f68-8tgl4\" (UID: \"6032a997-b795-4561-805c-110cf67a9d5d\") " pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4"
Oct 31 19:30:13.281392 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:13.281378    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6032a997-b795-4561-805c-110cf67a9d5d-tmp-dir\") pod \"premium-w5-5bbf75-kserve-79df6f5f68-8tgl4\" (UID: \"6032a997-b795-4561-805c-110cf67a9d5d\") " pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4"
Oct 31 19:30:13.382822 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:13.382787    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nxdb\" (UniqueName: \"kubernetes.io/projected/6032a997-b795-4561-805c-110cf67a9d5d-kube-api-access-9nxdb\") pod \"premium-w5-5bbf75-kserve-79df6f5f68-8tgl4\" (UID: \"6032a997-b795-4561-805c-110cf67a9d5d\") " pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4"
Oct 31 19:30:13.383023 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:13.382825    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6032a997-b795-4561-805c-110cf67a9d5d-tls-certs\") pod \"premium-w5-5bbf75-kserve-79df6f5f68-8tgl4\" (UID: \"6032a997-b795-4561-805c-110cf67a9d5d\") " pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4"
Oct 31 19:30:13.383023 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:13.382890    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6032a997-b795-4561-805c-110cf67a9d5d-dshm\") pod \"premium-w5-5bbf75-kserve-79df6f5f68-8tgl4\" (UID: \"6032a997-b795-4561-805c-110cf67a9d5d\") " pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4"
Oct 31 19:30:13.383023 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:13.382937    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6032a997-b795-4561-805c-110cf67a9d5d-model-cache\") pod \"premium-w5-5bbf75-kserve-79df6f5f68-8tgl4\" (UID: \"6032a997-b795-4561-805c-110cf67a9d5d\") " pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4"
Oct 31 19:30:13.383023 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:13.382967    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6032a997-b795-4561-805c-110cf67a9d5d-home\") pod \"premium-w5-5bbf75-kserve-79df6f5f68-8tgl4\" (UID: \"6032a997-b795-4561-805c-110cf67a9d5d\") " pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4"
Oct 31 19:30:13.383023 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:13.382997    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6032a997-b795-4561-805c-110cf67a9d5d-tmp-dir\") pod \"premium-w5-5bbf75-kserve-79df6f5f68-8tgl4\" (UID: \"6032a997-b795-4561-805c-110cf67a9d5d\") " pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4"
Oct 31 19:30:13.383485 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:13.383462    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6032a997-b795-4561-805c-110cf67a9d5d-model-cache\") pod \"premium-w5-5bbf75-kserve-79df6f5f68-8tgl4\" (UID: \"6032a997-b795-4561-805c-110cf67a9d5d\") " pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4"
Oct 31 19:30:13.383568 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:13.383513    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6032a997-b795-4561-805c-110cf67a9d5d-tmp-dir\") pod \"premium-w5-5bbf75-kserve-79df6f5f68-8tgl4\" (UID: \"6032a997-b795-4561-805c-110cf67a9d5d\") " pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4"
Oct 31 19:30:13.383568 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:13.383529    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6032a997-b795-4561-805c-110cf67a9d5d-home\") pod \"premium-w5-5bbf75-kserve-79df6f5f68-8tgl4\" (UID: \"6032a997-b795-4561-805c-110cf67a9d5d\") " pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4"
Oct 31 19:30:13.385338 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:13.385302    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6032a997-b795-4561-805c-110cf67a9d5d-dshm\") pod \"premium-w5-5bbf75-kserve-79df6f5f68-8tgl4\" (UID: \"6032a997-b795-4561-805c-110cf67a9d5d\") " pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4"
Oct 31 19:30:13.385442 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:13.385408    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6032a997-b795-4561-805c-110cf67a9d5d-tls-certs\") pod \"premium-w5-5bbf75-kserve-79df6f5f68-8tgl4\" (UID: \"6032a997-b795-4561-805c-110cf67a9d5d\") " pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4"
Oct 31 19:30:13.401943 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:13.401913    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nxdb\" (UniqueName: \"kubernetes.io/projected/6032a997-b795-4561-805c-110cf67a9d5d-kube-api-access-9nxdb\") pod \"premium-w5-5bbf75-kserve-79df6f5f68-8tgl4\" (UID: \"6032a997-b795-4561-805c-110cf67a9d5d\") " pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4"
Oct 31 19:30:13.481843 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:13.481785    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4"
Oct 31 19:30:13.586661 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:13.586628    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:13.759007 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:13.758979    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4"]
Oct 31 19:30:13.760699 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:30:13.760672    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6032a997_b795_4561_805c_110cf67a9d5d.slice/crio-335f8a49cc9e45e22fef97a198408dd433032c770bf87600d25dac647139a2b7 WatchSource:0}: Error finding container 335f8a49cc9e45e22fef97a198408dd433032c770bf87600d25dac647139a2b7: Status 404 returned error can't find the container with id 335f8a49cc9e45e22fef97a198408dd433032c770bf87600d25dac647139a2b7
Oct 31 19:30:13.906225 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:13.906187    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4" event={"ID":"6032a997-b795-4561-805c-110cf67a9d5d","Type":"ContainerStarted","Data":"304d3fda0c7387e793e8eef7d4cb8217c921aae2a0ae0b86994b47f2f32bdf70"}
Oct 31 19:30:13.906360 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:13.906237    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4" event={"ID":"6032a997-b795-4561-805c-110cf67a9d5d","Type":"ContainerStarted","Data":"335f8a49cc9e45e22fef97a198408dd433032c770bf87600d25dac647139a2b7"}
Oct 31 19:30:13.925171 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:13.925124    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4" podStartSLOduration=0.925110414 podStartE2EDuration="925.110414ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:30:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:30:13.92390693 +0000 UTC m=+644.497141060" watchObservedRunningTime="2026-10-31 19:30:13.925110414 +0000 UTC m=+644.498344560"
Oct 31 19:30:14.482550 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:14.482521    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4"
Oct 31 19:30:14.581228 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:14.581192    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:14.956809 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:14.956778    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:15.384198 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:15.384108    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:16.253729 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:16.253700    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:16.967291 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:16.967255    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:17.817779 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:17.817741    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:17.950285 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:17.950137    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:18.186700 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:18.186655    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs"]
Oct 31 19:30:18.190366 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:18.190342    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs"
Oct 31 19:30:18.198278 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:18.198095    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-23843f-openshift-default-dockercfg-27ptb\""
Oct 31 19:30:18.207858 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:18.207833    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs"]
Oct 31 19:30:18.234234 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:18.234185    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40240c3e-0f73-4ac9-80c1-4c581a6ee5a3-istio-token\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs\" (UID: \"40240c3e-0f73-4ac9-80c1-4c581a6ee5a3\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs"
Oct 31 19:30:18.234405 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:18.234235    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40240c3e-0f73-4ac9-80c1-4c581a6ee5a3-workload-certs\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs\" (UID: \"40240c3e-0f73-4ac9-80c1-4c581a6ee5a3\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs"
Oct 31 19:30:18.234405 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:18.234270    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40240c3e-0f73-4ac9-80c1-4c581a6ee5a3-istio-envoy\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs\" (UID: \"40240c3e-0f73-4ac9-80c1-4c581a6ee5a3\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs"
Oct 31 19:30:18.234405 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:18.234305    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40240c3e-0f73-4ac9-80c1-4c581a6ee5a3-istio-data\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs\" (UID: \"40240c3e-0f73-4ac9-80c1-4c581a6ee5a3\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs"
Oct 31 19:30:18.234405 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:18.234352    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40240c3e-0f73-4ac9-80c1-4c581a6ee5a3-istio-podinfo\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs\" (UID: \"40240c3e-0f73-4ac9-80c1-4c581a6ee5a3\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs"
Oct 31 19:30:18.234641 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:18.234409    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40240c3e-0f73-4ac9-80c1-4c581a6ee5a3-istiod-ca-cert\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs\" (UID: \"40240c3e-0f73-4ac9-80c1-4c581a6ee5a3\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs"
Oct 31 19:30:18.234641 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:18.234473    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40240c3e-0f73-4ac9-80c1-4c581a6ee5a3-workload-socket\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs\" (UID: \"40240c3e-0f73-4ac9-80c1-4c581a6ee5a3\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs"
Oct 31 19:30:18.234641 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:18.234509    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40240c3e-0f73-4ac9-80c1-4c581a6ee5a3-credential-socket\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs\" (UID: \"40240c3e-0f73-4ac9-80c1-4c581a6ee5a3\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs"
Oct 31 19:30:18.234641 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:18.234552    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs5dd\" (UniqueName: \"kubernetes.io/projected/40240c3e-0f73-4ac9-80c1-4c581a6ee5a3-kube-api-access-hs5dd\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs\" (UID: \"40240c3e-0f73-4ac9-80c1-4c581a6ee5a3\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs"
Oct 31 19:30:18.335284 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:18.335254    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40240c3e-0f73-4ac9-80c1-4c581a6ee5a3-istio-token\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs\" (UID: \"40240c3e-0f73-4ac9-80c1-4c581a6ee5a3\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs"
Oct 31 19:30:18.335457 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:18.335289    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40240c3e-0f73-4ac9-80c1-4c581a6ee5a3-workload-certs\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs\" (UID: \"40240c3e-0f73-4ac9-80c1-4c581a6ee5a3\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs"
Oct 31 19:30:18.335457 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:18.335312    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40240c3e-0f73-4ac9-80c1-4c581a6ee5a3-istio-envoy\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs\" (UID: \"40240c3e-0f73-4ac9-80c1-4c581a6ee5a3\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs"
Oct 31 19:30:18.335457 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:18.335352    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40240c3e-0f73-4ac9-80c1-4c581a6ee5a3-istio-data\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs\" (UID: \"40240c3e-0f73-4ac9-80c1-4c581a6ee5a3\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs"
Oct 31 19:30:18.335457 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:18.335377    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40240c3e-0f73-4ac9-80c1-4c581a6ee5a3-istio-podinfo\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs\" (UID: \"40240c3e-0f73-4ac9-80c1-4c581a6ee5a3\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs"
Oct 31 19:30:18.335457 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:18.335425    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40240c3e-0f73-4ac9-80c1-4c581a6ee5a3-istiod-ca-cert\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs\" (UID: \"40240c3e-0f73-4ac9-80c1-4c581a6ee5a3\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs"
Oct 31 19:30:18.335733 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:18.335482    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40240c3e-0f73-4ac9-80c1-4c581a6ee5a3-workload-socket\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs\" (UID: \"40240c3e-0f73-4ac9-80c1-4c581a6ee5a3\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs"
Oct 31 19:30:18.335733 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:18.335517    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40240c3e-0f73-4ac9-80c1-4c581a6ee5a3-credential-socket\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs\" (UID: \"40240c3e-0f73-4ac9-80c1-4c581a6ee5a3\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs"
Oct 31 19:30:18.335733 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:18.335599    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs5dd\" (UniqueName: \"kubernetes.io/projected/40240c3e-0f73-4ac9-80c1-4c581a6ee5a3-kube-api-access-hs5dd\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs\" (UID: \"40240c3e-0f73-4ac9-80c1-4c581a6ee5a3\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs"
Oct 31 19:30:18.335887 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:18.335757    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40240c3e-0f73-4ac9-80c1-4c581a6ee5a3-workload-certs\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs\" (UID: \"40240c3e-0f73-4ac9-80c1-4c581a6ee5a3\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs"
Oct 31 19:30:18.335941 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:18.335909    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40240c3e-0f73-4ac9-80c1-4c581a6ee5a3-istio-data\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs\" (UID: \"40240c3e-0f73-4ac9-80c1-4c581a6ee5a3\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs"
Oct 31 19:30:18.336043 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:18.336013    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40240c3e-0f73-4ac9-80c1-4c581a6ee5a3-workload-socket\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs\" (UID: \"40240c3e-0f73-4ac9-80c1-4c581a6ee5a3\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs"
Oct 31 19:30:18.336170 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:18.336052    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40240c3e-0f73-4ac9-80c1-4c581a6ee5a3-credential-socket\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs\" (UID: \"40240c3e-0f73-4ac9-80c1-4c581a6ee5a3\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs"
Oct 31 19:30:18.336366 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:18.336339    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40240c3e-0f73-4ac9-80c1-4c581a6ee5a3-istiod-ca-cert\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs\" (UID: \"40240c3e-0f73-4ac9-80c1-4c581a6ee5a3\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs"
Oct 31 19:30:18.337737 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:18.337717    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40240c3e-0f73-4ac9-80c1-4c581a6ee5a3-istio-envoy\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs\" (UID: \"40240c3e-0f73-4ac9-80c1-4c581a6ee5a3\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs"
Oct 31 19:30:18.337930 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:18.337911    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40240c3e-0f73-4ac9-80c1-4c581a6ee5a3-istio-podinfo\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs\" (UID: \"40240c3e-0f73-4ac9-80c1-4c581a6ee5a3\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs"
Oct 31 19:30:18.347023 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:18.346986    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40240c3e-0f73-4ac9-80c1-4c581a6ee5a3-istio-token\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs\" (UID: \"40240c3e-0f73-4ac9-80c1-4c581a6ee5a3\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs"
Oct 31 19:30:18.347406 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:18.347386    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs5dd\" (UniqueName: \"kubernetes.io/projected/40240c3e-0f73-4ac9-80c1-4c581a6ee5a3-kube-api-access-hs5dd\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs\" (UID: \"40240c3e-0f73-4ac9-80c1-4c581a6ee5a3\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs"
Oct 31 19:30:18.504984 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:18.504907    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs"
Oct 31 19:30:18.655449 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:18.655404    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:18.786644 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:18.786616    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs"]
Oct 31 19:30:18.788592 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:30:18.788557    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40240c3e_0f73_4ac9_80c1_4c581a6ee5a3.slice/crio-01831d7db0338d667ad23d1fc3e3dc504b098dca8012500f1faefda26216fb3f WatchSource:0}: Error finding container 01831d7db0338d667ad23d1fc3e3dc504b098dca8012500f1faefda26216fb3f: Status 404 returned error can't find the container with id 01831d7db0338d667ad23d1fc3e3dc504b098dca8012500f1faefda26216fb3f
Oct 31 19:30:18.790908 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:18.790871    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:30:18.790989 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:18.790939    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:30:18.790989 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:18.790974    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:30:18.955437 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:18.955391    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs" event={"ID":"40240c3e-0f73-4ac9-80c1-4c581a6ee5a3","Type":"ContainerStarted","Data":"64b7e4f082bf7683be8f37b11ef727b2e7aa2978220a43025ff6a3deee35315a"}
Oct 31 19:30:18.955437 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:18.955444    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs" event={"ID":"40240c3e-0f73-4ac9-80c1-4c581a6ee5a3","Type":"ContainerStarted","Data":"01831d7db0338d667ad23d1fc3e3dc504b098dca8012500f1faefda26216fb3f"}
Oct 31 19:30:18.981224 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:18.981176    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs" podStartSLOduration=0.981161961 podStartE2EDuration="981.161961ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:30:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:30:18.978118388 +0000 UTC m=+649.551352516" watchObservedRunningTime="2026-10-31 19:30:18.981161961 +0000 UTC m=+649.554396089"
Oct 31 19:30:19.244466 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:19.244430    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:19.505809 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:19.505729    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs"
Oct 31 19:30:19.507175 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:19.507144    4645 patch_prober.go:29] interesting pod/e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 31 19:30:19.507351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:19.507193    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs" podUID="40240c3e-0f73-4ac9-80c1-4c581a6ee5a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 31 19:30:19.580361 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:19.576967    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:20.507788 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:20.507745    4645 patch_prober.go:29] interesting pod/e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 31 19:30:20.508479 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:20.508444    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs" podUID="40240c3e-0f73-4ac9-80c1-4c581a6ee5a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 31 19:30:20.712399 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:20.712362    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:20.942301 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:20.941353    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:21.505907 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:21.505862    4645 patch_prober.go:29] interesting pod/e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 31 19:30:21.506094 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:21.505920    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs" podUID="40240c3e-0f73-4ac9-80c1-4c581a6ee5a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 31 19:30:21.670656 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:21.670579    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:22.505788 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:22.505749    4645 patch_prober.go:29] interesting pod/e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 31 19:30:22.506003 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:22.505820    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs" podUID="40240c3e-0f73-4ac9-80c1-4c581a6ee5a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 31 19:30:22.780436 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:22.777879    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:23.482122 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:23.482075    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4"
Oct 31 19:30:23.495273 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:23.495242    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4"
Oct 31 19:30:23.505713 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:23.505676    4645 patch_prober.go:29] interesting pod/e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 31 19:30:23.505872 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:23.505750    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs" podUID="40240c3e-0f73-4ac9-80c1-4c581a6ee5a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 31 19:30:24.032609 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:24.032572    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:24.211187 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:24.211153    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4"
Oct 31 19:30:24.506158 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:24.506124    4645 patch_prober.go:29] interesting pod/e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body=
Oct 31 19:30:24.506424 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:24.506183    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs" podUID="40240c3e-0f73-4ac9-80c1-4c581a6ee5a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused"
Oct 31 19:30:24.943230 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:24.943188    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:25.658967 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:25.658928    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:26.302921 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:26.302879    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs"
Oct 31 19:30:26.303408 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:26.303394    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs"
Oct 31 19:30:26.304461 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:26.304438    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs"
Oct 31 19:30:26.433819 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:26.433785    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:27.328555 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:27.328523    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:28.039610 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:28.039575    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:28.769984 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:28.769950    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:29.549512 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:29.549481    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:30.334527 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:30.334494    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:31.336221 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:31.336182    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:32.431877 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:32.431845    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:32.772498 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:32.772391    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd"]
Oct 31 19:30:32.785758 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:32.785728    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd"
Oct 31 19:30:32.786703 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:32.786677    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd"]
Oct 31 19:30:32.801351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:32.791661    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-8b083a-openshift-default-dockercfg-jr7vh\""
Oct 31 19:30:32.925351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:32.916722    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2907ee98-8c2b-456b-9308-16f5eb074e37-istio-data\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd\" (UID: \"2907ee98-8c2b-456b-9308-16f5eb074e37\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd"
Oct 31 19:30:32.925351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:32.916794    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2907ee98-8c2b-456b-9308-16f5eb074e37-istio-podinfo\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd\" (UID: \"2907ee98-8c2b-456b-9308-16f5eb074e37\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd"
Oct 31 19:30:32.925351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:32.916896    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2907ee98-8c2b-456b-9308-16f5eb074e37-istio-envoy\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd\" (UID: \"2907ee98-8c2b-456b-9308-16f5eb074e37\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd"
Oct 31 19:30:32.925351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:32.916928    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2907ee98-8c2b-456b-9308-16f5eb074e37-istiod-ca-cert\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd\" (UID: \"2907ee98-8c2b-456b-9308-16f5eb074e37\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd"
Oct 31 19:30:32.925351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:32.916985    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2907ee98-8c2b-456b-9308-16f5eb074e37-workload-socket\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd\" (UID: \"2907ee98-8c2b-456b-9308-16f5eb074e37\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd"
Oct 31 19:30:32.925351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:32.917019    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2907ee98-8c2b-456b-9308-16f5eb074e37-workload-certs\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd\" (UID: \"2907ee98-8c2b-456b-9308-16f5eb074e37\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd"
Oct 31 19:30:32.925351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:32.917044    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2907ee98-8c2b-456b-9308-16f5eb074e37-istio-token\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd\" (UID: \"2907ee98-8c2b-456b-9308-16f5eb074e37\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd"
Oct 31 19:30:32.925351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:32.917089    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2907ee98-8c2b-456b-9308-16f5eb074e37-credential-socket\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd\" (UID: \"2907ee98-8c2b-456b-9308-16f5eb074e37\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd"
Oct 31 19:30:32.925351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:32.917123    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwzl9\" (UniqueName: \"kubernetes.io/projected/2907ee98-8c2b-456b-9308-16f5eb074e37-kube-api-access-pwzl9\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd\" (UID: \"2907ee98-8c2b-456b-9308-16f5eb074e37\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd"
Oct 31 19:30:33.018614 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:33.018569    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2907ee98-8c2b-456b-9308-16f5eb074e37-istio-data\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd\" (UID: \"2907ee98-8c2b-456b-9308-16f5eb074e37\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd"
Oct 31 19:30:33.018806 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:33.018635    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2907ee98-8c2b-456b-9308-16f5eb074e37-istio-podinfo\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd\" (UID: \"2907ee98-8c2b-456b-9308-16f5eb074e37\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd"
Oct 31 19:30:33.018806 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:33.018733    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2907ee98-8c2b-456b-9308-16f5eb074e37-istio-envoy\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd\" (UID: \"2907ee98-8c2b-456b-9308-16f5eb074e37\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd"
Oct 31 19:30:33.018806 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:33.018764    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2907ee98-8c2b-456b-9308-16f5eb074e37-istiod-ca-cert\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd\" (UID: \"2907ee98-8c2b-456b-9308-16f5eb074e37\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd"
Oct 31 19:30:33.018977 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:33.018821    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2907ee98-8c2b-456b-9308-16f5eb074e37-workload-socket\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd\" (UID: \"2907ee98-8c2b-456b-9308-16f5eb074e37\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd"
Oct 31 19:30:33.018977 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:33.018880    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2907ee98-8c2b-456b-9308-16f5eb074e37-workload-certs\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd\" (UID: \"2907ee98-8c2b-456b-9308-16f5eb074e37\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd"
Oct 31 19:30:33.018977 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:33.018926    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2907ee98-8c2b-456b-9308-16f5eb074e37-istio-token\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd\" (UID: \"2907ee98-8c2b-456b-9308-16f5eb074e37\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd"
Oct 31 19:30:33.019118 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:33.018978    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2907ee98-8c2b-456b-9308-16f5eb074e37-credential-socket\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd\" (UID: \"2907ee98-8c2b-456b-9308-16f5eb074e37\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd"
Oct 31 19:30:33.019118 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:33.019013    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwzl9\" (UniqueName: \"kubernetes.io/projected/2907ee98-8c2b-456b-9308-16f5eb074e37-kube-api-access-pwzl9\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd\" (UID: \"2907ee98-8c2b-456b-9308-16f5eb074e37\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd"
Oct 31 19:30:33.019118 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:33.019074    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2907ee98-8c2b-456b-9308-16f5eb074e37-istio-data\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd\" (UID: \"2907ee98-8c2b-456b-9308-16f5eb074e37\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd"
Oct 31 19:30:33.019817 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:33.019742    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2907ee98-8c2b-456b-9308-16f5eb074e37-workload-certs\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd\" (UID: \"2907ee98-8c2b-456b-9308-16f5eb074e37\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd"
Oct 31 19:30:33.019817 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:33.019763    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2907ee98-8c2b-456b-9308-16f5eb074e37-workload-socket\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd\" (UID: \"2907ee98-8c2b-456b-9308-16f5eb074e37\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd"
Oct 31 19:30:33.019817 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:33.019798    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2907ee98-8c2b-456b-9308-16f5eb074e37-credential-socket\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd\" (UID: \"2907ee98-8c2b-456b-9308-16f5eb074e37\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd"
Oct 31 19:30:33.020061 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:33.019971    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2907ee98-8c2b-456b-9308-16f5eb074e37-istiod-ca-cert\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd\" (UID: \"2907ee98-8c2b-456b-9308-16f5eb074e37\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd"
Oct 31 19:30:33.021493 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:33.021471    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2907ee98-8c2b-456b-9308-16f5eb074e37-istio-podinfo\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd\" (UID: \"2907ee98-8c2b-456b-9308-16f5eb074e37\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd"
Oct 31 19:30:33.021896 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:33.021872    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2907ee98-8c2b-456b-9308-16f5eb074e37-istio-envoy\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd\" (UID: \"2907ee98-8c2b-456b-9308-16f5eb074e37\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd"
Oct 31 19:30:33.034164 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:33.034090    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn"]
Oct 31 19:30:33.034411 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:33.034242    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2907ee98-8c2b-456b-9308-16f5eb074e37-istio-token\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd\" (UID: \"2907ee98-8c2b-456b-9308-16f5eb074e37\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd"
Oct 31 19:30:33.036890 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:33.036865    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwzl9\" (UniqueName: \"kubernetes.io/projected/2907ee98-8c2b-456b-9308-16f5eb074e37-kube-api-access-pwzl9\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd\" (UID: \"2907ee98-8c2b-456b-9308-16f5eb074e37\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd"
Oct 31 19:30:33.039946 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:33.039926    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn"
Oct 31 19:30:33.046238 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:33.046220    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-996c32-openshift-default-dockercfg-lsfd2\""
Oct 31 19:30:33.049980 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:33.049933    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn"]
Oct 31 19:30:33.153218 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:33.153185    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:33.186589 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:33.186544    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd"
Oct 31 19:30:33.221517 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:33.221477    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgwz2\" (UniqueName: \"kubernetes.io/projected/15ee340d-0b42-4985-a3bd-fb0ba596bc9d-kube-api-access-mgwz2\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn\" (UID: \"15ee340d-0b42-4985-a3bd-fb0ba596bc9d\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn"
Oct 31 19:30:33.221693 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:33.221549    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15ee340d-0b42-4985-a3bd-fb0ba596bc9d-istio-data\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn\" (UID: \"15ee340d-0b42-4985-a3bd-fb0ba596bc9d\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn"
Oct 31 19:30:33.221693 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:33.221605    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15ee340d-0b42-4985-a3bd-fb0ba596bc9d-istio-token\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn\" (UID: \"15ee340d-0b42-4985-a3bd-fb0ba596bc9d\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn"
Oct 31 19:30:33.221693 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:33.221630    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15ee340d-0b42-4985-a3bd-fb0ba596bc9d-istio-envoy\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn\" (UID: \"15ee340d-0b42-4985-a3bd-fb0ba596bc9d\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn"
Oct 31 19:30:33.221693 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:33.221652    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15ee340d-0b42-4985-a3bd-fb0ba596bc9d-istio-podinfo\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn\" (UID: \"15ee340d-0b42-4985-a3bd-fb0ba596bc9d\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn"
Oct 31 19:30:33.221693 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:33.221679    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15ee340d-0b42-4985-a3bd-fb0ba596bc9d-workload-socket\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn\" (UID: \"15ee340d-0b42-4985-a3bd-fb0ba596bc9d\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn"
Oct 31 19:30:33.221988 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:33.221737    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15ee340d-0b42-4985-a3bd-fb0ba596bc9d-istiod-ca-cert\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn\" (UID: \"15ee340d-0b42-4985-a3bd-fb0ba596bc9d\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn"
Oct 31 19:30:33.221988 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:33.221817    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15ee340d-0b42-4985-a3bd-fb0ba596bc9d-credential-socket\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn\" (UID: \"15ee340d-0b42-4985-a3bd-fb0ba596bc9d\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn"
Oct 31 19:30:33.222106 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:33.222060    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15ee340d-0b42-4985-a3bd-fb0ba596bc9d-workload-certs\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn\" (UID: \"15ee340d-0b42-4985-a3bd-fb0ba596bc9d\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn"
Oct 31 19:30:33.323580 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:33.323496    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15ee340d-0b42-4985-a3bd-fb0ba596bc9d-istio-data\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn\" (UID: \"15ee340d-0b42-4985-a3bd-fb0ba596bc9d\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn"
Oct 31 19:30:33.323727 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:33.323590    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15ee340d-0b42-4985-a3bd-fb0ba596bc9d-istio-token\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn\" (UID: \"15ee340d-0b42-4985-a3bd-fb0ba596bc9d\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn"
Oct 31 19:30:33.323727 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:33.323621    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15ee340d-0b42-4985-a3bd-fb0ba596bc9d-istio-envoy\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn\" (UID: \"15ee340d-0b42-4985-a3bd-fb0ba596bc9d\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn"
Oct 31 19:30:33.323727 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:33.323649    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15ee340d-0b42-4985-a3bd-fb0ba596bc9d-istio-podinfo\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn\" (UID: \"15ee340d-0b42-4985-a3bd-fb0ba596bc9d\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn"
Oct 31 19:30:33.323727 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:33.323678    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15ee340d-0b42-4985-a3bd-fb0ba596bc9d-workload-socket\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn\" (UID: \"15ee340d-0b42-4985-a3bd-fb0ba596bc9d\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn"
Oct 31 19:30:33.323866 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:33.323742    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15ee340d-0b42-4985-a3bd-fb0ba596bc9d-istiod-ca-cert\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn\" (UID: \"15ee340d-0b42-4985-a3bd-fb0ba596bc9d\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn"
Oct 31 19:30:33.323866 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:33.323802    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15ee340d-0b42-4985-a3bd-fb0ba596bc9d-credential-socket\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn\" (UID: \"15ee340d-0b42-4985-a3bd-fb0ba596bc9d\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn"
Oct 31 19:30:33.323866 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:33.323835    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15ee340d-0b42-4985-a3bd-fb0ba596bc9d-istio-data\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn\" (UID: \"15ee340d-0b42-4985-a3bd-fb0ba596bc9d\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn"
Oct 31 19:30:33.323968 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:33.323912    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15ee340d-0b42-4985-a3bd-fb0ba596bc9d-workload-certs\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn\" (UID: \"15ee340d-0b42-4985-a3bd-fb0ba596bc9d\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn"
Oct 31 19:30:33.324002 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:33.323990    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgwz2\" (UniqueName: \"kubernetes.io/projected/15ee340d-0b42-4985-a3bd-fb0ba596bc9d-kube-api-access-mgwz2\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn\" (UID: \"15ee340d-0b42-4985-a3bd-fb0ba596bc9d\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn"
Oct 31 19:30:33.324217 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:33.324196    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15ee340d-0b42-4985-a3bd-fb0ba596bc9d-credential-socket\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn\" (UID: \"15ee340d-0b42-4985-a3bd-fb0ba596bc9d\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn"
Oct 31 19:30:33.324538 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:33.324521    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15ee340d-0b42-4985-a3bd-fb0ba596bc9d-workload-certs\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn\" (UID: \"15ee340d-0b42-4985-a3bd-fb0ba596bc9d\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn"
Oct 31 19:30:33.324861 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:33.324839    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15ee340d-0b42-4985-a3bd-fb0ba596bc9d-workload-socket\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn\" (UID: \"15ee340d-0b42-4985-a3bd-fb0ba596bc9d\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn"
Oct 31 19:30:33.325468 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:33.325444    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15ee340d-0b42-4985-a3bd-fb0ba596bc9d-istiod-ca-cert\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn\" (UID: \"15ee340d-0b42-4985-a3bd-fb0ba596bc9d\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn"
Oct 31 19:30:33.327730 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:33.327571    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15ee340d-0b42-4985-a3bd-fb0ba596bc9d-istio-envoy\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn\" (UID: \"15ee340d-0b42-4985-a3bd-fb0ba596bc9d\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn"
Oct 31 19:30:33.327730 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:33.327696    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15ee340d-0b42-4985-a3bd-fb0ba596bc9d-istio-podinfo\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn\" (UID: \"15ee340d-0b42-4985-a3bd-fb0ba596bc9d\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn"
Oct 31 19:30:33.339989 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:33.339953    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15ee340d-0b42-4985-a3bd-fb0ba596bc9d-istio-token\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn\" (UID: \"15ee340d-0b42-4985-a3bd-fb0ba596bc9d\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn"
Oct 31 19:30:33.340121 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:33.340042    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgwz2\" (UniqueName: \"kubernetes.io/projected/15ee340d-0b42-4985-a3bd-fb0ba596bc9d-kube-api-access-mgwz2\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn\" (UID: \"15ee340d-0b42-4985-a3bd-fb0ba596bc9d\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn"
Oct 31 19:30:33.376629 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:33.376581    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn"
Oct 31 19:30:33.497211 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:33.497184    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd"]
Oct 31 19:30:33.500056 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:30:33.500027    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2907ee98_8c2b_456b_9308_16f5eb074e37.slice/crio-86e9debbdf5f8cc152586cf64431f6404ee245caddd5f32137510fb730e7ae85 WatchSource:0}: Error finding container 86e9debbdf5f8cc152586cf64431f6404ee245caddd5f32137510fb730e7ae85: Status 404 returned error can't find the container with id 86e9debbdf5f8cc152586cf64431f6404ee245caddd5f32137510fb730e7ae85
Oct 31 19:30:33.503495 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:33.503433    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:30:33.503587 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:33.503549    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:30:33.503637 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:33.503620    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:30:33.668651 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:33.668618    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn"]
Oct 31 19:30:33.671211 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:30:33.671186    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15ee340d_0b42_4985_a3bd_fb0ba596bc9d.slice/crio-f91073323a171a4678273685be7c7fe2e8db1e6e248988f3e8421922885ecce0 WatchSource:0}: Error finding container f91073323a171a4678273685be7c7fe2e8db1e6e248988f3e8421922885ecce0: Status 404 returned error can't find the container with id f91073323a171a4678273685be7c7fe2e8db1e6e248988f3e8421922885ecce0
Oct 31 19:30:33.674967 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:33.674927    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:30:33.675079 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:33.675016    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:30:33.675079 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:33.675067    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:30:33.957312 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:33.957215    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:34.278413 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:34.278120    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd" event={"ID":"2907ee98-8c2b-456b-9308-16f5eb074e37","Type":"ContainerStarted","Data":"51b7d601382c1887a13615212f8695d89b91b011e9e9d7db175cacfb8671b5c0"}
Oct 31 19:30:34.278413 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:34.278183    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd" event={"ID":"2907ee98-8c2b-456b-9308-16f5eb074e37","Type":"ContainerStarted","Data":"86e9debbdf5f8cc152586cf64431f6404ee245caddd5f32137510fb730e7ae85"}
Oct 31 19:30:34.287353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:34.286936    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn" event={"ID":"15ee340d-0b42-4985-a3bd-fb0ba596bc9d","Type":"ContainerStarted","Data":"bc1463068f4e9e25a981a20fa1ddf217cc71add26b92cfd2e7e417f52a531629"}
Oct 31 19:30:34.287353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:34.287010    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn" event={"ID":"15ee340d-0b42-4985-a3bd-fb0ba596bc9d","Type":"ContainerStarted","Data":"f91073323a171a4678273685be7c7fe2e8db1e6e248988f3e8421922885ecce0"}
Oct 31 19:30:34.307354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:34.305472    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd" podStartSLOduration=2.305452689 podStartE2EDuration="2.305452689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:30:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:30:34.303871571 +0000 UTC m=+664.877105700" watchObservedRunningTime="2026-10-31 19:30:34.305452689 +0000 UTC m=+664.878686821"
Oct 31 19:30:34.346447 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:34.346382    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn" podStartSLOduration=1.3463662570000001 podStartE2EDuration="1.346366257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:30:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:30:34.33988923 +0000 UTC m=+664.913123362" watchObservedRunningTime="2026-10-31 19:30:34.346366257 +0000 UTC m=+664.919600383"
Oct 31 19:30:34.382353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:34.381712    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn"
Oct 31 19:30:34.397513 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:34.397469    4645 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 31 19:30:34.397787 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:34.397760    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn" podUID="15ee340d-0b42-4985-a3bd-fb0ba596bc9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 31 19:30:34.745716 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:34.745684    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:35.206356 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:35.205497    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd"
Oct 31 19:30:35.206586 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:35.206541    4645 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 31 19:30:35.206655 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:35.206597    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd" podUID="2907ee98-8c2b-456b-9308-16f5eb074e37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 31 19:30:35.386352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:35.383808    4645 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 31 19:30:35.386352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:35.383878    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn" podUID="15ee340d-0b42-4985-a3bd-fb0ba596bc9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 31 19:30:35.535677 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:35.535581    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:36.192352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:36.191826    4645 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 31 19:30:36.192352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:36.191893    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd" podUID="2907ee98-8c2b-456b-9308-16f5eb074e37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 31 19:30:36.246184 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:30:36.246136    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 19:30:36.379353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:36.377795    4645 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 31 19:30:36.379353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:36.377882    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn" podUID="15ee340d-0b42-4985-a3bd-fb0ba596bc9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 31 19:30:36.405937 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:36.405900    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:37.193806 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:37.193765    4645 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 31 19:30:37.194462 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:37.194417    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd" podUID="2907ee98-8c2b-456b-9308-16f5eb074e37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 31 19:30:37.349027 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:37.348994    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:37.383354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:37.377741    4645 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 31 19:30:37.383354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:37.377809    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn" podUID="15ee340d-0b42-4985-a3bd-fb0ba596bc9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 31 19:30:38.187875 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:38.187840    4645 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 31 19:30:38.188199 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:38.188168    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd" podUID="2907ee98-8c2b-456b-9308-16f5eb074e37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 31 19:30:38.334267 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:38.334235    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:38.380353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:38.379726    4645 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 31 19:30:38.380353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:38.379788    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn" podUID="15ee340d-0b42-4985-a3bd-fb0ba596bc9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 31 19:30:39.143249 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:39.143212    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:39.189363 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:39.188742    4645 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 31 19:30:39.189363 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:39.188820    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd" podUID="2907ee98-8c2b-456b-9308-16f5eb074e37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 31 19:30:39.379353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:39.378761    4645 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 31 19:30:39.379353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:39.378838    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn" podUID="15ee340d-0b42-4985-a3bd-fb0ba596bc9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 31 19:30:39.930362 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:39.928123    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:40.187978 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:40.187890    4645 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Oct 31 19:30:40.187978 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:40.187961    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd" podUID="2907ee98-8c2b-456b-9308-16f5eb074e37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Oct 31 19:30:40.377781 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:40.377742    4645 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Oct 31 19:30:40.378079 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:40.378046    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn" podUID="15ee340d-0b42-4985-a3bd-fb0ba596bc9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Oct 31 19:30:40.647428 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:40.646624    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:41.191828 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:41.191794    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd"
Oct 31 19:30:41.192133 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:41.192100    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd"
Oct 31 19:30:41.192779 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:41.192756    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd"
Oct 31 19:30:41.383110 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:41.383074    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn"
Oct 31 19:30:41.383478 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:41.383439    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn"
Oct 31 19:30:41.384279 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:41.384258    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn"
Oct 31 19:30:41.544258 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:41.544151    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:42.261672 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:42.261639    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:43.048065 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:43.048029    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:43.877080 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:43.877036    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:44.874354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:44.870972    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:45.768351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:45.764820    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:46.818954 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:46.818906    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:47.971237 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:47.971205    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:48.970863 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:48.970829    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:49.758112 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:49.758079    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:50.455320 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:50.455285    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:51.244855 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:51.244816    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:51.954215 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:51.954187    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:52.798108 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:52.798077    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:53.728596 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:53.727135    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:54.563157 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:54.563121    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:55.517130 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:55.517095    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:56.240912 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:56.240874    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:56.958468 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:56.958429    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:57.753663 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:57.753632    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:58.557353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:58.553752    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:30:59.353110 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:30:59.353082    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:00.260236 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:00.260192    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:01.057933 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:01.057892    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:01.940681 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:01.940646    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:02.738979 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:02.738948    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:03.556081 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:03.556043    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:04.503646 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:04.503611    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:05.490499 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:05.490470    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:06.246260 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:31:06.246222    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 19:31:06.669706 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:06.669680    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:07.592138 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:07.592097    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:08.622256 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:08.622220    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:09.605008 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:09.604976    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:10.442005 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:10.441971    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:11.256578 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:11.256343    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:12.081227 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:12.081195    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:13.069362 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:13.069301    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:13.962009 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:13.961977    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:14.895881 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:14.895843    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:15.106728 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:15.106693    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:15.788045 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:15.788014    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:16.835096 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:16.835054    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:17.954080 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:17.954044    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:18.754370 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:18.754318    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:19.656182 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:19.656145    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:20.559752 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:20.559721    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:21.395374 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:21.395342    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:22.404608 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:22.404566    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:23.370267 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:23.370233    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:23.499457 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:23.499427    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:24.488704 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:24.488675    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:25.392606 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:25.392574    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:26.456543 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:26.456515    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:27.431986 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:27.431941    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:28.610970 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:28.610935    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:29.747498 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:29.747464    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:30.927681 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:30.927645    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:32.005290 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:32.005251    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:33.098082 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:33.098042    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:33.332778 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:33.332746    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd"]
Oct 31 19:31:33.337076 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:33.337060    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd"
Oct 31 19:31:33.363756 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:33.363691    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd"]
Oct 31 19:31:33.410114 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:33.410084    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba844f48-656b-492b-8a60-f1e85d33d977-istio-podinfo\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd\" (UID: \"ba844f48-656b-492b-8a60-f1e85d33d977\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd"
Oct 31 19:31:33.410266 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:33.410119    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba844f48-656b-492b-8a60-f1e85d33d977-istio-data\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd\" (UID: \"ba844f48-656b-492b-8a60-f1e85d33d977\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd"
Oct 31 19:31:33.410266 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:33.410144    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba844f48-656b-492b-8a60-f1e85d33d977-istio-envoy\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd\" (UID: \"ba844f48-656b-492b-8a60-f1e85d33d977\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd"
Oct 31 19:31:33.410403 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:33.410282    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba844f48-656b-492b-8a60-f1e85d33d977-istio-token\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd\" (UID: \"ba844f48-656b-492b-8a60-f1e85d33d977\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd"
Oct 31 19:31:33.410463 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:33.410408    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5psls\" (UniqueName: \"kubernetes.io/projected/ba844f48-656b-492b-8a60-f1e85d33d977-kube-api-access-5psls\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd\" (UID: \"ba844f48-656b-492b-8a60-f1e85d33d977\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd"
Oct 31 19:31:33.410500 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:33.410482    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba844f48-656b-492b-8a60-f1e85d33d977-istiod-ca-cert\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd\" (UID: \"ba844f48-656b-492b-8a60-f1e85d33d977\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd"
Oct 31 19:31:33.410567 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:33.410548    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba844f48-656b-492b-8a60-f1e85d33d977-workload-certs\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd\" (UID: \"ba844f48-656b-492b-8a60-f1e85d33d977\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd"
Oct 31 19:31:33.410666 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:33.410648    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba844f48-656b-492b-8a60-f1e85d33d977-credential-socket\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd\" (UID: \"ba844f48-656b-492b-8a60-f1e85d33d977\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd"
Oct 31 19:31:33.410725 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:33.410708    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba844f48-656b-492b-8a60-f1e85d33d977-workload-socket\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd\" (UID: \"ba844f48-656b-492b-8a60-f1e85d33d977\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd"
Oct 31 19:31:33.511999 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:33.511971    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba844f48-656b-492b-8a60-f1e85d33d977-credential-socket\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd\" (UID: \"ba844f48-656b-492b-8a60-f1e85d33d977\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd"
Oct 31 19:31:33.512143 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:33.512015    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba844f48-656b-492b-8a60-f1e85d33d977-workload-socket\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd\" (UID: \"ba844f48-656b-492b-8a60-f1e85d33d977\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd"
Oct 31 19:31:33.512143 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:33.512067    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba844f48-656b-492b-8a60-f1e85d33d977-istio-podinfo\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd\" (UID: \"ba844f48-656b-492b-8a60-f1e85d33d977\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd"
Oct 31 19:31:33.512143 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:33.512087    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba844f48-656b-492b-8a60-f1e85d33d977-istio-data\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd\" (UID: \"ba844f48-656b-492b-8a60-f1e85d33d977\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd"
Oct 31 19:31:33.512143 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:33.512113    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba844f48-656b-492b-8a60-f1e85d33d977-istio-envoy\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd\" (UID: \"ba844f48-656b-492b-8a60-f1e85d33d977\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd"
Oct 31 19:31:33.512413 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:33.512154    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba844f48-656b-492b-8a60-f1e85d33d977-istio-token\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd\" (UID: \"ba844f48-656b-492b-8a60-f1e85d33d977\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd"
Oct 31 19:31:33.512413 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:33.512221    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5psls\" (UniqueName: \"kubernetes.io/projected/ba844f48-656b-492b-8a60-f1e85d33d977-kube-api-access-5psls\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd\" (UID: \"ba844f48-656b-492b-8a60-f1e85d33d977\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd"
Oct 31 19:31:33.512413 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:33.512251    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba844f48-656b-492b-8a60-f1e85d33d977-istiod-ca-cert\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd\" (UID: \"ba844f48-656b-492b-8a60-f1e85d33d977\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd"
Oct 31 19:31:33.512413 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:33.512298    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba844f48-656b-492b-8a60-f1e85d33d977-workload-certs\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd\" (UID: \"ba844f48-656b-492b-8a60-f1e85d33d977\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd"
Oct 31 19:31:33.512610 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:33.512508    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba844f48-656b-492b-8a60-f1e85d33d977-credential-socket\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd\" (UID: \"ba844f48-656b-492b-8a60-f1e85d33d977\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd"
Oct 31 19:31:33.512610 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:33.512547    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba844f48-656b-492b-8a60-f1e85d33d977-workload-socket\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd\" (UID: \"ba844f48-656b-492b-8a60-f1e85d33d977\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd"
Oct 31 19:31:33.512692 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:33.512647    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba844f48-656b-492b-8a60-f1e85d33d977-workload-certs\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd\" (UID: \"ba844f48-656b-492b-8a60-f1e85d33d977\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd"
Oct 31 19:31:33.512779 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:33.512758    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba844f48-656b-492b-8a60-f1e85d33d977-istio-data\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd\" (UID: \"ba844f48-656b-492b-8a60-f1e85d33d977\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd"
Oct 31 19:31:33.513060 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:33.513038    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba844f48-656b-492b-8a60-f1e85d33d977-istiod-ca-cert\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd\" (UID: \"ba844f48-656b-492b-8a60-f1e85d33d977\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd"
Oct 31 19:31:33.514429 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:33.514408    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba844f48-656b-492b-8a60-f1e85d33d977-istio-envoy\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd\" (UID: \"ba844f48-656b-492b-8a60-f1e85d33d977\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd"
Oct 31 19:31:33.514781 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:33.514764    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba844f48-656b-492b-8a60-f1e85d33d977-istio-podinfo\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd\" (UID: \"ba844f48-656b-492b-8a60-f1e85d33d977\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd"
Oct 31 19:31:33.525523 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:33.525502    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba844f48-656b-492b-8a60-f1e85d33d977-istio-token\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd\" (UID: \"ba844f48-656b-492b-8a60-f1e85d33d977\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd"
Oct 31 19:31:33.526849 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:33.526832    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5psls\" (UniqueName: \"kubernetes.io/projected/ba844f48-656b-492b-8a60-f1e85d33d977-kube-api-access-5psls\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd\" (UID: \"ba844f48-656b-492b-8a60-f1e85d33d977\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd"
Oct 31 19:31:33.652778 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:33.652728    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd"
Oct 31 19:31:33.940151 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:33.940113    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd"]
Oct 31 19:31:33.945211 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:31:33.945170    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba844f48_656b_492b_8a60_f1e85d33d977.slice/crio-850a79bf90284e00599330603d201ac141e9fb3c8179e32cefa818d8d29d9c14 WatchSource:0}: Error finding container 850a79bf90284e00599330603d201ac141e9fb3c8179e32cefa818d8d29d9c14: Status 404 returned error can't find the container with id 850a79bf90284e00599330603d201ac141e9fb3c8179e32cefa818d8d29d9c14
Oct 31 19:31:33.948107 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:33.948072    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:31:33.948216 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:33.948163    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:31:33.948279 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:33.948212    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:31:33.997766 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:33.997746    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l"]
Oct 31 19:31:34.001525 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:34.001509    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l"
Oct 31 19:31:34.012559 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:34.012540    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-8b083a\"/\"openshift-service-ca.crt\""
Oct 31 19:31:34.012810 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:34.012793    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-8b083a\"/\"default-dockercfg-55sdv\""
Oct 31 19:31:34.012961 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:34.012949    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-8b083a\"/\"kube-root-ca.crt\""
Oct 31 19:31:34.013047 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:34.013026    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-8b083a\"/\"distinct-w1-8b083a-kserve-self-signed-certs\""
Oct 31 19:31:34.022450 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:34.022431    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l"]
Oct 31 19:31:34.068015 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:34.067970    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:34.119547 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:34.119519    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12bea86c-61e5-465e-82fb-b4673a4d1be9-model-cache\") pod \"distinct-w1-8b083a-kserve-856d568d74-wvv9l\" (UID: \"12bea86c-61e5-465e-82fb-b4673a4d1be9\") " pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l"
Oct 31 19:31:34.119962 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:34.119559    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6hqt\" (UniqueName: \"kubernetes.io/projected/12bea86c-61e5-465e-82fb-b4673a4d1be9-kube-api-access-n6hqt\") pod \"distinct-w1-8b083a-kserve-856d568d74-wvv9l\" (UID: \"12bea86c-61e5-465e-82fb-b4673a4d1be9\") " pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l"
Oct 31 19:31:34.119962 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:34.119642    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12bea86c-61e5-465e-82fb-b4673a4d1be9-dshm\") pod \"distinct-w1-8b083a-kserve-856d568d74-wvv9l\" (UID: \"12bea86c-61e5-465e-82fb-b4673a4d1be9\") " pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l"
Oct 31 19:31:34.119962 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:34.119660    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12bea86c-61e5-465e-82fb-b4673a4d1be9-home\") pod \"distinct-w1-8b083a-kserve-856d568d74-wvv9l\" (UID: \"12bea86c-61e5-465e-82fb-b4673a4d1be9\") " pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l"
Oct 31 19:31:34.119962 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:34.119867    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12bea86c-61e5-465e-82fb-b4673a4d1be9-tmp-dir\") pod \"distinct-w1-8b083a-kserve-856d568d74-wvv9l\" (UID: \"12bea86c-61e5-465e-82fb-b4673a4d1be9\") " pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l"
Oct 31 19:31:34.119962 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:34.119911    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12bea86c-61e5-465e-82fb-b4673a4d1be9-tls-certs\") pod \"distinct-w1-8b083a-kserve-856d568d74-wvv9l\" (UID: \"12bea86c-61e5-465e-82fb-b4673a4d1be9\") " pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l"
Oct 31 19:31:34.220367 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:34.220240    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:34.221171 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:34.221120    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12bea86c-61e5-465e-82fb-b4673a4d1be9-tmp-dir\") pod \"distinct-w1-8b083a-kserve-856d568d74-wvv9l\" (UID: \"12bea86c-61e5-465e-82fb-b4673a4d1be9\") " pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l"
Oct 31 19:31:34.221324 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:34.221243    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12bea86c-61e5-465e-82fb-b4673a4d1be9-tls-certs\") pod \"distinct-w1-8b083a-kserve-856d568d74-wvv9l\" (UID: \"12bea86c-61e5-465e-82fb-b4673a4d1be9\") " pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l"
Oct 31 19:31:34.221626 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:34.221603    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12bea86c-61e5-465e-82fb-b4673a4d1be9-tmp-dir\") pod \"distinct-w1-8b083a-kserve-856d568d74-wvv9l\" (UID: \"12bea86c-61e5-465e-82fb-b4673a4d1be9\") " pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l"
Oct 31 19:31:34.221887 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:34.221863    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12bea86c-61e5-465e-82fb-b4673a4d1be9-model-cache\") pod \"distinct-w1-8b083a-kserve-856d568d74-wvv9l\" (UID: \"12bea86c-61e5-465e-82fb-b4673a4d1be9\") " pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l"
Oct 31 19:31:34.221973 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:34.221915    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6hqt\" (UniqueName: \"kubernetes.io/projected/12bea86c-61e5-465e-82fb-b4673a4d1be9-kube-api-access-n6hqt\") pod \"distinct-w1-8b083a-kserve-856d568d74-wvv9l\" (UID: \"12bea86c-61e5-465e-82fb-b4673a4d1be9\") " pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l"
Oct 31 19:31:34.222025 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:34.221995    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12bea86c-61e5-465e-82fb-b4673a4d1be9-dshm\") pod \"distinct-w1-8b083a-kserve-856d568d74-wvv9l\" (UID: \"12bea86c-61e5-465e-82fb-b4673a4d1be9\") " pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l"
Oct 31 19:31:34.222025 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:34.222019    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12bea86c-61e5-465e-82fb-b4673a4d1be9-home\") pod \"distinct-w1-8b083a-kserve-856d568d74-wvv9l\" (UID: \"12bea86c-61e5-465e-82fb-b4673a4d1be9\") " pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l"
Oct 31 19:31:34.226370 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:34.223088    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12bea86c-61e5-465e-82fb-b4673a4d1be9-model-cache\") pod \"distinct-w1-8b083a-kserve-856d568d74-wvv9l\" (UID: \"12bea86c-61e5-465e-82fb-b4673a4d1be9\") " pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l"
Oct 31 19:31:34.226370 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:34.223318    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12bea86c-61e5-465e-82fb-b4673a4d1be9-home\") pod \"distinct-w1-8b083a-kserve-856d568d74-wvv9l\" (UID: \"12bea86c-61e5-465e-82fb-b4673a4d1be9\") " pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l"
Oct 31 19:31:34.226370 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:34.224951    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12bea86c-61e5-465e-82fb-b4673a4d1be9-tls-certs\") pod \"distinct-w1-8b083a-kserve-856d568d74-wvv9l\" (UID: \"12bea86c-61e5-465e-82fb-b4673a4d1be9\") " pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l"
Oct 31 19:31:34.227107 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:34.226816    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12bea86c-61e5-465e-82fb-b4673a4d1be9-dshm\") pod \"distinct-w1-8b083a-kserve-856d568d74-wvv9l\" (UID: \"12bea86c-61e5-465e-82fb-b4673a4d1be9\") " pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l"
Oct 31 19:31:34.239756 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:34.239717    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6hqt\" (UniqueName: \"kubernetes.io/projected/12bea86c-61e5-465e-82fb-b4673a4d1be9-kube-api-access-n6hqt\") pod \"distinct-w1-8b083a-kserve-856d568d74-wvv9l\" (UID: \"12bea86c-61e5-465e-82fb-b4673a4d1be9\") " pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l"
Oct 31 19:31:34.314210 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:34.314169    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l"
Oct 31 19:31:34.603740 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:34.603708    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l"]
Oct 31 19:31:34.605629 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:31:34.605593    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod12bea86c_61e5_465e_82fb_b4673a4d1be9.slice/crio-e8a0f20b7f3a66c704a6cdd4913fa29f426990ea6c33f3bc5507d18daf639cb9 WatchSource:0}: Error finding container e8a0f20b7f3a66c704a6cdd4913fa29f426990ea6c33f3bc5507d18daf639cb9: Status 404 returned error can't find the container with id e8a0f20b7f3a66c704a6cdd4913fa29f426990ea6c33f3bc5507d18daf639cb9
Oct 31 19:31:34.690858 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:34.690830    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd" event={"ID":"ba844f48-656b-492b-8a60-f1e85d33d977","Type":"ContainerStarted","Data":"8aa46e2c2b335792a0224ab9f712b47604539d50f509b5ba4c83b5cf44720ee0"}
Oct 31 19:31:34.690999 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:34.690894    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd" event={"ID":"ba844f48-656b-492b-8a60-f1e85d33d977","Type":"ContainerStarted","Data":"850a79bf90284e00599330603d201ac141e9fb3c8179e32cefa818d8d29d9c14"}
Oct 31 19:31:34.692872 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:34.692843    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l" event={"ID":"12bea86c-61e5-465e-82fb-b4673a4d1be9","Type":"ContainerStarted","Data":"e8a0f20b7f3a66c704a6cdd4913fa29f426990ea6c33f3bc5507d18daf639cb9"}
Oct 31 19:31:35.381378 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:35.366292    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:35.653942 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:35.653861    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd"
Oct 31 19:31:35.655263 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:35.655237    4645 patch_prober.go:29] interesting pod/e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 31 19:31:35.655426 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:35.655287    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd" podUID="ba844f48-656b-492b-8a60-f1e85d33d977" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 31 19:31:35.704274 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:35.704248    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l" event={"ID":"12bea86c-61e5-465e-82fb-b4673a4d1be9","Type":"ContainerStarted","Data":"1669a4f702146b54e4ce8bbaf55868405d56a79c124e01e0282af8325d0047bc"}
Oct 31 19:31:35.738564 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:35.738313    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd" podStartSLOduration=2.738299913 podStartE2EDuration="2.738299913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:31:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:31:34.725580681 +0000 UTC m=+725.298814807" watchObservedRunningTime="2026-10-31 19:31:35.738299913 +0000 UTC m=+726.311534042"
Oct 31 19:31:35.738564 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:35.738553    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l" podStartSLOduration=2.738546558 podStartE2EDuration="2.738546558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:31:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:31:35.737243251 +0000 UTC m=+726.310477384" watchObservedRunningTime="2026-10-31 19:31:35.738546558 +0000 UTC m=+726.311780688"
Oct 31 19:31:36.246965 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:31:36.246928    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 19:31:36.315396 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:36.315362    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l"
Oct 31 19:31:36.654033 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:36.653953    4645 patch_prober.go:29] interesting pod/e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 31 19:31:36.654033 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:36.654020    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd" podUID="ba844f48-656b-492b-8a60-f1e85d33d977" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 31 19:31:37.040880 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:37.040841    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:37.653314 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:37.653278    4645 patch_prober.go:29] interesting pod/e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 31 19:31:37.653522 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:37.653367    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd" podUID="ba844f48-656b-492b-8a60-f1e85d33d977" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 31 19:31:38.653236 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:38.653201    4645 patch_prober.go:29] interesting pod/e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 31 19:31:38.653633 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:38.653264    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd" podUID="ba844f48-656b-492b-8a60-f1e85d33d977" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 31 19:31:38.681191 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:38.681039    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:39.653962 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:39.653933    4645 patch_prober.go:29] interesting pod/e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 31 19:31:39.654391 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:39.653980    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd" podUID="ba844f48-656b-492b-8a60-f1e85d33d977" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 31 19:31:39.885712 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:39.885677    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:40.708779 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:40.708735    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd"
Oct 31 19:31:40.709577 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:40.709552    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd"
Oct 31 19:31:40.710713 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:40.710691    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd"
Oct 31 19:31:41.036270 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:41.036183    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:41.162566 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:41.162530    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:41.996881 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:41.996848    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:43.197767 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:43.197731    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:44.317358 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:44.316314    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l"
Oct 31 19:31:44.616405 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:44.615413    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:44.799352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:44.794397    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw" podUID="d504b2d3-c26d-4dcf-a7eb-efe20732ae4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:31:44.818358 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:44.816978    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x" podUID="a774e286-dcb8-4a2f-8541-26bf231eb08d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:31:44.888399 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:44.885403    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n" podUID="dbb97168-db93-4757-9a1b-f9c5eb704fc5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:31:45.201470 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:45.201416    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4" podUID="6032a997-b795-4561-805c-110cf67a9d5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:31:45.317353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:45.316421    4645 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l" podUID="12bea86c-61e5-465e-82fb-b4673a4d1be9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": context deadline exceeded"
Oct 31 19:31:45.649405 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:45.648309    4645 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw" podUID="1287db54-a587-4c11-9eb6-1606de555894" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:31:45.658355 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:45.653799    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw" podUID="1287db54-a587-4c11-9eb6-1606de555894" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": context deadline exceeded"
Oct 31 19:31:45.833355 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:45.830394    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x" podUID="a774e286-dcb8-4a2f-8541-26bf231eb08d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:31:45.833355 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:45.830752    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x"
Oct 31 19:31:46.126352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:46.125621    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:46.507930 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:46.507902    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x"
Oct 31 19:31:46.898531 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:46.898444    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-9th7f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:31:46.899089 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:46.899055    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9th7f" podUID="76bc1f3a-74b3-49b3-bde1-cc3b305260d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:31:46.934355 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:46.930440    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n" podUID="dbb97168-db93-4757-9a1b-f9c5eb704fc5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:31:46.974315 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:46.974220    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:47.041352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:47.022475    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-6k6cd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:31:47.041352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:47.022561    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6k6cd" podUID="a8fca631-527d-46c0-9197-5edd7762adb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:31:47.366859 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:47.366418    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-msrfk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:31:47.366859 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:47.366480    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-msrfk" podUID="48b0949c-0c89-4c80-a1dc-64d14839f620" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:31:47.709366 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:47.709321    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:48.051141 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:48.051057    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4"]
Oct 31 19:31:48.061012 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:48.060974    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4"
Oct 31 19:31:48.073803 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:48.073777    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4"]
Oct 31 19:31:48.165734 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:48.165696    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adb9b9a5-acbc-4967-b115-ed4e68a84b7e-istiod-ca-cert\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4\" (UID: \"adb9b9a5-acbc-4967-b115-ed4e68a84b7e\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4"
Oct 31 19:31:48.165910 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:48.165747    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb9wg\" (UniqueName: \"kubernetes.io/projected/adb9b9a5-acbc-4967-b115-ed4e68a84b7e-kube-api-access-rb9wg\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4\" (UID: \"adb9b9a5-acbc-4967-b115-ed4e68a84b7e\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4"
Oct 31 19:31:48.165910 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:48.165856    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb9b9a5-acbc-4967-b115-ed4e68a84b7e-workload-socket\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4\" (UID: \"adb9b9a5-acbc-4967-b115-ed4e68a84b7e\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4"
Oct 31 19:31:48.165910 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:48.165894    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adb9b9a5-acbc-4967-b115-ed4e68a84b7e-istio-data\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4\" (UID: \"adb9b9a5-acbc-4967-b115-ed4e68a84b7e\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4"
Oct 31 19:31:48.166083 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:48.165995    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb9b9a5-acbc-4967-b115-ed4e68a84b7e-credential-socket\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4\" (UID: \"adb9b9a5-acbc-4967-b115-ed4e68a84b7e\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4"
Oct 31 19:31:48.166083 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:48.166027    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adb9b9a5-acbc-4967-b115-ed4e68a84b7e-istio-token\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4\" (UID: \"adb9b9a5-acbc-4967-b115-ed4e68a84b7e\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4"
Oct 31 19:31:48.166271 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:48.166242    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adb9b9a5-acbc-4967-b115-ed4e68a84b7e-workload-certs\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4\" (UID: \"adb9b9a5-acbc-4967-b115-ed4e68a84b7e\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4"
Oct 31 19:31:48.166376 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:48.166348    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adb9b9a5-acbc-4967-b115-ed4e68a84b7e-istio-envoy\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4\" (UID: \"adb9b9a5-acbc-4967-b115-ed4e68a84b7e\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4"
Oct 31 19:31:48.166440 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:48.166392    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adb9b9a5-acbc-4967-b115-ed4e68a84b7e-istio-podinfo\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4\" (UID: \"adb9b9a5-acbc-4967-b115-ed4e68a84b7e\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4"
Oct 31 19:31:48.267269 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:48.267231    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adb9b9a5-acbc-4967-b115-ed4e68a84b7e-workload-certs\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4\" (UID: \"adb9b9a5-acbc-4967-b115-ed4e68a84b7e\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4"
Oct 31 19:31:48.267269 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:48.267283    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adb9b9a5-acbc-4967-b115-ed4e68a84b7e-istio-envoy\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4\" (UID: \"adb9b9a5-acbc-4967-b115-ed4e68a84b7e\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4"
Oct 31 19:31:48.267522 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:48.267302    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adb9b9a5-acbc-4967-b115-ed4e68a84b7e-istio-podinfo\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4\" (UID: \"adb9b9a5-acbc-4967-b115-ed4e68a84b7e\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4"
Oct 31 19:31:48.267522 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:48.267370    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adb9b9a5-acbc-4967-b115-ed4e68a84b7e-istiod-ca-cert\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4\" (UID: \"adb9b9a5-acbc-4967-b115-ed4e68a84b7e\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4"
Oct 31 19:31:48.267522 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:48.267391    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb9wg\" (UniqueName: \"kubernetes.io/projected/adb9b9a5-acbc-4967-b115-ed4e68a84b7e-kube-api-access-rb9wg\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4\" (UID: \"adb9b9a5-acbc-4967-b115-ed4e68a84b7e\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4"
Oct 31 19:31:48.267522 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:48.267422    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb9b9a5-acbc-4967-b115-ed4e68a84b7e-workload-socket\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4\" (UID: \"adb9b9a5-acbc-4967-b115-ed4e68a84b7e\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4"
Oct 31 19:31:48.267753 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:48.267727    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adb9b9a5-acbc-4967-b115-ed4e68a84b7e-workload-certs\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4\" (UID: \"adb9b9a5-acbc-4967-b115-ed4e68a84b7e\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4"
Oct 31 19:31:48.267805 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:48.267778    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb9b9a5-acbc-4967-b115-ed4e68a84b7e-workload-socket\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4\" (UID: \"adb9b9a5-acbc-4967-b115-ed4e68a84b7e\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4"
Oct 31 19:31:48.267805 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:48.267787    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adb9b9a5-acbc-4967-b115-ed4e68a84b7e-istio-data\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4\" (UID: \"adb9b9a5-acbc-4967-b115-ed4e68a84b7e\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4"
Oct 31 19:31:48.267925 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:48.267842    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb9b9a5-acbc-4967-b115-ed4e68a84b7e-credential-socket\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4\" (UID: \"adb9b9a5-acbc-4967-b115-ed4e68a84b7e\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4"
Oct 31 19:31:48.267925 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:48.267874    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adb9b9a5-acbc-4967-b115-ed4e68a84b7e-istio-token\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4\" (UID: \"adb9b9a5-acbc-4967-b115-ed4e68a84b7e\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4"
Oct 31 19:31:48.268156 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:48.268124    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adb9b9a5-acbc-4967-b115-ed4e68a84b7e-istio-data\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4\" (UID: \"adb9b9a5-acbc-4967-b115-ed4e68a84b7e\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4"
Oct 31 19:31:48.268156 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:48.268150    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb9b9a5-acbc-4967-b115-ed4e68a84b7e-credential-socket\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4\" (UID: \"adb9b9a5-acbc-4967-b115-ed4e68a84b7e\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4"
Oct 31 19:31:48.268287 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:48.268160    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adb9b9a5-acbc-4967-b115-ed4e68a84b7e-istiod-ca-cert\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4\" (UID: \"adb9b9a5-acbc-4967-b115-ed4e68a84b7e\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4"
Oct 31 19:31:48.269870 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:48.269845    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adb9b9a5-acbc-4967-b115-ed4e68a84b7e-istio-envoy\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4\" (UID: \"adb9b9a5-acbc-4967-b115-ed4e68a84b7e\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4"
Oct 31 19:31:48.269966 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:48.269885    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adb9b9a5-acbc-4967-b115-ed4e68a84b7e-istio-podinfo\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4\" (UID: \"adb9b9a5-acbc-4967-b115-ed4e68a84b7e\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4"
Oct 31 19:31:48.279667 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:48.279641    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adb9b9a5-acbc-4967-b115-ed4e68a84b7e-istio-token\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4\" (UID: \"adb9b9a5-acbc-4967-b115-ed4e68a84b7e\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4"
Oct 31 19:31:48.279825 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:48.279802    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb9wg\" (UniqueName: \"kubernetes.io/projected/adb9b9a5-acbc-4967-b115-ed4e68a84b7e-kube-api-access-rb9wg\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4\" (UID: \"adb9b9a5-acbc-4967-b115-ed4e68a84b7e\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4"
Oct 31 19:31:48.377688 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:48.377627    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4"
Oct 31 19:31:48.483899 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:48.483867    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk"]
Oct 31 19:31:48.488061 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:48.488038    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk"
Oct 31 19:31:48.501075 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:48.501052    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk"]
Oct 31 19:31:48.571933 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:48.571906    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99-credential-socket\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk\" (UID: \"1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk"
Oct 31 19:31:48.572060 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:48.571979    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99-workload-certs\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk\" (UID: \"1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk"
Oct 31 19:31:48.572060 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:48.572023    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99-istio-podinfo\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk\" (UID: \"1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk"
Oct 31 19:31:48.572166 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:48.572109    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99-istio-envoy\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk\" (UID: \"1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk"
Oct 31 19:31:48.572215 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:48.572200    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99-istiod-ca-cert\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk\" (UID: \"1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk"
Oct 31 19:31:48.572272 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:48.572258    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99-workload-socket\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk\" (UID: \"1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk"
Oct 31 19:31:48.572303 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:48.572279    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99-istio-token\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk\" (UID: \"1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk"
Oct 31 19:31:48.572387 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:48.572375    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4b8r\" (UniqueName: \"kubernetes.io/projected/1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99-kube-api-access-n4b8r\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk\" (UID: \"1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk"
Oct 31 19:31:48.572474 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:48.572461    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99-istio-data\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk\" (UID: \"1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk"
Oct 31 19:31:48.652077 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:48.652009    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:48.657712 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:48.657691    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4"]
Oct 31 19:31:48.658841 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:31:48.658817    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadb9b9a5_acbc_4967_b115_ed4e68a84b7e.slice/crio-d29eb8b9e3aeb80456687f5b4b7617f0d678c6f780a8193f4db3995a0603a642 WatchSource:0}: Error finding container d29eb8b9e3aeb80456687f5b4b7617f0d678c6f780a8193f4db3995a0603a642: Status 404 returned error can't find the container with id d29eb8b9e3aeb80456687f5b4b7617f0d678c6f780a8193f4db3995a0603a642
Oct 31 19:31:48.661088 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:48.661059    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:31:48.661167 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:48.661117    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:31:48.661167 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:48.661146    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:31:48.673498 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:48.673472    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99-workload-socket\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk\" (UID: \"1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk"
Oct 31 19:31:48.673605 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:48.673506    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99-istio-token\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk\" (UID: \"1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk"
Oct 31 19:31:48.673661 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:48.673612    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4b8r\" (UniqueName: \"kubernetes.io/projected/1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99-kube-api-access-n4b8r\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk\" (UID: \"1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk"
Oct 31 19:31:48.673752 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:48.673730    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99-istio-data\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk\" (UID: \"1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk"
Oct 31 19:31:48.673815 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:48.673776    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99-credential-socket\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk\" (UID: \"1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk"
Oct 31 19:31:48.673902 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:48.673883    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99-workload-certs\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk\" (UID: \"1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk"
Oct 31 19:31:48.673962 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:48.673916    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99-istio-podinfo\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk\" (UID: \"1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk"
Oct 31 19:31:48.673962 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:48.673942    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99-istio-envoy\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk\" (UID: \"1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk"
Oct 31 19:31:48.674068 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:48.673884    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99-workload-socket\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk\" (UID: \"1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk"
Oct 31 19:31:48.674068 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:48.674021    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99-istiod-ca-cert\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk\" (UID: \"1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk"
Oct 31 19:31:48.674170 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:48.674068    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99-istio-data\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk\" (UID: \"1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk"
Oct 31 19:31:48.674284 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:48.674266    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99-workload-certs\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk\" (UID: \"1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk"
Oct 31 19:31:48.674813 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:48.674790    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99-credential-socket\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk\" (UID: \"1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk"
Oct 31 19:31:48.674813 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:48.674799    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99-istiod-ca-cert\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk\" (UID: \"1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk"
Oct 31 19:31:48.676315 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:48.676292    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99-istio-envoy\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk\" (UID: \"1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk"
Oct 31 19:31:48.676603 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:48.676586    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99-istio-podinfo\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk\" (UID: \"1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk"
Oct 31 19:31:48.685369 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:48.685344    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99-istio-token\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk\" (UID: \"1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk"
Oct 31 19:31:48.685857 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:48.685729    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4b8r\" (UniqueName: \"kubernetes.io/projected/1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99-kube-api-access-n4b8r\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk\" (UID: \"1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk"
Oct 31 19:31:48.808492 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:48.808458    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk"
Oct 31 19:31:48.954455 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:48.954414    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4" event={"ID":"adb9b9a5-acbc-4967-b115-ed4e68a84b7e","Type":"ContainerStarted","Data":"8687de1652c92cd6304e37f64d2cf017bdec67320d27f72c072cf9a6162a3879"}
Oct 31 19:31:48.954617 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:48.954484    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4" event={"ID":"adb9b9a5-acbc-4967-b115-ed4e68a84b7e","Type":"ContainerStarted","Data":"d29eb8b9e3aeb80456687f5b4b7617f0d678c6f780a8193f4db3995a0603a642"}
Oct 31 19:31:48.982054 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:48.982004    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4" podStartSLOduration=0.9819914 podStartE2EDuration="981.9914ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:31:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:31:48.979921698 +0000 UTC m=+739.553155828" watchObservedRunningTime="2026-10-31 19:31:48.9819914 +0000 UTC m=+739.555225528"
Oct 31 19:31:49.094594 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:49.094559    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk"]
Oct 31 19:31:49.095483 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:31:49.095457    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ba41e18_62b4_47d9_8a89_1b5a4e8a5a99.slice/crio-2db29144451663fb446b4a368a6cca61c6e28e44851271a81815a8216ce52a27 WatchSource:0}: Error finding container 2db29144451663fb446b4a368a6cca61c6e28e44851271a81815a8216ce52a27: Status 404 returned error can't find the container with id 2db29144451663fb446b4a368a6cca61c6e28e44851271a81815a8216ce52a27
Oct 31 19:31:49.097768 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:49.097738    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:31:49.097826 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:49.097799    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:31:49.097872 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:49.097831    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:31:49.275875 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:49.275788    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669"]
Oct 31 19:31:49.280480 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:49.280457    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669"
Oct 31 19:31:49.287516 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:49.287164    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-5bbf75\"/\"distinct-w5-5bbf75-kserve-self-signed-certs\""
Oct 31 19:31:49.291241 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:49.291219    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669"]
Oct 31 19:31:49.332748 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:49.332720    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:49.378660 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:49.378633    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4"
Oct 31 19:31:49.379920 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:49.379891    4645 patch_prober.go:29] interesting pod/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 31 19:31:49.380009 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:49.379952    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4" podUID="adb9b9a5-acbc-4967-b115-ed4e68a84b7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 31 19:31:49.383484 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:49.383461    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9285bee0-bf7c-4fd4-9da7-a1db021cae05-home\") pod \"distinct-w5-5bbf75-kserve-5cb5875f47-wp669\" (UID: \"9285bee0-bf7c-4fd4-9da7-a1db021cae05\") " pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669"
Oct 31 19:31:49.383574 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:49.383497    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9285bee0-bf7c-4fd4-9da7-a1db021cae05-tls-certs\") pod \"distinct-w5-5bbf75-kserve-5cb5875f47-wp669\" (UID: \"9285bee0-bf7c-4fd4-9da7-a1db021cae05\") " pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669"
Oct 31 19:31:49.383613 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:49.383581    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rghn4\" (UniqueName: \"kubernetes.io/projected/9285bee0-bf7c-4fd4-9da7-a1db021cae05-kube-api-access-rghn4\") pod \"distinct-w5-5bbf75-kserve-5cb5875f47-wp669\" (UID: \"9285bee0-bf7c-4fd4-9da7-a1db021cae05\") " pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669"
Oct 31 19:31:49.383672 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:49.383658    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9285bee0-bf7c-4fd4-9da7-a1db021cae05-tmp-dir\") pod \"distinct-w5-5bbf75-kserve-5cb5875f47-wp669\" (UID: \"9285bee0-bf7c-4fd4-9da7-a1db021cae05\") " pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669"
Oct 31 19:31:49.383745 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:49.383730    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9285bee0-bf7c-4fd4-9da7-a1db021cae05-dshm\") pod \"distinct-w5-5bbf75-kserve-5cb5875f47-wp669\" (UID: \"9285bee0-bf7c-4fd4-9da7-a1db021cae05\") " pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669"
Oct 31 19:31:49.383789 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:49.383768    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9285bee0-bf7c-4fd4-9da7-a1db021cae05-model-cache\") pod \"distinct-w5-5bbf75-kserve-5cb5875f47-wp669\" (UID: \"9285bee0-bf7c-4fd4-9da7-a1db021cae05\") " pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669"
Oct 31 19:31:49.485639 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:49.485604    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9285bee0-bf7c-4fd4-9da7-a1db021cae05-tmp-dir\") pod \"distinct-w5-5bbf75-kserve-5cb5875f47-wp669\" (UID: \"9285bee0-bf7c-4fd4-9da7-a1db021cae05\") " pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669"
Oct 31 19:31:49.485832 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:49.485736    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9285bee0-bf7c-4fd4-9da7-a1db021cae05-dshm\") pod \"distinct-w5-5bbf75-kserve-5cb5875f47-wp669\" (UID: \"9285bee0-bf7c-4fd4-9da7-a1db021cae05\") " pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669"
Oct 31 19:31:49.485832 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:49.485795    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9285bee0-bf7c-4fd4-9da7-a1db021cae05-model-cache\") pod \"distinct-w5-5bbf75-kserve-5cb5875f47-wp669\" (UID: \"9285bee0-bf7c-4fd4-9da7-a1db021cae05\") " pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669"
Oct 31 19:31:49.485968 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:49.485892    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9285bee0-bf7c-4fd4-9da7-a1db021cae05-home\") pod \"distinct-w5-5bbf75-kserve-5cb5875f47-wp669\" (UID: \"9285bee0-bf7c-4fd4-9da7-a1db021cae05\") " pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669"
Oct 31 19:31:49.485968 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:49.485917    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9285bee0-bf7c-4fd4-9da7-a1db021cae05-tls-certs\") pod \"distinct-w5-5bbf75-kserve-5cb5875f47-wp669\" (UID: \"9285bee0-bf7c-4fd4-9da7-a1db021cae05\") " pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669"
Oct 31 19:31:49.485968 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:49.485957    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rghn4\" (UniqueName: \"kubernetes.io/projected/9285bee0-bf7c-4fd4-9da7-a1db021cae05-kube-api-access-rghn4\") pod \"distinct-w5-5bbf75-kserve-5cb5875f47-wp669\" (UID: \"9285bee0-bf7c-4fd4-9da7-a1db021cae05\") " pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669"
Oct 31 19:31:49.486121 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:49.486059    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9285bee0-bf7c-4fd4-9da7-a1db021cae05-tmp-dir\") pod \"distinct-w5-5bbf75-kserve-5cb5875f47-wp669\" (UID: \"9285bee0-bf7c-4fd4-9da7-a1db021cae05\") " pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669"
Oct 31 19:31:49.486246 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:49.486226    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9285bee0-bf7c-4fd4-9da7-a1db021cae05-home\") pod \"distinct-w5-5bbf75-kserve-5cb5875f47-wp669\" (UID: \"9285bee0-bf7c-4fd4-9da7-a1db021cae05\") " pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669"
Oct 31 19:31:49.486462 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:49.486442    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9285bee0-bf7c-4fd4-9da7-a1db021cae05-model-cache\") pod \"distinct-w5-5bbf75-kserve-5cb5875f47-wp669\" (UID: \"9285bee0-bf7c-4fd4-9da7-a1db021cae05\") " pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669"
Oct 31 19:31:49.488214 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:49.488191    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9285bee0-bf7c-4fd4-9da7-a1db021cae05-dshm\") pod \"distinct-w5-5bbf75-kserve-5cb5875f47-wp669\" (UID: \"9285bee0-bf7c-4fd4-9da7-a1db021cae05\") " pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669"
Oct 31 19:31:49.488537 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:49.488506    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9285bee0-bf7c-4fd4-9da7-a1db021cae05-tls-certs\") pod \"distinct-w5-5bbf75-kserve-5cb5875f47-wp669\" (UID: \"9285bee0-bf7c-4fd4-9da7-a1db021cae05\") " pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669"
Oct 31 19:31:49.504919 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:49.500213    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rghn4\" (UniqueName: \"kubernetes.io/projected/9285bee0-bf7c-4fd4-9da7-a1db021cae05-kube-api-access-rghn4\") pod \"distinct-w5-5bbf75-kserve-5cb5875f47-wp669\" (UID: \"9285bee0-bf7c-4fd4-9da7-a1db021cae05\") " pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669"
Oct 31 19:31:49.610402 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:49.610288    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669"
Oct 31 19:31:49.732311 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:49.732282    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:49.914410 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:49.914382    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669"]
Oct 31 19:31:49.915614 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:31:49.915588    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9285bee0_bf7c_4fd4_9da7_a1db021cae05.slice/crio-e108f5ab597b3f8176df1527f7997be4cc61674360ecb579fba7593f9d0e77e2 WatchSource:0}: Error finding container e108f5ab597b3f8176df1527f7997be4cc61674360ecb579fba7593f9d0e77e2: Status 404 returned error can't find the container with id e108f5ab597b3f8176df1527f7997be4cc61674360ecb579fba7593f9d0e77e2
Oct 31 19:31:49.987903 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:49.987863    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669" event={"ID":"9285bee0-bf7c-4fd4-9da7-a1db021cae05","Type":"ContainerStarted","Data":"e108f5ab597b3f8176df1527f7997be4cc61674360ecb579fba7593f9d0e77e2"}
Oct 31 19:31:49.990735 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:49.990711    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk" event={"ID":"1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99","Type":"ContainerStarted","Data":"9764f894f229b36d3d6d8d7485ec8d3f328d69d6524026ca88a4fd6d1b51a530"}
Oct 31 19:31:49.990864 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:49.990751    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk" event={"ID":"1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99","Type":"ContainerStarted","Data":"2db29144451663fb446b4a368a6cca61c6e28e44851271a81815a8216ce52a27"}
Oct 31 19:31:50.014181 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:50.014122    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk" podStartSLOduration=2.014106435 podStartE2EDuration="2.014106435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:31:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:31:50.012067749 +0000 UTC m=+740.585301877" watchObservedRunningTime="2026-10-31 19:31:50.014106435 +0000 UTC m=+740.587340564"
Oct 31 19:31:50.385548 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:50.385448    4645 patch_prober.go:29] interesting pod/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 31 19:31:50.386134 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:50.386104    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4" podUID="adb9b9a5-acbc-4967-b115-ed4e68a84b7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 31 19:31:50.803352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:50.802321    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:50.809437 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:50.809406    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk"
Oct 31 19:31:50.810868 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:50.810838    4645 patch_prober.go:29] interesting pod/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 31 19:31:50.810974 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:50.810895    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk" podUID="1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 31 19:31:51.010763 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:51.010732    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669" event={"ID":"9285bee0-bf7c-4fd4-9da7-a1db021cae05","Type":"ContainerStarted","Data":"cd72eae84adda09d32a83ce40769c2177eda422d27bd920117ae51111a429827"}
Oct 31 19:31:51.031908 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:51.031860    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669" podStartSLOduration=2.031848087 podStartE2EDuration="2.031848087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:31:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:31:51.02835641 +0000 UTC m=+741.601590538" watchObservedRunningTime="2026-10-31 19:31:51.031848087 +0000 UTC m=+741.605082215"
Oct 31 19:31:51.382351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:51.381684    4645 patch_prober.go:29] interesting pod/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 31 19:31:51.382351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:51.381743    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4" podUID="adb9b9a5-acbc-4967-b115-ed4e68a84b7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 31 19:31:51.610838 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:51.610808    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669"
Oct 31 19:31:51.810080 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:51.809643    4645 patch_prober.go:29] interesting pod/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 31 19:31:51.810080 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:51.809700    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk" podUID="1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 31 19:31:51.834115 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:51.834088    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:52.382353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:52.379680    4645 patch_prober.go:29] interesting pod/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 31 19:31:52.382353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:52.379740    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4" podUID="adb9b9a5-acbc-4967-b115-ed4e68a84b7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 31 19:31:52.812139 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:52.811683    4645 patch_prober.go:29] interesting pod/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 31 19:31:52.812139 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:52.811743    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk" podUID="1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 31 19:31:52.869726 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:52.869696    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:53.382350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:53.380796    4645 patch_prober.go:29] interesting pod/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 31 19:31:53.382350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:53.380866    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4" podUID="adb9b9a5-acbc-4967-b115-ed4e68a84b7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 31 19:31:53.834859 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:53.834806    4645 patch_prober.go:29] interesting pod/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 31 19:31:53.835409 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:53.834903    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk" podUID="1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 31 19:31:54.379353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:54.378865    4645 patch_prober.go:29] interesting pod/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 31 19:31:54.379353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:54.378939    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4" podUID="adb9b9a5-acbc-4967-b115-ed4e68a84b7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 31 19:31:54.490353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:54.487401    4645 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4" podUID="6032a997-b795-4561-805c-110cf67a9d5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": context deadline exceeded"
Oct 31 19:31:54.519349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:54.512736    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x" podUID="a774e286-dcb8-4a2f-8541-26bf231eb08d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:31:54.656411 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:54.655471    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw" podUID="1287db54-a587-4c11-9eb6-1606de555894" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:31:54.687350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:54.687285    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:54.782468 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:54.782419    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw" podUID="d504b2d3-c26d-4dcf-a7eb-efe20732ae4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:31:54.822350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:54.821686    4645 patch_prober.go:29] interesting pod/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 31 19:31:54.822350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:54.821747    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk" podUID="1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 31 19:31:54.886353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:54.868579    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n" podUID="dbb97168-db93-4757-9a1b-f9c5eb704fc5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:31:55.325349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:55.324610    4645 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l" podUID="12bea86c-61e5-465e-82fb-b4673a4d1be9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:31:55.381176 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:55.379746    4645 patch_prober.go:29] interesting pod/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 31 19:31:55.381176 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:55.379818    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4" podUID="adb9b9a5-acbc-4967-b115-ed4e68a84b7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 31 19:31:55.526356 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:55.514408    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x" podUID="a774e286-dcb8-4a2f-8541-26bf231eb08d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 19:31:55.526356 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:55.515088    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x"
Oct 31 19:31:55.549946 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:55.549916    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x"
Oct 31 19:31:55.809909 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:55.809866    4645 patch_prober.go:29] interesting pod/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Oct 31 19:31:55.810080 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:55.809926    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk" podUID="1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Oct 31 19:31:55.910278 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:55.910242    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:56.478544 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:56.478507    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4"
Oct 31 19:31:56.478998 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:56.478962    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4"
Oct 31 19:31:56.479987 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:56.479965    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4"
Oct 31 19:31:56.833463 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:56.833370    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:57.182474 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:57.182445    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk"
Oct 31 19:31:57.184496 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:57.184474    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk"
Oct 31 19:31:57.185623 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:57.185604    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk"
Oct 31 19:31:57.208608 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:57.208582    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:57.771605 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:57.771572    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:58.710862 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:58.710829    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:59.449748 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:59.449714    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:31:59.618442 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:59.618415    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669"
Oct 31 19:31:59.623355 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:31:59.623308    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669"
Oct 31 19:32:00.122369 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:00.122340    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669"
Oct 31 19:32:00.241127 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:00.241096    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:00.745354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:00.745312    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:01.186758 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:01.186724    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:02.054145 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:02.054114    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:02.791992 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:02.791961    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm"]
Oct 31 19:32:02.796454 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:02.796436    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm"
Oct 31 19:32:02.823624 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:02.823176    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm"]
Oct 31 19:32:02.824849 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:02.824822    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f482c94-43ba-4570-9b2a-ef4886627127-istio-podinfo\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm\" (UID: \"0f482c94-43ba-4570-9b2a-ef4886627127\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm"
Oct 31 19:32:02.824962 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:02.824856    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f482c94-43ba-4570-9b2a-ef4886627127-istiod-ca-cert\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm\" (UID: \"0f482c94-43ba-4570-9b2a-ef4886627127\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm"
Oct 31 19:32:02.825044 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:02.825029    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f482c94-43ba-4570-9b2a-ef4886627127-istio-envoy\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm\" (UID: \"0f482c94-43ba-4570-9b2a-ef4886627127\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm"
Oct 31 19:32:02.825092 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:02.825050    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f482c94-43ba-4570-9b2a-ef4886627127-istio-data\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm\" (UID: \"0f482c94-43ba-4570-9b2a-ef4886627127\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm"
Oct 31 19:32:02.825153 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:02.825093    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f482c94-43ba-4570-9b2a-ef4886627127-workload-certs\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm\" (UID: \"0f482c94-43ba-4570-9b2a-ef4886627127\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm"
Oct 31 19:32:02.825238 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:02.825206    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f482c94-43ba-4570-9b2a-ef4886627127-istio-token\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm\" (UID: \"0f482c94-43ba-4570-9b2a-ef4886627127\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm"
Oct 31 19:32:02.825379 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:02.825279    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f482c94-43ba-4570-9b2a-ef4886627127-workload-socket\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm\" (UID: \"0f482c94-43ba-4570-9b2a-ef4886627127\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm"
Oct 31 19:32:02.825379 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:02.825323    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f482c94-43ba-4570-9b2a-ef4886627127-credential-socket\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm\" (UID: \"0f482c94-43ba-4570-9b2a-ef4886627127\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm"
Oct 31 19:32:02.825517 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:02.825427    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62tnz\" (UniqueName: \"kubernetes.io/projected/0f482c94-43ba-4570-9b2a-ef4886627127-kube-api-access-62tnz\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm\" (UID: \"0f482c94-43ba-4570-9b2a-ef4886627127\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm"
Oct 31 19:32:02.926821 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:02.926785    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f482c94-43ba-4570-9b2a-ef4886627127-workload-certs\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm\" (UID: \"0f482c94-43ba-4570-9b2a-ef4886627127\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm"
Oct 31 19:32:02.927030 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:02.926838    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f482c94-43ba-4570-9b2a-ef4886627127-istio-token\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm\" (UID: \"0f482c94-43ba-4570-9b2a-ef4886627127\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm"
Oct 31 19:32:02.927030 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:02.926902    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f482c94-43ba-4570-9b2a-ef4886627127-workload-socket\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm\" (UID: \"0f482c94-43ba-4570-9b2a-ef4886627127\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm"
Oct 31 19:32:02.927030 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:02.926945    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f482c94-43ba-4570-9b2a-ef4886627127-credential-socket\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm\" (UID: \"0f482c94-43ba-4570-9b2a-ef4886627127\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm"
Oct 31 19:32:02.927030 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:02.926977    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62tnz\" (UniqueName: \"kubernetes.io/projected/0f482c94-43ba-4570-9b2a-ef4886627127-kube-api-access-62tnz\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm\" (UID: \"0f482c94-43ba-4570-9b2a-ef4886627127\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm"
Oct 31 19:32:02.927030 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:02.927020    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f482c94-43ba-4570-9b2a-ef4886627127-istio-podinfo\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm\" (UID: \"0f482c94-43ba-4570-9b2a-ef4886627127\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm"
Oct 31 19:32:02.927294 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:02.927048    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f482c94-43ba-4570-9b2a-ef4886627127-istiod-ca-cert\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm\" (UID: \"0f482c94-43ba-4570-9b2a-ef4886627127\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm"
Oct 31 19:32:02.927294 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:02.927165    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f482c94-43ba-4570-9b2a-ef4886627127-workload-certs\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm\" (UID: \"0f482c94-43ba-4570-9b2a-ef4886627127\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm"
Oct 31 19:32:02.927294 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:02.927195    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f482c94-43ba-4570-9b2a-ef4886627127-istio-envoy\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm\" (UID: \"0f482c94-43ba-4570-9b2a-ef4886627127\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm"
Oct 31 19:32:02.927294 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:02.927222    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f482c94-43ba-4570-9b2a-ef4886627127-istio-data\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm\" (UID: \"0f482c94-43ba-4570-9b2a-ef4886627127\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm"
Oct 31 19:32:02.927294 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:02.927274    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f482c94-43ba-4570-9b2a-ef4886627127-workload-socket\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm\" (UID: \"0f482c94-43ba-4570-9b2a-ef4886627127\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm"
Oct 31 19:32:02.927596 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:02.927478    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f482c94-43ba-4570-9b2a-ef4886627127-credential-socket\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm\" (UID: \"0f482c94-43ba-4570-9b2a-ef4886627127\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm"
Oct 31 19:32:02.927649 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:02.927603    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f482c94-43ba-4570-9b2a-ef4886627127-istio-data\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm\" (UID: \"0f482c94-43ba-4570-9b2a-ef4886627127\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm"
Oct 31 19:32:02.927831 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:02.927813    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f482c94-43ba-4570-9b2a-ef4886627127-istiod-ca-cert\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm\" (UID: \"0f482c94-43ba-4570-9b2a-ef4886627127\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm"
Oct 31 19:32:02.929507 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:02.929489    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f482c94-43ba-4570-9b2a-ef4886627127-istio-envoy\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm\" (UID: \"0f482c94-43ba-4570-9b2a-ef4886627127\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm"
Oct 31 19:32:02.929568 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:02.929536    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f482c94-43ba-4570-9b2a-ef4886627127-istio-podinfo\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm\" (UID: \"0f482c94-43ba-4570-9b2a-ef4886627127\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm"
Oct 31 19:32:02.939689 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:02.939665    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f482c94-43ba-4570-9b2a-ef4886627127-istio-token\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm\" (UID: \"0f482c94-43ba-4570-9b2a-ef4886627127\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm"
Oct 31 19:32:02.940187 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:02.940156    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62tnz\" (UniqueName: \"kubernetes.io/projected/0f482c94-43ba-4570-9b2a-ef4886627127-kube-api-access-62tnz\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm\" (UID: \"0f482c94-43ba-4570-9b2a-ef4886627127\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm"
Oct 31 19:32:02.941755 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:02.941736    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84"]
Oct 31 19:32:02.946707 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:02.946688    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84"
Oct 31 19:32:02.961047 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:02.961023    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84"]
Oct 31 19:32:03.028702 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:03.028677    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fdda8be-d1cd-4235-a995-a6f5af5b7fff-istio-podinfo\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84\" (UID: \"1fdda8be-d1cd-4235-a995-a6f5af5b7fff\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84"
Oct 31 19:32:03.028800 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:03.028732    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fdda8be-d1cd-4235-a995-a6f5af5b7fff-workload-certs\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84\" (UID: \"1fdda8be-d1cd-4235-a995-a6f5af5b7fff\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84"
Oct 31 19:32:03.028846 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:03.028804    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fdda8be-d1cd-4235-a995-a6f5af5b7fff-istio-envoy\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84\" (UID: \"1fdda8be-d1cd-4235-a995-a6f5af5b7fff\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84"
Oct 31 19:32:03.028846 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:03.028835    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fdda8be-d1cd-4235-a995-a6f5af5b7fff-istio-token\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84\" (UID: \"1fdda8be-d1cd-4235-a995-a6f5af5b7fff\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84"
Oct 31 19:32:03.028914 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:03.028878    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msqpb\" (UniqueName: \"kubernetes.io/projected/1fdda8be-d1cd-4235-a995-a6f5af5b7fff-kube-api-access-msqpb\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84\" (UID: \"1fdda8be-d1cd-4235-a995-a6f5af5b7fff\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84"
Oct 31 19:32:03.028947 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:03.028930    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fdda8be-d1cd-4235-a995-a6f5af5b7fff-workload-socket\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84\" (UID: \"1fdda8be-d1cd-4235-a995-a6f5af5b7fff\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84"
Oct 31 19:32:03.029021 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:03.029005    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fdda8be-d1cd-4235-a995-a6f5af5b7fff-istiod-ca-cert\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84\" (UID: \"1fdda8be-d1cd-4235-a995-a6f5af5b7fff\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84"
Oct 31 19:32:03.029058 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:03.029051    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fdda8be-d1cd-4235-a995-a6f5af5b7fff-istio-data\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84\" (UID: \"1fdda8be-d1cd-4235-a995-a6f5af5b7fff\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84"
Oct 31 19:32:03.029123 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:03.029111    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fdda8be-d1cd-4235-a995-a6f5af5b7fff-credential-socket\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84\" (UID: \"1fdda8be-d1cd-4235-a995-a6f5af5b7fff\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84"
Oct 31 19:32:03.110322 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:03.110267    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm"
Oct 31 19:32:03.130320 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:03.130294    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fdda8be-d1cd-4235-a995-a6f5af5b7fff-workload-certs\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84\" (UID: \"1fdda8be-d1cd-4235-a995-a6f5af5b7fff\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84"
Oct 31 19:32:03.130438 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:03.130349    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fdda8be-d1cd-4235-a995-a6f5af5b7fff-istio-envoy\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84\" (UID: \"1fdda8be-d1cd-4235-a995-a6f5af5b7fff\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84"
Oct 31 19:32:03.130438 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:03.130377    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fdda8be-d1cd-4235-a995-a6f5af5b7fff-istio-token\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84\" (UID: \"1fdda8be-d1cd-4235-a995-a6f5af5b7fff\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84"
Oct 31 19:32:03.130563 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:03.130537    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msqpb\" (UniqueName: \"kubernetes.io/projected/1fdda8be-d1cd-4235-a995-a6f5af5b7fff-kube-api-access-msqpb\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84\" (UID: \"1fdda8be-d1cd-4235-a995-a6f5af5b7fff\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84"
Oct 31 19:32:03.130631 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:03.130617    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fdda8be-d1cd-4235-a995-a6f5af5b7fff-workload-socket\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84\" (UID: \"1fdda8be-d1cd-4235-a995-a6f5af5b7fff\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84"
Oct 31 19:32:03.130689 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:03.130680    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fdda8be-d1cd-4235-a995-a6f5af5b7fff-istiod-ca-cert\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84\" (UID: \"1fdda8be-d1cd-4235-a995-a6f5af5b7fff\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84"
Oct 31 19:32:03.130756 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:03.130736    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fdda8be-d1cd-4235-a995-a6f5af5b7fff-istio-data\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84\" (UID: \"1fdda8be-d1cd-4235-a995-a6f5af5b7fff\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84"
Oct 31 19:32:03.130812 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:03.130791    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fdda8be-d1cd-4235-a995-a6f5af5b7fff-credential-socket\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84\" (UID: \"1fdda8be-d1cd-4235-a995-a6f5af5b7fff\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84"
Oct 31 19:32:03.131043 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:03.131020    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fdda8be-d1cd-4235-a995-a6f5af5b7fff-workload-socket\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84\" (UID: \"1fdda8be-d1cd-4235-a995-a6f5af5b7fff\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84"
Oct 31 19:32:03.131119 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:03.130739    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fdda8be-d1cd-4235-a995-a6f5af5b7fff-workload-certs\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84\" (UID: \"1fdda8be-d1cd-4235-a995-a6f5af5b7fff\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84"
Oct 31 19:32:03.131119 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:03.131096    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fdda8be-d1cd-4235-a995-a6f5af5b7fff-istio-podinfo\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84\" (UID: \"1fdda8be-d1cd-4235-a995-a6f5af5b7fff\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84"
Oct 31 19:32:03.131119 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:03.131104    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fdda8be-d1cd-4235-a995-a6f5af5b7fff-credential-socket\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84\" (UID: \"1fdda8be-d1cd-4235-a995-a6f5af5b7fff\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84"
Oct 31 19:32:03.131253 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:03.131176    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fdda8be-d1cd-4235-a995-a6f5af5b7fff-istio-data\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84\" (UID: \"1fdda8be-d1cd-4235-a995-a6f5af5b7fff\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84"
Oct 31 19:32:03.131566 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:03.131545    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fdda8be-d1cd-4235-a995-a6f5af5b7fff-istiod-ca-cert\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84\" (UID: \"1fdda8be-d1cd-4235-a995-a6f5af5b7fff\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84"
Oct 31 19:32:03.133071 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:03.133050    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fdda8be-d1cd-4235-a995-a6f5af5b7fff-istio-envoy\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84\" (UID: \"1fdda8be-d1cd-4235-a995-a6f5af5b7fff\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84"
Oct 31 19:32:03.133487 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:03.133471    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fdda8be-d1cd-4235-a995-a6f5af5b7fff-istio-podinfo\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84\" (UID: \"1fdda8be-d1cd-4235-a995-a6f5af5b7fff\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84"
Oct 31 19:32:03.143453 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:03.143429    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fdda8be-d1cd-4235-a995-a6f5af5b7fff-istio-token\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84\" (UID: \"1fdda8be-d1cd-4235-a995-a6f5af5b7fff\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84"
Oct 31 19:32:03.144375 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:03.144354    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msqpb\" (UniqueName: \"kubernetes.io/projected/1fdda8be-d1cd-4235-a995-a6f5af5b7fff-kube-api-access-msqpb\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84\" (UID: \"1fdda8be-d1cd-4235-a995-a6f5af5b7fff\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84"
Oct 31 19:32:03.167043 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:03.167015    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:03.266203 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:03.266163    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84"
Oct 31 19:32:03.395495 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:03.395469    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm"]
Oct 31 19:32:03.395974 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:32:03.395954    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f482c94_43ba_4570_9b2a_ef4886627127.slice/crio-f49f4fa577c3cdf3a2cc3020d2cdf095c730b59e842de80990c07268ea35067d WatchSource:0}: Error finding container f49f4fa577c3cdf3a2cc3020d2cdf095c730b59e842de80990c07268ea35067d: Status 404 returned error can't find the container with id f49f4fa577c3cdf3a2cc3020d2cdf095c730b59e842de80990c07268ea35067d
Oct 31 19:32:03.398535 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:03.398503    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:32:03.398623 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:03.398565    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:32:03.398623 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:03.398595    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:32:03.556189 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:03.556102    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84"]
Oct 31 19:32:03.561869 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:32:03.561838    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fdda8be_d1cd_4235_a995_a6f5af5b7fff.slice/crio-93f2b6958c98229184f836e500ea312400ddf584daf9592962d5d300e0af389a WatchSource:0}: Error finding container 93f2b6958c98229184f836e500ea312400ddf584daf9592962d5d300e0af389a: Status 404 returned error can't find the container with id 93f2b6958c98229184f836e500ea312400ddf584daf9592962d5d300e0af389a
Oct 31 19:32:03.564544 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:03.564504    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:32:03.564659 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:03.564594    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:32:03.564659 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:03.564646    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:32:03.773904 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:03.773750    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w"]
Oct 31 19:32:03.779915 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:03.778438    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w"
Oct 31 19:32:03.803693 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:03.803664    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w"]
Oct 31 19:32:03.839268 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:03.839238    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/981d31f6-7349-4835-81d3-8bbf0408f0d0-workload-socket\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w\" (UID: \"981d31f6-7349-4835-81d3-8bbf0408f0d0\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w"
Oct 31 19:32:03.839478 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:03.839453    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/981d31f6-7349-4835-81d3-8bbf0408f0d0-credential-socket\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w\" (UID: \"981d31f6-7349-4835-81d3-8bbf0408f0d0\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w"
Oct 31 19:32:03.839561 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:03.839536    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/981d31f6-7349-4835-81d3-8bbf0408f0d0-istio-token\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w\" (UID: \"981d31f6-7349-4835-81d3-8bbf0408f0d0\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w"
Oct 31 19:32:03.839648 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:03.839624    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/981d31f6-7349-4835-81d3-8bbf0408f0d0-workload-certs\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w\" (UID: \"981d31f6-7349-4835-81d3-8bbf0408f0d0\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w"
Oct 31 19:32:03.839744 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:03.839727    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/981d31f6-7349-4835-81d3-8bbf0408f0d0-istio-podinfo\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w\" (UID: \"981d31f6-7349-4835-81d3-8bbf0408f0d0\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w"
Oct 31 19:32:03.839843 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:03.839772    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jccxn\" (UniqueName: \"kubernetes.io/projected/981d31f6-7349-4835-81d3-8bbf0408f0d0-kube-api-access-jccxn\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w\" (UID: \"981d31f6-7349-4835-81d3-8bbf0408f0d0\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w"
Oct 31 19:32:03.839843 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:03.839803    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/981d31f6-7349-4835-81d3-8bbf0408f0d0-istio-envoy\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w\" (UID: \"981d31f6-7349-4835-81d3-8bbf0408f0d0\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w"
Oct 31 19:32:03.839843 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:03.839830    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/981d31f6-7349-4835-81d3-8bbf0408f0d0-istio-data\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w\" (UID: \"981d31f6-7349-4835-81d3-8bbf0408f0d0\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w"
Oct 31 19:32:03.840003 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:03.839854    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/981d31f6-7349-4835-81d3-8bbf0408f0d0-istiod-ca-cert\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w\" (UID: \"981d31f6-7349-4835-81d3-8bbf0408f0d0\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w"
Oct 31 19:32:03.940754 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:03.940725    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/981d31f6-7349-4835-81d3-8bbf0408f0d0-istio-podinfo\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w\" (UID: \"981d31f6-7349-4835-81d3-8bbf0408f0d0\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w"
Oct 31 19:32:03.940904 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:03.940767    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jccxn\" (UniqueName: \"kubernetes.io/projected/981d31f6-7349-4835-81d3-8bbf0408f0d0-kube-api-access-jccxn\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w\" (UID: \"981d31f6-7349-4835-81d3-8bbf0408f0d0\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w"
Oct 31 19:32:03.940904 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:03.940788    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/981d31f6-7349-4835-81d3-8bbf0408f0d0-istio-envoy\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w\" (UID: \"981d31f6-7349-4835-81d3-8bbf0408f0d0\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w"
Oct 31 19:32:03.940904 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:03.940812    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/981d31f6-7349-4835-81d3-8bbf0408f0d0-istio-data\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w\" (UID: \"981d31f6-7349-4835-81d3-8bbf0408f0d0\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w"
Oct 31 19:32:03.940904 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:03.940856    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/981d31f6-7349-4835-81d3-8bbf0408f0d0-istiod-ca-cert\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w\" (UID: \"981d31f6-7349-4835-81d3-8bbf0408f0d0\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w"
Oct 31 19:32:03.941123 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:03.940994    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/981d31f6-7349-4835-81d3-8bbf0408f0d0-workload-socket\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w\" (UID: \"981d31f6-7349-4835-81d3-8bbf0408f0d0\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w"
Oct 31 19:32:03.941123 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:03.941078    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/981d31f6-7349-4835-81d3-8bbf0408f0d0-credential-socket\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w\" (UID: \"981d31f6-7349-4835-81d3-8bbf0408f0d0\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w"
Oct 31 19:32:03.941225 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:03.941124    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/981d31f6-7349-4835-81d3-8bbf0408f0d0-istio-token\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w\" (UID: \"981d31f6-7349-4835-81d3-8bbf0408f0d0\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w"
Oct 31 19:32:03.941225 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:03.941171    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/981d31f6-7349-4835-81d3-8bbf0408f0d0-workload-certs\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w\" (UID: \"981d31f6-7349-4835-81d3-8bbf0408f0d0\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w"
Oct 31 19:32:03.942058 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:03.941835    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/981d31f6-7349-4835-81d3-8bbf0408f0d0-istiod-ca-cert\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w\" (UID: \"981d31f6-7349-4835-81d3-8bbf0408f0d0\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w"
Oct 31 19:32:03.943193 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:03.943139    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/981d31f6-7349-4835-81d3-8bbf0408f0d0-credential-socket\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w\" (UID: \"981d31f6-7349-4835-81d3-8bbf0408f0d0\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w"
Oct 31 19:32:03.943559 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:03.943534    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/981d31f6-7349-4835-81d3-8bbf0408f0d0-istio-data\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w\" (UID: \"981d31f6-7349-4835-81d3-8bbf0408f0d0\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w"
Oct 31 19:32:03.943672 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:03.943643    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/981d31f6-7349-4835-81d3-8bbf0408f0d0-workload-socket\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w\" (UID: \"981d31f6-7349-4835-81d3-8bbf0408f0d0\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w"
Oct 31 19:32:03.944186 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:03.944164    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/981d31f6-7349-4835-81d3-8bbf0408f0d0-istio-podinfo\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w\" (UID: \"981d31f6-7349-4835-81d3-8bbf0408f0d0\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w"
Oct 31 19:32:03.944570 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:03.944544    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/981d31f6-7349-4835-81d3-8bbf0408f0d0-workload-certs\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w\" (UID: \"981d31f6-7349-4835-81d3-8bbf0408f0d0\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w"
Oct 31 19:32:03.947009 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:03.946990    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/981d31f6-7349-4835-81d3-8bbf0408f0d0-istio-envoy\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w\" (UID: \"981d31f6-7349-4835-81d3-8bbf0408f0d0\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w"
Oct 31 19:32:03.953194 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:03.953171    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jccxn\" (UniqueName: \"kubernetes.io/projected/981d31f6-7349-4835-81d3-8bbf0408f0d0-kube-api-access-jccxn\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w\" (UID: \"981d31f6-7349-4835-81d3-8bbf0408f0d0\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w"
Oct 31 19:32:03.953678 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:03.953663    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/981d31f6-7349-4835-81d3-8bbf0408f0d0-istio-token\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w\" (UID: \"981d31f6-7349-4835-81d3-8bbf0408f0d0\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w"
Oct 31 19:32:04.106801 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:04.106708    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w"
Oct 31 19:32:04.145857 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:04.145594    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84" event={"ID":"1fdda8be-d1cd-4235-a995-a6f5af5b7fff","Type":"ContainerStarted","Data":"663cfae531af8a2ac21c01f2a5fcbaa95a17b99955660cf249732bace82e8a79"}
Oct 31 19:32:04.145857 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:04.145648    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84" event={"ID":"1fdda8be-d1cd-4235-a995-a6f5af5b7fff","Type":"ContainerStarted","Data":"93f2b6958c98229184f836e500ea312400ddf584daf9592962d5d300e0af389a"}
Oct 31 19:32:04.147608 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:04.147583    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm" event={"ID":"0f482c94-43ba-4570-9b2a-ef4886627127","Type":"ContainerStarted","Data":"185cdf71b9e21f8f49d2323c2f6de1ed0c6a9f1d6544be04ca5065dd4c992af1"}
Oct 31 19:32:04.147715 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:04.147631    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm" event={"ID":"0f482c94-43ba-4570-9b2a-ef4886627127","Type":"ContainerStarted","Data":"f49f4fa577c3cdf3a2cc3020d2cdf095c730b59e842de80990c07268ea35067d"}
Oct 31 19:32:04.150083 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:04.150055    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:04.229714 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:04.229650    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm" podStartSLOduration=2.229628922 podStartE2EDuration="2.229628922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:32:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:32:04.226786007 +0000 UTC m=+754.800020139" watchObservedRunningTime="2026-10-31 19:32:04.229628922 +0000 UTC m=+754.802863053"
Oct 31 19:32:04.231093 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:04.230821    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84" podStartSLOduration=2.230804308 podStartE2EDuration="2.230804308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:32:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:32:04.184651266 +0000 UTC m=+754.757885396" watchObservedRunningTime="2026-10-31 19:32:04.230804308 +0000 UTC m=+754.804038439"
Oct 31 19:32:04.287693 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:04.287491    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84"
Oct 31 19:32:04.291224 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:04.291185    4645 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 31 19:32:04.291431 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:04.291245    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84" podUID="1fdda8be-d1cd-4235-a995-a6f5af5b7fff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 31 19:32:04.331155 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:04.331127    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l"
Oct 31 19:32:04.344153 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:04.344125    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l"
Oct 31 19:32:04.437592 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:04.437565    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w"]
Oct 31 19:32:04.438377 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:32:04.438351    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod981d31f6_7349_4835_81d3_8bbf0408f0d0.slice/crio-7f352ef3150f230937aa1424290ae09e7b862bd63ef9b705b9f4734a8f4da679 WatchSource:0}: Error finding container 7f352ef3150f230937aa1424290ae09e7b862bd63ef9b705b9f4734a8f4da679: Status 404 returned error can't find the container with id 7f352ef3150f230937aa1424290ae09e7b862bd63ef9b705b9f4734a8f4da679
Oct 31 19:32:04.441362 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:04.441315    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:32:04.441438 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:04.441409    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:32:04.441493 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:04.441442    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:32:05.045528 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:05.044496    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:05.116599 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:05.116567    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm"
Oct 31 19:32:05.116995 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:05.116974    4645 patch_prober.go:29] interesting pod/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 31 19:32:05.117151 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:05.117126    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm" podUID="0f482c94-43ba-4570-9b2a-ef4886627127" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 31 19:32:05.175016 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:05.174981    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w" event={"ID":"981d31f6-7349-4835-81d3-8bbf0408f0d0","Type":"ContainerStarted","Data":"d96ae52208eacbcc7609c5b75fee45de5cc01d64ba1eb915cec770ea57d8f827"}
Oct 31 19:32:05.175276 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:05.175253    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w" event={"ID":"981d31f6-7349-4835-81d3-8bbf0408f0d0","Type":"ContainerStarted","Data":"7f352ef3150f230937aa1424290ae09e7b862bd63ef9b705b9f4734a8f4da679"}
Oct 31 19:32:05.278353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:05.272675    4645 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 31 19:32:05.278353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:05.272745    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84" podUID="1fdda8be-d1cd-4235-a995-a6f5af5b7fff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 31 19:32:05.790381 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:05.787605    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:06.119411 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:06.118840    4645 patch_prober.go:29] interesting pod/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 31 19:32:06.119411 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:06.118931    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm" podUID="0f482c94-43ba-4570-9b2a-ef4886627127" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 31 19:32:06.122350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:06.122306    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w"
Oct 31 19:32:06.123087 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:06.123066    4645 patch_prober.go:29] interesting pod/e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 31 19:32:06.123258 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:06.123235    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w" podUID="981d31f6-7349-4835-81d3-8bbf0408f0d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 31 19:32:06.134777 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:06.134750    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:06.267953 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:06.267921    4645 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 31 19:32:06.268282 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:06.268232    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84" podUID="1fdda8be-d1cd-4235-a995-a6f5af5b7fff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 31 19:32:06.445246 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:32:06.445041    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 19:32:07.115353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:07.114643    4645 patch_prober.go:29] interesting pod/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 31 19:32:07.115353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:07.114703    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm" podUID="0f482c94-43ba-4570-9b2a-ef4886627127" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 31 19:32:07.115353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:07.114643    4645 patch_prober.go:29] interesting pod/e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 31 19:32:07.115353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:07.115092    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w" podUID="981d31f6-7349-4835-81d3-8bbf0408f0d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 31 19:32:07.180392 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:07.180360    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:07.284533 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:07.284501    4645 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 31 19:32:07.285000 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:07.284970    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84" podUID="1fdda8be-d1cd-4235-a995-a6f5af5b7fff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 31 19:32:08.142354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:08.129003    4645 patch_prober.go:29] interesting pod/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 31 19:32:08.142962 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:08.142926    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm" podUID="0f482c94-43ba-4570-9b2a-ef4886627127" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 31 19:32:08.143508 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:08.142864    4645 patch_prober.go:29] interesting pod/e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 31 19:32:08.143682 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:08.143660    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w" podUID="981d31f6-7349-4835-81d3-8bbf0408f0d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 31 19:32:08.294353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:08.275696    4645 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 31 19:32:08.294353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:08.275761    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84" podUID="1fdda8be-d1cd-4235-a995-a6f5af5b7fff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 31 19:32:08.299289 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:08.298873    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:09.129352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:09.128875    4645 patch_prober.go:29] interesting pod/e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 31 19:32:09.129352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:09.128939    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w" podUID="981d31f6-7349-4835-81d3-8bbf0408f0d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 31 19:32:09.129633 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:09.129566    4645 patch_prober.go:29] interesting pod/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 31 19:32:09.129633 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:09.129607    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm" podUID="0f482c94-43ba-4570-9b2a-ef4886627127" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 31 19:32:09.268734 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:09.267049    4645 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 31 19:32:09.268734 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:09.267137    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84" podUID="1fdda8be-d1cd-4235-a995-a6f5af5b7fff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 31 19:32:09.525126 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:09.525091    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:09.579521 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:09.579472    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x" podUID="a774e286-dcb8-4a2f-8541-26bf231eb08d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:32:09.661735 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:09.661417    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw" podUID="1287db54-a587-4c11-9eb6-1606de555894" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:32:09.793401 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:09.777699    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw" podUID="d504b2d3-c26d-4dcf-a7eb-efe20732ae4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 31 19:32:09.912350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:09.912022    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n" podUID="dbb97168-db93-4757-9a1b-f9c5eb704fc5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:32:10.109412 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:10.108754    4645 patch_prober.go:29] interesting pod/e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 31 19:32:10.109412 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:10.108831    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w" podUID="981d31f6-7349-4835-81d3-8bbf0408f0d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 31 19:32:10.113713 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:10.113681    4645 patch_prober.go:29] interesting pod/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 31 19:32:10.113965 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:10.113934    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm" podUID="0f482c94-43ba-4570-9b2a-ef4886627127" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 31 19:32:10.124440 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:10.124403    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669" podUID="9285bee0-bf7c-4fd4-9da7-a1db021cae05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:32:10.225363 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:10.220409    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4" podUID="6032a997-b795-4561-805c-110cf67a9d5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:32:10.269099 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:10.269049    4645 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 31 19:32:10.269884 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:10.269836    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84" podUID="1fdda8be-d1cd-4235-a995-a6f5af5b7fff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 31 19:32:10.347352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:10.346399    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l" podUID="12bea86c-61e5-465e-82fb-b4673a4d1be9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:32:10.458353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:10.458223    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:10.580533 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:10.580469    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x" podUID="a774e286-dcb8-4a2f-8541-26bf231eb08d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:32:10.581846 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:10.581824    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x"
Oct 31 19:32:10.615222 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:10.615185    4645 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669" podUID="9285bee0-bf7c-4fd4-9da7-a1db021cae05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:32:10.667293 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:10.667237    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w" podStartSLOduration=7.667219368 podStartE2EDuration="7.667219368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:32:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:32:05.223629848 +0000 UTC m=+755.796864016" watchObservedRunningTime="2026-10-31 19:32:10.667219368 +0000 UTC m=+761.240453498"
Oct 31 19:32:10.834017 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:10.833912    4645 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw" podUID="d504b2d3-c26d-4dcf-a7eb-efe20732ae4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:32:10.834550 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:10.834521    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw" podUID="d504b2d3-c26d-4dcf-a7eb-efe20732ae4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 31 19:32:10.835632 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:10.835614    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw"
Oct 31 19:32:10.923366 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:10.922520    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n" podUID="dbb97168-db93-4757-9a1b-f9c5eb704fc5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:32:10.923366 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:10.922941    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n"
Oct 31 19:32:11.119900 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:11.119809    4645 patch_prober.go:29] interesting pod/e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 31 19:32:11.120138 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:11.120103    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w" podUID="981d31f6-7349-4835-81d3-8bbf0408f0d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 31 19:32:11.131768 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:11.131398    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669" podUID="9285bee0-bf7c-4fd4-9da7-a1db021cae05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:32:11.131930 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:11.131788    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669"
Oct 31 19:32:11.161371 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:11.149501    4645 patch_prober.go:29] interesting pod/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 31 19:32:11.161371 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:11.149651    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm" podUID="0f482c94-43ba-4570-9b2a-ef4886627127" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 31 19:32:11.161371 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:11.151190    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x"
Oct 31 19:32:11.281451 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:11.281195    4645 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 31 19:32:11.281865 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:11.281832    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84" podUID="1fdda8be-d1cd-4235-a995-a6f5af5b7fff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 31 19:32:11.282781 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:11.282696    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669"
Oct 31 19:32:11.325063 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:11.325028    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4" podUID="6032a997-b795-4561-805c-110cf67a9d5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:32:11.330122 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:11.325797    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4"
Oct 31 19:32:11.482486 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:11.482454    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw"
Oct 31 19:32:11.483861 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:11.482791    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4"
Oct 31 19:32:11.543989 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:11.543962    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n"
Oct 31 19:32:11.779075 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:11.778988    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:12.108256 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:12.107979    4645 patch_prober.go:29] interesting pod/e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 31 19:32:12.108256 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:12.108061    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w" podUID="981d31f6-7349-4835-81d3-8bbf0408f0d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 31 19:32:12.245958 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:12.245923    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm"
Oct 31 19:32:12.246490 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:12.246463    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm"
Oct 31 19:32:12.248175 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:12.248149    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm"
Oct 31 19:32:12.271351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:12.270659    4645 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body=
Oct 31 19:32:12.271351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:12.270723    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84" podUID="1fdda8be-d1cd-4235-a995-a6f5af5b7fff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused"
Oct 31 19:32:12.751591 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:12.751524    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:13.117825 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:13.117735    4645 patch_prober.go:29] interesting pod/e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 31 19:32:13.118397 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:13.118363    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w" podUID="981d31f6-7349-4835-81d3-8bbf0408f0d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 31 19:32:13.271532 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:13.271498    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84"
Oct 31 19:32:13.271990 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:13.271967    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84"
Oct 31 19:32:13.272812 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:13.272787    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84"
Oct 31 19:32:13.547846 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:13.547814    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:14.402221 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:14.402195    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w"
Oct 31 19:32:14.402728 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:14.402666    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w"
Oct 31 19:32:14.403556 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:14.403537    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w"
Oct 31 19:32:14.439853 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:14.439802    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:14.676986 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:14.676954    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2q74f"]
Oct 31 19:32:14.682067 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:14.682041    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2q74f"
Oct 31 19:32:14.685590 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:14.685521    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 31 19:32:14.685590 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:14.685523    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 31 19:32:14.685939 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:14.685923    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 31 19:32:14.686056 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:14.685956    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-7z5vd\""
Oct 31 19:32:14.695309 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:14.695286    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2q74f"]
Oct 31 19:32:14.787690 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:14.787658    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ab41024e-f7d2-4167-be83-b7dae45cb624-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2q74f\" (UID: \"ab41024e-f7d2-4167-be83-b7dae45cb624\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2q74f"
Oct 31 19:32:14.787809 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:14.787791    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ab41024e-f7d2-4167-be83-b7dae45cb624-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2q74f\" (UID: \"ab41024e-f7d2-4167-be83-b7dae45cb624\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2q74f"
Oct 31 19:32:14.787855 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:14.787845    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab41024e-f7d2-4167-be83-b7dae45cb624-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2q74f\" (UID: \"ab41024e-f7d2-4167-be83-b7dae45cb624\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2q74f"
Oct 31 19:32:14.787903 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:14.787863    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ab41024e-f7d2-4167-be83-b7dae45cb624-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2q74f\" (UID: \"ab41024e-f7d2-4167-be83-b7dae45cb624\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2q74f"
Oct 31 19:32:14.787903 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:14.787885    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab41024e-f7d2-4167-be83-b7dae45cb624-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2q74f\" (UID: \"ab41024e-f7d2-4167-be83-b7dae45cb624\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2q74f"
Oct 31 19:32:14.788036 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:14.788015    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrw7d\" (UniqueName: \"kubernetes.io/projected/ab41024e-f7d2-4167-be83-b7dae45cb624-kube-api-access-qrw7d\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2q74f\" (UID: \"ab41024e-f7d2-4167-be83-b7dae45cb624\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2q74f"
Oct 31 19:32:14.788081 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:14.788052    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ab41024e-f7d2-4167-be83-b7dae45cb624-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2q74f\" (UID: \"ab41024e-f7d2-4167-be83-b7dae45cb624\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2q74f"
Oct 31 19:32:14.889429 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:14.889402    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab41024e-f7d2-4167-be83-b7dae45cb624-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2q74f\" (UID: \"ab41024e-f7d2-4167-be83-b7dae45cb624\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2q74f"
Oct 31 19:32:14.889527 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:14.889435    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ab41024e-f7d2-4167-be83-b7dae45cb624-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2q74f\" (UID: \"ab41024e-f7d2-4167-be83-b7dae45cb624\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2q74f"
Oct 31 19:32:14.889527 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:14.889454    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab41024e-f7d2-4167-be83-b7dae45cb624-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2q74f\" (UID: \"ab41024e-f7d2-4167-be83-b7dae45cb624\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2q74f"
Oct 31 19:32:14.889527 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:14.889512    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrw7d\" (UniqueName: \"kubernetes.io/projected/ab41024e-f7d2-4167-be83-b7dae45cb624-kube-api-access-qrw7d\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2q74f\" (UID: \"ab41024e-f7d2-4167-be83-b7dae45cb624\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2q74f"
Oct 31 19:32:14.889624 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:14.889549    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ab41024e-f7d2-4167-be83-b7dae45cb624-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2q74f\" (UID: \"ab41024e-f7d2-4167-be83-b7dae45cb624\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2q74f"
Oct 31 19:32:14.889624 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:14.889609    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ab41024e-f7d2-4167-be83-b7dae45cb624-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2q74f\" (UID: \"ab41024e-f7d2-4167-be83-b7dae45cb624\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2q74f"
Oct 31 19:32:14.889731 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:14.889714    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ab41024e-f7d2-4167-be83-b7dae45cb624-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2q74f\" (UID: \"ab41024e-f7d2-4167-be83-b7dae45cb624\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2q74f"
Oct 31 19:32:14.890459 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:14.890431    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ab41024e-f7d2-4167-be83-b7dae45cb624-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2q74f\" (UID: \"ab41024e-f7d2-4167-be83-b7dae45cb624\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2q74f"
Oct 31 19:32:14.891773 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:14.891743    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab41024e-f7d2-4167-be83-b7dae45cb624-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2q74f\" (UID: \"ab41024e-f7d2-4167-be83-b7dae45cb624\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2q74f"
Oct 31 19:32:14.891871 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:14.891844    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ab41024e-f7d2-4167-be83-b7dae45cb624-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2q74f\" (UID: \"ab41024e-f7d2-4167-be83-b7dae45cb624\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2q74f"
Oct 31 19:32:14.891939 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:14.891894    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ab41024e-f7d2-4167-be83-b7dae45cb624-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2q74f\" (UID: \"ab41024e-f7d2-4167-be83-b7dae45cb624\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2q74f"
Oct 31 19:32:14.891939 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:14.891909    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ab41024e-f7d2-4167-be83-b7dae45cb624-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2q74f\" (UID: \"ab41024e-f7d2-4167-be83-b7dae45cb624\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2q74f"
Oct 31 19:32:14.902639 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:14.902615    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab41024e-f7d2-4167-be83-b7dae45cb624-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2q74f\" (UID: \"ab41024e-f7d2-4167-be83-b7dae45cb624\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2q74f"
Oct 31 19:32:14.903192 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:14.903166    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrw7d\" (UniqueName: \"kubernetes.io/projected/ab41024e-f7d2-4167-be83-b7dae45cb624-kube-api-access-qrw7d\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2q74f\" (UID: \"ab41024e-f7d2-4167-be83-b7dae45cb624\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2q74f"
Oct 31 19:32:15.008543 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:15.008492    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2q74f"
Oct 31 19:32:15.094808 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:15.094769    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:15.282132 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:15.282066    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:15.289097 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:15.289069    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2q74f"]
Oct 31 19:32:15.289742 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:32:15.289719    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab41024e_f7d2_4167_be83_b7dae45cb624.slice/crio-5b45961fef5c106db67580e5ee5cfd6f4580224cbeaf09a1b6c1ad732387cd21 WatchSource:0}: Error finding container 5b45961fef5c106db67580e5ee5cfd6f4580224cbeaf09a1b6c1ad732387cd21: Status 404 returned error can't find the container with id 5b45961fef5c106db67580e5ee5cfd6f4580224cbeaf09a1b6c1ad732387cd21
Oct 31 19:32:15.292305 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:15.292267    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:32:15.292402 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:15.292384    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:32:15.533800 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:15.533714    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2q74f" event={"ID":"ab41024e-f7d2-4167-be83-b7dae45cb624","Type":"ContainerStarted","Data":"ae7178d8a40ffd5a4aa2fa56d51ac6443c1cbea8e924cef567f4632d835d44d8"}
Oct 31 19:32:15.533800 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:15.533767    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2q74f" event={"ID":"ab41024e-f7d2-4167-be83-b7dae45cb624","Type":"ContainerStarted","Data":"5b45961fef5c106db67580e5ee5cfd6f4580224cbeaf09a1b6c1ad732387cd21"}
Oct 31 19:32:15.580257 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:15.580178    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2q74f" podStartSLOduration=1.580142661 podStartE2EDuration="1.580142661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:32:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:32:15.573681615 +0000 UTC m=+766.146915745" watchObservedRunningTime="2026-10-31 19:32:15.580142661 +0000 UTC m=+766.153376790"
Oct 31 19:32:15.589498 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:15.589470    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:16.307273 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:16.307235    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:16.538918 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:16.538888    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2q74f"
Oct 31 19:32:16.540708 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:16.540685    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2q74f"
Oct 31 19:32:17.480769 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:17.480734    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:18.224286 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:18.224255    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:19.280233 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:19.280195    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:20.394272 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:20.394239    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:21.561309 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:21.561267    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:22.964996 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:22.964971    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:23.808203 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:23.808131    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:24.602212 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:24.602180    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:25.843176 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:25.843121    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:26.959492 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:26.959463    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:27.859224 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:27.859193    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:28.638988 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:28.638950    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:29.554644 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:29.554608    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:30.347205 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:30.347166    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:31.411348 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:31.407593    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:32.650777 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:32.650736    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:33.634432 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:33.634388    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr"]
Oct 31 19:32:33.640114 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:33.640090    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr"
Oct 31 19:32:33.661670 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:33.661644    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr"]
Oct 31 19:32:33.690661 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:33.690631    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbd78997-222e-4952-84fe-1f33dab54c0c-istio-token\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr\" (UID: \"fbd78997-222e-4952-84fe-1f33dab54c0c\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr"
Oct 31 19:32:33.690812 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:33.690678    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbd78997-222e-4952-84fe-1f33dab54c0c-istiod-ca-cert\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr\" (UID: \"fbd78997-222e-4952-84fe-1f33dab54c0c\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr"
Oct 31 19:32:33.690812 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:33.690705    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbd78997-222e-4952-84fe-1f33dab54c0c-istio-podinfo\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr\" (UID: \"fbd78997-222e-4952-84fe-1f33dab54c0c\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr"
Oct 31 19:32:33.690925 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:33.690810    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbd78997-222e-4952-84fe-1f33dab54c0c-workload-socket\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr\" (UID: \"fbd78997-222e-4952-84fe-1f33dab54c0c\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr"
Oct 31 19:32:33.690925 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:33.690883    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbd78997-222e-4952-84fe-1f33dab54c0c-workload-certs\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr\" (UID: \"fbd78997-222e-4952-84fe-1f33dab54c0c\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr"
Oct 31 19:32:33.691030 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:33.690939    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbd78997-222e-4952-84fe-1f33dab54c0c-istio-envoy\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr\" (UID: \"fbd78997-222e-4952-84fe-1f33dab54c0c\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr"
Oct 31 19:32:33.691103 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:33.691089    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbd78997-222e-4952-84fe-1f33dab54c0c-istio-data\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr\" (UID: \"fbd78997-222e-4952-84fe-1f33dab54c0c\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr"
Oct 31 19:32:33.691235 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:33.691221    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbd78997-222e-4952-84fe-1f33dab54c0c-credential-socket\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr\" (UID: \"fbd78997-222e-4952-84fe-1f33dab54c0c\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr"
Oct 31 19:32:33.691292 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:33.691251    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm729\" (UniqueName: \"kubernetes.io/projected/fbd78997-222e-4952-84fe-1f33dab54c0c-kube-api-access-cm729\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr\" (UID: \"fbd78997-222e-4952-84fe-1f33dab54c0c\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr"
Oct 31 19:32:33.701401 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:33.700340    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:33.792634 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:33.792602    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbd78997-222e-4952-84fe-1f33dab54c0c-istio-data\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr\" (UID: \"fbd78997-222e-4952-84fe-1f33dab54c0c\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr"
Oct 31 19:32:33.792813 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:33.792723    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbd78997-222e-4952-84fe-1f33dab54c0c-credential-socket\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr\" (UID: \"fbd78997-222e-4952-84fe-1f33dab54c0c\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr"
Oct 31 19:32:33.792813 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:33.792745    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm729\" (UniqueName: \"kubernetes.io/projected/fbd78997-222e-4952-84fe-1f33dab54c0c-kube-api-access-cm729\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr\" (UID: \"fbd78997-222e-4952-84fe-1f33dab54c0c\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr"
Oct 31 19:32:33.792813 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:33.792772    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbd78997-222e-4952-84fe-1f33dab54c0c-istio-token\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr\" (UID: \"fbd78997-222e-4952-84fe-1f33dab54c0c\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr"
Oct 31 19:32:33.792813 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:33.792792    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbd78997-222e-4952-84fe-1f33dab54c0c-istiod-ca-cert\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr\" (UID: \"fbd78997-222e-4952-84fe-1f33dab54c0c\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr"
Oct 31 19:32:33.793035 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:33.792815    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbd78997-222e-4952-84fe-1f33dab54c0c-istio-podinfo\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr\" (UID: \"fbd78997-222e-4952-84fe-1f33dab54c0c\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr"
Oct 31 19:32:33.793035 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:33.792983    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbd78997-222e-4952-84fe-1f33dab54c0c-workload-socket\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr\" (UID: \"fbd78997-222e-4952-84fe-1f33dab54c0c\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr"
Oct 31 19:32:33.793172 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:33.793078    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbd78997-222e-4952-84fe-1f33dab54c0c-workload-certs\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr\" (UID: \"fbd78997-222e-4952-84fe-1f33dab54c0c\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr"
Oct 31 19:32:33.793172 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:33.793084    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbd78997-222e-4952-84fe-1f33dab54c0c-istio-data\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr\" (UID: \"fbd78997-222e-4952-84fe-1f33dab54c0c\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr"
Oct 31 19:32:33.793172 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:33.793113    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbd78997-222e-4952-84fe-1f33dab54c0c-credential-socket\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr\" (UID: \"fbd78997-222e-4952-84fe-1f33dab54c0c\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr"
Oct 31 19:32:33.793364 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:33.793171    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbd78997-222e-4952-84fe-1f33dab54c0c-istio-envoy\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr\" (UID: \"fbd78997-222e-4952-84fe-1f33dab54c0c\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr"
Oct 31 19:32:33.793364 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:33.793268    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbd78997-222e-4952-84fe-1f33dab54c0c-workload-socket\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr\" (UID: \"fbd78997-222e-4952-84fe-1f33dab54c0c\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr"
Oct 31 19:32:33.793491 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:33.793407    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbd78997-222e-4952-84fe-1f33dab54c0c-workload-certs\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr\" (UID: \"fbd78997-222e-4952-84fe-1f33dab54c0c\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr"
Oct 31 19:32:33.793619 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:33.793599    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbd78997-222e-4952-84fe-1f33dab54c0c-istiod-ca-cert\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr\" (UID: \"fbd78997-222e-4952-84fe-1f33dab54c0c\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr"
Oct 31 19:32:33.795232 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:33.795216    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbd78997-222e-4952-84fe-1f33dab54c0c-istio-envoy\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr\" (UID: \"fbd78997-222e-4952-84fe-1f33dab54c0c\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr"
Oct 31 19:32:33.795398 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:33.795382    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbd78997-222e-4952-84fe-1f33dab54c0c-istio-podinfo\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr\" (UID: \"fbd78997-222e-4952-84fe-1f33dab54c0c\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr"
Oct 31 19:32:33.804478 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:33.804457    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbd78997-222e-4952-84fe-1f33dab54c0c-istio-token\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr\" (UID: \"fbd78997-222e-4952-84fe-1f33dab54c0c\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr"
Oct 31 19:32:33.804641 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:33.804621    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm729\" (UniqueName: \"kubernetes.io/projected/fbd78997-222e-4952-84fe-1f33dab54c0c-kube-api-access-cm729\") pod \"e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr\" (UID: \"fbd78997-222e-4952-84fe-1f33dab54c0c\") " pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr"
Oct 31 19:32:33.957166 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:33.957138    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr"
Oct 31 19:32:34.239945 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:34.239914    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr"]
Oct 31 19:32:34.240457 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:32:34.240432    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbd78997_222e_4952_84fe_1f33dab54c0c.slice/crio-5af469c9949eff9bc9e188d683e667898f740e532131e282547f192643320734 WatchSource:0}: Error finding container 5af469c9949eff9bc9e188d683e667898f740e532131e282547f192643320734: Status 404 returned error can't find the container with id 5af469c9949eff9bc9e188d683e667898f740e532131e282547f192643320734
Oct 31 19:32:34.243362 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:34.243302    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:32:34.243432 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:34.243394    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:32:34.243432 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:34.243425    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:32:34.720735 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:34.720697    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr" event={"ID":"fbd78997-222e-4952-84fe-1f33dab54c0c","Type":"ContainerStarted","Data":"5364b424b2114ec610c9b26810324f27f80f430b71e99baec2911f585a296972"}
Oct 31 19:32:34.721285 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:34.720761    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr" event={"ID":"fbd78997-222e-4952-84fe-1f33dab54c0c","Type":"ContainerStarted","Data":"5af469c9949eff9bc9e188d683e667898f740e532131e282547f192643320734"}
Oct 31 19:32:34.745513 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:34.745462    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr" podStartSLOduration=1.745449101 podStartE2EDuration="1.745449101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:32:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:32:34.742255273 +0000 UTC m=+785.315489401" watchObservedRunningTime="2026-10-31 19:32:34.745449101 +0000 UTC m=+785.318683230"
Oct 31 19:32:34.764686 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:34.764655    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:34.964441 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:34.964408    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr"
Oct 31 19:32:35.190462 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:35.190425    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:35.981980 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:35.981947    4645 patch_prober.go:29] interesting pod/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 31 19:32:35.982408 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:35.982000    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr" podUID="fbd78997-222e-4952-84fe-1f33dab54c0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 31 19:32:36.113354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:36.113301    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:36.247340 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:32:36.247235    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 19:32:36.978362 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:36.969631    4645 patch_prober.go:29] interesting pod/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 31 19:32:36.978362 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:36.969687    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr" podUID="fbd78997-222e-4952-84fe-1f33dab54c0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 31 19:32:37.391875 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:37.391782    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:37.963354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:37.959658    4645 patch_prober.go:29] interesting pod/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 31 19:32:37.963354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:37.959731    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr" podUID="fbd78997-222e-4952-84fe-1f33dab54c0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 31 19:32:38.724022 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:38.723986    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:38.962781 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:38.960677    4645 patch_prober.go:29] interesting pod/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 31 19:32:38.962781 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:38.960736    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr" podUID="fbd78997-222e-4952-84fe-1f33dab54c0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 31 19:32:39.978395 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:39.971667    4645 patch_prober.go:29] interesting pod/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 31 19:32:39.978395 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:39.971723    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr" podUID="fbd78997-222e-4952-84fe-1f33dab54c0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 31 19:32:40.007484 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:40.007448    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:40.958014 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:40.957980    4645 patch_prober.go:29] interesting pod/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 31 19:32:40.958193 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:40.958050    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr" podUID="fbd78997-222e-4952-84fe-1f33dab54c0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 31 19:32:41.169373 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:41.169313    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:41.962760 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:41.962731    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr"
Oct 31 19:32:41.963146 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:41.963110    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr"
Oct 31 19:32:41.964190 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:41.964168    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr"
Oct 31 19:32:42.040023 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:42.039979    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:43.010586 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:43.010548    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:43.145674 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:43.145632    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:43.853392 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:43.853357    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:44.741357 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:44.741286    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:45.335353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:45.335282    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:45.557740 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:45.557705    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:45.676017 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:45.675983    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:46.361527 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:46.361485    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:47.463659 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:47.463624    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:48.192558 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.192518    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2"]
Oct 31 19:32:48.267352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.267240    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2"]
Oct 31 19:32:48.267352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.267294    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc"]
Oct 31 19:32:48.273534 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.272436    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2"
Oct 31 19:32:48.301357 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.295697    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8332946b-f51b-45e0-befe-8d8ce8a46c7d-istio-envoy\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2\" (UID: \"8332946b-f51b-45e0-befe-8d8ce8a46c7d\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2"
Oct 31 19:32:48.301357 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.295762    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8332946b-f51b-45e0-befe-8d8ce8a46c7d-istio-podinfo\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2\" (UID: \"8332946b-f51b-45e0-befe-8d8ce8a46c7d\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2"
Oct 31 19:32:48.301357 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.295817    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8332946b-f51b-45e0-befe-8d8ce8a46c7d-istiod-ca-cert\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2\" (UID: \"8332946b-f51b-45e0-befe-8d8ce8a46c7d\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2"
Oct 31 19:32:48.301357 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.295866    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xlt4\" (UniqueName: \"kubernetes.io/projected/8332946b-f51b-45e0-befe-8d8ce8a46c7d-kube-api-access-9xlt4\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2\" (UID: \"8332946b-f51b-45e0-befe-8d8ce8a46c7d\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2"
Oct 31 19:32:48.310357 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.301708    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8332946b-f51b-45e0-befe-8d8ce8a46c7d-workload-socket\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2\" (UID: \"8332946b-f51b-45e0-befe-8d8ce8a46c7d\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2"
Oct 31 19:32:48.310357 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.301810    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8332946b-f51b-45e0-befe-8d8ce8a46c7d-workload-certs\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2\" (UID: \"8332946b-f51b-45e0-befe-8d8ce8a46c7d\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2"
Oct 31 19:32:48.310357 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.301883    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8332946b-f51b-45e0-befe-8d8ce8a46c7d-credential-socket\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2\" (UID: \"8332946b-f51b-45e0-befe-8d8ce8a46c7d\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2"
Oct 31 19:32:48.310357 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.301975    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8332946b-f51b-45e0-befe-8d8ce8a46c7d-istio-token\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2\" (UID: \"8332946b-f51b-45e0-befe-8d8ce8a46c7d\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2"
Oct 31 19:32:48.310357 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.302012    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8332946b-f51b-45e0-befe-8d8ce8a46c7d-istio-data\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2\" (UID: \"8332946b-f51b-45e0-befe-8d8ce8a46c7d\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2"
Oct 31 19:32:48.383377 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.381487    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc"]
Oct 31 19:32:48.383377 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.381541    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf"]
Oct 31 19:32:48.385250 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.385222    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc"
Oct 31 19:32:48.409569 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.408399    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8332946b-f51b-45e0-befe-8d8ce8a46c7d-istio-token\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2\" (UID: \"8332946b-f51b-45e0-befe-8d8ce8a46c7d\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2"
Oct 31 19:32:48.430347 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.421955    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8332946b-f51b-45e0-befe-8d8ce8a46c7d-istio-data\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2\" (UID: \"8332946b-f51b-45e0-befe-8d8ce8a46c7d\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2"
Oct 31 19:32:48.430347 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.422033    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8332946b-f51b-45e0-befe-8d8ce8a46c7d-istio-envoy\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2\" (UID: \"8332946b-f51b-45e0-befe-8d8ce8a46c7d\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2"
Oct 31 19:32:48.430347 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.422067    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7-workload-socket\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc\" (UID: \"a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc"
Oct 31 19:32:48.430347 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.422123    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8332946b-f51b-45e0-befe-8d8ce8a46c7d-istio-podinfo\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2\" (UID: \"8332946b-f51b-45e0-befe-8d8ce8a46c7d\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2"
Oct 31 19:32:48.430347 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.422196    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8332946b-f51b-45e0-befe-8d8ce8a46c7d-istiod-ca-cert\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2\" (UID: \"8332946b-f51b-45e0-befe-8d8ce8a46c7d\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2"
Oct 31 19:32:48.430347 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.422231    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzpd6\" (UniqueName: \"kubernetes.io/projected/a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7-kube-api-access-jzpd6\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc\" (UID: \"a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc"
Oct 31 19:32:48.430347 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.422290    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xlt4\" (UniqueName: \"kubernetes.io/projected/8332946b-f51b-45e0-befe-8d8ce8a46c7d-kube-api-access-9xlt4\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2\" (UID: \"8332946b-f51b-45e0-befe-8d8ce8a46c7d\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2"
Oct 31 19:32:48.430347 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.423606    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf"]
Oct 31 19:32:48.430347 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.423705    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf"
Oct 31 19:32:48.430347 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.425551    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8332946b-f51b-45e0-befe-8d8ce8a46c7d-workload-socket\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2\" (UID: \"8332946b-f51b-45e0-befe-8d8ce8a46c7d\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2"
Oct 31 19:32:48.430347 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.425617    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7-credential-socket\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc\" (UID: \"a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc"
Oct 31 19:32:48.430347 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.425658    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8332946b-f51b-45e0-befe-8d8ce8a46c7d-workload-certs\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2\" (UID: \"8332946b-f51b-45e0-befe-8d8ce8a46c7d\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2"
Oct 31 19:32:48.430347 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.425683    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7-istiod-ca-cert\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc\" (UID: \"a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc"
Oct 31 19:32:48.430347 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.425764    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7-istio-podinfo\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc\" (UID: \"a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc"
Oct 31 19:32:48.430347 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.425795    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8332946b-f51b-45e0-befe-8d8ce8a46c7d-credential-socket\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2\" (UID: \"8332946b-f51b-45e0-befe-8d8ce8a46c7d\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2"
Oct 31 19:32:48.430347 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.425821    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7-istio-envoy\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc\" (UID: \"a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc"
Oct 31 19:32:48.430347 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.425853    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7-istio-token\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc\" (UID: \"a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc"
Oct 31 19:32:48.430347 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.425926    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7-workload-certs\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc\" (UID: \"a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc"
Oct 31 19:32:48.430347 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.425987    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7-istio-data\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc\" (UID: \"a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc"
Oct 31 19:32:48.430347 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.426505    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8332946b-f51b-45e0-befe-8d8ce8a46c7d-istio-data\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2\" (UID: \"8332946b-f51b-45e0-befe-8d8ce8a46c7d\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2"
Oct 31 19:32:48.473427 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.442093    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8332946b-f51b-45e0-befe-8d8ce8a46c7d-istiod-ca-cert\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2\" (UID: \"8332946b-f51b-45e0-befe-8d8ce8a46c7d\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2"
Oct 31 19:32:48.473427 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.442101    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8332946b-f51b-45e0-befe-8d8ce8a46c7d-credential-socket\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2\" (UID: \"8332946b-f51b-45e0-befe-8d8ce8a46c7d\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2"
Oct 31 19:32:48.473427 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.442387    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8332946b-f51b-45e0-befe-8d8ce8a46c7d-workload-certs\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2\" (UID: \"8332946b-f51b-45e0-befe-8d8ce8a46c7d\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2"
Oct 31 19:32:48.473427 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.442767    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8332946b-f51b-45e0-befe-8d8ce8a46c7d-workload-socket\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2\" (UID: \"8332946b-f51b-45e0-befe-8d8ce8a46c7d\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2"
Oct 31 19:32:48.473427 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.460087    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xlt4\" (UniqueName: \"kubernetes.io/projected/8332946b-f51b-45e0-befe-8d8ce8a46c7d-kube-api-access-9xlt4\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2\" (UID: \"8332946b-f51b-45e0-befe-8d8ce8a46c7d\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2"
Oct 31 19:32:48.521836 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.519764    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:48.521836 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.521194    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8332946b-f51b-45e0-befe-8d8ce8a46c7d-istio-token\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2\" (UID: \"8332946b-f51b-45e0-befe-8d8ce8a46c7d\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2"
Oct 31 19:32:48.526350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.522403    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8332946b-f51b-45e0-befe-8d8ce8a46c7d-istio-envoy\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2\" (UID: \"8332946b-f51b-45e0-befe-8d8ce8a46c7d\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2"
Oct 31 19:32:48.590770 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.531618    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/506d452a-2e17-44c0-9b4e-c7ea88e7e5bf-istio-podinfo\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf\" (UID: \"506d452a-2e17-44c0-9b4e-c7ea88e7e5bf\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf"
Oct 31 19:32:48.590770 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.531664    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/506d452a-2e17-44c0-9b4e-c7ea88e7e5bf-istio-token\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf\" (UID: \"506d452a-2e17-44c0-9b4e-c7ea88e7e5bf\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf"
Oct 31 19:32:48.590770 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.531696    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/506d452a-2e17-44c0-9b4e-c7ea88e7e5bf-istio-envoy\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf\" (UID: \"506d452a-2e17-44c0-9b4e-c7ea88e7e5bf\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf"
Oct 31 19:32:48.590770 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.531738    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7-workload-certs\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc\" (UID: \"a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc"
Oct 31 19:32:48.590770 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.531766    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/506d452a-2e17-44c0-9b4e-c7ea88e7e5bf-istiod-ca-cert\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf\" (UID: \"506d452a-2e17-44c0-9b4e-c7ea88e7e5bf\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf"
Oct 31 19:32:48.590770 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.531805    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/506d452a-2e17-44c0-9b4e-c7ea88e7e5bf-credential-socket\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf\" (UID: \"506d452a-2e17-44c0-9b4e-c7ea88e7e5bf\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf"
Oct 31 19:32:48.590770 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.531846    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7-istio-data\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc\" (UID: \"a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc"
Oct 31 19:32:48.590770 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.531886    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/506d452a-2e17-44c0-9b4e-c7ea88e7e5bf-workload-socket\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf\" (UID: \"506d452a-2e17-44c0-9b4e-c7ea88e7e5bf\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf"
Oct 31 19:32:48.590770 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.531960    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7-workload-socket\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc\" (UID: \"a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc"
Oct 31 19:32:48.590770 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.532027    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jghk8\" (UniqueName: \"kubernetes.io/projected/506d452a-2e17-44c0-9b4e-c7ea88e7e5bf-kube-api-access-jghk8\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf\" (UID: \"506d452a-2e17-44c0-9b4e-c7ea88e7e5bf\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf"
Oct 31 19:32:48.590770 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.532082    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/506d452a-2e17-44c0-9b4e-c7ea88e7e5bf-istio-data\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf\" (UID: \"506d452a-2e17-44c0-9b4e-c7ea88e7e5bf\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf"
Oct 31 19:32:48.590770 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.532133    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzpd6\" (UniqueName: \"kubernetes.io/projected/a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7-kube-api-access-jzpd6\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc\" (UID: \"a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc"
Oct 31 19:32:48.590770 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.532263    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7-credential-socket\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc\" (UID: \"a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc"
Oct 31 19:32:48.590770 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.532303    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7-istiod-ca-cert\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc\" (UID: \"a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc"
Oct 31 19:32:48.590770 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.532416    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7-istio-podinfo\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc\" (UID: \"a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc"
Oct 31 19:32:48.590770 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.532449    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7-istio-envoy\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc\" (UID: \"a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc"
Oct 31 19:32:48.590770 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.532480    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7-istio-token\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc\" (UID: \"a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc"
Oct 31 19:32:48.590770 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.532508    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/506d452a-2e17-44c0-9b4e-c7ea88e7e5bf-workload-certs\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf\" (UID: \"506d452a-2e17-44c0-9b4e-c7ea88e7e5bf\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf"
Oct 31 19:32:48.590770 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.533000    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7-workload-certs\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc\" (UID: \"a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc"
Oct 31 19:32:48.590770 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.533303    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7-istio-data\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc\" (UID: \"a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc"
Oct 31 19:32:48.590770 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.533591    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7-workload-socket\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc\" (UID: \"a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc"
Oct 31 19:32:48.590770 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.534153    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7-credential-socket\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc\" (UID: \"a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc"
Oct 31 19:32:48.590770 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.540939    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7-istiod-ca-cert\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc\" (UID: \"a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc"
Oct 31 19:32:48.637351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.634342    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/506d452a-2e17-44c0-9b4e-c7ea88e7e5bf-workload-certs\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf\" (UID: \"506d452a-2e17-44c0-9b4e-c7ea88e7e5bf\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf"
Oct 31 19:32:48.637351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.634418    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/506d452a-2e17-44c0-9b4e-c7ea88e7e5bf-istio-podinfo\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf\" (UID: \"506d452a-2e17-44c0-9b4e-c7ea88e7e5bf\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf"
Oct 31 19:32:48.662352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.634451    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/506d452a-2e17-44c0-9b4e-c7ea88e7e5bf-istio-token\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf\" (UID: \"506d452a-2e17-44c0-9b4e-c7ea88e7e5bf\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf"
Oct 31 19:32:48.662352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.656524    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/506d452a-2e17-44c0-9b4e-c7ea88e7e5bf-istio-envoy\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf\" (UID: \"506d452a-2e17-44c0-9b4e-c7ea88e7e5bf\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf"
Oct 31 19:32:48.662352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.656584    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/506d452a-2e17-44c0-9b4e-c7ea88e7e5bf-istiod-ca-cert\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf\" (UID: \"506d452a-2e17-44c0-9b4e-c7ea88e7e5bf\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf"
Oct 31 19:32:48.662352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.656633    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/506d452a-2e17-44c0-9b4e-c7ea88e7e5bf-credential-socket\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf\" (UID: \"506d452a-2e17-44c0-9b4e-c7ea88e7e5bf\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf"
Oct 31 19:32:48.662352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.656692    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/506d452a-2e17-44c0-9b4e-c7ea88e7e5bf-workload-socket\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf\" (UID: \"506d452a-2e17-44c0-9b4e-c7ea88e7e5bf\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf"
Oct 31 19:32:48.662352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.656832    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jghk8\" (UniqueName: \"kubernetes.io/projected/506d452a-2e17-44c0-9b4e-c7ea88e7e5bf-kube-api-access-jghk8\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf\" (UID: \"506d452a-2e17-44c0-9b4e-c7ea88e7e5bf\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf"
Oct 31 19:32:48.662352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.656894    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/506d452a-2e17-44c0-9b4e-c7ea88e7e5bf-istio-data\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf\" (UID: \"506d452a-2e17-44c0-9b4e-c7ea88e7e5bf\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf"
Oct 31 19:32:48.662352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.661157    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzpd6\" (UniqueName: \"kubernetes.io/projected/a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7-kube-api-access-jzpd6\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc\" (UID: \"a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc"
Oct 31 19:32:48.662352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.661975    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8332946b-f51b-45e0-befe-8d8ce8a46c7d-istio-podinfo\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2\" (UID: \"8332946b-f51b-45e0-befe-8d8ce8a46c7d\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2"
Oct 31 19:32:48.662352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.635772    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/506d452a-2e17-44c0-9b4e-c7ea88e7e5bf-workload-certs\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf\" (UID: \"506d452a-2e17-44c0-9b4e-c7ea88e7e5bf\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf"
Oct 31 19:32:48.663007 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.662577    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7-istio-podinfo\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc\" (UID: \"a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc"
Oct 31 19:32:48.679356 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.664862    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7-istio-token\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc\" (UID: \"a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc"
Oct 31 19:32:48.679356 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.669018    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/506d452a-2e17-44c0-9b4e-c7ea88e7e5bf-istio-podinfo\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf\" (UID: \"506d452a-2e17-44c0-9b4e-c7ea88e7e5bf\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf"
Oct 31 19:32:48.679356 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.675193    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/506d452a-2e17-44c0-9b4e-c7ea88e7e5bf-credential-socket\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf\" (UID: \"506d452a-2e17-44c0-9b4e-c7ea88e7e5bf\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf"
Oct 31 19:32:48.679356 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.676024    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/506d452a-2e17-44c0-9b4e-c7ea88e7e5bf-istiod-ca-cert\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf\" (UID: \"506d452a-2e17-44c0-9b4e-c7ea88e7e5bf\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf"
Oct 31 19:32:48.679356 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.677156    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7-istio-envoy\") pod \"e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc\" (UID: \"a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7\") " pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc"
Oct 31 19:32:48.679722 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.679499    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/506d452a-2e17-44c0-9b4e-c7ea88e7e5bf-workload-socket\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf\" (UID: \"506d452a-2e17-44c0-9b4e-c7ea88e7e5bf\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf"
Oct 31 19:32:48.679722 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.679619    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/506d452a-2e17-44c0-9b4e-c7ea88e7e5bf-istio-data\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf\" (UID: \"506d452a-2e17-44c0-9b4e-c7ea88e7e5bf\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf"
Oct 31 19:32:48.703388 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.683548    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2"
Oct 31 19:32:48.722489 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.721919    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/506d452a-2e17-44c0-9b4e-c7ea88e7e5bf-istio-token\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf\" (UID: \"506d452a-2e17-44c0-9b4e-c7ea88e7e5bf\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf"
Oct 31 19:32:48.752083 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.751706    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc"
Oct 31 19:32:48.752083 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.751983    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/506d452a-2e17-44c0-9b4e-c7ea88e7e5bf-istio-envoy\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf\" (UID: \"506d452a-2e17-44c0-9b4e-c7ea88e7e5bf\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf"
Oct 31 19:32:48.753043 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.752988    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jghk8\" (UniqueName: \"kubernetes.io/projected/506d452a-2e17-44c0-9b4e-c7ea88e7e5bf-kube-api-access-jghk8\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf\" (UID: \"506d452a-2e17-44c0-9b4e-c7ea88e7e5bf\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf"
Oct 31 19:32:48.849352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.835823    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4"]
Oct 31 19:32:48.948693 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.947583    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf"
Oct 31 19:32:48.972367 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.961521    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4"]
Oct 31 19:32:48.972367 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:48.961647    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4"
Oct 31 19:32:49.154670 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:49.150164    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/452d967b-8434-4766-9b19-690c49adc197-istio-podinfo\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4\" (UID: \"452d967b-8434-4766-9b19-690c49adc197\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4"
Oct 31 19:32:49.154670 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:49.150209    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66r4p\" (UniqueName: \"kubernetes.io/projected/452d967b-8434-4766-9b19-690c49adc197-kube-api-access-66r4p\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4\" (UID: \"452d967b-8434-4766-9b19-690c49adc197\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4"
Oct 31 19:32:49.154670 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:49.150263    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/452d967b-8434-4766-9b19-690c49adc197-istiod-ca-cert\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4\" (UID: \"452d967b-8434-4766-9b19-690c49adc197\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4"
Oct 31 19:32:49.154670 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:49.150361    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/452d967b-8434-4766-9b19-690c49adc197-istio-token\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4\" (UID: \"452d967b-8434-4766-9b19-690c49adc197\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4"
Oct 31 19:32:49.154670 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:49.150401    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/452d967b-8434-4766-9b19-690c49adc197-workload-socket\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4\" (UID: \"452d967b-8434-4766-9b19-690c49adc197\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4"
Oct 31 19:32:49.154670 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:49.150454    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/452d967b-8434-4766-9b19-690c49adc197-istio-data\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4\" (UID: \"452d967b-8434-4766-9b19-690c49adc197\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4"
Oct 31 19:32:49.154670 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:49.150567    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/452d967b-8434-4766-9b19-690c49adc197-workload-certs\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4\" (UID: \"452d967b-8434-4766-9b19-690c49adc197\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4"
Oct 31 19:32:49.154670 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:49.150679    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/452d967b-8434-4766-9b19-690c49adc197-istio-envoy\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4\" (UID: \"452d967b-8434-4766-9b19-690c49adc197\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4"
Oct 31 19:32:49.154670 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:49.150738    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/452d967b-8434-4766-9b19-690c49adc197-credential-socket\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4\" (UID: \"452d967b-8434-4766-9b19-690c49adc197\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4"
Oct 31 19:32:49.252354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:49.252147    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/452d967b-8434-4766-9b19-690c49adc197-istiod-ca-cert\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4\" (UID: \"452d967b-8434-4766-9b19-690c49adc197\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4"
Oct 31 19:32:49.252354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:49.252242    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/452d967b-8434-4766-9b19-690c49adc197-istio-token\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4\" (UID: \"452d967b-8434-4766-9b19-690c49adc197\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4"
Oct 31 19:32:49.252354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:49.252280    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/452d967b-8434-4766-9b19-690c49adc197-workload-socket\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4\" (UID: \"452d967b-8434-4766-9b19-690c49adc197\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4"
Oct 31 19:32:49.252649 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:49.252368    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/452d967b-8434-4766-9b19-690c49adc197-istio-data\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4\" (UID: \"452d967b-8434-4766-9b19-690c49adc197\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4"
Oct 31 19:32:49.252649 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:49.252527    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/452d967b-8434-4766-9b19-690c49adc197-workload-certs\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4\" (UID: \"452d967b-8434-4766-9b19-690c49adc197\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4"
Oct 31 19:32:49.252649 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:49.252631    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/452d967b-8434-4766-9b19-690c49adc197-istio-envoy\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4\" (UID: \"452d967b-8434-4766-9b19-690c49adc197\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4"
Oct 31 19:32:49.252820 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:49.252683    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/452d967b-8434-4766-9b19-690c49adc197-credential-socket\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4\" (UID: \"452d967b-8434-4766-9b19-690c49adc197\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4"
Oct 31 19:32:49.252820 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:49.252748    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/452d967b-8434-4766-9b19-690c49adc197-istio-podinfo\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4\" (UID: \"452d967b-8434-4766-9b19-690c49adc197\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4"
Oct 31 19:32:49.252820 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:49.252773    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66r4p\" (UniqueName: \"kubernetes.io/projected/452d967b-8434-4766-9b19-690c49adc197-kube-api-access-66r4p\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4\" (UID: \"452d967b-8434-4766-9b19-690c49adc197\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4"
Oct 31 19:32:49.257350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:49.253924    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/452d967b-8434-4766-9b19-690c49adc197-istiod-ca-cert\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4\" (UID: \"452d967b-8434-4766-9b19-690c49adc197\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4"
Oct 31 19:32:49.257350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:49.254504    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/452d967b-8434-4766-9b19-690c49adc197-workload-socket\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4\" (UID: \"452d967b-8434-4766-9b19-690c49adc197\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4"
Oct 31 19:32:49.257350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:49.255055    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/452d967b-8434-4766-9b19-690c49adc197-credential-socket\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4\" (UID: \"452d967b-8434-4766-9b19-690c49adc197\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4"
Oct 31 19:32:49.257350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:49.255647    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/452d967b-8434-4766-9b19-690c49adc197-istio-data\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4\" (UID: \"452d967b-8434-4766-9b19-690c49adc197\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4"
Oct 31 19:32:49.257350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:49.255900    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/452d967b-8434-4766-9b19-690c49adc197-workload-certs\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4\" (UID: \"452d967b-8434-4766-9b19-690c49adc197\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4"
Oct 31 19:32:49.321393 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:49.321099    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66r4p\" (UniqueName: \"kubernetes.io/projected/452d967b-8434-4766-9b19-690c49adc197-kube-api-access-66r4p\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4\" (UID: \"452d967b-8434-4766-9b19-690c49adc197\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4"
Oct 31 19:32:49.338353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:49.321611    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/452d967b-8434-4766-9b19-690c49adc197-istio-token\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4\" (UID: \"452d967b-8434-4766-9b19-690c49adc197\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4"
Oct 31 19:32:49.402369 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:49.377844    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/452d967b-8434-4766-9b19-690c49adc197-istio-podinfo\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4\" (UID: \"452d967b-8434-4766-9b19-690c49adc197\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4"
Oct 31 19:32:49.405694 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:49.405623    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/452d967b-8434-4766-9b19-690c49adc197-istio-envoy\") pod \"e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4\" (UID: \"452d967b-8434-4766-9b19-690c49adc197\") " pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4"
Oct 31 19:32:49.808355 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:49.807486    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4"
Oct 31 19:32:49.851256 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:49.841889    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:50.048351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:50.039228    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf"]
Oct 31 19:32:50.048351 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:32:50.042055    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod506d452a_2e17_44c0_9b4e_c7ea88e7e5bf.slice/crio-0618340424f1ba1e6ca6b2934f037b519dd5166898361ba92285750f916ef180 WatchSource:0}: Error finding container 0618340424f1ba1e6ca6b2934f037b519dd5166898361ba92285750f916ef180: Status 404 returned error can't find the container with id 0618340424f1ba1e6ca6b2934f037b519dd5166898361ba92285750f916ef180
Oct 31 19:32:50.048351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:50.044897    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:32:50.048351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:50.045005    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:32:50.048351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:50.045068    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:32:50.130931 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:50.128075    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc"]
Oct 31 19:32:50.134136 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:50.134090    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:32:50.134285 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:50.134185    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:32:50.134285 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:50.134230    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:32:50.195127 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:50.194802    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4"]
Oct 31 19:32:50.200418 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:50.200390    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2"]
Oct 31 19:32:50.209377 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:32:50.207591    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod452d967b_8434_4766_9b19_690c49adc197.slice/crio-0464259c471388e0e11dcf72a2832bfd707edf68a59f8e617c5c3f59f791d646 WatchSource:0}: Error finding container 0464259c471388e0e11dcf72a2832bfd707edf68a59f8e617c5c3f59f791d646: Status 404 returned error can't find the container with id 0464259c471388e0e11dcf72a2832bfd707edf68a59f8e617c5c3f59f791d646
Oct 31 19:32:50.211376 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:50.211257    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:32:50.212064 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:32:50.212037    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8332946b_f51b_45e0_befe_8d8ce8a46c7d.slice/crio-8a8dc77b3a1a0f71c972669b797cef8f86d1d4767638bb6557a65a40411118ae WatchSource:0}: Error finding container 8a8dc77b3a1a0f71c972669b797cef8f86d1d4767638bb6557a65a40411118ae: Status 404 returned error can't find the container with id 8a8dc77b3a1a0f71c972669b797cef8f86d1d4767638bb6557a65a40411118ae
Oct 31 19:32:50.212996 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:50.212539    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:32:50.212996 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:50.212602    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:32:50.216624 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:50.215761    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:32:50.216624 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:50.215839    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:32:50.216624 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:50.215887    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:32:50.756629 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:50.756594    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:50.923886 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:50.923847    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2" event={"ID":"8332946b-f51b-45e0-befe-8d8ce8a46c7d","Type":"ContainerStarted","Data":"ab576a191d5cace8de5e3988efae6e1f50c60884e16cab8ffa6e1a2c5a9591aa"}
Oct 31 19:32:50.924508 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:50.924479    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2" event={"ID":"8332946b-f51b-45e0-befe-8d8ce8a46c7d","Type":"ContainerStarted","Data":"8a8dc77b3a1a0f71c972669b797cef8f86d1d4767638bb6557a65a40411118ae"}
Oct 31 19:32:50.928082 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:50.926774    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf" event={"ID":"506d452a-2e17-44c0-9b4e-c7ea88e7e5bf","Type":"ContainerStarted","Data":"3b2ed82f1f816debce75a6206b7bc06a278d5d8a913b9a23776168803d3123a9"}
Oct 31 19:32:50.928082 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:50.926822    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf" event={"ID":"506d452a-2e17-44c0-9b4e-c7ea88e7e5bf","Type":"ContainerStarted","Data":"0618340424f1ba1e6ca6b2934f037b519dd5166898361ba92285750f916ef180"}
Oct 31 19:32:50.930407 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:50.930376    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc" event={"ID":"a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7","Type":"ContainerStarted","Data":"fa9954d53d791144c7a3b2dfdfe4bfa44b08b62e95fb0fcb5d1835f950c47e74"}
Oct 31 19:32:50.930502 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:50.930459    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc" event={"ID":"a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7","Type":"ContainerStarted","Data":"fdf15a3d116dc80e9753e35af90abe589ba5ea49c5d8d33cbf9e00375d6fddbe"}
Oct 31 19:32:50.932303 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:50.932281    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4" event={"ID":"452d967b-8434-4766-9b19-690c49adc197","Type":"ContainerStarted","Data":"28b5fbd07c27a91c35c6eda7f308ee48d6e6d3c9ab9b1ad6f44b700acf30459b"}
Oct 31 19:32:50.932443 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:50.932318    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4" event={"ID":"452d967b-8434-4766-9b19-690c49adc197","Type":"ContainerStarted","Data":"0464259c471388e0e11dcf72a2832bfd707edf68a59f8e617c5c3f59f791d646"}
Oct 31 19:32:50.956390 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:50.951502    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2" podStartSLOduration=2.951485362 podStartE2EDuration="2.951485362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:32:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:32:50.947599222 +0000 UTC m=+801.520833352" watchObservedRunningTime="2026-10-31 19:32:50.951485362 +0000 UTC m=+801.524719494"
Oct 31 19:32:50.956390 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:50.955638    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf"
Oct 31 19:32:50.977630 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:50.977559    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc" podStartSLOduration=2.977544423 podStartE2EDuration="2.977544423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:32:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:32:50.976457355 +0000 UTC m=+801.549691484" watchObservedRunningTime="2026-10-31 19:32:50.977544423 +0000 UTC m=+801.550778551"
Oct 31 19:32:51.010388 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:51.010208    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf" podStartSLOduration=3.01018533 podStartE2EDuration="3.01018533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:32:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:32:51.000774304 +0000 UTC m=+801.574008454" watchObservedRunningTime="2026-10-31 19:32:51.01018533 +0000 UTC m=+801.583419460"
Oct 31 19:32:51.029949 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:51.029880    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4" podStartSLOduration=3.029862127 podStartE2EDuration="3.029862127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:32:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:32:51.028055325 +0000 UTC m=+801.601289454" watchObservedRunningTime="2026-10-31 19:32:51.029862127 +0000 UTC m=+801.603096256"
Oct 31 19:32:51.231741 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:51.231685    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:51.381414 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:51.371262    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:51.702355 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:51.692970    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2"
Oct 31 19:32:51.706355 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:51.703185    4645 patch_prober.go:29] interesting pod/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 31 19:32:51.706355 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:51.703244    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2" podUID="8332946b-f51b-45e0-befe-8d8ce8a46c7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 31 19:32:51.763353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:51.754605    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc"
Oct 31 19:32:51.763353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:51.754821    4645 patch_prober.go:29] interesting pod/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 31 19:32:51.763353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:51.754884    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc" podUID="a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 31 19:32:51.816356 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:51.815844    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:51.816570 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:51.816482    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4"
Oct 31 19:32:51.829353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:51.825508    4645 patch_prober.go:29] interesting pod/e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 31 19:32:51.829353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:51.825578    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4" podUID="452d967b-8434-4766-9b19-690c49adc197" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 31 19:32:51.978548 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:51.973511    4645 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 31 19:32:51.978548 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:51.973573    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf" podUID="506d452a-2e17-44c0-9b4e-c7ea88e7e5bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 31 19:32:52.664354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:52.657119    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw" podUID="1287db54-a587-4c11-9eb6-1606de555894" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:32:52.700717 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:52.700677    4645 patch_prober.go:29] interesting pod/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 31 19:32:52.742919 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:52.726431    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2" podUID="8332946b-f51b-45e0-befe-8d8ce8a46c7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 31 19:32:52.780396 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:52.770285    4645 patch_prober.go:29] interesting pod/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 31 19:32:52.780396 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:52.770374    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc" podUID="a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 31 19:32:52.879427 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:52.820376    4645 patch_prober.go:29] interesting pod/e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 31 19:32:52.879784 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:52.879742    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4" podUID="452d967b-8434-4766-9b19-690c49adc197" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 31 19:32:52.912855 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:52.911483    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:52.990497 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:52.964426    4645 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 31 19:32:52.990497 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:52.964508    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf" podUID="506d452a-2e17-44c0-9b4e-c7ea88e7e5bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 31 19:32:53.357411 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:53.333257    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669" podUID="9285bee0-bf7c-4fd4-9da7-a1db021cae05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:32:53.394357 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:53.344065    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l" podUID="12bea86c-61e5-465e-82fb-b4673a4d1be9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 31 19:32:53.404988 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:53.404950    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:53.483612 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:53.483564    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4" podUID="6032a997-b795-4561-805c-110cf67a9d5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:32:53.484215 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:53.484183    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw" podUID="d504b2d3-c26d-4dcf-a7eb-efe20732ae4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 31 19:32:53.590490 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:53.590427    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n" podUID="dbb97168-db93-4757-9a1b-f9c5eb704fc5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:32:53.700375 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:53.700294    4645 patch_prober.go:29] interesting pod/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 31 19:32:53.700690 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:53.700662    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2" podUID="8332946b-f51b-45e0-befe-8d8ce8a46c7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 31 19:32:53.794148 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:53.794109    4645 patch_prober.go:29] interesting pod/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 31 19:32:53.798446 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:53.798401    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc" podUID="a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 31 19:32:53.813244 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:53.813159    4645 patch_prober.go:29] interesting pod/e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 31 19:32:53.813538 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:53.813509    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4" podUID="452d967b-8434-4766-9b19-690c49adc197" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 31 19:32:53.961413 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:53.958220    4645 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 31 19:32:53.961413 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:53.958280    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf" podUID="506d452a-2e17-44c0-9b4e-c7ea88e7e5bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 31 19:32:54.170355 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:54.169463    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x" podUID="a774e286-dcb8-4a2f-8541-26bf231eb08d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:32:54.194355 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:54.191247    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:54.364533 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:54.364423    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669" podUID="9285bee0-bf7c-4fd4-9da7-a1db021cae05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 31 19:32:54.365398 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:54.365373    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669"
Oct 31 19:32:54.399215 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:54.397398    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l" podUID="12bea86c-61e5-465e-82fb-b4673a4d1be9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:32:54.427161 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:54.427130    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l"
Oct 31 19:32:54.488359 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:54.486720    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4" podUID="6032a997-b795-4561-805c-110cf67a9d5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:32:54.488359 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:54.487232    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4"
Oct 31 19:32:54.580353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:54.536491    4645 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4" podUID="6032a997-b795-4561-805c-110cf67a9d5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": context deadline exceeded"
Oct 31 19:32:54.581040 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:54.541422    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw" podUID="d504b2d3-c26d-4dcf-a7eb-efe20732ae4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 31 19:32:54.585349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:54.581607    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw"
Oct 31 19:32:54.654525 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:54.654415    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw" podUID="1287db54-a587-4c11-9eb6-1606de555894" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:32:54.726826 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:54.726788    4645 patch_prober.go:29] interesting pod/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 31 19:32:54.727123 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:54.727088    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2" podUID="8332946b-f51b-45e0-befe-8d8ce8a46c7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 31 19:32:54.786223 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:54.786185    4645 patch_prober.go:29] interesting pod/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 31 19:32:54.828584 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:54.809450    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc" podUID="a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 31 19:32:54.899355 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:54.844814    4645 patch_prober.go:29] interesting pod/e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 31 19:32:54.899355 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:54.894176    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4" podUID="452d967b-8434-4766-9b19-690c49adc197" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 31 19:32:54.990055 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:54.990018    4645 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 31 19:32:54.990375 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:54.990344    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf" podUID="506d452a-2e17-44c0-9b4e-c7ea88e7e5bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 31 19:32:55.187356 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:55.177456    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x" podUID="a774e286-dcb8-4a2f-8541-26bf231eb08d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:32:55.187356 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:55.179400    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x"
Oct 31 19:32:55.335417 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:55.331831    4645 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l" podUID="12bea86c-61e5-465e-82fb-b4673a4d1be9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:32:55.366091 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:55.366046    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669" podUID="9285bee0-bf7c-4fd4-9da7-a1db021cae05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:32:55.456354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:55.448935    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l" podUID="12bea86c-61e5-465e-82fb-b4673a4d1be9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:32:55.494835 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:55.493447    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4" podUID="6032a997-b795-4561-805c-110cf67a9d5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:32:55.503467 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:55.503367    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:55.548477 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:55.548423    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n" podUID="dbb97168-db93-4757-9a1b-f9c5eb704fc5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:32:55.587794 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:55.587683    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw" podUID="d504b2d3-c26d-4dcf-a7eb-efe20732ae4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:32:55.653516 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:55.653476    4645 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw" podUID="1287db54-a587-4c11-9eb6-1606de555894" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/health\": context deadline exceeded"
Oct 31 19:32:55.655001 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:55.654968    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw" podUID="1287db54-a587-4c11-9eb6-1606de555894" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:32:55.731355 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:55.706912    4645 patch_prober.go:29] interesting pod/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 31 19:32:55.758363 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:55.743742    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2" podUID="8332946b-f51b-45e0-befe-8d8ce8a46c7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 31 19:32:55.758698 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:55.758668    4645 patch_prober.go:29] interesting pod/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 31 19:32:55.758906 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:55.758870    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc" podUID="a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 31 19:32:55.888121 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:55.888024    4645 patch_prober.go:29] interesting pod/e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 31 19:32:55.892442 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:55.892405    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4" podUID="452d967b-8434-4766-9b19-690c49adc197" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 31 19:32:55.903352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:55.898901    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw"
Oct 31 19:32:55.956410 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:55.956277    4645 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 31 19:32:55.956410 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:55.956352    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf" podUID="506d452a-2e17-44c0-9b4e-c7ea88e7e5bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 31 19:32:56.186356 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:56.182627    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x" podUID="a774e286-dcb8-4a2f-8541-26bf231eb08d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 19:32:56.372836 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:56.371424    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669" podUID="9285bee0-bf7c-4fd4-9da7-a1db021cae05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:32:56.502412 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:56.501970    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4" podUID="6032a997-b795-4561-805c-110cf67a9d5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 31 19:32:56.506531 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:56.506478    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l" podUID="12bea86c-61e5-465e-82fb-b4673a4d1be9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:32:56.569492 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:56.559024    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n" podUID="dbb97168-db93-4757-9a1b-f9c5eb704fc5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 19:32:56.569492 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:56.559667    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n"
Oct 31 19:32:56.591401 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:56.591360    4645 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x" podUID="a774e286-dcb8-4a2f-8541-26bf231eb08d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:32:56.592172 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:56.592140    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw" podUID="d504b2d3-c26d-4dcf-a7eb-efe20732ae4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:32:56.659352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:56.657436    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw" podUID="1287db54-a587-4c11-9eb6-1606de555894" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:32:56.760748 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:56.732226    4645 patch_prober.go:29] interesting pod/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 31 19:32:56.760748 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:56.732285    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2" podUID="8332946b-f51b-45e0-befe-8d8ce8a46c7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 31 19:32:56.760748 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:56.742800    4645 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n" podUID="dbb97168-db93-4757-9a1b-f9c5eb704fc5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:32:56.785450 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:56.778775    4645 patch_prober.go:29] interesting pod/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 31 19:32:56.785450 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:56.778838    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc" podUID="a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 31 19:32:56.815042 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:56.815005    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:56.899853 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:56.899455    4645 patch_prober.go:29] interesting pod/e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 31 19:32:56.899853 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:56.899623    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4" podUID="452d967b-8434-4766-9b19-690c49adc197" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 31 19:32:57.000355 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:56.999686    4645 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 31 19:32:57.000355 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:56.999758    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf" podUID="506d452a-2e17-44c0-9b4e-c7ea88e7e5bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 31 19:32:57.193503 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:57.193443    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x" podUID="a774e286-dcb8-4a2f-8541-26bf231eb08d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:32:57.373241 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:57.373193    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669" podUID="9285bee0-bf7c-4fd4-9da7-a1db021cae05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:32:57.503734 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:57.503537    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4" podUID="6032a997-b795-4561-805c-110cf67a9d5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:32:57.520461 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:57.520402    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l" podUID="12bea86c-61e5-465e-82fb-b4673a4d1be9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 31 19:32:57.568484 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:57.568430    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n" podUID="dbb97168-db93-4757-9a1b-f9c5eb704fc5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:32:57.606354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:57.604104    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw" podUID="d504b2d3-c26d-4dcf-a7eb-efe20732ae4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:32:57.681298 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:57.681252    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw" podUID="1287db54-a587-4c11-9eb6-1606de555894" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": context deadline exceeded"
Oct 31 19:32:57.707291 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:57.698451    4645 patch_prober.go:29] interesting pod/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 31 19:32:57.708134 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:57.708088    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2" podUID="8332946b-f51b-45e0-befe-8d8ce8a46c7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 31 19:32:57.715363 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:57.700885    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:57.763050 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:57.762961    4645 patch_prober.go:29] interesting pod/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 31 19:32:57.763295 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:57.763267    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc" podUID="a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 31 19:32:57.820010 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:57.819976    4645 patch_prober.go:29] interesting pod/e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 31 19:32:57.820312 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:57.820279    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4" podUID="452d967b-8434-4766-9b19-690c49adc197" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 31 19:32:57.953765 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:57.953728    4645 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 31 19:32:57.954064 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:57.954034    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf" podUID="506d452a-2e17-44c0-9b4e-c7ea88e7e5bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 31 19:32:58.119410 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:58.119306    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4"
Oct 31 19:32:58.211359 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:58.209442    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x" podUID="a774e286-dcb8-4a2f-8541-26bf231eb08d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:32:58.211359 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:58.210348    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw"
Oct 31 19:32:58.233536 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:58.226034    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669"
Oct 31 19:32:58.252291 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:58.252263    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l"
Oct 31 19:32:58.307370 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:58.306577    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw"
Oct 31 19:32:58.313160 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:58.313136    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n"
Oct 31 19:32:58.332013 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:58.331983    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x"
Oct 31 19:32:58.600269 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:58.600230    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:58.689650 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:58.689618    4645 patch_prober.go:29] interesting pod/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 31 19:32:58.691379 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:58.689922    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2" podUID="8332946b-f51b-45e0-befe-8d8ce8a46c7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 31 19:32:58.753028 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:58.752983    4645 patch_prober.go:29] interesting pod/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 31 19:32:58.753236 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:58.753057    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc" podUID="a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 31 19:32:58.815748 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:58.815712    4645 patch_prober.go:29] interesting pod/e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Oct 31 19:32:58.816821 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:58.816001    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4" podUID="452d967b-8434-4766-9b19-690c49adc197" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Oct 31 19:32:58.951844 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:58.951801    4645 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 31 19:32:58.952022 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:58.951881    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf" podUID="506d452a-2e17-44c0-9b4e-c7ea88e7e5bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 31 19:32:59.606457 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:59.606421    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:32:59.699257 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:59.699224    4645 patch_prober.go:29] interesting pod/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Oct 31 19:32:59.723704 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:59.723215    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2" podUID="8332946b-f51b-45e0-befe-8d8ce8a46c7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Oct 31 19:32:59.817735 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:59.817707    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4"
Oct 31 19:32:59.818248 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:59.818121    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4"
Oct 31 19:32:59.819300 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:59.819276    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4"
Oct 31 19:32:59.958525 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:59.958489    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf"
Oct 31 19:32:59.959116 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:59.959095    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf"
Oct 31 19:32:59.959757 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:32:59.959739    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf"
Oct 31 19:33:00.024357 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:00.023709    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc"
Oct 31 19:33:00.047354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:00.034302    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc"
Oct 31 19:33:00.047354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:00.037563    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc"
Oct 31 19:33:00.370363 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:00.370252    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:00.693144 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:00.693113    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2"
Oct 31 19:33:00.693637 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:00.693468    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2"
Oct 31 19:33:00.695069 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:00.695047    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2"
Oct 31 19:33:01.241533 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:01.241497    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:02.083572 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:02.083538    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:02.859593 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:02.859563    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:02.932972 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:02.932938    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf"]
Oct 31 19:33:02.943616 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:02.943578    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf"
Oct 31 19:33:02.959079 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:02.959051    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl"]
Oct 31 19:33:02.966477 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:02.966443    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl"
Oct 31 19:33:02.967607 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:02.967583    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf"]
Oct 31 19:33:02.975210 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:02.975182    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl"]
Oct 31 19:33:03.059050 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.059019    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2a68f87-af40-4d21-9ef0-620f397e13be-istio-envoy\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf\" (UID: \"b2a68f87-af40-4d21-9ef0-620f397e13be\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf"
Oct 31 19:33:03.059219 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.059059    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m2zf\" (UniqueName: \"kubernetes.io/projected/b2a68f87-af40-4d21-9ef0-620f397e13be-kube-api-access-6m2zf\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf\" (UID: \"b2a68f87-af40-4d21-9ef0-620f397e13be\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf"
Oct 31 19:33:03.059219 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.059127    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x7tg\" (UniqueName: \"kubernetes.io/projected/619ab3b5-8258-4aea-90fe-a351097d91cd-kube-api-access-4x7tg\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl\" (UID: \"619ab3b5-8258-4aea-90fe-a351097d91cd\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl"
Oct 31 19:33:03.059297 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.059231    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/619ab3b5-8258-4aea-90fe-a351097d91cd-istio-data\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl\" (UID: \"619ab3b5-8258-4aea-90fe-a351097d91cd\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl"
Oct 31 19:33:03.059297 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.059262    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/619ab3b5-8258-4aea-90fe-a351097d91cd-istiod-ca-cert\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl\" (UID: \"619ab3b5-8258-4aea-90fe-a351097d91cd\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl"
Oct 31 19:33:03.059297 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.059283    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/619ab3b5-8258-4aea-90fe-a351097d91cd-workload-socket\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl\" (UID: \"619ab3b5-8258-4aea-90fe-a351097d91cd\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl"
Oct 31 19:33:03.059484 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.059461    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2a68f87-af40-4d21-9ef0-620f397e13be-istiod-ca-cert\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf\" (UID: \"b2a68f87-af40-4d21-9ef0-620f397e13be\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf"
Oct 31 19:33:03.059532 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.059521    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/619ab3b5-8258-4aea-90fe-a351097d91cd-istio-envoy\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl\" (UID: \"619ab3b5-8258-4aea-90fe-a351097d91cd\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl"
Oct 31 19:33:03.059589 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.059538    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2a68f87-af40-4d21-9ef0-620f397e13be-istio-data\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf\" (UID: \"b2a68f87-af40-4d21-9ef0-620f397e13be\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf"
Oct 31 19:33:03.059589 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.059574    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2a68f87-af40-4d21-9ef0-620f397e13be-workload-certs\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf\" (UID: \"b2a68f87-af40-4d21-9ef0-620f397e13be\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf"
Oct 31 19:33:03.059680 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.059666    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2a68f87-af40-4d21-9ef0-620f397e13be-credential-socket\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf\" (UID: \"b2a68f87-af40-4d21-9ef0-620f397e13be\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf"
Oct 31 19:33:03.059711 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.059692    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/619ab3b5-8258-4aea-90fe-a351097d91cd-credential-socket\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl\" (UID: \"619ab3b5-8258-4aea-90fe-a351097d91cd\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl"
Oct 31 19:33:03.059747 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.059710    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/619ab3b5-8258-4aea-90fe-a351097d91cd-workload-certs\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl\" (UID: \"619ab3b5-8258-4aea-90fe-a351097d91cd\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl"
Oct 31 19:33:03.059781 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.059757    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/619ab3b5-8258-4aea-90fe-a351097d91cd-istio-podinfo\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl\" (UID: \"619ab3b5-8258-4aea-90fe-a351097d91cd\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl"
Oct 31 19:33:03.059814 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.059782    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/619ab3b5-8258-4aea-90fe-a351097d91cd-istio-token\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl\" (UID: \"619ab3b5-8258-4aea-90fe-a351097d91cd\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl"
Oct 31 19:33:03.059848 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.059814    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2a68f87-af40-4d21-9ef0-620f397e13be-workload-socket\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf\" (UID: \"b2a68f87-af40-4d21-9ef0-620f397e13be\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf"
Oct 31 19:33:03.059919 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.059905    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2a68f87-af40-4d21-9ef0-620f397e13be-istio-podinfo\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf\" (UID: \"b2a68f87-af40-4d21-9ef0-620f397e13be\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf"
Oct 31 19:33:03.059956 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.059931    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2a68f87-af40-4d21-9ef0-620f397e13be-istio-token\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf\" (UID: \"b2a68f87-af40-4d21-9ef0-620f397e13be\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf"
Oct 31 19:33:03.161522 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.161490    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2a68f87-af40-4d21-9ef0-620f397e13be-workload-socket\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf\" (UID: \"b2a68f87-af40-4d21-9ef0-620f397e13be\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf"
Oct 31 19:33:03.162002 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.161558    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2a68f87-af40-4d21-9ef0-620f397e13be-istio-podinfo\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf\" (UID: \"b2a68f87-af40-4d21-9ef0-620f397e13be\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf"
Oct 31 19:33:03.162002 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.161581    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2a68f87-af40-4d21-9ef0-620f397e13be-istio-token\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf\" (UID: \"b2a68f87-af40-4d21-9ef0-620f397e13be\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf"
Oct 31 19:33:03.162002 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.161605    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2a68f87-af40-4d21-9ef0-620f397e13be-istio-envoy\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf\" (UID: \"b2a68f87-af40-4d21-9ef0-620f397e13be\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf"
Oct 31 19:33:03.162002 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.161625    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m2zf\" (UniqueName: \"kubernetes.io/projected/b2a68f87-af40-4d21-9ef0-620f397e13be-kube-api-access-6m2zf\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf\" (UID: \"b2a68f87-af40-4d21-9ef0-620f397e13be\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf"
Oct 31 19:33:03.162002 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.161654    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x7tg\" (UniqueName: \"kubernetes.io/projected/619ab3b5-8258-4aea-90fe-a351097d91cd-kube-api-access-4x7tg\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl\" (UID: \"619ab3b5-8258-4aea-90fe-a351097d91cd\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl"
Oct 31 19:33:03.162002 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.161858    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/619ab3b5-8258-4aea-90fe-a351097d91cd-istio-data\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl\" (UID: \"619ab3b5-8258-4aea-90fe-a351097d91cd\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl"
Oct 31 19:33:03.162002 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.161919    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/619ab3b5-8258-4aea-90fe-a351097d91cd-istiod-ca-cert\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl\" (UID: \"619ab3b5-8258-4aea-90fe-a351097d91cd\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl"
Oct 31 19:33:03.162002 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.161968    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/619ab3b5-8258-4aea-90fe-a351097d91cd-workload-socket\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl\" (UID: \"619ab3b5-8258-4aea-90fe-a351097d91cd\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl"
Oct 31 19:33:03.162002 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.161975    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2a68f87-af40-4d21-9ef0-620f397e13be-workload-socket\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf\" (UID: \"b2a68f87-af40-4d21-9ef0-620f397e13be\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf"
Oct 31 19:33:03.162492 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.162071    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2a68f87-af40-4d21-9ef0-620f397e13be-istiod-ca-cert\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf\" (UID: \"b2a68f87-af40-4d21-9ef0-620f397e13be\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf"
Oct 31 19:33:03.162492 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.162146    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/619ab3b5-8258-4aea-90fe-a351097d91cd-istio-envoy\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl\" (UID: \"619ab3b5-8258-4aea-90fe-a351097d91cd\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl"
Oct 31 19:33:03.162492 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.162167    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2a68f87-af40-4d21-9ef0-620f397e13be-istio-data\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf\" (UID: \"b2a68f87-af40-4d21-9ef0-620f397e13be\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf"
Oct 31 19:33:03.162492 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.162222    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2a68f87-af40-4d21-9ef0-620f397e13be-workload-certs\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf\" (UID: \"b2a68f87-af40-4d21-9ef0-620f397e13be\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf"
Oct 31 19:33:03.162492 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.162225    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/619ab3b5-8258-4aea-90fe-a351097d91cd-istio-data\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl\" (UID: \"619ab3b5-8258-4aea-90fe-a351097d91cd\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl"
Oct 31 19:33:03.162492 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.162309    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2a68f87-af40-4d21-9ef0-620f397e13be-credential-socket\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf\" (UID: \"b2a68f87-af40-4d21-9ef0-620f397e13be\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf"
Oct 31 19:33:03.162492 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.162372    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/619ab3b5-8258-4aea-90fe-a351097d91cd-credential-socket\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl\" (UID: \"619ab3b5-8258-4aea-90fe-a351097d91cd\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl"
Oct 31 19:33:03.162492 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.162398    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/619ab3b5-8258-4aea-90fe-a351097d91cd-workload-certs\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl\" (UID: \"619ab3b5-8258-4aea-90fe-a351097d91cd\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl"
Oct 31 19:33:03.162492 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.162442    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/619ab3b5-8258-4aea-90fe-a351097d91cd-istio-podinfo\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl\" (UID: \"619ab3b5-8258-4aea-90fe-a351097d91cd\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl"
Oct 31 19:33:03.162492 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.162484    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/619ab3b5-8258-4aea-90fe-a351097d91cd-istio-token\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl\" (UID: \"619ab3b5-8258-4aea-90fe-a351097d91cd\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl"
Oct 31 19:33:03.163048 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.162581    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2a68f87-af40-4d21-9ef0-620f397e13be-workload-certs\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf\" (UID: \"b2a68f87-af40-4d21-9ef0-620f397e13be\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf"
Oct 31 19:33:03.163048 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.162704    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/619ab3b5-8258-4aea-90fe-a351097d91cd-istiod-ca-cert\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl\" (UID: \"619ab3b5-8258-4aea-90fe-a351097d91cd\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl"
Oct 31 19:33:03.163048 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.162760    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/619ab3b5-8258-4aea-90fe-a351097d91cd-workload-socket\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl\" (UID: \"619ab3b5-8258-4aea-90fe-a351097d91cd\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl"
Oct 31 19:33:03.163048 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.162873    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2a68f87-af40-4d21-9ef0-620f397e13be-istiod-ca-cert\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf\" (UID: \"b2a68f87-af40-4d21-9ef0-620f397e13be\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf"
Oct 31 19:33:03.163048 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.162894    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2a68f87-af40-4d21-9ef0-620f397e13be-istio-data\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf\" (UID: \"b2a68f87-af40-4d21-9ef0-620f397e13be\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf"
Oct 31 19:33:03.163048 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.163044    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2a68f87-af40-4d21-9ef0-620f397e13be-credential-socket\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf\" (UID: \"b2a68f87-af40-4d21-9ef0-620f397e13be\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf"
Oct 31 19:33:03.163314 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.163158    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/619ab3b5-8258-4aea-90fe-a351097d91cd-workload-certs\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl\" (UID: \"619ab3b5-8258-4aea-90fe-a351097d91cd\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl"
Oct 31 19:33:03.163314 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.163180    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/619ab3b5-8258-4aea-90fe-a351097d91cd-credential-socket\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl\" (UID: \"619ab3b5-8258-4aea-90fe-a351097d91cd\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl"
Oct 31 19:33:03.164494 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.164471    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2a68f87-af40-4d21-9ef0-620f397e13be-istio-envoy\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf\" (UID: \"b2a68f87-af40-4d21-9ef0-620f397e13be\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf"
Oct 31 19:33:03.164594 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.164541    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2a68f87-af40-4d21-9ef0-620f397e13be-istio-podinfo\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf\" (UID: \"b2a68f87-af40-4d21-9ef0-620f397e13be\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf"
Oct 31 19:33:03.164850 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.164828    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/619ab3b5-8258-4aea-90fe-a351097d91cd-istio-envoy\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl\" (UID: \"619ab3b5-8258-4aea-90fe-a351097d91cd\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl"
Oct 31 19:33:03.165155 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.165139    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/619ab3b5-8258-4aea-90fe-a351097d91cd-istio-podinfo\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl\" (UID: \"619ab3b5-8258-4aea-90fe-a351097d91cd\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl"
Oct 31 19:33:03.176820 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.176791    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2a68f87-af40-4d21-9ef0-620f397e13be-istio-token\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf\" (UID: \"b2a68f87-af40-4d21-9ef0-620f397e13be\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf"
Oct 31 19:33:03.176970 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.176954    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m2zf\" (UniqueName: \"kubernetes.io/projected/b2a68f87-af40-4d21-9ef0-620f397e13be-kube-api-access-6m2zf\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf\" (UID: \"b2a68f87-af40-4d21-9ef0-620f397e13be\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf"
Oct 31 19:33:03.177521 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.177503    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/619ab3b5-8258-4aea-90fe-a351097d91cd-istio-token\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl\" (UID: \"619ab3b5-8258-4aea-90fe-a351097d91cd\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl"
Oct 31 19:33:03.178034 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.178015    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x7tg\" (UniqueName: \"kubernetes.io/projected/619ab3b5-8258-4aea-90fe-a351097d91cd-kube-api-access-4x7tg\") pod \"e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl\" (UID: \"619ab3b5-8258-4aea-90fe-a351097d91cd\") " pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl"
Oct 31 19:33:03.297161 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.297125    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf"
Oct 31 19:33:03.330102 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.330072    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl"
Oct 31 19:33:03.474585 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.474542    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5"]
Oct 31 19:33:03.480970 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.480937    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5"
Oct 31 19:33:03.494280 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.494243    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5"]
Oct 31 19:33:03.569171 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.569137    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bf496d2-b4cd-417b-b5dc-3afbac91f26b-istio-envoy\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5\" (UID: \"2bf496d2-b4cd-417b-b5dc-3afbac91f26b\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5"
Oct 31 19:33:03.569411 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.569271    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bf496d2-b4cd-417b-b5dc-3afbac91f26b-istio-token\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5\" (UID: \"2bf496d2-b4cd-417b-b5dc-3afbac91f26b\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5"
Oct 31 19:33:03.569470 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.569429    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bf496d2-b4cd-417b-b5dc-3afbac91f26b-workload-certs\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5\" (UID: \"2bf496d2-b4cd-417b-b5dc-3afbac91f26b\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5"
Oct 31 19:33:03.569581 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.569565    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m88qn\" (UniqueName: \"kubernetes.io/projected/2bf496d2-b4cd-417b-b5dc-3afbac91f26b-kube-api-access-m88qn\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5\" (UID: \"2bf496d2-b4cd-417b-b5dc-3afbac91f26b\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5"
Oct 31 19:33:03.569677 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.569656    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bf496d2-b4cd-417b-b5dc-3afbac91f26b-istiod-ca-cert\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5\" (UID: \"2bf496d2-b4cd-417b-b5dc-3afbac91f26b\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5"
Oct 31 19:33:03.569741 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.569696    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bf496d2-b4cd-417b-b5dc-3afbac91f26b-istio-podinfo\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5\" (UID: \"2bf496d2-b4cd-417b-b5dc-3afbac91f26b\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5"
Oct 31 19:33:03.569791 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.569748    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bf496d2-b4cd-417b-b5dc-3afbac91f26b-istio-data\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5\" (UID: \"2bf496d2-b4cd-417b-b5dc-3afbac91f26b\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5"
Oct 31 19:33:03.569849 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.569790    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bf496d2-b4cd-417b-b5dc-3afbac91f26b-workload-socket\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5\" (UID: \"2bf496d2-b4cd-417b-b5dc-3afbac91f26b\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5"
Oct 31 19:33:03.569849 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.569813    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bf496d2-b4cd-417b-b5dc-3afbac91f26b-credential-socket\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5\" (UID: \"2bf496d2-b4cd-417b-b5dc-3afbac91f26b\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5"
Oct 31 19:33:03.611743 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:33:03.611708    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2a68f87_af40_4d21_9ef0_620f397e13be.slice/crio-257c7275cf3b4f16f7ece3ef65a7350b60731681b35ff06cb32a443a520a5b5c WatchSource:0}: Error finding container 257c7275cf3b4f16f7ece3ef65a7350b60731681b35ff06cb32a443a520a5b5c: Status 404 returned error can't find the container with id 257c7275cf3b4f16f7ece3ef65a7350b60731681b35ff06cb32a443a520a5b5c
Oct 31 19:33:03.619287 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.615273    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:33:03.619287 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.615526    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:33:03.619287 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.615623    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:33:03.631190 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.631159    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf"]
Oct 31 19:33:03.636602 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.636578    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl"]
Oct 31 19:33:03.637006 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:33:03.636985    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod619ab3b5_8258_4aea_90fe_a351097d91cd.slice/crio-7e9d64b66e013c70b54ff818553434604cb00c55cd25fe70542be78cec55cd92 WatchSource:0}: Error finding container 7e9d64b66e013c70b54ff818553434604cb00c55cd25fe70542be78cec55cd92: Status 404 returned error can't find the container with id 7e9d64b66e013c70b54ff818553434604cb00c55cd25fe70542be78cec55cd92
Oct 31 19:33:03.639757 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.639714    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:33:03.639850 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.639807    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:33:03.639891 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.639851    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:33:03.670940 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.670912    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bf496d2-b4cd-417b-b5dc-3afbac91f26b-workload-certs\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5\" (UID: \"2bf496d2-b4cd-417b-b5dc-3afbac91f26b\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5"
Oct 31 19:33:03.671045 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.671034    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m88qn\" (UniqueName: \"kubernetes.io/projected/2bf496d2-b4cd-417b-b5dc-3afbac91f26b-kube-api-access-m88qn\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5\" (UID: \"2bf496d2-b4cd-417b-b5dc-3afbac91f26b\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5"
Oct 31 19:33:03.671093 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.671073    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bf496d2-b4cd-417b-b5dc-3afbac91f26b-istiod-ca-cert\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5\" (UID: \"2bf496d2-b4cd-417b-b5dc-3afbac91f26b\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5"
Oct 31 19:33:03.671149 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.671103    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bf496d2-b4cd-417b-b5dc-3afbac91f26b-istio-podinfo\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5\" (UID: \"2bf496d2-b4cd-417b-b5dc-3afbac91f26b\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5"
Oct 31 19:33:03.671149 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.671135    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bf496d2-b4cd-417b-b5dc-3afbac91f26b-istio-data\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5\" (UID: \"2bf496d2-b4cd-417b-b5dc-3afbac91f26b\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5"
Oct 31 19:33:03.671246 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.671191    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bf496d2-b4cd-417b-b5dc-3afbac91f26b-workload-socket\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5\" (UID: \"2bf496d2-b4cd-417b-b5dc-3afbac91f26b\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5"
Oct 31 19:33:03.671246 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.671226    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bf496d2-b4cd-417b-b5dc-3afbac91f26b-credential-socket\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5\" (UID: \"2bf496d2-b4cd-417b-b5dc-3afbac91f26b\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5"
Oct 31 19:33:03.671379 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.671287    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bf496d2-b4cd-417b-b5dc-3afbac91f26b-istio-envoy\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5\" (UID: \"2bf496d2-b4cd-417b-b5dc-3afbac91f26b\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5"
Oct 31 19:33:03.671440 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.671405    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bf496d2-b4cd-417b-b5dc-3afbac91f26b-istio-token\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5\" (UID: \"2bf496d2-b4cd-417b-b5dc-3afbac91f26b\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5"
Oct 31 19:33:03.671497 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.671478    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bf496d2-b4cd-417b-b5dc-3afbac91f26b-workload-certs\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5\" (UID: \"2bf496d2-b4cd-417b-b5dc-3afbac91f26b\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5"
Oct 31 19:33:03.671757 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.671733    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bf496d2-b4cd-417b-b5dc-3afbac91f26b-workload-socket\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5\" (UID: \"2bf496d2-b4cd-417b-b5dc-3afbac91f26b\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5"
Oct 31 19:33:03.671912 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.671888    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bf496d2-b4cd-417b-b5dc-3afbac91f26b-credential-socket\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5\" (UID: \"2bf496d2-b4cd-417b-b5dc-3afbac91f26b\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5"
Oct 31 19:33:03.672111 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.672082    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bf496d2-b4cd-417b-b5dc-3afbac91f26b-istio-data\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5\" (UID: \"2bf496d2-b4cd-417b-b5dc-3afbac91f26b\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5"
Oct 31 19:33:03.672111 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.672096    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bf496d2-b4cd-417b-b5dc-3afbac91f26b-istiod-ca-cert\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5\" (UID: \"2bf496d2-b4cd-417b-b5dc-3afbac91f26b\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5"
Oct 31 19:33:03.676377 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.676348    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bf496d2-b4cd-417b-b5dc-3afbac91f26b-istio-envoy\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5\" (UID: \"2bf496d2-b4cd-417b-b5dc-3afbac91f26b\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5"
Oct 31 19:33:03.677012 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.676982    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bf496d2-b4cd-417b-b5dc-3afbac91f26b-istio-podinfo\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5\" (UID: \"2bf496d2-b4cd-417b-b5dc-3afbac91f26b\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5"
Oct 31 19:33:03.686079 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.686019    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bf496d2-b4cd-417b-b5dc-3afbac91f26b-istio-token\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5\" (UID: \"2bf496d2-b4cd-417b-b5dc-3afbac91f26b\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5"
Oct 31 19:33:03.686197 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.686163    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m88qn\" (UniqueName: \"kubernetes.io/projected/2bf496d2-b4cd-417b-b5dc-3afbac91f26b-kube-api-access-m88qn\") pod \"e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5\" (UID: \"2bf496d2-b4cd-417b-b5dc-3afbac91f26b\") " pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5"
Oct 31 19:33:03.755199 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.755157    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf" event={"ID":"b2a68f87-af40-4d21-9ef0-620f397e13be","Type":"ContainerStarted","Data":"aaa54faf936afb8ee998be89fa312e75432531e53d0905dcca6e7f2fe700885c"}
Oct 31 19:33:03.755414 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.755225    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf" event={"ID":"b2a68f87-af40-4d21-9ef0-620f397e13be","Type":"ContainerStarted","Data":"257c7275cf3b4f16f7ece3ef65a7350b60731681b35ff06cb32a443a520a5b5c"}
Oct 31 19:33:03.758431 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.758401    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl" event={"ID":"619ab3b5-8258-4aea-90fe-a351097d91cd","Type":"ContainerStarted","Data":"91eaff5584a21d4367366fb3491e9729eeaad0a898f0406a181141ad0977f780"}
Oct 31 19:33:03.758563 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.758456    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl" event={"ID":"619ab3b5-8258-4aea-90fe-a351097d91cd","Type":"ContainerStarted","Data":"7e9d64b66e013c70b54ff818553434604cb00c55cd25fe70542be78cec55cd92"}
Oct 31 19:33:03.763114 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.763088    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:03.782102 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.782040    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf" podStartSLOduration=1.7820224850000002 podStartE2EDuration="1.782022485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:33:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:33:03.777255431 +0000 UTC m=+814.350489561" watchObservedRunningTime="2026-10-31 19:33:03.782022485 +0000 UTC m=+814.355256617"
Oct 31 19:33:03.804355 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.800965    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl" podStartSLOduration=1.800947539 podStartE2EDuration="1.800947539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:33:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:33:03.799838239 +0000 UTC m=+814.373072371" watchObservedRunningTime="2026-10-31 19:33:03.800947539 +0000 UTC m=+814.374181670"
Oct 31 19:33:03.815175 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:03.814779    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5"
Oct 31 19:33:04.162143 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:33:04.162105    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bf496d2_b4cd_417b_b5dc_3afbac91f26b.slice/crio-a68085ff91a5b2f4be6d84f9f5c159e47d0ab3e0913f471fcb71e133320783cf WatchSource:0}: Error finding container a68085ff91a5b2f4be6d84f9f5c159e47d0ab3e0913f471fcb71e133320783cf: Status 404 returned error can't find the container with id a68085ff91a5b2f4be6d84f9f5c159e47d0ab3e0913f471fcb71e133320783cf
Oct 31 19:33:04.162143 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:04.162128    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5"]
Oct 31 19:33:04.165223 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:04.165183    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:33:04.165434 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:04.165268    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:33:04.165434 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:04.165308    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:33:04.298046 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:04.298008    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf"
Oct 31 19:33:04.299717 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:04.299684    4645 patch_prober.go:29] interesting pod/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 31 19:33:04.299877 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:04.299741    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf" podUID="b2a68f87-af40-4d21-9ef0-620f397e13be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 31 19:33:04.330950 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:04.330917    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl"
Oct 31 19:33:04.332438 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:04.332411    4645 patch_prober.go:29] interesting pod/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 31 19:33:04.332569 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:04.332461    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl" podUID="619ab3b5-8258-4aea-90fe-a351097d91cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 31 19:33:04.694354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:04.678221    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:04.848917 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:04.848880    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5" event={"ID":"2bf496d2-b4cd-417b-b5dc-3afbac91f26b","Type":"ContainerStarted","Data":"a5b0e166ca64ced17af9ede634b7c94b9c98be1c40972abd9ac1e175bcf6acf7"}
Oct 31 19:33:04.849157 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:04.848937    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5" event={"ID":"2bf496d2-b4cd-417b-b5dc-3afbac91f26b","Type":"ContainerStarted","Data":"a68085ff91a5b2f4be6d84f9f5c159e47d0ab3e0913f471fcb71e133320783cf"}
Oct 31 19:33:05.308876 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:05.308838    4645 patch_prober.go:29] interesting pod/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 31 19:33:05.313455 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:05.313410    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf" podUID="b2a68f87-af40-4d21-9ef0-620f397e13be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 31 19:33:05.331492 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:05.331464    4645 patch_prober.go:29] interesting pod/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 31 19:33:05.331665 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:05.331514    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl" podUID="619ab3b5-8258-4aea-90fe-a351097d91cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 31 19:33:05.448806 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:05.448769    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:05.841653 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:05.839935    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5"
Oct 31 19:33:05.841653 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:05.840612    4645 patch_prober.go:29] interesting pod/e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 31 19:33:05.841653 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:05.840677    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5" podUID="2bf496d2-b4cd-417b-b5dc-3afbac91f26b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 31 19:33:06.299355 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:06.297753    4645 patch_prober.go:29] interesting pod/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 31 19:33:06.299355 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:06.297818    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf" podUID="b2a68f87-af40-4d21-9ef0-620f397e13be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 31 19:33:06.330894 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:06.330725    4645 patch_prober.go:29] interesting pod/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 31 19:33:06.330894 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:06.330785    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl" podUID="619ab3b5-8258-4aea-90fe-a351097d91cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 31 19:33:06.333505 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:33:06.333470    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 19:33:06.524063 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:06.524024    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:06.818713 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:06.818674    4645 patch_prober.go:29] interesting pod/e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 31 19:33:06.819234 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:06.819200    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5" podUID="2bf496d2-b4cd-417b-b5dc-3afbac91f26b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 31 19:33:07.313355 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:07.313147    4645 patch_prober.go:29] interesting pod/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 31 19:33:07.313355 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:07.313223    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf" podUID="b2a68f87-af40-4d21-9ef0-620f397e13be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 31 19:33:07.336465 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:07.336430    4645 patch_prober.go:29] interesting pod/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 31 19:33:07.337541 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:07.337504    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl" podUID="619ab3b5-8258-4aea-90fe-a351097d91cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 31 19:33:07.412353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:07.408909    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:07.818353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:07.817855    4645 patch_prober.go:29] interesting pod/e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 31 19:33:07.818353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:07.817926    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5" podUID="2bf496d2-b4cd-417b-b5dc-3afbac91f26b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 31 19:33:08.301355 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:08.297761    4645 patch_prober.go:29] interesting pod/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 31 19:33:08.301355 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:08.297824    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf" podUID="b2a68f87-af40-4d21-9ef0-620f397e13be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 31 19:33:08.334351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:08.333802    4645 patch_prober.go:29] interesting pod/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 31 19:33:08.334351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:08.333861    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl" podUID="619ab3b5-8258-4aea-90fe-a351097d91cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 31 19:33:08.343435 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:08.343301    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:08.831355 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:08.824449    4645 patch_prober.go:29] interesting pod/e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 31 19:33:08.831355 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:08.824508    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5" podUID="2bf496d2-b4cd-417b-b5dc-3afbac91f26b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 31 19:33:09.216353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:09.213867    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:09.305379 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:09.299274    4645 patch_prober.go:29] interesting pod/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 31 19:33:09.305379 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:09.303889    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf" podUID="b2a68f87-af40-4d21-9ef0-620f397e13be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 31 19:33:09.376902 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:09.376868    4645 patch_prober.go:29] interesting pod/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 31 19:33:09.387545 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:09.387502    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl" podUID="619ab3b5-8258-4aea-90fe-a351097d91cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 31 19:33:09.816528 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:09.815716    4645 patch_prober.go:29] interesting pod/e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 31 19:33:09.816528 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:09.815780    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5" podUID="2bf496d2-b4cd-417b-b5dc-3afbac91f26b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 31 19:33:10.307413 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:10.285041    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:10.331308 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:10.331273    4645 patch_prober.go:29] interesting pod/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 31 19:33:10.331657 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:10.331623    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl" podUID="619ab3b5-8258-4aea-90fe-a351097d91cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 31 19:33:10.332410 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:10.331514    4645 patch_prober.go:29] interesting pod/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 31 19:33:10.332603 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:10.332583    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf" podUID="b2a68f87-af40-4d21-9ef0-620f397e13be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 31 19:33:10.816311 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:10.816276    4645 patch_prober.go:29] interesting pod/e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body=
Oct 31 19:33:10.816725 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:10.816346    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5" podUID="2bf496d2-b4cd-417b-b5dc-3afbac91f26b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused"
Oct 31 19:33:11.159436 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:11.159307    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:11.298776 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:11.298737    4645 patch_prober.go:29] interesting pod/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 31 19:33:11.299051 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:11.299020    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf" podUID="b2a68f87-af40-4d21-9ef0-620f397e13be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 31 19:33:11.331166 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:11.331133    4645 patch_prober.go:29] interesting pod/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 31 19:33:11.331430 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:11.331399    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl" podUID="619ab3b5-8258-4aea-90fe-a351097d91cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 31 19:33:11.987085 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:11.987011    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:12.231302 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:12.231264    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5"
Oct 31 19:33:12.232953 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:12.232929    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5"
Oct 31 19:33:12.233959 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:12.233926    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5"
Oct 31 19:33:12.256962 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:12.256867    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5" podStartSLOduration=9.25685154 podStartE2EDuration="9.25685154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:33:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:33:04.882650122 +0000 UTC m=+815.455884252" watchObservedRunningTime="2026-10-31 19:33:12.25685154 +0000 UTC m=+822.830085669"
Oct 31 19:33:12.304428 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:12.304394    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf"
Oct 31 19:33:12.305408 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:12.305382    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf"
Oct 31 19:33:12.306160 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:12.306139    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf"
Oct 31 19:33:12.336590 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:12.336553    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl"
Oct 31 19:33:12.337698 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:12.337672    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl"
Oct 31 19:33:12.338515 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:12.338487    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl"
Oct 31 19:33:12.687540 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:12.687486    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:13.214456 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:13.214420    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:15.325450 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:15.325403    4645 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l" podUID="12bea86c-61e5-465e-82fb-b4673a4d1be9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:15.340511 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:15.340473    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x" podUID="a774e286-dcb8-4a2f-8541-26bf231eb08d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:15.375441 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:15.375403    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669" podUID="9285bee0-bf7c-4fd4-9da7-a1db021cae05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:15.416338 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:15.416287    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:15.468489 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:15.468441    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l" podUID="12bea86c-61e5-465e-82fb-b4673a4d1be9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:15.499917 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:15.499846    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4" podUID="6032a997-b795-4561-805c-110cf67a9d5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:15.573512 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:15.573462    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n" podUID="dbb97168-db93-4757-9a1b-f9c5eb704fc5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:15.606920 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:15.606818    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw" podUID="d504b2d3-c26d-4dcf-a7eb-efe20732ae4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:15.650455 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:15.650407    4645 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw" podUID="1287db54-a587-4c11-9eb6-1606de555894" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/health\": context deadline exceeded"
Oct 31 19:33:15.684458 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:15.684413    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw" podUID="1287db54-a587-4c11-9eb6-1606de555894" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": context deadline exceeded"
Oct 31 19:33:15.732742 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:15.732652    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:16.070354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:16.069553    4645 patch_prober.go:29] interesting pod/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:33:16.095353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:16.074393    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc" podUID="a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:16.354413 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:16.350740    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x" podUID="a774e286-dcb8-4a2f-8541-26bf231eb08d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:16.357724 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:16.357695    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x"
Oct 31 19:33:16.404354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:16.403914    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669" podUID="9285bee0-bf7c-4fd4-9da7-a1db021cae05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 31 19:33:16.429353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:16.406149    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669"
Oct 31 19:33:16.531488 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:16.531429    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4" podUID="6032a997-b795-4561-805c-110cf67a9d5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 31 19:33:16.675454 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:16.615406    4645 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x" podUID="a774e286-dcb8-4a2f-8541-26bf231eb08d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": context deadline exceeded"
Oct 31 19:33:16.675454 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:16.615544    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4"
Oct 31 19:33:16.675454 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:16.627978    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n" podUID="dbb97168-db93-4757-9a1b-f9c5eb704fc5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 19:33:16.675454 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:16.628269    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n"
Oct 31 19:33:16.675454 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:16.628764    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw" podUID="d504b2d3-c26d-4dcf-a7eb-efe20732ae4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:16.675454 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:16.628971    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw"
Oct 31 19:33:16.675454 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:16.656653    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l" podUID="12bea86c-61e5-465e-82fb-b4673a4d1be9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 31 19:33:16.686359 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:16.682916    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l"
Oct 31 19:33:16.880361 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:16.877621    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw" podUID="1287db54-a587-4c11-9eb6-1606de555894" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": context deadline exceeded"
Oct 31 19:33:16.906361 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:16.900517    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw"
Oct 31 19:33:16.906361 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:16.902003    4645 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n" podUID="dbb97168-db93-4757-9a1b-f9c5eb704fc5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:17.026412 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:16.984926    4645 patch_prober.go:29] interesting pod/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:33:17.026412 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:16.984987    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2" podUID="8332946b-f51b-45e0-befe-8d8ce8a46c7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:17.390431 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:17.390315    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x" podUID="a774e286-dcb8-4a2f-8541-26bf231eb08d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 19:33:17.490353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:17.406413    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669" podUID="9285bee0-bf7c-4fd4-9da7-a1db021cae05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:17.942354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:17.631481    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4" podUID="6032a997-b795-4561-805c-110cf67a9d5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 31 19:33:17.942354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:17.875159    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n" podUID="dbb97168-db93-4757-9a1b-f9c5eb704fc5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:17.942354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:17.901999    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw" podUID="1287db54-a587-4c11-9eb6-1606de555894" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:18.075895 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:18.075849    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw" podUID="d504b2d3-c26d-4dcf-a7eb-efe20732ae4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 31 19:33:18.102242 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:18.102200    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l" podUID="12bea86c-61e5-465e-82fb-b4673a4d1be9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 31 19:33:18.298422 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:18.288700    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:18.549261 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:18.549095    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x" podUID="a774e286-dcb8-4a2f-8541-26bf231eb08d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:18.950545 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:18.950486    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw" podUID="1287db54-a587-4c11-9eb6-1606de555894" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:18.989022 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:18.962512    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669" podUID="9285bee0-bf7c-4fd4-9da7-a1db021cae05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 31 19:33:18.989365 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:18.987504    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4" podUID="6032a997-b795-4561-805c-110cf67a9d5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 31 19:33:18.989790 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:18.987557    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n" podUID="dbb97168-db93-4757-9a1b-f9c5eb704fc5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 19:33:19.085354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:19.081470    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw" podUID="d504b2d3-c26d-4dcf-a7eb-efe20732ae4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 31 19:33:19.316054 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:19.315966    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l" podUID="12bea86c-61e5-465e-82fb-b4673a4d1be9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 31 19:33:19.422007 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:19.421972    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:19.434710 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:19.434673    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x"
Oct 31 19:33:19.458100 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:19.458069    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw"
Oct 31 19:33:19.461108 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:19.461085    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669"
Oct 31 19:33:19.573629 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:19.573546    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n"
Oct 31 19:33:19.579315 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:19.579295    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4"
Oct 31 19:33:19.580847 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:19.580834    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l"
Oct 31 19:33:19.781558 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:19.781517    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw"
Oct 31 19:33:20.213576 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:20.213536    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:21.163242 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:21.163207    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:22.059584 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:22.059549    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:23.041518 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:23.041488    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:23.959533 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:23.959496    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:24.838606 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:24.838571    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:25.758192 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:25.758151    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:26.224059 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:26.224024    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:26.558648 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:26.558564    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:27.301959 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:27.301925    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:28.151723 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:28.151692    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:28.900375 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:28.900313    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:29.734171 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:29.734141    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:30.551609 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:30.551575    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:31.659894 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:31.659865    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:32.542171 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:32.542132    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:33.178998 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.178966    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd"]
Oct 31 19:33:33.184754 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.184730    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd"
Oct 31 19:33:33.206607 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.206573    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd"]
Oct 31 19:33:33.212041 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.212010    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd"]
Oct 31 19:33:33.217729 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.217703    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd"
Oct 31 19:33:33.229610 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.229582    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd"]
Oct 31 19:33:33.311887 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.311850    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b36d1635-5796-4ff5-83e3-7d63ff424c78-istio-data\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd\" (UID: \"b36d1635-5796-4ff5-83e3-7d63ff424c78\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd"
Oct 31 19:33:33.312080 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.311914    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b36d1635-5796-4ff5-83e3-7d63ff424c78-istio-envoy\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd\" (UID: \"b36d1635-5796-4ff5-83e3-7d63ff424c78\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd"
Oct 31 19:33:33.312080 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.312010    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sf6v\" (UniqueName: \"kubernetes.io/projected/a28d9602-371e-4d21-b4f4-0ba0966dd9f0-kube-api-access-7sf6v\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd\" (UID: \"a28d9602-371e-4d21-b4f4-0ba0966dd9f0\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd"
Oct 31 19:33:33.312219 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.312084    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b36d1635-5796-4ff5-83e3-7d63ff424c78-istio-token\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd\" (UID: \"b36d1635-5796-4ff5-83e3-7d63ff424c78\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd"
Oct 31 19:33:33.312219 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.312137    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a28d9602-371e-4d21-b4f4-0ba0966dd9f0-istio-podinfo\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd\" (UID: \"a28d9602-371e-4d21-b4f4-0ba0966dd9f0\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd"
Oct 31 19:33:33.312219 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.312204    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b36d1635-5796-4ff5-83e3-7d63ff424c78-istio-podinfo\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd\" (UID: \"b36d1635-5796-4ff5-83e3-7d63ff424c78\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd"
Oct 31 19:33:33.312383 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.312253    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a28d9602-371e-4d21-b4f4-0ba0966dd9f0-istiod-ca-cert\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd\" (UID: \"a28d9602-371e-4d21-b4f4-0ba0966dd9f0\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd"
Oct 31 19:33:33.312383 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.312364    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b36d1635-5796-4ff5-83e3-7d63ff424c78-credential-socket\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd\" (UID: \"b36d1635-5796-4ff5-83e3-7d63ff424c78\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd"
Oct 31 19:33:33.312482 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.312410    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a28d9602-371e-4d21-b4f4-0ba0966dd9f0-istio-data\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd\" (UID: \"a28d9602-371e-4d21-b4f4-0ba0966dd9f0\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd"
Oct 31 19:33:33.312571 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.312554    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a28d9602-371e-4d21-b4f4-0ba0966dd9f0-credential-socket\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd\" (UID: \"a28d9602-371e-4d21-b4f4-0ba0966dd9f0\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd"
Oct 31 19:33:33.312630 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.312598    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b36d1635-5796-4ff5-83e3-7d63ff424c78-istiod-ca-cert\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd\" (UID: \"b36d1635-5796-4ff5-83e3-7d63ff424c78\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd"
Oct 31 19:33:33.312671 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.312625    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsxh4\" (UniqueName: \"kubernetes.io/projected/b36d1635-5796-4ff5-83e3-7d63ff424c78-kube-api-access-hsxh4\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd\" (UID: \"b36d1635-5796-4ff5-83e3-7d63ff424c78\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd"
Oct 31 19:33:33.312721 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.312704    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b36d1635-5796-4ff5-83e3-7d63ff424c78-workload-socket\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd\" (UID: \"b36d1635-5796-4ff5-83e3-7d63ff424c78\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd"
Oct 31 19:33:33.313013 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.312769    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a28d9602-371e-4d21-b4f4-0ba0966dd9f0-istio-envoy\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd\" (UID: \"a28d9602-371e-4d21-b4f4-0ba0966dd9f0\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd"
Oct 31 19:33:33.313013 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.312810    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b36d1635-5796-4ff5-83e3-7d63ff424c78-workload-certs\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd\" (UID: \"b36d1635-5796-4ff5-83e3-7d63ff424c78\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd"
Oct 31 19:33:33.313013 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.312894    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a28d9602-371e-4d21-b4f4-0ba0966dd9f0-istio-token\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd\" (UID: \"a28d9602-371e-4d21-b4f4-0ba0966dd9f0\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd"
Oct 31 19:33:33.313013 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.312939    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a28d9602-371e-4d21-b4f4-0ba0966dd9f0-workload-socket\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd\" (UID: \"a28d9602-371e-4d21-b4f4-0ba0966dd9f0\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd"
Oct 31 19:33:33.313152 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.313015    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a28d9602-371e-4d21-b4f4-0ba0966dd9f0-workload-certs\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd\" (UID: \"a28d9602-371e-4d21-b4f4-0ba0966dd9f0\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd"
Oct 31 19:33:33.414902 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.414862    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b36d1635-5796-4ff5-83e3-7d63ff424c78-credential-socket\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd\" (UID: \"b36d1635-5796-4ff5-83e3-7d63ff424c78\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd"
Oct 31 19:33:33.414902 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.414902    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a28d9602-371e-4d21-b4f4-0ba0966dd9f0-istio-data\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd\" (UID: \"a28d9602-371e-4d21-b4f4-0ba0966dd9f0\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd"
Oct 31 19:33:33.415153 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.414961    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a28d9602-371e-4d21-b4f4-0ba0966dd9f0-credential-socket\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd\" (UID: \"a28d9602-371e-4d21-b4f4-0ba0966dd9f0\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd"
Oct 31 19:33:33.415153 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.414987    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b36d1635-5796-4ff5-83e3-7d63ff424c78-istiod-ca-cert\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd\" (UID: \"b36d1635-5796-4ff5-83e3-7d63ff424c78\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd"
Oct 31 19:33:33.415153 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.415003    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsxh4\" (UniqueName: \"kubernetes.io/projected/b36d1635-5796-4ff5-83e3-7d63ff424c78-kube-api-access-hsxh4\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd\" (UID: \"b36d1635-5796-4ff5-83e3-7d63ff424c78\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd"
Oct 31 19:33:33.415153 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.415030    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b36d1635-5796-4ff5-83e3-7d63ff424c78-workload-socket\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd\" (UID: \"b36d1635-5796-4ff5-83e3-7d63ff424c78\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd"
Oct 31 19:33:33.415580 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.415104    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a28d9602-371e-4d21-b4f4-0ba0966dd9f0-istio-envoy\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd\" (UID: \"a28d9602-371e-4d21-b4f4-0ba0966dd9f0\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd"
Oct 31 19:33:33.415705 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.415631    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b36d1635-5796-4ff5-83e3-7d63ff424c78-workload-certs\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd\" (UID: \"b36d1635-5796-4ff5-83e3-7d63ff424c78\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd"
Oct 31 19:33:33.415705 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.415702    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a28d9602-371e-4d21-b4f4-0ba0966dd9f0-istio-token\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd\" (UID: \"a28d9602-371e-4d21-b4f4-0ba0966dd9f0\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd"
Oct 31 19:33:33.415822 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.415742    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b36d1635-5796-4ff5-83e3-7d63ff424c78-istiod-ca-cert\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd\" (UID: \"b36d1635-5796-4ff5-83e3-7d63ff424c78\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd"
Oct 31 19:33:33.415822 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.415404    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b36d1635-5796-4ff5-83e3-7d63ff424c78-credential-socket\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd\" (UID: \"b36d1635-5796-4ff5-83e3-7d63ff424c78\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd"
Oct 31 19:33:33.415822 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.415756    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a28d9602-371e-4d21-b4f4-0ba0966dd9f0-workload-socket\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd\" (UID: \"a28d9602-371e-4d21-b4f4-0ba0966dd9f0\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd"
Oct 31 19:33:33.415987 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.415404    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a28d9602-371e-4d21-b4f4-0ba0966dd9f0-istio-data\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd\" (UID: \"a28d9602-371e-4d21-b4f4-0ba0966dd9f0\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd"
Oct 31 19:33:33.415987 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.415856    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a28d9602-371e-4d21-b4f4-0ba0966dd9f0-workload-certs\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd\" (UID: \"a28d9602-371e-4d21-b4f4-0ba0966dd9f0\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd"
Oct 31 19:33:33.415987 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.415946    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b36d1635-5796-4ff5-83e3-7d63ff424c78-istio-data\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd\" (UID: \"b36d1635-5796-4ff5-83e3-7d63ff424c78\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd"
Oct 31 19:33:33.416211 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.416020    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a28d9602-371e-4d21-b4f4-0ba0966dd9f0-workload-socket\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd\" (UID: \"a28d9602-371e-4d21-b4f4-0ba0966dd9f0\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd"
Oct 31 19:33:33.416211 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.416026    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b36d1635-5796-4ff5-83e3-7d63ff424c78-istio-envoy\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd\" (UID: \"b36d1635-5796-4ff5-83e3-7d63ff424c78\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd"
Oct 31 19:33:33.416211 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.416088    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sf6v\" (UniqueName: \"kubernetes.io/projected/a28d9602-371e-4d21-b4f4-0ba0966dd9f0-kube-api-access-7sf6v\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd\" (UID: \"a28d9602-371e-4d21-b4f4-0ba0966dd9f0\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd"
Oct 31 19:33:33.416211 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.416138    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b36d1635-5796-4ff5-83e3-7d63ff424c78-istio-token\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd\" (UID: \"b36d1635-5796-4ff5-83e3-7d63ff424c78\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd"
Oct 31 19:33:33.416211 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.416161    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a28d9602-371e-4d21-b4f4-0ba0966dd9f0-workload-certs\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd\" (UID: \"a28d9602-371e-4d21-b4f4-0ba0966dd9f0\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd"
Oct 31 19:33:33.416211 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.416181    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a28d9602-371e-4d21-b4f4-0ba0966dd9f0-istio-podinfo\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd\" (UID: \"a28d9602-371e-4d21-b4f4-0ba0966dd9f0\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd"
Oct 31 19:33:33.416211 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.416210    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b36d1635-5796-4ff5-83e3-7d63ff424c78-istio-podinfo\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd\" (UID: \"b36d1635-5796-4ff5-83e3-7d63ff424c78\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd"
Oct 31 19:33:33.416602 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.416262    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a28d9602-371e-4d21-b4f4-0ba0966dd9f0-istiod-ca-cert\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd\" (UID: \"a28d9602-371e-4d21-b4f4-0ba0966dd9f0\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd"
Oct 31 19:33:33.416602 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.415442    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a28d9602-371e-4d21-b4f4-0ba0966dd9f0-credential-socket\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd\" (UID: \"a28d9602-371e-4d21-b4f4-0ba0966dd9f0\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd"
Oct 31 19:33:33.416602 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.415537    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b36d1635-5796-4ff5-83e3-7d63ff424c78-workload-socket\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd\" (UID: \"b36d1635-5796-4ff5-83e3-7d63ff424c78\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd"
Oct 31 19:33:33.416762 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.416721    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b36d1635-5796-4ff5-83e3-7d63ff424c78-workload-certs\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd\" (UID: \"b36d1635-5796-4ff5-83e3-7d63ff424c78\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd"
Oct 31 19:33:33.416944 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.416892    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b36d1635-5796-4ff5-83e3-7d63ff424c78-istio-data\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd\" (UID: \"b36d1635-5796-4ff5-83e3-7d63ff424c78\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd"
Oct 31 19:33:33.417054 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.417037    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a28d9602-371e-4d21-b4f4-0ba0966dd9f0-istiod-ca-cert\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd\" (UID: \"a28d9602-371e-4d21-b4f4-0ba0966dd9f0\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd"
Oct 31 19:33:33.418822 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.418781    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a28d9602-371e-4d21-b4f4-0ba0966dd9f0-istio-envoy\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd\" (UID: \"a28d9602-371e-4d21-b4f4-0ba0966dd9f0\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd"
Oct 31 19:33:33.419374 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.419349    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a28d9602-371e-4d21-b4f4-0ba0966dd9f0-istio-podinfo\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd\" (UID: \"a28d9602-371e-4d21-b4f4-0ba0966dd9f0\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd"
Oct 31 19:33:33.419536 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.419515    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b36d1635-5796-4ff5-83e3-7d63ff424c78-istio-podinfo\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd\" (UID: \"b36d1635-5796-4ff5-83e3-7d63ff424c78\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd"
Oct 31 19:33:33.419807 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.419783    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b36d1635-5796-4ff5-83e3-7d63ff424c78-istio-envoy\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd\" (UID: \"b36d1635-5796-4ff5-83e3-7d63ff424c78\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd"
Oct 31 19:33:33.456822 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.456742    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a28d9602-371e-4d21-b4f4-0ba0966dd9f0-istio-token\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd\" (UID: \"a28d9602-371e-4d21-b4f4-0ba0966dd9f0\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd"
Oct 31 19:33:33.456822 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.456755    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsxh4\" (UniqueName: \"kubernetes.io/projected/b36d1635-5796-4ff5-83e3-7d63ff424c78-kube-api-access-hsxh4\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd\" (UID: \"b36d1635-5796-4ff5-83e3-7d63ff424c78\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd"
Oct 31 19:33:33.458191 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.458161    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b36d1635-5796-4ff5-83e3-7d63ff424c78-istio-token\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd\" (UID: \"b36d1635-5796-4ff5-83e3-7d63ff424c78\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd"
Oct 31 19:33:33.458348 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.458287    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sf6v\" (UniqueName: \"kubernetes.io/projected/a28d9602-371e-4d21-b4f4-0ba0966dd9f0-kube-api-access-7sf6v\") pod \"e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd\" (UID: \"a28d9602-371e-4d21-b4f4-0ba0966dd9f0\") " pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd"
Oct 31 19:33:33.504988 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.504950    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422"]
Oct 31 19:33:33.511847 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.511417    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422"
Oct 31 19:33:33.526223 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.513548    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd"
Oct 31 19:33:33.532022 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.531997    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:33.538411 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.536485    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422"]
Oct 31 19:33:33.551776 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.551743    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd"
Oct 31 19:33:33.636370 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.633597    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bc08948-e33b-4897-b931-021255991026-istio-token\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422\" (UID: \"2bc08948-e33b-4897-b931-021255991026\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422"
Oct 31 19:33:33.636855 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.636826    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bc08948-e33b-4897-b931-021255991026-istio-podinfo\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422\" (UID: \"2bc08948-e33b-4897-b931-021255991026\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422"
Oct 31 19:33:33.636935 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.636887    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bc08948-e33b-4897-b931-021255991026-istio-data\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422\" (UID: \"2bc08948-e33b-4897-b931-021255991026\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422"
Oct 31 19:33:33.636935 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.636912    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xfwd\" (UniqueName: \"kubernetes.io/projected/2bc08948-e33b-4897-b931-021255991026-kube-api-access-2xfwd\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422\" (UID: \"2bc08948-e33b-4897-b931-021255991026\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422"
Oct 31 19:33:33.637048 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.637037    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bc08948-e33b-4897-b931-021255991026-istio-envoy\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422\" (UID: \"2bc08948-e33b-4897-b931-021255991026\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422"
Oct 31 19:33:33.637127 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.637112    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bc08948-e33b-4897-b931-021255991026-credential-socket\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422\" (UID: \"2bc08948-e33b-4897-b931-021255991026\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422"
Oct 31 19:33:33.637228 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.637210    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bc08948-e33b-4897-b931-021255991026-istiod-ca-cert\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422\" (UID: \"2bc08948-e33b-4897-b931-021255991026\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422"
Oct 31 19:33:33.637286 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.637276    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bc08948-e33b-4897-b931-021255991026-workload-certs\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422\" (UID: \"2bc08948-e33b-4897-b931-021255991026\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422"
Oct 31 19:33:33.637366 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.637354    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bc08948-e33b-4897-b931-021255991026-workload-socket\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422\" (UID: \"2bc08948-e33b-4897-b931-021255991026\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422"
Oct 31 19:33:33.852281 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.852144    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bc08948-e33b-4897-b931-021255991026-istio-envoy\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422\" (UID: \"2bc08948-e33b-4897-b931-021255991026\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422"
Oct 31 19:33:33.875049 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.874973    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bc08948-e33b-4897-b931-021255991026-credential-socket\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422\" (UID: \"2bc08948-e33b-4897-b931-021255991026\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422"
Oct 31 19:33:33.875661 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.875439    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bc08948-e33b-4897-b931-021255991026-istio-envoy\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422\" (UID: \"2bc08948-e33b-4897-b931-021255991026\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422"
Oct 31 19:33:33.895062 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.874512    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bc08948-e33b-4897-b931-021255991026-credential-socket\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422\" (UID: \"2bc08948-e33b-4897-b931-021255991026\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422"
Oct 31 19:33:33.895062 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.893498    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bc08948-e33b-4897-b931-021255991026-istiod-ca-cert\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422\" (UID: \"2bc08948-e33b-4897-b931-021255991026\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422"
Oct 31 19:33:33.895062 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.893549    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bc08948-e33b-4897-b931-021255991026-workload-certs\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422\" (UID: \"2bc08948-e33b-4897-b931-021255991026\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422"
Oct 31 19:33:33.895062 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.893698    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bc08948-e33b-4897-b931-021255991026-workload-socket\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422\" (UID: \"2bc08948-e33b-4897-b931-021255991026\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422"
Oct 31 19:33:33.895062 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.893854    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bc08948-e33b-4897-b931-021255991026-istio-token\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422\" (UID: \"2bc08948-e33b-4897-b931-021255991026\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422"
Oct 31 19:33:33.895062 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.893964    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bc08948-e33b-4897-b931-021255991026-istio-podinfo\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422\" (UID: \"2bc08948-e33b-4897-b931-021255991026\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422"
Oct 31 19:33:33.895062 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.893996    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bc08948-e33b-4897-b931-021255991026-istio-data\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422\" (UID: \"2bc08948-e33b-4897-b931-021255991026\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422"
Oct 31 19:33:33.895062 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.894024    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xfwd\" (UniqueName: \"kubernetes.io/projected/2bc08948-e33b-4897-b931-021255991026-kube-api-access-2xfwd\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422\" (UID: \"2bc08948-e33b-4897-b931-021255991026\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422"
Oct 31 19:33:33.898407 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.898341    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bc08948-e33b-4897-b931-021255991026-istio-data\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422\" (UID: \"2bc08948-e33b-4897-b931-021255991026\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422"
Oct 31 19:33:33.900548 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.899859    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bc08948-e33b-4897-b931-021255991026-workload-socket\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422\" (UID: \"2bc08948-e33b-4897-b931-021255991026\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422"
Oct 31 19:33:33.900548 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.900105    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bc08948-e33b-4897-b931-021255991026-workload-certs\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422\" (UID: \"2bc08948-e33b-4897-b931-021255991026\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422"
Oct 31 19:33:33.901648 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.901584    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bc08948-e33b-4897-b931-021255991026-istiod-ca-cert\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422\" (UID: \"2bc08948-e33b-4897-b931-021255991026\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422"
Oct 31 19:33:33.962041 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.962006    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bc08948-e33b-4897-b931-021255991026-istio-podinfo\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422\" (UID: \"2bc08948-e33b-4897-b931-021255991026\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422"
Oct 31 19:33:33.962616 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.962564    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd"]
Oct 31 19:33:33.985186 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.984823    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bc08948-e33b-4897-b931-021255991026-istio-token\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422\" (UID: \"2bc08948-e33b-4897-b931-021255991026\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422"
Oct 31 19:33:33.999577 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.999099    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:33:33.999577 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.999194    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:33:33.999577 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:33.999238    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:33:34.002863 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:34.002833    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xfwd\" (UniqueName: \"kubernetes.io/projected/2bc08948-e33b-4897-b931-021255991026-kube-api-access-2xfwd\") pod \"e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422\" (UID: \"2bc08948-e33b-4897-b931-021255991026\") " pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422"
Oct 31 19:33:34.174451 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:34.174418    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422"
Oct 31 19:33:34.180766 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:34.180552    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd"]
Oct 31 19:33:34.278934 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:34.278884    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:33:34.279183 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:34.279157    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:33:34.279311 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:34.279293    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:33:34.388965 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:34.388679    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd" event={"ID":"b36d1635-5796-4ff5-83e3-7d63ff424c78","Type":"ContainerStarted","Data":"3d8363b34ad0e3628026dce6fdd30305241213411799e24cb4243cec5f32e8df"}
Oct 31 19:33:34.391156 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:34.391128    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd" event={"ID":"a28d9602-371e-4d21-b4f4-0ba0966dd9f0","Type":"ContainerStarted","Data":"69aeca40a4d178f613216a310e9de5d818f79541a8205d8e6a6bcc38ea7f31af"}
Oct 31 19:33:34.686597 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:34.686522    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:35.595353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:35.592214    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:35.707359 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:35.706986    4645 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw" podUID="1287db54-a587-4c11-9eb6-1606de555894" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/health\": context deadline exceeded"
Oct 31 19:33:35.970353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:35.966741    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4" podUID="6032a997-b795-4561-805c-110cf67a9d5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 31 19:33:36.067493 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:36.067452    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669" podUID="9285bee0-bf7c-4fd4-9da7-a1db021cae05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:36.256404 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:36.255267    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw" podUID="d504b2d3-c26d-4dcf-a7eb-efe20732ae4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:36.374350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:36.255422    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n" podUID="dbb97168-db93-4757-9a1b-f9c5eb704fc5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:36.479351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:36.471432    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l" podUID="12bea86c-61e5-465e-82fb-b4673a4d1be9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:36.479351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:36.478797    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x" podUID="a774e286-dcb8-4a2f-8541-26bf231eb08d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:36.550354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:36.543225    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:33:36.663889 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:36.663841    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:33:36.664521 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:36.664490    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:33:36.665127 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:36.596587    4645 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x" podUID="a774e286-dcb8-4a2f-8541-26bf231eb08d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": context deadline exceeded"
Oct 31 19:33:36.665682 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:36.599319    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd" event={"ID":"b36d1635-5796-4ff5-83e3-7d63ff424c78","Type":"ContainerStarted","Data":"be1177ce28e491d0d1df526c206ecfffdd7c102f847f8a052b306f944a2dcc4a"}
Oct 31 19:33:36.665890 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:36.665868    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422"]
Oct 31 19:33:36.666022 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:36.625701    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd" podStartSLOduration=3.625681751 podStartE2EDuration="3.625681751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:33:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:33:36.623825447 +0000 UTC m=+847.197059579" watchObservedRunningTime="2026-10-31 19:33:36.625681751 +0000 UTC m=+847.198915882"
Oct 31 19:33:36.733358 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:36.718689    4645 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n" podUID="dbb97168-db93-4757-9a1b-f9c5eb704fc5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": context deadline exceeded"
Oct 31 19:33:36.808419 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:36.807438    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd" event={"ID":"a28d9602-371e-4d21-b4f4-0ba0966dd9f0","Type":"ContainerStarted","Data":"7027ae75e445f2fddca99bbdbd51052db08abe34861186cff0c60f0742982ab4"}
Oct 31 19:33:36.997594 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:36.997377    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4" podUID="6032a997-b795-4561-805c-110cf67a9d5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:37.021518 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:37.003352    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4"
Oct 31 19:33:37.122373 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:36.600318    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw" podUID="1287db54-a587-4c11-9eb6-1606de555894" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:37.122373 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:37.083070    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669" podUID="9285bee0-bf7c-4fd4-9da7-a1db021cae05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:37.142548 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:37.136782    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669"
Oct 31 19:33:37.246467 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:37.225043    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd" podStartSLOduration=4.22501825 podStartE2EDuration="4.22501825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:33:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:33:37.169776236 +0000 UTC m=+847.743010412" watchObservedRunningTime="2026-10-31 19:33:37.22501825 +0000 UTC m=+847.798252379"
Oct 31 19:33:37.364646 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:37.364528    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n" podUID="dbb97168-db93-4757-9a1b-f9c5eb704fc5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:37.366312 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:37.366270    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw" podUID="d504b2d3-c26d-4dcf-a7eb-efe20732ae4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 31 19:33:37.367726 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:37.367709    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n"
Oct 31 19:33:37.394502 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:37.394465    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw"
Oct 31 19:33:37.524354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:37.500470    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x" podUID="a774e286-dcb8-4a2f-8541-26bf231eb08d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:37.524354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:37.501800    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x"
Oct 31 19:33:37.574355 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:37.550488    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l" podUID="12bea86c-61e5-465e-82fb-b4673a4d1be9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:37.596353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:37.571643    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd"
Oct 31 19:33:37.596353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:37.592646    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l"
Oct 31 19:33:37.596353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:37.592669    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd"
Oct 31 19:33:37.596353 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:33:37.594036    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 19:33:37.651420 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:37.622711    4645 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 31 19:33:37.651420 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:37.622772    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd" podUID="b36d1635-5796-4ff5-83e3-7d63ff424c78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 31 19:33:37.651420 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:37.623041    4645 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 31 19:33:37.651420 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:37.623074    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd" podUID="a28d9602-371e-4d21-b4f4-0ba0966dd9f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 31 19:33:38.005361 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:38.000446    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4" podUID="6032a997-b795-4561-805c-110cf67a9d5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 31 19:33:38.108892 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:38.102139    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw" podUID="1287db54-a587-4c11-9eb6-1606de555894" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": context deadline exceeded"
Oct 31 19:33:38.136359 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:38.129481    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669" podUID="9285bee0-bf7c-4fd4-9da7-a1db021cae05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 31 19:33:38.166352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:38.147494    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw"
Oct 31 19:33:38.398538 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:38.398478    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n" podUID="dbb97168-db93-4757-9a1b-f9c5eb704fc5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 19:33:38.414390 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:38.410486    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw" podUID="d504b2d3-c26d-4dcf-a7eb-efe20732ae4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:38.523353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:38.519546    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x" podUID="a774e286-dcb8-4a2f-8541-26bf231eb08d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:38.603437 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:38.591981    4645 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 31 19:33:38.603437 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:38.592033    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd" podUID="a28d9602-371e-4d21-b4f4-0ba0966dd9f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 31 19:33:38.634867 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:38.609862    4645 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 31 19:33:38.634867 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:38.609922    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd" podUID="b36d1635-5796-4ff5-83e3-7d63ff424c78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 31 19:33:38.634867 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:38.610063    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l" podUID="12bea86c-61e5-465e-82fb-b4673a4d1be9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 31 19:33:38.832031 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:38.831991    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422" event={"ID":"2bc08948-e33b-4897-b931-021255991026","Type":"ContainerStarted","Data":"543245d4d93f5662f8b4b70791a55b493a8893a7734997b0ad3cb21fd821d5f0"}
Oct 31 19:33:39.021472 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:39.021421    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4" podUID="6032a997-b795-4561-805c-110cf67a9d5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:39.164383 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:39.164320    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669" podUID="9285bee0-bf7c-4fd4-9da7-a1db021cae05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:39.165128 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:39.163991    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw" podUID="1287db54-a587-4c11-9eb6-1606de555894" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:39.233123 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:39.233081    4645 patch_prober.go:29] interesting pod/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:33:39.233434 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:39.233403    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln" podUID="f3b22b1e-9d49-4a54-b016-bc504f03390c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:39.422359 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:39.420834    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n" podUID="dbb97168-db93-4757-9a1b-f9c5eb704fc5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:39.461724 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:39.449560    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw" podUID="d504b2d3-c26d-4dcf-a7eb-efe20732ae4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:39.535624 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:39.535571    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x" podUID="a774e286-dcb8-4a2f-8541-26bf231eb08d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:39.549480 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:39.549440    4645 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 31 19:33:39.549757 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:39.549723    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd" podUID="b36d1635-5796-4ff5-83e3-7d63ff424c78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 31 19:33:39.748414 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:39.736824    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l" podUID="12bea86c-61e5-465e-82fb-b4673a4d1be9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 31 19:33:39.752352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:39.748795    4645 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 31 19:33:39.752352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:39.748852    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd" podUID="a28d9602-371e-4d21-b4f4-0ba0966dd9f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 31 19:33:40.108028 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:40.107827    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4" podUID="6032a997-b795-4561-805c-110cf67a9d5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 31 19:33:40.218237 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:40.218193    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw" podUID="1287db54-a587-4c11-9eb6-1606de555894" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:40.218684 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:40.218654    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669" podUID="9285bee0-bf7c-4fd4-9da7-a1db021cae05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:40.461356 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:40.459099    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n" podUID="dbb97168-db93-4757-9a1b-f9c5eb704fc5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:40.558882 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:40.558664    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x" podUID="a774e286-dcb8-4a2f-8541-26bf231eb08d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 19:33:40.564766 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:40.560357    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw" podUID="d504b2d3-c26d-4dcf-a7eb-efe20732ae4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:40.800526 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:40.794060    4645 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 31 19:33:40.825395 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:40.812960    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd" podUID="a28d9602-371e-4d21-b4f4-0ba0966dd9f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 31 19:33:40.825395 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:33:40.812905    4645 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.12s"
Oct 31 19:33:40.825395 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:40.813323    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422" event={"ID":"2bc08948-e33b-4897-b931-021255991026","Type":"ContainerStarted","Data":"62b3fbf61bfd0d8884480ba17cffc179991acb864b1dd1e4d6ab09ff82b2ac72"}
Oct 31 19:33:40.856809 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:40.856767    4645 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669" podUID="9285bee0-bf7c-4fd4-9da7-a1db021cae05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": context deadline exceeded"
Oct 31 19:33:40.857216 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:40.857156    4645 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 31 19:33:40.985888 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:40.985818    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd" podUID="b36d1635-5796-4ff5-83e3-7d63ff424c78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 31 19:33:40.985888 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:40.888994    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422" podStartSLOduration=7.888972287 podStartE2EDuration="7.888972287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:33:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:33:40.885888639 +0000 UTC m=+851.459122769" watchObservedRunningTime="2026-10-31 19:33:40.888972287 +0000 UTC m=+851.462206418"
Oct 31 19:33:40.989408 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:40.885066    4645 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw" podUID="d504b2d3-c26d-4dcf-a7eb-efe20732ae4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": context deadline exceeded"
Oct 31 19:33:40.989625 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:40.955123    4645 patch_prober.go:29] interesting pod/e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:33:40.989748 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:40.989725    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt" podUID="3fa97d00-d3af-4c53-bfce-83746cdd6ce3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:40.990237 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:40.901690    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l" podUID="12bea86c-61e5-465e-82fb-b4673a4d1be9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 31 19:33:41.192623 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:41.185414    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422"
Oct 31 19:33:41.195189 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:41.195159    4645 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 31 19:33:41.200245 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:41.200213    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422" podUID="2bc08948-e33b-4897-b931-021255991026" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 31 19:33:41.200627 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:41.195667    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l"
Oct 31 19:33:41.373354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:41.349168    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4"
Oct 31 19:33:41.373354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:41.351409    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw"
Oct 31 19:33:41.505447 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:41.495876    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:41.637362 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:41.614775    4645 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 31 19:33:41.637680 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:41.637631    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd" podUID="a28d9602-371e-4d21-b4f4-0ba0966dd9f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 31 19:33:41.648354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:41.638298    4645 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 31 19:33:41.665046 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:41.648584    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd" podUID="b36d1635-5796-4ff5-83e3-7d63ff424c78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 31 19:33:41.787412 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:41.774506    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669" podUID="9285bee0-bf7c-4fd4-9da7-a1db021cae05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 31 19:33:41.967864 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:41.967768    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n" podUID="dbb97168-db93-4757-9a1b-f9c5eb704fc5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 19:33:41.989778 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:41.989736    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x" podUID="a774e286-dcb8-4a2f-8541-26bf231eb08d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 19:33:42.049404 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:42.024592    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw" podUID="d504b2d3-c26d-4dcf-a7eb-efe20732ae4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 31 19:33:42.201237 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:42.201198    4645 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 31 19:33:42.201891 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:42.201855    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422" podUID="2bc08948-e33b-4897-b931-021255991026" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 31 19:33:42.228360 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:42.211188    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw"
Oct 31 19:33:42.523510 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:42.500307    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x"
Oct 31 19:33:42.609820 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:42.609770    4645 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 31 19:33:42.610138 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:42.610089    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd" podUID="b36d1635-5796-4ff5-83e3-7d63ff424c78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 31 19:33:42.615520 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:42.615471    4645 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 31 19:33:42.623378 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:42.619395    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd" podUID="a28d9602-371e-4d21-b4f4-0ba0966dd9f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 31 19:33:42.647914 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:42.647868    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l" podUID="12bea86c-61e5-465e-82fb-b4673a4d1be9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 31 19:33:42.946281 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:42.946243    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n"
Oct 31 19:33:42.974904 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:42.974849    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669" podUID="9285bee0-bf7c-4fd4-9da7-a1db021cae05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:33:43.408961 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:43.408861    4645 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 31 19:33:43.468239 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:43.468195    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422" podUID="2bc08948-e33b-4897-b931-021255991026" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 31 19:33:43.469018 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:43.434671    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:43.629352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:43.619237    4645 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 31 19:33:43.629352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:43.619301    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd" podUID="b36d1635-5796-4ff5-83e3-7d63ff424c78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 31 19:33:43.629352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:43.619556    4645 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 31 19:33:43.629352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:43.619585    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd" podUID="a28d9602-371e-4d21-b4f4-0ba0966dd9f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 31 19:33:43.852374 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:43.852301    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669"
Oct 31 19:33:44.187712 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:44.186443    4645 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 31 19:33:44.187712 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:44.186511    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422" podUID="2bc08948-e33b-4897-b931-021255991026" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 31 19:33:44.518396 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:44.518284    4645 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 31 19:33:44.523473 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:44.523429    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd" podUID="b36d1635-5796-4ff5-83e3-7d63ff424c78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 31 19:33:44.553250 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:44.553221    4645 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 31 19:33:44.553489 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:44.553469    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd" podUID="a28d9602-371e-4d21-b4f4-0ba0966dd9f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 31 19:33:45.183350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:45.182680    4645 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 31 19:33:45.183350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:45.182745    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422" podUID="2bc08948-e33b-4897-b931-021255991026" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 31 19:33:45.512437 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:45.508907    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:45.516562 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:45.516424    4645 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 31 19:33:45.516840 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:45.516813    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd" podUID="b36d1635-5796-4ff5-83e3-7d63ff424c78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 31 19:33:45.552267 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:45.552231    4645 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 31 19:33:45.552676 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:45.552299    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd" podUID="a28d9602-371e-4d21-b4f4-0ba0966dd9f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 31 19:33:46.197389 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:46.193047    4645 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 31 19:33:46.200628 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:46.197651    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422" podUID="2bc08948-e33b-4897-b931-021255991026" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 31 19:33:46.515342 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:46.515231    4645 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 31 19:33:46.515342 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:46.515299    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd" podUID="b36d1635-5796-4ff5-83e3-7d63ff424c78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 31 19:33:46.607858 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:46.607819    4645 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 31 19:33:46.608465 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:46.608430    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd" podUID="a28d9602-371e-4d21-b4f4-0ba0966dd9f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 31 19:33:47.123941 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:47.123815    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:47.180758 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:47.180720    4645 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Oct 31 19:33:47.181011 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:47.180980    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422" podUID="2bc08948-e33b-4897-b931-021255991026" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Oct 31 19:33:47.649454 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:47.649420    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd"
Oct 31 19:33:47.652467 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:47.650119    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd"
Oct 31 19:33:47.653468 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:47.653445    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd"
Oct 31 19:33:47.667205 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:47.667170    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd"
Oct 31 19:33:47.669049 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:47.669012    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd"
Oct 31 19:33:47.669861 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:47.669833    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd"
Oct 31 19:33:48.553151 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:48.553110    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422"
Oct 31 19:33:48.553922 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:48.553888    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422"
Oct 31 19:33:48.554757 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:48.554733    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422"
Oct 31 19:33:48.793941 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:48.793906    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:50.425096 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:50.425058    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:51.398958 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:51.398918    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:52.393695 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:52.393660    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:53.324429 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:53.324392    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:54.290066 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:54.290034    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:55.063099 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:55.063069    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:55.981579 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:55.981544    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:56.878418 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:56.878363    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:57.630449 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:57.630412    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:58.644783 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:58.644751    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:33:59.577560 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:33:59.577525    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:00.491895 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:00.491862    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:01.504907 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:01.504873    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:02.356933 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:02.356898    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:03.242140 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:03.242098    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:04.202160 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:04.202122    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:04.749060 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:04.749024    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:05.567266 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:05.567237    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:06.248661 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:34:06.248626    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 19:34:06.464840 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:06.462723    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:07.167617 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:07.167577    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:08.038360 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:08.038300    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:08.867527 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:08.867490    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:10.565521 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:10.565473    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x" podUID="a774e286-dcb8-4a2f-8541-26bf231eb08d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:10.864094 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:10.863990    4645 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669" podUID="9285bee0-bf7c-4fd4-9da7-a1db021cae05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": context deadline exceeded"
Oct 31 19:34:10.889218 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:10.889170    4645 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw" podUID="d504b2d3-c26d-4dcf-a7eb-efe20732ae4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:10.974744 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:10.974705    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n" podUID="dbb97168-db93-4757-9a1b-f9c5eb704fc5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 19:34:11.054187 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:11.054143    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4" podUID="6032a997-b795-4561-805c-110cf67a9d5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:11.192354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:11.171904    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw" podUID="1287db54-a587-4c11-9eb6-1606de555894" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:11.194036 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:11.176437    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669" podUID="9285bee0-bf7c-4fd4-9da7-a1db021cae05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 31 19:34:11.423966 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:11.423923    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw" podUID="d504b2d3-c26d-4dcf-a7eb-efe20732ae4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 31 19:34:11.569525 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:11.569405    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x" podUID="a774e286-dcb8-4a2f-8541-26bf231eb08d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 19:34:11.571435 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:11.571408    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x"
Oct 31 19:34:11.858069 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:11.857966    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l" podUID="12bea86c-61e5-465e-82fb-b4673a4d1be9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:12.041502 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:12.041459    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n" podUID="dbb97168-db93-4757-9a1b-f9c5eb704fc5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:12.042468 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:12.042445    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n"
Oct 31 19:34:12.072926 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:12.072885    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4" podUID="6032a997-b795-4561-805c-110cf67a9d5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 31 19:34:12.073559 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:12.073536    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4"
Oct 31 19:34:12.400354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:12.243389    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw" podUID="1287db54-a587-4c11-9eb6-1606de555894" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": context deadline exceeded"
Oct 31 19:34:12.489353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:12.450048    4645 patch_prober.go:29] interesting pod/e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:34:12.489353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:12.459910    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs" podUID="40240c3e-0f73-4ac9-80c1-4c581a6ee5a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:12.489353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:12.453309    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669" podUID="9285bee0-bf7c-4fd4-9da7-a1db021cae05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 31 19:34:12.489353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:12.460552    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw"
Oct 31 19:34:12.489353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:12.460609    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669"
Oct 31 19:34:12.489353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:12.450355    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw" podUID="d504b2d3-c26d-4dcf-a7eb-efe20732ae4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:12.489353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:12.464603    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw"
Oct 31 19:34:12.546354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:12.542482    4645 patch_prober.go:29] interesting pod/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:34:12.546354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:12.542547    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4" podUID="adb9b9a5-acbc-4967-b115-ed4e68a84b7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:12.704194 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:12.703837    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x" podUID="a774e286-dcb8-4a2f-8541-26bf231eb08d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:12.863363 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:12.863307    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:12.920131 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:12.887551    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l" podUID="12bea86c-61e5-465e-82fb-b4673a4d1be9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:12.920131 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:12.918932    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l"
Oct 31 19:34:13.042529 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:13.042416    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n" podUID="dbb97168-db93-4757-9a1b-f9c5eb704fc5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 19:34:13.159353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:13.148340    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4" podUID="6032a997-b795-4561-805c-110cf67a9d5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 31 19:34:13.444385 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:13.443371    4645 patch_prober.go:29] interesting pod/e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:34:13.444385 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:13.443436    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5" podUID="2bf496d2-b4cd-417b-b5dc-3afbac91f26b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:13.468364 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:13.467753    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw" podUID="d504b2d3-c26d-4dcf-a7eb-efe20732ae4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 31 19:34:13.476629 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:13.476481    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw" podUID="1287db54-a587-4c11-9eb6-1606de555894" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": context deadline exceeded"
Oct 31 19:34:13.477115 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:13.476848    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669" podUID="9285bee0-bf7c-4fd4-9da7-a1db021cae05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 31 19:34:13.704756 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:13.704661    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x" podUID="a774e286-dcb8-4a2f-8541-26bf231eb08d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:13.919449 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:13.919404    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l" podUID="12bea86c-61e5-465e-82fb-b4673a4d1be9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 31 19:34:14.047748 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:14.047624    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n" podUID="dbb97168-db93-4757-9a1b-f9c5eb704fc5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 19:34:14.152280 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:14.151512    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4" podUID="6032a997-b795-4561-805c-110cf67a9d5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 31 19:34:14.468671 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:14.468091    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw" podUID="d504b2d3-c26d-4dcf-a7eb-efe20732ae4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 31 19:34:14.516007 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:14.514077    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw" podUID="1287db54-a587-4c11-9eb6-1606de555894" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": context deadline exceeded"
Oct 31 19:34:14.516007 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:14.514479    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669" podUID="9285bee0-bf7c-4fd4-9da7-a1db021cae05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 31 19:34:14.516007 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:14.515928    4645 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4" podUID="6032a997-b795-4561-805c-110cf67a9d5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:14.713156 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:14.712661    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x" podUID="a774e286-dcb8-4a2f-8541-26bf231eb08d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:15.052354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:14.985170    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l" podUID="12bea86c-61e5-465e-82fb-b4673a4d1be9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:15.088375 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:15.058585    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n" podUID="dbb97168-db93-4757-9a1b-f9c5eb704fc5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:15.176614 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:15.165940    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4" podUID="6032a997-b795-4561-805c-110cf67a9d5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:15.331267 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:15.321526    4645 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l" podUID="12bea86c-61e5-465e-82fb-b4673a4d1be9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:15.470049 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:15.468749    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw" podUID="d504b2d3-c26d-4dcf-a7eb-efe20732ae4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:15.498845 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:15.497256    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw"
Oct 31 19:34:15.501546 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:15.501432    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw"
Oct 31 19:34:15.517459 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:15.517425    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669" podUID="9285bee0-bf7c-4fd4-9da7-a1db021cae05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 31 19:34:15.536222 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:15.533637    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n"
Oct 31 19:34:15.548343 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:15.543625    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l"
Oct 31 19:34:15.630452 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:15.630375    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x"
Oct 31 19:34:15.641501 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:15.632757    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669"
Oct 31 19:34:15.641501 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:15.633418    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4"
Oct 31 19:34:15.779739 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:15.779709    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:16.700589 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:16.700552    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:17.540029 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:17.539987    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:18.703262 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:18.703227    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:19.464939 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:19.464909    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:20.275473 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:20.275442    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:21.054705 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:21.054674    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:21.193662 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:21.193623    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:21.874110 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:21.874078    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:22.259418 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:22.259387    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:23.001484 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:23.001450    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:23.125402 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:23.124476    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:23.287776 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:23.287747    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:23.699629 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:23.695860    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-85be723d-gw-openshift-default-76d9b68fb5-2lbp5"]
Oct 31 19:34:23.709966 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:23.709940    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-85be723d-gw-openshift-default-76d9b68fb5-2lbp5"]
Oct 31 19:34:23.710128 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:23.710047    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-85be723d-gw-openshift-default-76d9b68fb5-2lbp5"
Oct 31 19:34:23.714897 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:23.714852    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-85be723d-gw-openshift-default-dockercfg-9ldbw\""
Oct 31 19:34:23.755008 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:23.754979    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:23.799012 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:23.798987    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw968\" (UniqueName: \"kubernetes.io/projected/9744a5bd-39a7-4449-b5d4-2ff35e08084b-kube-api-access-zw968\") pod \"e2e-migrate-85be723d-gw-openshift-default-76d9b68fb5-2lbp5\" (UID: \"9744a5bd-39a7-4449-b5d4-2ff35e08084b\") " pod="openshift-ingress/e2e-migrate-85be723d-gw-openshift-default-76d9b68fb5-2lbp5"
Oct 31 19:34:23.799124 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:23.799096    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9744a5bd-39a7-4449-b5d4-2ff35e08084b-istiod-ca-cert\") pod \"e2e-migrate-85be723d-gw-openshift-default-76d9b68fb5-2lbp5\" (UID: \"9744a5bd-39a7-4449-b5d4-2ff35e08084b\") " pod="openshift-ingress/e2e-migrate-85be723d-gw-openshift-default-76d9b68fb5-2lbp5"
Oct 31 19:34:23.799168 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:23.799120    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9744a5bd-39a7-4449-b5d4-2ff35e08084b-credential-socket\") pod \"e2e-migrate-85be723d-gw-openshift-default-76d9b68fb5-2lbp5\" (UID: \"9744a5bd-39a7-4449-b5d4-2ff35e08084b\") " pod="openshift-ingress/e2e-migrate-85be723d-gw-openshift-default-76d9b68fb5-2lbp5"
Oct 31 19:34:23.799273 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:23.799254    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9744a5bd-39a7-4449-b5d4-2ff35e08084b-workload-certs\") pod \"e2e-migrate-85be723d-gw-openshift-default-76d9b68fb5-2lbp5\" (UID: \"9744a5bd-39a7-4449-b5d4-2ff35e08084b\") " pod="openshift-ingress/e2e-migrate-85be723d-gw-openshift-default-76d9b68fb5-2lbp5"
Oct 31 19:34:23.799309 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:23.799293    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9744a5bd-39a7-4449-b5d4-2ff35e08084b-istio-data\") pod \"e2e-migrate-85be723d-gw-openshift-default-76d9b68fb5-2lbp5\" (UID: \"9744a5bd-39a7-4449-b5d4-2ff35e08084b\") " pod="openshift-ingress/e2e-migrate-85be723d-gw-openshift-default-76d9b68fb5-2lbp5"
Oct 31 19:34:23.799374 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:23.799363    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9744a5bd-39a7-4449-b5d4-2ff35e08084b-istio-envoy\") pod \"e2e-migrate-85be723d-gw-openshift-default-76d9b68fb5-2lbp5\" (UID: \"9744a5bd-39a7-4449-b5d4-2ff35e08084b\") " pod="openshift-ingress/e2e-migrate-85be723d-gw-openshift-default-76d9b68fb5-2lbp5"
Oct 31 19:34:23.799484 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:23.799470    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9744a5bd-39a7-4449-b5d4-2ff35e08084b-workload-socket\") pod \"e2e-migrate-85be723d-gw-openshift-default-76d9b68fb5-2lbp5\" (UID: \"9744a5bd-39a7-4449-b5d4-2ff35e08084b\") " pod="openshift-ingress/e2e-migrate-85be723d-gw-openshift-default-76d9b68fb5-2lbp5"
Oct 31 19:34:23.799531 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:23.799521    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9744a5bd-39a7-4449-b5d4-2ff35e08084b-istio-podinfo\") pod \"e2e-migrate-85be723d-gw-openshift-default-76d9b68fb5-2lbp5\" (UID: \"9744a5bd-39a7-4449-b5d4-2ff35e08084b\") " pod="openshift-ingress/e2e-migrate-85be723d-gw-openshift-default-76d9b68fb5-2lbp5"
Oct 31 19:34:23.799566 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:23.799543    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9744a5bd-39a7-4449-b5d4-2ff35e08084b-istio-token\") pod \"e2e-migrate-85be723d-gw-openshift-default-76d9b68fb5-2lbp5\" (UID: \"9744a5bd-39a7-4449-b5d4-2ff35e08084b\") " pod="openshift-ingress/e2e-migrate-85be723d-gw-openshift-default-76d9b68fb5-2lbp5"
Oct 31 19:34:23.901350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:23.900354    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:23.910382 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:23.901851    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw968\" (UniqueName: \"kubernetes.io/projected/9744a5bd-39a7-4449-b5d4-2ff35e08084b-kube-api-access-zw968\") pod \"e2e-migrate-85be723d-gw-openshift-default-76d9b68fb5-2lbp5\" (UID: \"9744a5bd-39a7-4449-b5d4-2ff35e08084b\") " pod="openshift-ingress/e2e-migrate-85be723d-gw-openshift-default-76d9b68fb5-2lbp5"
Oct 31 19:34:23.910382 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:23.902368    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9744a5bd-39a7-4449-b5d4-2ff35e08084b-istiod-ca-cert\") pod \"e2e-migrate-85be723d-gw-openshift-default-76d9b68fb5-2lbp5\" (UID: \"9744a5bd-39a7-4449-b5d4-2ff35e08084b\") " pod="openshift-ingress/e2e-migrate-85be723d-gw-openshift-default-76d9b68fb5-2lbp5"
Oct 31 19:34:23.910382 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:23.902401    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9744a5bd-39a7-4449-b5d4-2ff35e08084b-credential-socket\") pod \"e2e-migrate-85be723d-gw-openshift-default-76d9b68fb5-2lbp5\" (UID: \"9744a5bd-39a7-4449-b5d4-2ff35e08084b\") " pod="openshift-ingress/e2e-migrate-85be723d-gw-openshift-default-76d9b68fb5-2lbp5"
Oct 31 19:34:23.910382 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:23.903941    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9744a5bd-39a7-4449-b5d4-2ff35e08084b-workload-certs\") pod \"e2e-migrate-85be723d-gw-openshift-default-76d9b68fb5-2lbp5\" (UID: \"9744a5bd-39a7-4449-b5d4-2ff35e08084b\") " pod="openshift-ingress/e2e-migrate-85be723d-gw-openshift-default-76d9b68fb5-2lbp5"
Oct 31 19:34:23.910382 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:23.903988    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9744a5bd-39a7-4449-b5d4-2ff35e08084b-istio-data\") pod \"e2e-migrate-85be723d-gw-openshift-default-76d9b68fb5-2lbp5\" (UID: \"9744a5bd-39a7-4449-b5d4-2ff35e08084b\") " pod="openshift-ingress/e2e-migrate-85be723d-gw-openshift-default-76d9b68fb5-2lbp5"
Oct 31 19:34:23.910382 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:23.904045    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9744a5bd-39a7-4449-b5d4-2ff35e08084b-istio-envoy\") pod \"e2e-migrate-85be723d-gw-openshift-default-76d9b68fb5-2lbp5\" (UID: \"9744a5bd-39a7-4449-b5d4-2ff35e08084b\") " pod="openshift-ingress/e2e-migrate-85be723d-gw-openshift-default-76d9b68fb5-2lbp5"
Oct 31 19:34:23.910382 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:23.904145    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9744a5bd-39a7-4449-b5d4-2ff35e08084b-workload-socket\") pod \"e2e-migrate-85be723d-gw-openshift-default-76d9b68fb5-2lbp5\" (UID: \"9744a5bd-39a7-4449-b5d4-2ff35e08084b\") " pod="openshift-ingress/e2e-migrate-85be723d-gw-openshift-default-76d9b68fb5-2lbp5"
Oct 31 19:34:23.910382 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:23.903801    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9744a5bd-39a7-4449-b5d4-2ff35e08084b-istiod-ca-cert\") pod \"e2e-migrate-85be723d-gw-openshift-default-76d9b68fb5-2lbp5\" (UID: \"9744a5bd-39a7-4449-b5d4-2ff35e08084b\") " pod="openshift-ingress/e2e-migrate-85be723d-gw-openshift-default-76d9b68fb5-2lbp5"
Oct 31 19:34:23.910382 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:23.903143    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9744a5bd-39a7-4449-b5d4-2ff35e08084b-credential-socket\") pod \"e2e-migrate-85be723d-gw-openshift-default-76d9b68fb5-2lbp5\" (UID: \"9744a5bd-39a7-4449-b5d4-2ff35e08084b\") " pod="openshift-ingress/e2e-migrate-85be723d-gw-openshift-default-76d9b68fb5-2lbp5"
Oct 31 19:34:23.910382 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:23.904580    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9744a5bd-39a7-4449-b5d4-2ff35e08084b-workload-socket\") pod \"e2e-migrate-85be723d-gw-openshift-default-76d9b68fb5-2lbp5\" (UID: \"9744a5bd-39a7-4449-b5d4-2ff35e08084b\") " pod="openshift-ingress/e2e-migrate-85be723d-gw-openshift-default-76d9b68fb5-2lbp5"
Oct 31 19:34:23.910382 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:23.904809    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9744a5bd-39a7-4449-b5d4-2ff35e08084b-workload-certs\") pod \"e2e-migrate-85be723d-gw-openshift-default-76d9b68fb5-2lbp5\" (UID: \"9744a5bd-39a7-4449-b5d4-2ff35e08084b\") " pod="openshift-ingress/e2e-migrate-85be723d-gw-openshift-default-76d9b68fb5-2lbp5"
Oct 31 19:34:23.910382 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:23.905033    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9744a5bd-39a7-4449-b5d4-2ff35e08084b-istio-data\") pod \"e2e-migrate-85be723d-gw-openshift-default-76d9b68fb5-2lbp5\" (UID: \"9744a5bd-39a7-4449-b5d4-2ff35e08084b\") " pod="openshift-ingress/e2e-migrate-85be723d-gw-openshift-default-76d9b68fb5-2lbp5"
Oct 31 19:34:23.910382 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:23.905662    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9744a5bd-39a7-4449-b5d4-2ff35e08084b-istio-podinfo\") pod \"e2e-migrate-85be723d-gw-openshift-default-76d9b68fb5-2lbp5\" (UID: \"9744a5bd-39a7-4449-b5d4-2ff35e08084b\") " pod="openshift-ingress/e2e-migrate-85be723d-gw-openshift-default-76d9b68fb5-2lbp5"
Oct 31 19:34:23.910382 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:23.905702    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9744a5bd-39a7-4449-b5d4-2ff35e08084b-istio-token\") pod \"e2e-migrate-85be723d-gw-openshift-default-76d9b68fb5-2lbp5\" (UID: \"9744a5bd-39a7-4449-b5d4-2ff35e08084b\") " pod="openshift-ingress/e2e-migrate-85be723d-gw-openshift-default-76d9b68fb5-2lbp5"
Oct 31 19:34:23.949353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:23.948869    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9744a5bd-39a7-4449-b5d4-2ff35e08084b-istio-token\") pod \"e2e-migrate-85be723d-gw-openshift-default-76d9b68fb5-2lbp5\" (UID: \"9744a5bd-39a7-4449-b5d4-2ff35e08084b\") " pod="openshift-ingress/e2e-migrate-85be723d-gw-openshift-default-76d9b68fb5-2lbp5"
Oct 31 19:34:23.953395 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:23.950419    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw968\" (UniqueName: \"kubernetes.io/projected/9744a5bd-39a7-4449-b5d4-2ff35e08084b-kube-api-access-zw968\") pod \"e2e-migrate-85be723d-gw-openshift-default-76d9b68fb5-2lbp5\" (UID: \"9744a5bd-39a7-4449-b5d4-2ff35e08084b\") " pod="openshift-ingress/e2e-migrate-85be723d-gw-openshift-default-76d9b68fb5-2lbp5"
Oct 31 19:34:23.953395 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:23.951200    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9744a5bd-39a7-4449-b5d4-2ff35e08084b-istio-podinfo\") pod \"e2e-migrate-85be723d-gw-openshift-default-76d9b68fb5-2lbp5\" (UID: \"9744a5bd-39a7-4449-b5d4-2ff35e08084b\") " pod="openshift-ingress/e2e-migrate-85be723d-gw-openshift-default-76d9b68fb5-2lbp5"
Oct 31 19:34:23.953395 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:23.951452    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9744a5bd-39a7-4449-b5d4-2ff35e08084b-istio-envoy\") pod \"e2e-migrate-85be723d-gw-openshift-default-76d9b68fb5-2lbp5\" (UID: \"9744a5bd-39a7-4449-b5d4-2ff35e08084b\") " pod="openshift-ingress/e2e-migrate-85be723d-gw-openshift-default-76d9b68fb5-2lbp5"
Oct 31 19:34:24.040353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:24.039788    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-85be723d-gw-openshift-default-76d9b68fb5-2lbp5"
Oct 31 19:34:24.344898 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:24.344779    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:24.436530 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:24.436494    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-85be723d-gw-openshift-default-76d9b68fb5-2lbp5"]
Oct 31 19:34:24.440110 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:24.440027    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:34:24.440232 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:24.440120    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:34:24.440232 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:24.440177    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:34:24.571168 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:24.571133    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-85be723d-gw-openshift-default-76d9b68fb5-2lbp5" event={"ID":"9744a5bd-39a7-4449-b5d4-2ff35e08084b","Type":"ContainerStarted","Data":"b71e4b898a7511285f44ac1515214284218a6f7cbbc547d8761d47387ee2a12e"}
Oct 31 19:34:24.571398 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:24.571189    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-85be723d-gw-openshift-default-76d9b68fb5-2lbp5" event={"ID":"9744a5bd-39a7-4449-b5d4-2ff35e08084b","Type":"ContainerStarted","Data":"dd490a089e1b3bff64d4d4a2f5d708465cc3506082beea02205052cc75cdadb9"}
Oct 31 19:34:24.595107 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:24.595004    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-85be723d-gw-openshift-default-76d9b68fb5-2lbp5" podStartSLOduration=1.594989247 podStartE2EDuration="1.594989247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:34:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:34:24.592588816 +0000 UTC m=+895.165822969" watchObservedRunningTime="2026-10-31 19:34:24.594989247 +0000 UTC m=+895.168223375"
Oct 31 19:34:25.041113 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:25.041080    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-85be723d-gw-openshift-default-76d9b68fb5-2lbp5"
Oct 31 19:34:25.046689 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:25.046665    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-85be723d-gw-openshift-default-76d9b68fb5-2lbp5"
Oct 31 19:34:25.408518 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:25.408444    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:25.580473 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:25.580443    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-85be723d-gw-openshift-default-76d9b68fb5-2lbp5"
Oct 31 19:34:25.581299 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:25.581278    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-85be723d-gw-openshift-default-76d9b68fb5-2lbp5"
Oct 31 19:34:26.404420 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:26.404379    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:26.524389 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:26.524353    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:26.861457 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:26.861422    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:27.029748 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:27.029713    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:27.141214 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:27.141132    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:27.988739 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:27.988707    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:29.140687 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:29.140655    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:29.700939 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:29.700901    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:29.802100 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:29.802072    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:30.033850 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:30.033757    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:30.169653 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:30.169616    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:30.314923 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:30.314842    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:30.800562 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:30.800347    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:30.929450 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:30.929415    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:31.508688 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:31.508657    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:31.627934 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:31.627898    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:32.359950 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:32.359919    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:33.432484 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:33.432446    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:34.292182 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:34.292139    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:34.520682 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:34.520644    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:34.717447 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:34.717413    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:35.168961 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:35.168931    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:35.735230 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:35.735137    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:35.792840 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:35.792804    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1cc6560c-58795b5bc5-hq6bn"]
Oct 31 19:34:35.798218 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:35.798194    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1cc6560c-58795b5bc5-hq6bn"
Oct 31 19:34:35.801816 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:35.801794    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-1cc6560c\""
Oct 31 19:34:35.805110 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:35.805087    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1cc6560c-58795b5bc5-hq6bn"]
Oct 31 19:34:35.822890 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:35.822860    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-1cc6560c-dockercfg-pz86b\""
Oct 31 19:34:35.846825 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:35.846774    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:35.951293 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:35.951259    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wns8h\" (UniqueName: \"kubernetes.io/projected/a6ac25a9-5887-4497-88d8-db5162badbd5-kube-api-access-wns8h\") pod \"payload-pre-processing-e2e-ait-1cc6560c-58795b5bc5-hq6bn\" (UID: \"a6ac25a9-5887-4497-88d8-db5162badbd5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1cc6560c-58795b5bc5-hq6bn"
Oct 31 19:34:35.951502 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:35.951467    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6ac25a9-5887-4497-88d8-db5162badbd5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1cc6560c-58795b5bc5-hq6bn\" (UID: \"a6ac25a9-5887-4497-88d8-db5162badbd5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1cc6560c-58795b5bc5-hq6bn"
Oct 31 19:34:35.966841 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:35.966812    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:36.061870 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:36.061762    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6ac25a9-5887-4497-88d8-db5162badbd5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1cc6560c-58795b5bc5-hq6bn\" (UID: \"a6ac25a9-5887-4497-88d8-db5162badbd5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1cc6560c-58795b5bc5-hq6bn"
Oct 31 19:34:36.062453 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:36.062427    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wns8h\" (UniqueName: \"kubernetes.io/projected/a6ac25a9-5887-4497-88d8-db5162badbd5-kube-api-access-wns8h\") pod \"payload-pre-processing-e2e-ait-1cc6560c-58795b5bc5-hq6bn\" (UID: \"a6ac25a9-5887-4497-88d8-db5162badbd5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1cc6560c-58795b5bc5-hq6bn"
Oct 31 19:34:36.064894 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:36.064873    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6ac25a9-5887-4497-88d8-db5162badbd5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1cc6560c-58795b5bc5-hq6bn\" (UID: \"a6ac25a9-5887-4497-88d8-db5162badbd5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1cc6560c-58795b5bc5-hq6bn"
Oct 31 19:34:36.087365 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:36.083021    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wns8h\" (UniqueName: \"kubernetes.io/projected/a6ac25a9-5887-4497-88d8-db5162badbd5-kube-api-access-wns8h\") pod \"payload-pre-processing-e2e-ait-1cc6560c-58795b5bc5-hq6bn\" (UID: \"a6ac25a9-5887-4497-88d8-db5162badbd5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1cc6560c-58795b5bc5-hq6bn"
Oct 31 19:34:36.135867 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:36.134616    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1cc6560c-58795b5bc5-hq6bn"
Oct 31 19:34:36.364137 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:34:36.364102    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 19:34:36.576193 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:36.576157    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1cc6560c-58795b5bc5-hq6bn"]
Oct 31 19:34:36.580522 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:34:36.580497    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6ac25a9_5887_4497_88d8_db5162badbd5.slice/crio-94ca5fbc80b0e413eb235f061173787180ae0157de380fd4c0522790ac616109 WatchSource:0}: Error finding container 94ca5fbc80b0e413eb235f061173787180ae0157de380fd4c0522790ac616109: Status 404 returned error can't find the container with id 94ca5fbc80b0e413eb235f061173787180ae0157de380fd4c0522790ac616109
Oct 31 19:34:36.739287 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:36.739253    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1cc6560c-58795b5bc5-hq6bn" event={"ID":"a6ac25a9-5887-4497-88d8-db5162badbd5","Type":"ContainerStarted","Data":"94ca5fbc80b0e413eb235f061173787180ae0157de380fd4c0522790ac616109"}
Oct 31 19:34:37.090926 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:37.090878    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:37.747616 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:37.747581    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1cc6560c-58795b5bc5-hq6bn" event={"ID":"a6ac25a9-5887-4497-88d8-db5162badbd5","Type":"ContainerStarted","Data":"7d3cb91322f149e2a35267047c7107736e9d96e7644fe3599581e157394cf64c"}
Oct 31 19:34:37.748028 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:37.747642    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1cc6560c-58795b5bc5-hq6bn"
Oct 31 19:34:37.769667 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:37.769621    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-1cc6560c-58795b5bc5-hq6bn" podStartSLOduration=2.769609149 podStartE2EDuration="2.769609149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:34:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:34:37.765452798 +0000 UTC m=+908.338686928" watchObservedRunningTime="2026-10-31 19:34:37.769609149 +0000 UTC m=+908.342843279"
Oct 31 19:34:38.134764 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:38.134668    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:39.050209 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:39.050176    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:39.901584 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:39.901548    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:40.016474 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:40.016434    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:40.958002 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:40.957967    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:41.785440 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:41.785409    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:42.663128 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:42.663097    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:43.432376 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:43.432307    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:43.565379 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:43.565291    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:43.806791 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:43.806684    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:44.095512 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:44.095426    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:45.482486 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:45.482433    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw" podUID="1287db54-a587-4c11-9eb6-1606de555894" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:45.506446 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:45.506407    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw" podUID="d504b2d3-c26d-4dcf-a7eb-efe20732ae4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:45.521461 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:45.521419    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669" podUID="9285bee0-bf7c-4fd4-9da7-a1db021cae05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:45.645379 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:45.644481    4645 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw" podUID="1287db54-a587-4c11-9eb6-1606de555894" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:45.726355 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:45.722681    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x" podUID="a774e286-dcb8-4a2f-8541-26bf231eb08d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 19:34:46.083468 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:46.083406    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l" podUID="12bea86c-61e5-465e-82fb-b4673a4d1be9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:46.118472 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:46.118427    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n" podUID="dbb97168-db93-4757-9a1b-f9c5eb704fc5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:46.172470 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:46.172417    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4" podUID="6032a997-b795-4561-805c-110cf67a9d5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:46.484474 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:46.484419    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw" podUID="1287db54-a587-4c11-9eb6-1606de555894" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": context deadline exceeded"
Oct 31 19:34:46.485659 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:46.485633    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw"
Oct 31 19:34:46.528386 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:46.525389    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw" podUID="d504b2d3-c26d-4dcf-a7eb-efe20732ae4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:46.528386 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:46.525989    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw"
Oct 31 19:34:46.528386 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:46.527772    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669" podUID="9285bee0-bf7c-4fd4-9da7-a1db021cae05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 31 19:34:46.528386 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:46.528114    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669"
Oct 31 19:34:46.605517 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:46.605461    4645 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x" podUID="a774e286-dcb8-4a2f-8541-26bf231eb08d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:46.725478 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:46.725428    4645 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n" podUID="dbb97168-db93-4757-9a1b-f9c5eb704fc5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:46.725973 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:46.725938    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x" podUID="a774e286-dcb8-4a2f-8541-26bf231eb08d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:46.745993 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:46.745910    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x"
Oct 31 19:34:47.095990 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:47.088038    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l" podUID="12bea86c-61e5-465e-82fb-b4673a4d1be9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:47.102703 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:47.102662    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l"
Oct 31 19:34:47.165532 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:47.127313    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n" podUID="dbb97168-db93-4757-9a1b-f9c5eb704fc5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:47.165532 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:47.128861    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n"
Oct 31 19:34:47.351473 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:47.349468    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4" podUID="6032a997-b795-4561-805c-110cf67a9d5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 31 19:34:47.385354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:47.357412    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4"
Oct 31 19:34:47.548348 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:47.522997    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw" podUID="1287db54-a587-4c11-9eb6-1606de555894" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:47.630789 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:47.630689    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669" podUID="9285bee0-bf7c-4fd4-9da7-a1db021cae05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:47.799353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:47.786935    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x" podUID="a774e286-dcb8-4a2f-8541-26bf231eb08d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 19:34:47.921171 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:47.921128    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw" podUID="d504b2d3-c26d-4dcf-a7eb-efe20732ae4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 31 19:34:48.128941 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:48.128887    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n" podUID="dbb97168-db93-4757-9a1b-f9c5eb704fc5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:48.285702 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:48.175257    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l" podUID="12bea86c-61e5-465e-82fb-b4673a4d1be9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:48.353353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:48.352455    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4" podUID="6032a997-b795-4561-805c-110cf67a9d5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:48.573421 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:48.570103    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw" podUID="1287db54-a587-4c11-9eb6-1606de555894" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:48.683759 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:48.683717    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669" podUID="9285bee0-bf7c-4fd4-9da7-a1db021cae05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 31 19:34:48.792354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:48.789150    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x" podUID="a774e286-dcb8-4a2f-8541-26bf231eb08d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:48.891423 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:48.886481    4645 patch_prober.go:29] interesting pod/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:34:48.891423 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:48.886572    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s" podUID="426c8fc3-1152-4a03-81e6-ada81e4d0007" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:48.959353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:48.937404    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw" podUID="d504b2d3-c26d-4dcf-a7eb-efe20732ae4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:48.959353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:48.955727    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1cc6560c-58795b5bc5-hq6bn"
Oct 31 19:34:49.023152 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:49.023106    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:49.256360 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:49.245609    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n" podUID="dbb97168-db93-4757-9a1b-f9c5eb704fc5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:49.410610 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:49.409059    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l" podUID="12bea86c-61e5-465e-82fb-b4673a4d1be9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 31 19:34:49.411183 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:49.409102    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4" podUID="6032a997-b795-4561-805c-110cf67a9d5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:49.701554 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:49.701512    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669" podUID="9285bee0-bf7c-4fd4-9da7-a1db021cae05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:49.870621 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:49.734442    4645 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:34:49.870621 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:49.869834    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422" podUID="2bc08948-e33b-4897-b931-021255991026" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:49.870621 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:49.772572    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw" podUID="1287db54-a587-4c11-9eb6-1606de555894" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": context deadline exceeded"
Oct 31 19:34:49.874563 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:49.753029    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw"
Oct 31 19:34:49.874563 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:49.798699    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x" podUID="a774e286-dcb8-4a2f-8541-26bf231eb08d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:34:49.932338 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:49.931208    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:49.942466 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:49.942434    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l"
Oct 31 19:34:49.943051 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:49.943030    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4"
Oct 31 19:34:49.944293 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:49.944273    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n"
Oct 31 19:34:49.946866 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:49.945689    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw"
Oct 31 19:34:49.947090 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:49.947056    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x"
Oct 31 19:34:49.947692 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:49.947666    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669"
Oct 31 19:34:50.066278 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:50.066156    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:51.128089 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:51.128033    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:52.129968 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:52.129929    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:53.071895 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:53.071860    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:53.965892 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:53.965851    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:54.766484 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:54.766433    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:55.566036 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:55.566006    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:56.707277 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:56.707236    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:57.750186 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:57.750153    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:58.556807 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:58.556773    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:34:59.408745 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:34:59.408707    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:00.621311 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:00.621277    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:01.544456 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:01.544424    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:01.664914 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:01.664888    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:02.301078 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:02.301046    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:02.688709 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:02.688678    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:02.864130 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:02.864085    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:02.982155 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:02.982059    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:03.671629 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:03.671589    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:04.584648 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:04.584606    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:05.346579 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:05.346543    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:06.247247 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:06.247214    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:06.366342 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:35:06.366294    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 19:35:06.959406 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:06.959355    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:07.167351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:07.155512    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:07.363425 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:07.363344    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:08.147254 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:08.147223    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:09.055017 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:09.054984    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:09.172750 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:09.172713    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:09.886080 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:09.886050    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:10.164100 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:10.164065    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:10.322165 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:10.322138    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:10.503034 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:10.502952    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:10.637819 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:10.637792    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:10.811298 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:10.811233    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:10.939386 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:10.939322    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:11.670934 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:11.670901    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:12.028972 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:12.028870    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:12.136941 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:12.136905    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:12.524995 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:12.524963    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:12.923611 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:12.923579    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:13.030024 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:13.029992    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:13.139111 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:13.139077    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:13.151530 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:13.151503    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-85be723d-gw-openshift-default-76d9b68fb5-2lbp5"]
Oct 31 19:35:13.151763 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:13.151720    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-85be723d-gw-openshift-default-76d9b68fb5-2lbp5" podUID="9744a5bd-39a7-4449-b5d4-2ff35e08084b" containerName="istio-proxy" containerID="cri-o://b71e4b898a7511285f44ac1515214284218a6f7cbbc547d8761d47387ee2a12e" gracePeriod=30
Oct 31 19:35:13.264587 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:13.264497    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:13.858681 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:13.858649    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:14.044615 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:14.044572    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:14.405575 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:14.405539    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-63eab93e-openshift-default-6d4c965c9c-lwwjn"]
Oct 31 19:35:14.427533 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:14.424262    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-63eab93e-openshift-default-6d4c965c9c-lwwjn"
Oct 31 19:35:14.436209 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:14.436168    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71e3bcf-7168-42d4-a719-69d77cb2d362-credential-socket\") pod \"e2e-ait-63eab93e-openshift-default-6d4c965c9c-lwwjn\" (UID: \"e71e3bcf-7168-42d4-a719-69d77cb2d362\") " pod="openshift-ingress/e2e-ait-63eab93e-openshift-default-6d4c965c9c-lwwjn"
Oct 31 19:35:14.436554 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:14.436530    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e71e3bcf-7168-42d4-a719-69d77cb2d362-istio-envoy\") pod \"e2e-ait-63eab93e-openshift-default-6d4c965c9c-lwwjn\" (UID: \"e71e3bcf-7168-42d4-a719-69d77cb2d362\") " pod="openshift-ingress/e2e-ait-63eab93e-openshift-default-6d4c965c9c-lwwjn"
Oct 31 19:35:14.436687 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:14.436671    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t2fb\" (UniqueName: \"kubernetes.io/projected/e71e3bcf-7168-42d4-a719-69d77cb2d362-kube-api-access-7t2fb\") pod \"e2e-ait-63eab93e-openshift-default-6d4c965c9c-lwwjn\" (UID: \"e71e3bcf-7168-42d4-a719-69d77cb2d362\") " pod="openshift-ingress/e2e-ait-63eab93e-openshift-default-6d4c965c9c-lwwjn"
Oct 31 19:35:14.439010 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:14.438978    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e71e3bcf-7168-42d4-a719-69d77cb2d362-istio-token\") pod \"e2e-ait-63eab93e-openshift-default-6d4c965c9c-lwwjn\" (UID: \"e71e3bcf-7168-42d4-a719-69d77cb2d362\") " pod="openshift-ingress/e2e-ait-63eab93e-openshift-default-6d4c965c9c-lwwjn"
Oct 31 19:35:14.439175 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:14.439157    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71e3bcf-7168-42d4-a719-69d77cb2d362-workload-socket\") pod \"e2e-ait-63eab93e-openshift-default-6d4c965c9c-lwwjn\" (UID: \"e71e3bcf-7168-42d4-a719-69d77cb2d362\") " pod="openshift-ingress/e2e-ait-63eab93e-openshift-default-6d4c965c9c-lwwjn"
Oct 31 19:35:14.439294 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:14.439278    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e71e3bcf-7168-42d4-a719-69d77cb2d362-istio-data\") pod \"e2e-ait-63eab93e-openshift-default-6d4c965c9c-lwwjn\" (UID: \"e71e3bcf-7168-42d4-a719-69d77cb2d362\") " pod="openshift-ingress/e2e-ait-63eab93e-openshift-default-6d4c965c9c-lwwjn"
Oct 31 19:35:14.439434 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:14.439418    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e71e3bcf-7168-42d4-a719-69d77cb2d362-workload-certs\") pod \"e2e-ait-63eab93e-openshift-default-6d4c965c9c-lwwjn\" (UID: \"e71e3bcf-7168-42d4-a719-69d77cb2d362\") " pod="openshift-ingress/e2e-ait-63eab93e-openshift-default-6d4c965c9c-lwwjn"
Oct 31 19:35:14.439552 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:14.439535    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e71e3bcf-7168-42d4-a719-69d77cb2d362-istiod-ca-cert\") pod \"e2e-ait-63eab93e-openshift-default-6d4c965c9c-lwwjn\" (UID: \"e71e3bcf-7168-42d4-a719-69d77cb2d362\") " pod="openshift-ingress/e2e-ait-63eab93e-openshift-default-6d4c965c9c-lwwjn"
Oct 31 19:35:14.439804 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:14.439791    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e71e3bcf-7168-42d4-a719-69d77cb2d362-istio-podinfo\") pod \"e2e-ait-63eab93e-openshift-default-6d4c965c9c-lwwjn\" (UID: \"e71e3bcf-7168-42d4-a719-69d77cb2d362\") " pod="openshift-ingress/e2e-ait-63eab93e-openshift-default-6d4c965c9c-lwwjn"
Oct 31 19:35:14.477443 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:14.452741    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-63eab93e-openshift-default-dockercfg-g9g6q\""
Oct 31 19:35:14.477443 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:14.468711    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-63eab93e-openshift-default-6d4c965c9c-lwwjn"]
Oct 31 19:35:14.500455 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:14.500027    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:14.541413 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:14.541379    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71e3bcf-7168-42d4-a719-69d77cb2d362-credential-socket\") pod \"e2e-ait-63eab93e-openshift-default-6d4c965c9c-lwwjn\" (UID: \"e71e3bcf-7168-42d4-a719-69d77cb2d362\") " pod="openshift-ingress/e2e-ait-63eab93e-openshift-default-6d4c965c9c-lwwjn"
Oct 31 19:35:14.542142 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:14.542030    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71e3bcf-7168-42d4-a719-69d77cb2d362-credential-socket\") pod \"e2e-ait-63eab93e-openshift-default-6d4c965c9c-lwwjn\" (UID: \"e71e3bcf-7168-42d4-a719-69d77cb2d362\") " pod="openshift-ingress/e2e-ait-63eab93e-openshift-default-6d4c965c9c-lwwjn"
Oct 31 19:35:14.542454 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:14.542431    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e71e3bcf-7168-42d4-a719-69d77cb2d362-istio-envoy\") pod \"e2e-ait-63eab93e-openshift-default-6d4c965c9c-lwwjn\" (UID: \"e71e3bcf-7168-42d4-a719-69d77cb2d362\") " pod="openshift-ingress/e2e-ait-63eab93e-openshift-default-6d4c965c9c-lwwjn"
Oct 31 19:35:14.556971 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:14.556933    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t2fb\" (UniqueName: \"kubernetes.io/projected/e71e3bcf-7168-42d4-a719-69d77cb2d362-kube-api-access-7t2fb\") pod \"e2e-ait-63eab93e-openshift-default-6d4c965c9c-lwwjn\" (UID: \"e71e3bcf-7168-42d4-a719-69d77cb2d362\") " pod="openshift-ingress/e2e-ait-63eab93e-openshift-default-6d4c965c9c-lwwjn"
Oct 31 19:35:14.557529 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:14.557506    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e71e3bcf-7168-42d4-a719-69d77cb2d362-istio-token\") pod \"e2e-ait-63eab93e-openshift-default-6d4c965c9c-lwwjn\" (UID: \"e71e3bcf-7168-42d4-a719-69d77cb2d362\") " pod="openshift-ingress/e2e-ait-63eab93e-openshift-default-6d4c965c9c-lwwjn"
Oct 31 19:35:14.557949 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:14.557931    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71e3bcf-7168-42d4-a719-69d77cb2d362-workload-socket\") pod \"e2e-ait-63eab93e-openshift-default-6d4c965c9c-lwwjn\" (UID: \"e71e3bcf-7168-42d4-a719-69d77cb2d362\") " pod="openshift-ingress/e2e-ait-63eab93e-openshift-default-6d4c965c9c-lwwjn"
Oct 31 19:35:14.558470 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:14.558448    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e71e3bcf-7168-42d4-a719-69d77cb2d362-istio-data\") pod \"e2e-ait-63eab93e-openshift-default-6d4c965c9c-lwwjn\" (UID: \"e71e3bcf-7168-42d4-a719-69d77cb2d362\") " pod="openshift-ingress/e2e-ait-63eab93e-openshift-default-6d4c965c9c-lwwjn"
Oct 31 19:35:14.558913 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:14.558895    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e71e3bcf-7168-42d4-a719-69d77cb2d362-workload-certs\") pod \"e2e-ait-63eab93e-openshift-default-6d4c965c9c-lwwjn\" (UID: \"e71e3bcf-7168-42d4-a719-69d77cb2d362\") " pod="openshift-ingress/e2e-ait-63eab93e-openshift-default-6d4c965c9c-lwwjn"
Oct 31 19:35:14.559042 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:14.559025    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e71e3bcf-7168-42d4-a719-69d77cb2d362-istiod-ca-cert\") pod \"e2e-ait-63eab93e-openshift-default-6d4c965c9c-lwwjn\" (UID: \"e71e3bcf-7168-42d4-a719-69d77cb2d362\") " pod="openshift-ingress/e2e-ait-63eab93e-openshift-default-6d4c965c9c-lwwjn"
Oct 31 19:35:14.562304 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:14.558387    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71e3bcf-7168-42d4-a719-69d77cb2d362-workload-socket\") pod \"e2e-ait-63eab93e-openshift-default-6d4c965c9c-lwwjn\" (UID: \"e71e3bcf-7168-42d4-a719-69d77cb2d362\") " pod="openshift-ingress/e2e-ait-63eab93e-openshift-default-6d4c965c9c-lwwjn"
Oct 31 19:35:14.562506 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:14.558837    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e71e3bcf-7168-42d4-a719-69d77cb2d362-istio-data\") pod \"e2e-ait-63eab93e-openshift-default-6d4c965c9c-lwwjn\" (UID: \"e71e3bcf-7168-42d4-a719-69d77cb2d362\") " pod="openshift-ingress/e2e-ait-63eab93e-openshift-default-6d4c965c9c-lwwjn"
Oct 31 19:35:14.563548 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:14.562144    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e71e3bcf-7168-42d4-a719-69d77cb2d362-istiod-ca-cert\") pod \"e2e-ait-63eab93e-openshift-default-6d4c965c9c-lwwjn\" (UID: \"e71e3bcf-7168-42d4-a719-69d77cb2d362\") " pod="openshift-ingress/e2e-ait-63eab93e-openshift-default-6d4c965c9c-lwwjn"
Oct 31 19:35:14.563853 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:14.563836    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e71e3bcf-7168-42d4-a719-69d77cb2d362-workload-certs\") pod \"e2e-ait-63eab93e-openshift-default-6d4c965c9c-lwwjn\" (UID: \"e71e3bcf-7168-42d4-a719-69d77cb2d362\") " pod="openshift-ingress/e2e-ait-63eab93e-openshift-default-6d4c965c9c-lwwjn"
Oct 31 19:35:14.564174 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:14.564148    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e71e3bcf-7168-42d4-a719-69d77cb2d362-istio-podinfo\") pod \"e2e-ait-63eab93e-openshift-default-6d4c965c9c-lwwjn\" (UID: \"e71e3bcf-7168-42d4-a719-69d77cb2d362\") " pod="openshift-ingress/e2e-ait-63eab93e-openshift-default-6d4c965c9c-lwwjn"
Oct 31 19:35:14.568851 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:14.567470    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e71e3bcf-7168-42d4-a719-69d77cb2d362-istio-envoy\") pod \"e2e-ait-63eab93e-openshift-default-6d4c965c9c-lwwjn\" (UID: \"e71e3bcf-7168-42d4-a719-69d77cb2d362\") " pod="openshift-ingress/e2e-ait-63eab93e-openshift-default-6d4c965c9c-lwwjn"
Oct 31 19:35:14.578084 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:14.578055    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e71e3bcf-7168-42d4-a719-69d77cb2d362-istio-token\") pod \"e2e-ait-63eab93e-openshift-default-6d4c965c9c-lwwjn\" (UID: \"e71e3bcf-7168-42d4-a719-69d77cb2d362\") " pod="openshift-ingress/e2e-ait-63eab93e-openshift-default-6d4c965c9c-lwwjn"
Oct 31 19:35:14.678270 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:14.678241    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e71e3bcf-7168-42d4-a719-69d77cb2d362-istio-podinfo\") pod \"e2e-ait-63eab93e-openshift-default-6d4c965c9c-lwwjn\" (UID: \"e71e3bcf-7168-42d4-a719-69d77cb2d362\") " pod="openshift-ingress/e2e-ait-63eab93e-openshift-default-6d4c965c9c-lwwjn"
Oct 31 19:35:14.682750 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:14.682697    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t2fb\" (UniqueName: \"kubernetes.io/projected/e71e3bcf-7168-42d4-a719-69d77cb2d362-kube-api-access-7t2fb\") pod \"e2e-ait-63eab93e-openshift-default-6d4c965c9c-lwwjn\" (UID: \"e71e3bcf-7168-42d4-a719-69d77cb2d362\") " pod="openshift-ingress/e2e-ait-63eab93e-openshift-default-6d4c965c9c-lwwjn"
Oct 31 19:35:14.813249 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:14.813213    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:14.924678 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:14.924644    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-63eab93e-openshift-default-6d4c965c9c-lwwjn"
Oct 31 19:35:15.220714 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:35:15.220679    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode71e3bcf_7168_42d4_a719_69d77cb2d362.slice/crio-86c75b8821012291c3b4dfbab2de30050aa0994b87c6366dadfcf979aa9cba14 WatchSource:0}: Error finding container 86c75b8821012291c3b4dfbab2de30050aa0994b87c6366dadfcf979aa9cba14: Status 404 returned error can't find the container with id 86c75b8821012291c3b4dfbab2de30050aa0994b87c6366dadfcf979aa9cba14
Oct 31 19:35:15.221027 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:15.220773    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-63eab93e-openshift-default-6d4c965c9c-lwwjn"]
Oct 31 19:35:15.223299 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:15.223269    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:35:15.223406 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:15.223368    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:35:15.223451 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:15.223403    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:35:15.234488 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:15.234454    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:15.754987 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:15.754454    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1cc6560c-58795b5bc5-hq6bn"]
Oct 31 19:35:15.825945 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:15.825907    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:15.938798 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:15.938765    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:16.070439 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:16.068317    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:16.173534 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:16.172488    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-63eab93e-openshift-default-6d4c965c9c-lwwjn" event={"ID":"e71e3bcf-7168-42d4-a719-69d77cb2d362","Type":"ContainerStarted","Data":"1167df67295e61f33f0c97df63eb0765b354b2829c585eee0f0670814d5f8a0f"}
Oct 31 19:35:16.173534 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:16.172559    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-63eab93e-openshift-default-6d4c965c9c-lwwjn" event={"ID":"e71e3bcf-7168-42d4-a719-69d77cb2d362","Type":"ContainerStarted","Data":"86c75b8821012291c3b4dfbab2de30050aa0994b87c6366dadfcf979aa9cba14"}
Oct 31 19:35:16.202355 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:16.196221    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:16.231351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:16.219905    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-63eab93e-openshift-default-6d4c965c9c-lwwjn" podStartSLOduration=2.219884943 podStartE2EDuration="2.219884943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:35:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:35:16.21881623 +0000 UTC m=+946.792050362" watchObservedRunningTime="2026-10-31 19:35:16.219884943 +0000 UTC m=+946.793119077"
Oct 31 19:35:16.314293 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:16.314255    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:16.430537 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:16.430488    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:16.593593 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:16.577846    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:16.926773 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:16.925472    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-63eab93e-openshift-default-6d4c965c9c-lwwjn"
Oct 31 19:35:16.933228 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:16.933200    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-63eab93e-openshift-default-6d4c965c9c-lwwjn"
Oct 31 19:35:17.196410 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:17.193779    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:17.209211 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:17.209183    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-63eab93e-openshift-default-6d4c965c9c-lwwjn"
Oct 31 19:35:17.210951 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:17.210929    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-63eab93e-openshift-default-6d4c965c9c-lwwjn"
Oct 31 19:35:17.744840 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:17.744805    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:17.922465 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:17.922432    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:18.042237 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:18.042158    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:18.158181 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:18.158150    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-0b9ff685-openshift-default-648b65bf89-k5qjt"]
Oct 31 19:35:18.182801 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:18.182776    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-0b9ff685-openshift-default-648b65bf89-k5qjt"
Oct 31 19:35:18.186694 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:18.186653    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-0b9ff685-openshift-default-648b65bf89-k5qjt"]
Oct 31 19:35:18.201366 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:18.201341    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-0b9ff685-openshift-default-dockercfg-t5vnw\""
Oct 31 19:35:18.226245 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:18.226210    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:18.287692 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:18.287661    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ec649cc-bc05-452a-95b8-efaf6c891a4c-workload-certs\") pod \"e2e-derive-0b9ff685-openshift-default-648b65bf89-k5qjt\" (UID: \"6ec649cc-bc05-452a-95b8-efaf6c891a4c\") " pod="openshift-ingress/e2e-derive-0b9ff685-openshift-default-648b65bf89-k5qjt"
Oct 31 19:35:18.287796 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:18.287737    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ec649cc-bc05-452a-95b8-efaf6c891a4c-istio-envoy\") pod \"e2e-derive-0b9ff685-openshift-default-648b65bf89-k5qjt\" (UID: \"6ec649cc-bc05-452a-95b8-efaf6c891a4c\") " pod="openshift-ingress/e2e-derive-0b9ff685-openshift-default-648b65bf89-k5qjt"
Oct 31 19:35:18.287892 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:18.287842    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ec649cc-bc05-452a-95b8-efaf6c891a4c-workload-socket\") pod \"e2e-derive-0b9ff685-openshift-default-648b65bf89-k5qjt\" (UID: \"6ec649cc-bc05-452a-95b8-efaf6c891a4c\") " pod="openshift-ingress/e2e-derive-0b9ff685-openshift-default-648b65bf89-k5qjt"
Oct 31 19:35:18.287965 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:18.287942    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ec649cc-bc05-452a-95b8-efaf6c891a4c-credential-socket\") pod \"e2e-derive-0b9ff685-openshift-default-648b65bf89-k5qjt\" (UID: \"6ec649cc-bc05-452a-95b8-efaf6c891a4c\") " pod="openshift-ingress/e2e-derive-0b9ff685-openshift-default-648b65bf89-k5qjt"
Oct 31 19:35:18.288007 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:18.287982    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ec649cc-bc05-452a-95b8-efaf6c891a4c-istio-token\") pod \"e2e-derive-0b9ff685-openshift-default-648b65bf89-k5qjt\" (UID: \"6ec649cc-bc05-452a-95b8-efaf6c891a4c\") " pod="openshift-ingress/e2e-derive-0b9ff685-openshift-default-648b65bf89-k5qjt"
Oct 31 19:35:18.288082 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:18.288066    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ec649cc-bc05-452a-95b8-efaf6c891a4c-istio-podinfo\") pod \"e2e-derive-0b9ff685-openshift-default-648b65bf89-k5qjt\" (UID: \"6ec649cc-bc05-452a-95b8-efaf6c891a4c\") " pod="openshift-ingress/e2e-derive-0b9ff685-openshift-default-648b65bf89-k5qjt"
Oct 31 19:35:18.288184 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:18.288166    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-929rg\" (UniqueName: \"kubernetes.io/projected/6ec649cc-bc05-452a-95b8-efaf6c891a4c-kube-api-access-929rg\") pod \"e2e-derive-0b9ff685-openshift-default-648b65bf89-k5qjt\" (UID: \"6ec649cc-bc05-452a-95b8-efaf6c891a4c\") " pod="openshift-ingress/e2e-derive-0b9ff685-openshift-default-648b65bf89-k5qjt"
Oct 31 19:35:18.288322 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:18.288308    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ec649cc-bc05-452a-95b8-efaf6c891a4c-istio-data\") pod \"e2e-derive-0b9ff685-openshift-default-648b65bf89-k5qjt\" (UID: \"6ec649cc-bc05-452a-95b8-efaf6c891a4c\") " pod="openshift-ingress/e2e-derive-0b9ff685-openshift-default-648b65bf89-k5qjt"
Oct 31 19:35:18.288427 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:18.288394    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ec649cc-bc05-452a-95b8-efaf6c891a4c-istiod-ca-cert\") pod \"e2e-derive-0b9ff685-openshift-default-648b65bf89-k5qjt\" (UID: \"6ec649cc-bc05-452a-95b8-efaf6c891a4c\") " pod="openshift-ingress/e2e-derive-0b9ff685-openshift-default-648b65bf89-k5qjt"
Oct 31 19:35:18.354365 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:18.351580    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:18.435352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:18.434473    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ec649cc-bc05-452a-95b8-efaf6c891a4c-istio-data\") pod \"e2e-derive-0b9ff685-openshift-default-648b65bf89-k5qjt\" (UID: \"6ec649cc-bc05-452a-95b8-efaf6c891a4c\") " pod="openshift-ingress/e2e-derive-0b9ff685-openshift-default-648b65bf89-k5qjt"
Oct 31 19:35:18.435352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:18.434561    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ec649cc-bc05-452a-95b8-efaf6c891a4c-istiod-ca-cert\") pod \"e2e-derive-0b9ff685-openshift-default-648b65bf89-k5qjt\" (UID: \"6ec649cc-bc05-452a-95b8-efaf6c891a4c\") " pod="openshift-ingress/e2e-derive-0b9ff685-openshift-default-648b65bf89-k5qjt"
Oct 31 19:35:18.435352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:18.434702    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ec649cc-bc05-452a-95b8-efaf6c891a4c-workload-certs\") pod \"e2e-derive-0b9ff685-openshift-default-648b65bf89-k5qjt\" (UID: \"6ec649cc-bc05-452a-95b8-efaf6c891a4c\") " pod="openshift-ingress/e2e-derive-0b9ff685-openshift-default-648b65bf89-k5qjt"
Oct 31 19:35:18.435352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:18.434731    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ec649cc-bc05-452a-95b8-efaf6c891a4c-istio-envoy\") pod \"e2e-derive-0b9ff685-openshift-default-648b65bf89-k5qjt\" (UID: \"6ec649cc-bc05-452a-95b8-efaf6c891a4c\") " pod="openshift-ingress/e2e-derive-0b9ff685-openshift-default-648b65bf89-k5qjt"
Oct 31 19:35:18.435352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:18.434839    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ec649cc-bc05-452a-95b8-efaf6c891a4c-workload-socket\") pod \"e2e-derive-0b9ff685-openshift-default-648b65bf89-k5qjt\" (UID: \"6ec649cc-bc05-452a-95b8-efaf6c891a4c\") " pod="openshift-ingress/e2e-derive-0b9ff685-openshift-default-648b65bf89-k5qjt"
Oct 31 19:35:18.435352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:18.434901    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ec649cc-bc05-452a-95b8-efaf6c891a4c-credential-socket\") pod \"e2e-derive-0b9ff685-openshift-default-648b65bf89-k5qjt\" (UID: \"6ec649cc-bc05-452a-95b8-efaf6c891a4c\") " pod="openshift-ingress/e2e-derive-0b9ff685-openshift-default-648b65bf89-k5qjt"
Oct 31 19:35:18.435352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:18.434933    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ec649cc-bc05-452a-95b8-efaf6c891a4c-istio-token\") pod \"e2e-derive-0b9ff685-openshift-default-648b65bf89-k5qjt\" (UID: \"6ec649cc-bc05-452a-95b8-efaf6c891a4c\") " pod="openshift-ingress/e2e-derive-0b9ff685-openshift-default-648b65bf89-k5qjt"
Oct 31 19:35:18.435352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:18.434976    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ec649cc-bc05-452a-95b8-efaf6c891a4c-istio-podinfo\") pod \"e2e-derive-0b9ff685-openshift-default-648b65bf89-k5qjt\" (UID: \"6ec649cc-bc05-452a-95b8-efaf6c891a4c\") " pod="openshift-ingress/e2e-derive-0b9ff685-openshift-default-648b65bf89-k5qjt"
Oct 31 19:35:18.435352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:18.435035    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-929rg\" (UniqueName: \"kubernetes.io/projected/6ec649cc-bc05-452a-95b8-efaf6c891a4c-kube-api-access-929rg\") pod \"e2e-derive-0b9ff685-openshift-default-648b65bf89-k5qjt\" (UID: \"6ec649cc-bc05-452a-95b8-efaf6c891a4c\") " pod="openshift-ingress/e2e-derive-0b9ff685-openshift-default-648b65bf89-k5qjt"
Oct 31 19:35:18.457896 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:18.436093    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ec649cc-bc05-452a-95b8-efaf6c891a4c-istio-data\") pod \"e2e-derive-0b9ff685-openshift-default-648b65bf89-k5qjt\" (UID: \"6ec649cc-bc05-452a-95b8-efaf6c891a4c\") " pod="openshift-ingress/e2e-derive-0b9ff685-openshift-default-648b65bf89-k5qjt"
Oct 31 19:35:18.457896 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:18.436817    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ec649cc-bc05-452a-95b8-efaf6c891a4c-istiod-ca-cert\") pod \"e2e-derive-0b9ff685-openshift-default-648b65bf89-k5qjt\" (UID: \"6ec649cc-bc05-452a-95b8-efaf6c891a4c\") " pod="openshift-ingress/e2e-derive-0b9ff685-openshift-default-648b65bf89-k5qjt"
Oct 31 19:35:18.457896 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:18.437079    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ec649cc-bc05-452a-95b8-efaf6c891a4c-workload-certs\") pod \"e2e-derive-0b9ff685-openshift-default-648b65bf89-k5qjt\" (UID: \"6ec649cc-bc05-452a-95b8-efaf6c891a4c\") " pod="openshift-ingress/e2e-derive-0b9ff685-openshift-default-648b65bf89-k5qjt"
Oct 31 19:35:18.461349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:18.459401    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ec649cc-bc05-452a-95b8-efaf6c891a4c-credential-socket\") pod \"e2e-derive-0b9ff685-openshift-default-648b65bf89-k5qjt\" (UID: \"6ec649cc-bc05-452a-95b8-efaf6c891a4c\") " pod="openshift-ingress/e2e-derive-0b9ff685-openshift-default-648b65bf89-k5qjt"
Oct 31 19:35:18.461349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:18.459676    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ec649cc-bc05-452a-95b8-efaf6c891a4c-workload-socket\") pod \"e2e-derive-0b9ff685-openshift-default-648b65bf89-k5qjt\" (UID: \"6ec649cc-bc05-452a-95b8-efaf6c891a4c\") " pod="openshift-ingress/e2e-derive-0b9ff685-openshift-default-648b65bf89-k5qjt"
Oct 31 19:35:18.564811 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:18.564194    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ec649cc-bc05-452a-95b8-efaf6c891a4c-istio-envoy\") pod \"e2e-derive-0b9ff685-openshift-default-648b65bf89-k5qjt\" (UID: \"6ec649cc-bc05-452a-95b8-efaf6c891a4c\") " pod="openshift-ingress/e2e-derive-0b9ff685-openshift-default-648b65bf89-k5qjt"
Oct 31 19:35:18.567112 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:18.567083    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:18.576120 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:18.576059    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ec649cc-bc05-452a-95b8-efaf6c891a4c-istio-token\") pod \"e2e-derive-0b9ff685-openshift-default-648b65bf89-k5qjt\" (UID: \"6ec649cc-bc05-452a-95b8-efaf6c891a4c\") " pod="openshift-ingress/e2e-derive-0b9ff685-openshift-default-648b65bf89-k5qjt"
Oct 31 19:35:18.589212 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:18.589073    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-929rg\" (UniqueName: \"kubernetes.io/projected/6ec649cc-bc05-452a-95b8-efaf6c891a4c-kube-api-access-929rg\") pod \"e2e-derive-0b9ff685-openshift-default-648b65bf89-k5qjt\" (UID: \"6ec649cc-bc05-452a-95b8-efaf6c891a4c\") " pod="openshift-ingress/e2e-derive-0b9ff685-openshift-default-648b65bf89-k5qjt"
Oct 31 19:35:18.607754 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:18.607289    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ec649cc-bc05-452a-95b8-efaf6c891a4c-istio-podinfo\") pod \"e2e-derive-0b9ff685-openshift-default-648b65bf89-k5qjt\" (UID: \"6ec649cc-bc05-452a-95b8-efaf6c891a4c\") " pod="openshift-ingress/e2e-derive-0b9ff685-openshift-default-648b65bf89-k5qjt"
Oct 31 19:35:18.646132 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:18.646098    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-85be723d-gw-openshift-default-76d9b68fb5-2lbp5"
Oct 31 19:35:18.766301 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:18.766267    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9744a5bd-39a7-4449-b5d4-2ff35e08084b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9744a5bd-39a7-4449-b5d4-2ff35e08084b-workload-certs\") pod \"9744a5bd-39a7-4449-b5d4-2ff35e08084b\" (UID: \"9744a5bd-39a7-4449-b5d4-2ff35e08084b\") "
Oct 31 19:35:18.766794 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:18.766397    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9744a5bd-39a7-4449-b5d4-2ff35e08084b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9744a5bd-39a7-4449-b5d4-2ff35e08084b-istio-podinfo\") pod \"9744a5bd-39a7-4449-b5d4-2ff35e08084b\" (UID: \"9744a5bd-39a7-4449-b5d4-2ff35e08084b\") "
Oct 31 19:35:18.766794 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:18.766448    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9744a5bd-39a7-4449-b5d4-2ff35e08084b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9744a5bd-39a7-4449-b5d4-2ff35e08084b-istio-envoy\") pod \"9744a5bd-39a7-4449-b5d4-2ff35e08084b\" (UID: \"9744a5bd-39a7-4449-b5d4-2ff35e08084b\") "
Oct 31 19:35:18.766794 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:18.766491    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9744a5bd-39a7-4449-b5d4-2ff35e08084b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9744a5bd-39a7-4449-b5d4-2ff35e08084b-credential-socket\") pod \"9744a5bd-39a7-4449-b5d4-2ff35e08084b\" (UID: \"9744a5bd-39a7-4449-b5d4-2ff35e08084b\") "
Oct 31 19:35:18.766794 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:18.766596    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9744a5bd-39a7-4449-b5d4-2ff35e08084b-workload-certs" pod "9744a5bd-39a7-4449-b5d4-2ff35e08084b" (UID: "9744a5bd-39a7-4449-b5d4-2ff35e08084b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:35:18.766794 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:18.766636    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9744a5bd-39a7-4449-b5d4-2ff35e08084b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9744a5bd-39a7-4449-b5d4-2ff35e08084b-workload-socket\") pod \"9744a5bd-39a7-4449-b5d4-2ff35e08084b\" (UID: \"9744a5bd-39a7-4449-b5d4-2ff35e08084b\") "
Oct 31 19:35:18.766794 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:18.766703    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9744a5bd-39a7-4449-b5d4-2ff35e08084b-istio-token\" (UniqueName: \"kubernetes.io/projected/9744a5bd-39a7-4449-b5d4-2ff35e08084b-istio-token\") pod \"9744a5bd-39a7-4449-b5d4-2ff35e08084b\" (UID: \"9744a5bd-39a7-4449-b5d4-2ff35e08084b\") "
Oct 31 19:35:18.767096 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:18.766875    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9744a5bd-39a7-4449-b5d4-2ff35e08084b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9744a5bd-39a7-4449-b5d4-2ff35e08084b-istiod-ca-cert\") pod \"9744a5bd-39a7-4449-b5d4-2ff35e08084b\" (UID: \"9744a5bd-39a7-4449-b5d4-2ff35e08084b\") "
Oct 31 19:35:18.767096 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:18.766854    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9744a5bd-39a7-4449-b5d4-2ff35e08084b-credential-socket" pod "9744a5bd-39a7-4449-b5d4-2ff35e08084b" (UID: "9744a5bd-39a7-4449-b5d4-2ff35e08084b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:35:18.767096 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:18.766904    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9744a5bd-39a7-4449-b5d4-2ff35e08084b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9744a5bd-39a7-4449-b5d4-2ff35e08084b-istio-data\") pod \"9744a5bd-39a7-4449-b5d4-2ff35e08084b\" (UID: \"9744a5bd-39a7-4449-b5d4-2ff35e08084b\") "
Oct 31 19:35:18.767096 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:18.766923    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9744a5bd-39a7-4449-b5d4-2ff35e08084b-workload-socket" pod "9744a5bd-39a7-4449-b5d4-2ff35e08084b" (UID: "9744a5bd-39a7-4449-b5d4-2ff35e08084b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:35:18.767096 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:18.766960    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9744a5bd-39a7-4449-b5d4-2ff35e08084b-kube-api-access-zw968\" (UniqueName: \"kubernetes.io/projected/9744a5bd-39a7-4449-b5d4-2ff35e08084b-kube-api-access-zw968\") pod \"9744a5bd-39a7-4449-b5d4-2ff35e08084b\" (UID: \"9744a5bd-39a7-4449-b5d4-2ff35e08084b\") "
Oct 31 19:35:18.767305 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:18.767128    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9744a5bd-39a7-4449-b5d4-2ff35e08084b-istio-data" pod "9744a5bd-39a7-4449-b5d4-2ff35e08084b" (UID: "9744a5bd-39a7-4449-b5d4-2ff35e08084b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:35:18.767305 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:18.767185    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9744a5bd-39a7-4449-b5d4-2ff35e08084b-istiod-ca-cert" pod "9744a5bd-39a7-4449-b5d4-2ff35e08084b" (UID: "9744a5bd-39a7-4449-b5d4-2ff35e08084b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:35:18.768088 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:18.768065    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9744a5bd-39a7-4449-b5d4-2ff35e08084b-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:35:18.768225 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:18.768090    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9744a5bd-39a7-4449-b5d4-2ff35e08084b-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:35:18.768225 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:18.768108    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9744a5bd-39a7-4449-b5d4-2ff35e08084b-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:35:18.768225 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:18.768124    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9744a5bd-39a7-4449-b5d4-2ff35e08084b-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:35:18.768225 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:18.768139    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9744a5bd-39a7-4449-b5d4-2ff35e08084b-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:35:18.769579 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:18.769550    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9744a5bd-39a7-4449-b5d4-2ff35e08084b-istio-token" pod "9744a5bd-39a7-4449-b5d4-2ff35e08084b" (UID: "9744a5bd-39a7-4449-b5d4-2ff35e08084b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:35:18.769684 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:18.769655    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9744a5bd-39a7-4449-b5d4-2ff35e08084b-kube-api-access-zw968" pod "9744a5bd-39a7-4449-b5d4-2ff35e08084b" (UID: "9744a5bd-39a7-4449-b5d4-2ff35e08084b"). InnerVolumeSpecName "kube-api-access-zw968". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:35:18.769864 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:18.769845    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9744a5bd-39a7-4449-b5d4-2ff35e08084b-istio-podinfo" pod "9744a5bd-39a7-4449-b5d4-2ff35e08084b" (UID: "9744a5bd-39a7-4449-b5d4-2ff35e08084b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:35:18.769921 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:18.769904    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9744a5bd-39a7-4449-b5d4-2ff35e08084b-istio-envoy" pod "9744a5bd-39a7-4449-b5d4-2ff35e08084b" (UID: "9744a5bd-39a7-4449-b5d4-2ff35e08084b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:35:18.848058 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:18.848003    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-0b9ff685-openshift-default-648b65bf89-k5qjt"
Oct 31 19:35:18.869955 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:18.869933    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9744a5bd-39a7-4449-b5d4-2ff35e08084b-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:35:18.870047 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:18.869959    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9744a5bd-39a7-4449-b5d4-2ff35e08084b-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:35:18.870047 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:18.869973    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9744a5bd-39a7-4449-b5d4-2ff35e08084b-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:35:18.870047 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:18.869988    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zw968\" (UniqueName: \"kubernetes.io/projected/9744a5bd-39a7-4449-b5d4-2ff35e08084b-kube-api-access-zw968\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:35:19.085147 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:19.085112    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:19.180519 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:19.180482    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-0b9ff685-openshift-default-648b65bf89-k5qjt"]
Oct 31 19:35:19.182293 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:35:19.182265    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ec649cc_bc05_452a_95b8_efaf6c891a4c.slice/crio-d8962c2a71c1f3eb5c5e32fd7ba0daa353e105e579e958519b2cb896d9f92a23 WatchSource:0}: Error finding container d8962c2a71c1f3eb5c5e32fd7ba0daa353e105e579e958519b2cb896d9f92a23: Status 404 returned error can't find the container with id d8962c2a71c1f3eb5c5e32fd7ba0daa353e105e579e958519b2cb896d9f92a23
Oct 31 19:35:19.185425 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:19.185391    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:35:19.185527 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:19.185472    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:35:19.185527 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:19.185516    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:35:19.231022 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:19.230999    4645 generic.go:356] "Generic (PLEG): container finished" podID="9744a5bd-39a7-4449-b5d4-2ff35e08084b" containerID="b71e4b898a7511285f44ac1515214284218a6f7cbbc547d8761d47387ee2a12e" exitCode=0
Oct 31 19:35:19.231131 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:19.231043    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-85be723d-gw-openshift-default-76d9b68fb5-2lbp5"
Oct 31 19:35:19.231131 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:19.231080    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-85be723d-gw-openshift-default-76d9b68fb5-2lbp5" event={"ID":"9744a5bd-39a7-4449-b5d4-2ff35e08084b","Type":"ContainerDied","Data":"b71e4b898a7511285f44ac1515214284218a6f7cbbc547d8761d47387ee2a12e"}
Oct 31 19:35:19.231241 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:19.231132    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-85be723d-gw-openshift-default-76d9b68fb5-2lbp5" event={"ID":"9744a5bd-39a7-4449-b5d4-2ff35e08084b","Type":"ContainerDied","Data":"dd490a089e1b3bff64d4d4a2f5d708465cc3506082beea02205052cc75cdadb9"}
Oct 31 19:35:19.231241 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:19.231156    4645 scope.go:122] "RemoveContainer" containerID="b71e4b898a7511285f44ac1515214284218a6f7cbbc547d8761d47387ee2a12e"
Oct 31 19:35:19.232481 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:19.232458    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0b9ff685-openshift-default-648b65bf89-k5qjt" event={"ID":"6ec649cc-bc05-452a-95b8-efaf6c891a4c","Type":"ContainerStarted","Data":"d8962c2a71c1f3eb5c5e32fd7ba0daa353e105e579e958519b2cb896d9f92a23"}
Oct 31 19:35:19.240952 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:19.240937    4645 scope.go:122] "RemoveContainer" containerID="b71e4b898a7511285f44ac1515214284218a6f7cbbc547d8761d47387ee2a12e"
Oct 31 19:35:19.241230 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:35:19.241201    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b71e4b898a7511285f44ac1515214284218a6f7cbbc547d8761d47387ee2a12e\": container with ID starting with b71e4b898a7511285f44ac1515214284218a6f7cbbc547d8761d47387ee2a12e not found: ID does not exist" containerID="b71e4b898a7511285f44ac1515214284218a6f7cbbc547d8761d47387ee2a12e"
Oct 31 19:35:19.241300 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:19.241240    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b71e4b898a7511285f44ac1515214284218a6f7cbbc547d8761d47387ee2a12e"} err="failed to get container status \"b71e4b898a7511285f44ac1515214284218a6f7cbbc547d8761d47387ee2a12e\": rpc error: code = NotFound desc = could not find container \"b71e4b898a7511285f44ac1515214284218a6f7cbbc547d8761d47387ee2a12e\": container with ID starting with b71e4b898a7511285f44ac1515214284218a6f7cbbc547d8761d47387ee2a12e not found: ID does not exist"
Oct 31 19:35:19.308123 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:19.308091    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-85be723d-gw-openshift-default-76d9b68fb5-2lbp5"]
Oct 31 19:35:19.317446 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:19.317417    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-85be723d-gw-openshift-default-76d9b68fb5-2lbp5"]
Oct 31 19:35:19.331319 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:19.331294    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:19.442715 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:19.442641    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:19.626626 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:19.626557    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9744a5bd-39a7-4449-b5d4-2ff35e08084b" path="/var/lib/kubelet/pods/9744a5bd-39a7-4449-b5d4-2ff35e08084b/volumes"
Oct 31 19:35:19.737486 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:19.737307    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:20.246164 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:20.246133    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0b9ff685-openshift-default-648b65bf89-k5qjt" event={"ID":"6ec649cc-bc05-452a-95b8-efaf6c891a4c","Type":"ContainerStarted","Data":"e52c2c269e5f6b3e4adfaab3907d07922c229c377ea17a1460ffbe26970a1b21"}
Oct 31 19:35:20.274257 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:20.272191    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-0b9ff685-openshift-default-648b65bf89-k5qjt" podStartSLOduration=2.272178589 podStartE2EDuration="2.272178589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:35:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:35:20.271414261 +0000 UTC m=+950.844648394" watchObservedRunningTime="2026-10-31 19:35:20.272178589 +0000 UTC m=+950.845412718"
Oct 31 19:35:20.775390 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:20.773473    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-1cc6560c-58795b5bc5-hq6bn" podUID="a6ac25a9-5887-4497-88d8-db5162badbd5" containerName="payload-pre-processing" containerID="cri-o://7d3cb91322f149e2a35267047c7107736e9d96e7644fe3599581e157394cf64c" gracePeriod=25
Oct 31 19:35:20.852669 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:20.852429    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-0b9ff685-openshift-default-648b65bf89-k5qjt"
Oct 31 19:35:20.892348 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:20.891584    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-0b9ff685-openshift-default-648b65bf89-k5qjt"
Oct 31 19:35:20.892348 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:20.892319    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:21.303815 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:21.299669    4645 generic.go:356] "Generic (PLEG): container finished" podID="a6ac25a9-5887-4497-88d8-db5162badbd5" containerID="7d3cb91322f149e2a35267047c7107736e9d96e7644fe3599581e157394cf64c" exitCode=0
Oct 31 19:35:21.303815 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:21.301054    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1cc6560c-58795b5bc5-hq6bn" event={"ID":"a6ac25a9-5887-4497-88d8-db5162badbd5","Type":"ContainerDied","Data":"7d3cb91322f149e2a35267047c7107736e9d96e7644fe3599581e157394cf64c"}
Oct 31 19:35:21.303815 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:21.301116    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-0b9ff685-openshift-default-648b65bf89-k5qjt"
Oct 31 19:35:21.303815 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:21.302198    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-0b9ff685-openshift-default-648b65bf89-k5qjt"
Oct 31 19:35:21.437371 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:21.435470    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1cc6560c-58795b5bc5-hq6bn"
Oct 31 19:35:21.581662 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:21.569261    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6ac25a9-5887-4497-88d8-db5162badbd5-kube-api-access-wns8h\" (UniqueName: \"kubernetes.io/projected/a6ac25a9-5887-4497-88d8-db5162badbd5-kube-api-access-wns8h\") pod \"a6ac25a9-5887-4497-88d8-db5162badbd5\" (UID: \"a6ac25a9-5887-4497-88d8-db5162badbd5\") "
Oct 31 19:35:21.581662 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:21.569357    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6ac25a9-5887-4497-88d8-db5162badbd5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6ac25a9-5887-4497-88d8-db5162badbd5-plugins-config-volume\") pod \"a6ac25a9-5887-4497-88d8-db5162badbd5\" (UID: \"a6ac25a9-5887-4497-88d8-db5162badbd5\") "
Oct 31 19:35:21.651350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:21.649974    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6ac25a9-5887-4497-88d8-db5162badbd5-kube-api-access-wns8h" pod "a6ac25a9-5887-4497-88d8-db5162badbd5" (UID: "a6ac25a9-5887-4497-88d8-db5162badbd5"). InnerVolumeSpecName "kube-api-access-wns8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:35:21.662179 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:21.653391    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6ac25a9-5887-4497-88d8-db5162badbd5-plugins-config-volume" pod "a6ac25a9-5887-4497-88d8-db5162badbd5" (UID: "a6ac25a9-5887-4497-88d8-db5162badbd5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:35:21.686352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:21.685908    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:21.692358 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:21.689224    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wns8h\" (UniqueName: \"kubernetes.io/projected/a6ac25a9-5887-4497-88d8-db5162badbd5-kube-api-access-wns8h\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:35:21.692358 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:21.689259    4645 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6ac25a9-5887-4497-88d8-db5162badbd5-plugins-config-volume\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:35:21.887862 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:21.887475    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:22.397354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:22.303271    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:22.606365 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:22.602938    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:22.771347 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:22.760220    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:22.796349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:22.786517    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1cc6560c-58795b5bc5-hq6bn"
Oct 31 19:35:22.796349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:22.789427    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1cc6560c-58795b5bc5-hq6bn" event={"ID":"a6ac25a9-5887-4497-88d8-db5162badbd5","Type":"ContainerDied","Data":"94ca5fbc80b0e413eb235f061173787180ae0157de380fd4c0522790ac616109"}
Oct 31 19:35:22.796349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:22.789491    4645 scope.go:122] "RemoveContainer" containerID="7d3cb91322f149e2a35267047c7107736e9d96e7644fe3599581e157394cf64c"
Oct 31 19:35:22.910359 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:22.907938    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1cc6560c-58795b5bc5-hq6bn"]
Oct 31 19:35:22.934106 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:22.934074    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1cc6560c-58795b5bc5-hq6bn"]
Oct 31 19:35:22.946517 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:22.946465    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw" podUID="d504b2d3-c26d-4dcf-a7eb-efe20732ae4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:35:23.137524 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:23.137420    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n" podUID="dbb97168-db93-4757-9a1b-f9c5eb704fc5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:35:23.250415 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:23.250303    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:23.401542 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:23.401443    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l" podUID="12bea86c-61e5-465e-82fb-b4673a4d1be9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:35:23.502376 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:23.502307    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:23.787354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:23.781587    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6ac25a9-5887-4497-88d8-db5162badbd5" path="/var/lib/kubelet/pods/a6ac25a9-5887-4497-88d8-db5162badbd5/volumes"
Oct 31 19:35:23.811354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:23.805493    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x" podUID="a774e286-dcb8-4a2f-8541-26bf231eb08d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 19:35:23.947299 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:23.947256    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw" podUID="d504b2d3-c26d-4dcf-a7eb-efe20732ae4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:35:23.948208 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:23.948171    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw"
Oct 31 19:35:24.205413 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:24.175200    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n" podUID="dbb97168-db93-4757-9a1b-f9c5eb704fc5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:35:24.205413 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:24.175753    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n"
Oct 31 19:35:24.466419 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:24.458275    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4" podUID="6032a997-b795-4561-805c-110cf67a9d5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:35:24.466419 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:24.458632    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l" podUID="12bea86c-61e5-465e-82fb-b4673a4d1be9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 31 19:35:24.466419 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:24.459182    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l"
Oct 31 19:35:24.524279 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:24.494898    4645 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4" podUID="6032a997-b795-4561-805c-110cf67a9d5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:35:24.858410 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:24.848777    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n"
Oct 31 19:35:24.858410 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:24.850037    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l"
Oct 31 19:35:24.903513 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:24.859500    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669" podUID="9285bee0-bf7c-4fd4-9da7-a1db021cae05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:35:24.903513 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:24.880366    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:24.903513 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:24.895609    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x" podUID="a774e286-dcb8-4a2f-8541-26bf231eb08d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 19:35:24.903513 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:24.895943    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x"
Oct 31 19:35:25.036782 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:24.946778    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw" podUID="1287db54-a587-4c11-9eb6-1606de555894" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:35:25.036782 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:24.972449    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw" podUID="d504b2d3-c26d-4dcf-a7eb-efe20732ae4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 31 19:35:25.323352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:25.316009    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:25.495355 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:25.488754    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4" podUID="6032a997-b795-4561-805c-110cf67a9d5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:35:25.495355 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:25.490121    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4"
Oct 31 19:35:25.754352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:25.747249    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x"
Oct 31 19:35:26.007040 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:26.006839    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw" podUID="d504b2d3-c26d-4dcf-a7eb-efe20732ae4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:35:26.626353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:26.605505    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4" podUID="6032a997-b795-4561-805c-110cf67a9d5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 31 19:35:26.650390 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:26.641467    4645 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x" podUID="a774e286-dcb8-4a2f-8541-26bf231eb08d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": context deadline exceeded"
Oct 31 19:35:27.175353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:26.719140    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669" podUID="9285bee0-bf7c-4fd4-9da7-a1db021cae05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 31 19:35:27.175353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:26.820582    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x" podUID="a774e286-dcb8-4a2f-8541-26bf231eb08d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 19:35:27.191499 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:26.962781    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:27.191824 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:26.991656    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l" podUID="12bea86c-61e5-465e-82fb-b4673a4d1be9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 31 19:35:27.193571 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:27.028811    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw" podUID="1287db54-a587-4c11-9eb6-1606de555894" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": context deadline exceeded"
Oct 31 19:35:27.194036 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:27.077752    4645 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n" podUID="dbb97168-db93-4757-9a1b-f9c5eb704fc5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:35:27.194164 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:27.097652    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n" podUID="dbb97168-db93-4757-9a1b-f9c5eb704fc5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:35:27.194806 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:27.138200    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw" podUID="d504b2d3-c26d-4dcf-a7eb-efe20732ae4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:35:27.332795 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:27.332759    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:27.607883 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:27.607278    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4" podUID="6032a997-b795-4561-805c-110cf67a9d5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:35:27.997015 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:27.996982    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4"
Oct 31 19:35:27.999393 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:27.999373    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw"
Oct 31 19:35:28.126355 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:28.125727    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:29.270170 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:29.270130    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:30.133936 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:30.133856    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:30.949602 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:30.949569    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:31.058719 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:31.058675    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:32.069973 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:32.069936    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:33.046382 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:33.046313    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:33.943788 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:33.943753    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:34.826934 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:34.826902    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:34.946976 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:34.946938    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:35.060282 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:35.060241    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:35.757723 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:35.757675    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:36.245832 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:35:36.245800    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 19:35:36.696933 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:36.696900    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:37.749304 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:37.749272    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:38.861991 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:38.861950    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:39.684338 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:39.684288    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:40.399908 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:40.399872    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:41.575776 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:41.575726    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:42.337204 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:42.337172    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:42.596381 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:42.596273    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:42.707155 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:42.707121    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:43.171033 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:43.170997    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:43.955876 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:43.955848    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:45.158937 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:45.158900    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:46.235083 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:46.235054    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:46.968348 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:46.968301    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:47.809703 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:47.809669    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:48.641299 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:48.641268    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:49.448060 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:49.448022    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:50.147087 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:50.147054    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:50.848288 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:50.848255    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:51.702753 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:51.702715    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:52.515201 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:52.515167    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:53.166169 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:53.166115    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:53.945810 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:53.945781    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:54.573965 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:54.573930    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:54.691661 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:54.691631    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:54.805167 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:54.802346    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:55.656983 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:55.656952    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:57.479353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:57.478980    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:57.590999 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:57.590967    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:57.650503 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:57.616223    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4" podUID="6032a997-b795-4561-805c-110cf67a9d5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 31 19:35:57.778429 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:57.717301    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669" podUID="9285bee0-bf7c-4fd4-9da7-a1db021cae05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:35:57.781532 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:57.776620    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x" podUID="a774e286-dcb8-4a2f-8541-26bf231eb08d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:35:58.002356 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:57.889891    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n" podUID="dbb97168-db93-4757-9a1b-f9c5eb704fc5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:35:58.025133 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:57.899455    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l" podUID="12bea86c-61e5-465e-82fb-b4673a4d1be9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 31 19:35:58.025929 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:57.933502    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw" podUID="1287db54-a587-4c11-9eb6-1606de555894" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": context deadline exceeded"
Oct 31 19:35:58.026354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:57.943872    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:58.213354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:58.207274    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw" podUID="d504b2d3-c26d-4dcf-a7eb-efe20732ae4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:35:58.284468 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:58.284429    4645 patch_prober.go:29] interesting pod/e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:35:58.292357 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:58.284717    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5" podUID="2bf496d2-b4cd-417b-b5dc-3afbac91f26b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:35:58.653816 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:58.651420    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4" podUID="6032a997-b795-4561-805c-110cf67a9d5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:35:58.653816 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:58.652023    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4"
Oct 31 19:35:58.788607 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:58.788562    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669" podUID="9285bee0-bf7c-4fd4-9da7-a1db021cae05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:35:58.807350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:58.802377    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669"
Oct 31 19:35:58.886196 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:58.885982    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x" podUID="a774e286-dcb8-4a2f-8541-26bf231eb08d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:35:58.927413 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:58.917059    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x"
Oct 31 19:35:59.108451 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:59.108407    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n" podUID="dbb97168-db93-4757-9a1b-f9c5eb704fc5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 19:35:59.139252 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:59.109002    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l" podUID="12bea86c-61e5-465e-82fb-b4673a4d1be9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 31 19:35:59.140370 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:59.140312    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw" podUID="1287db54-a587-4c11-9eb6-1606de555894" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": context deadline exceeded"
Oct 31 19:35:59.372406 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:59.185981    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n"
Oct 31 19:35:59.372406 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:59.362424    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l"
Oct 31 19:35:59.443288 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:59.426398    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw"
Oct 31 19:35:59.477360 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:59.253459    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:59.521353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:59.239925    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw" podUID="d504b2d3-c26d-4dcf-a7eb-efe20732ae4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:35:59.522928 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:59.522898    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw"
Oct 31 19:35:59.716350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:59.699955    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4" podUID="6032a997-b795-4561-805c-110cf67a9d5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:35:59.749472 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:59.734891    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:35:59.948371 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:59.926429    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669" podUID="9285bee0-bf7c-4fd4-9da7-a1db021cae05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 31 19:35:59.948371 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:59.934429    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x" podUID="a774e286-dcb8-4a2f-8541-26bf231eb08d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:35:59.948371 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:35:59.941904    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:00.231564 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:00.231500    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n" podUID="dbb97168-db93-4757-9a1b-f9c5eb704fc5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 19:36:00.331354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:00.307455    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l" podUID="12bea86c-61e5-465e-82fb-b4673a4d1be9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 31 19:36:00.412049 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:00.411500    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw" podUID="1287db54-a587-4c11-9eb6-1606de555894" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:36:00.534647 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:00.534539    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw" podUID="d504b2d3-c26d-4dcf-a7eb-efe20732ae4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:36:00.689193 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:00.689152    4645 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669" podUID="9285bee0-bf7c-4fd4-9da7-a1db021cae05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": context deadline exceeded"
Oct 31 19:36:00.717560 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:00.717512    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4" podUID="6032a997-b795-4561-805c-110cf67a9d5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:36:00.885881 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:00.885779    4645 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw" podUID="d504b2d3-c26d-4dcf-a7eb-efe20732ae4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:36:01.173410 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:01.024131    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669" podUID="9285bee0-bf7c-4fd4-9da7-a1db021cae05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:36:01.173410 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:01.024425    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:01.335411 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:01.319670    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l" podUID="12bea86c-61e5-465e-82fb-b4673a4d1be9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:36:01.336081 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:01.319628    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n" podUID="dbb97168-db93-4757-9a1b-f9c5eb704fc5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 19:36:01.499422 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:01.496166    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x" podUID="a774e286-dcb8-4a2f-8541-26bf231eb08d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 19:36:01.580352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:01.571604    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw" podUID="d504b2d3-c26d-4dcf-a7eb-efe20732ae4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:36:01.580352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:01.571744    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw" podUID="1287db54-a587-4c11-9eb6-1606de555894" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:36:01.716723 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:01.716497    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:01.731499 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:01.731137    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4" podUID="6032a997-b795-4561-805c-110cf67a9d5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:36:02.088514 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:02.088479    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-9th7f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:36:02.088887 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:02.088857    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9th7f" podUID="76bc1f3a-74b3-49b3-bde1-cc3b305260d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:36:02.089707 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:02.089666    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-6k6cd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:36:02.089932 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:02.089909    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6k6cd" podUID="a8fca631-527d-46c0-9197-5edd7762adb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:36:02.373010 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:02.372889    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669" podUID="9285bee0-bf7c-4fd4-9da7-a1db021cae05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 31 19:36:02.407469 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:02.407437    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:02.496495 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:02.496450    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x" podUID="a774e286-dcb8-4a2f-8541-26bf231eb08d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 19:36:02.560261 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:02.560195    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:02.560632 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:02.560599    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l" podUID="12bea86c-61e5-465e-82fb-b4673a4d1be9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:36:02.635464 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:02.635372    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-63eab93e-openshift-default-6d4c965c9c-lwwjn"]
Oct 31 19:36:02.635917 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:02.635882    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-63eab93e-openshift-default-6d4c965c9c-lwwjn" podUID="e71e3bcf-7168-42d4-a719-69d77cb2d362" containerName="istio-proxy" containerID="cri-o://1167df67295e61f33f0c97df63eb0765b354b2829c585eee0f0670814d5f8a0f" gracePeriod=30
Oct 31 19:36:02.683358 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:02.682587    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw" podUID="1287db54-a587-4c11-9eb6-1606de555894" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:36:02.683358 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:02.682786    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw" podUID="d504b2d3-c26d-4dcf-a7eb-efe20732ae4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 31 19:36:02.748353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:02.736005    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4" podUID="6032a997-b795-4561-805c-110cf67a9d5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:36:02.769352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:02.765929    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n" podUID="dbb97168-db93-4757-9a1b-f9c5eb704fc5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:36:02.971203 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:02.971167    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:03.404125 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:03.404031    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll"]
Oct 31 19:36:03.405316 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:03.405285    4645 cpu_manager.go:414] "containerMap: removing container" podUID="9744a5bd-39a7-4449-b5d4-2ff35e08084b" containerName="istio-proxy"
Oct 31 19:36:03.405489 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:03.405472    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9744a5bd-39a7-4449-b5d4-2ff35e08084b" containerName="istio-proxy"
Oct 31 19:36:03.405632 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:03.405621    4645 cpu_manager.go:414] "containerMap: removing container" podUID="a6ac25a9-5887-4497-88d8-db5162badbd5" containerName="payload-pre-processing"
Oct 31 19:36:03.413628 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:03.413595    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6ac25a9-5887-4497-88d8-db5162badbd5" containerName="payload-pre-processing"
Oct 31 19:36:03.414100 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:03.414079    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="9744a5bd-39a7-4449-b5d4-2ff35e08084b" containerName="istio-proxy"
Oct 31 19:36:03.414228 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:03.414217    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6ac25a9-5887-4497-88d8-db5162badbd5" containerName="payload-pre-processing"
Oct 31 19:36:03.427360 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:03.417688    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669" podUID="9285bee0-bf7c-4fd4-9da7-a1db021cae05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:36:03.430656 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:03.430630    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll"
Oct 31 19:36:03.441185 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:03.441157    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-cd9729-openshift-default-dockercfg-4d2qm\""
Oct 31 19:36:03.458171 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:03.458144    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll"]
Oct 31 19:36:03.497057 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:03.497006    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bffa80e8-d986-43ac-a6d5-8d828ca2a35f-workload-socket\") pod \"e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll\" (UID: \"bffa80e8-d986-43ac-a6d5-8d828ca2a35f\") " pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll"
Oct 31 19:36:03.497472 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:03.497446    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bffa80e8-d986-43ac-a6d5-8d828ca2a35f-credential-socket\") pod \"e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll\" (UID: \"bffa80e8-d986-43ac-a6d5-8d828ca2a35f\") " pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll"
Oct 31 19:36:03.497710 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:03.497688    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bffa80e8-d986-43ac-a6d5-8d828ca2a35f-istio-data\") pod \"e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll\" (UID: \"bffa80e8-d986-43ac-a6d5-8d828ca2a35f\") " pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll"
Oct 31 19:36:03.498061 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:03.498025    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bffa80e8-d986-43ac-a6d5-8d828ca2a35f-istio-envoy\") pod \"e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll\" (UID: \"bffa80e8-d986-43ac-a6d5-8d828ca2a35f\") " pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll"
Oct 31 19:36:03.498263 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:03.498248    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bffa80e8-d986-43ac-a6d5-8d828ca2a35f-workload-certs\") pod \"e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll\" (UID: \"bffa80e8-d986-43ac-a6d5-8d828ca2a35f\") " pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll"
Oct 31 19:36:03.498466 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:03.498429    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bffa80e8-d986-43ac-a6d5-8d828ca2a35f-istio-podinfo\") pod \"e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll\" (UID: \"bffa80e8-d986-43ac-a6d5-8d828ca2a35f\") " pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll"
Oct 31 19:36:03.498690 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:03.498673    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bffa80e8-d986-43ac-a6d5-8d828ca2a35f-istio-token\") pod \"e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll\" (UID: \"bffa80e8-d986-43ac-a6d5-8d828ca2a35f\") " pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll"
Oct 31 19:36:03.498921 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:03.498900    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnr7j\" (UniqueName: \"kubernetes.io/projected/bffa80e8-d986-43ac-a6d5-8d828ca2a35f-kube-api-access-rnr7j\") pod \"e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll\" (UID: \"bffa80e8-d986-43ac-a6d5-8d828ca2a35f\") " pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll"
Oct 31 19:36:03.499133 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:03.499115    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bffa80e8-d986-43ac-a6d5-8d828ca2a35f-istiod-ca-cert\") pod \"e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll\" (UID: \"bffa80e8-d986-43ac-a6d5-8d828ca2a35f\") " pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll"
Oct 31 19:36:03.519485 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:03.519438    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x" podUID="a774e286-dcb8-4a2f-8541-26bf231eb08d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:36:03.563983 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:03.562424    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l" podUID="12bea86c-61e5-465e-82fb-b4673a4d1be9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 31 19:36:03.628455 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:03.621813    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bffa80e8-d986-43ac-a6d5-8d828ca2a35f-istio-envoy\") pod \"e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll\" (UID: \"bffa80e8-d986-43ac-a6d5-8d828ca2a35f\") " pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll"
Oct 31 19:36:03.628455 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:03.621895    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bffa80e8-d986-43ac-a6d5-8d828ca2a35f-workload-certs\") pod \"e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll\" (UID: \"bffa80e8-d986-43ac-a6d5-8d828ca2a35f\") " pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll"
Oct 31 19:36:03.628455 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:03.621924    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bffa80e8-d986-43ac-a6d5-8d828ca2a35f-istio-podinfo\") pod \"e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll\" (UID: \"bffa80e8-d986-43ac-a6d5-8d828ca2a35f\") " pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll"
Oct 31 19:36:03.628455 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:03.621978    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bffa80e8-d986-43ac-a6d5-8d828ca2a35f-istio-token\") pod \"e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll\" (UID: \"bffa80e8-d986-43ac-a6d5-8d828ca2a35f\") " pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll"
Oct 31 19:36:03.628455 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:03.622043    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnr7j\" (UniqueName: \"kubernetes.io/projected/bffa80e8-d986-43ac-a6d5-8d828ca2a35f-kube-api-access-rnr7j\") pod \"e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll\" (UID: \"bffa80e8-d986-43ac-a6d5-8d828ca2a35f\") " pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll"
Oct 31 19:36:03.628455 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:03.622080    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bffa80e8-d986-43ac-a6d5-8d828ca2a35f-istiod-ca-cert\") pod \"e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll\" (UID: \"bffa80e8-d986-43ac-a6d5-8d828ca2a35f\") " pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll"
Oct 31 19:36:03.628455 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:03.622264    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bffa80e8-d986-43ac-a6d5-8d828ca2a35f-workload-socket\") pod \"e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll\" (UID: \"bffa80e8-d986-43ac-a6d5-8d828ca2a35f\") " pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll"
Oct 31 19:36:03.628455 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:03.622366    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bffa80e8-d986-43ac-a6d5-8d828ca2a35f-credential-socket\") pod \"e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll\" (UID: \"bffa80e8-d986-43ac-a6d5-8d828ca2a35f\") " pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll"
Oct 31 19:36:03.628455 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:03.622397    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bffa80e8-d986-43ac-a6d5-8d828ca2a35f-istio-data\") pod \"e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll\" (UID: \"bffa80e8-d986-43ac-a6d5-8d828ca2a35f\") " pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll"
Oct 31 19:36:03.628455 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:03.622932    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bffa80e8-d986-43ac-a6d5-8d828ca2a35f-istio-data\") pod \"e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll\" (UID: \"bffa80e8-d986-43ac-a6d5-8d828ca2a35f\") " pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll"
Oct 31 19:36:03.631245 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:03.631215    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bffa80e8-d986-43ac-a6d5-8d828ca2a35f-workload-certs\") pod \"e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll\" (UID: \"bffa80e8-d986-43ac-a6d5-8d828ca2a35f\") " pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll"
Oct 31 19:36:03.646794 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:03.633425    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bffa80e8-d986-43ac-a6d5-8d828ca2a35f-workload-socket\") pod \"e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll\" (UID: \"bffa80e8-d986-43ac-a6d5-8d828ca2a35f\") " pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll"
Oct 31 19:36:03.646996 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:03.645706    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bffa80e8-d986-43ac-a6d5-8d828ca2a35f-credential-socket\") pod \"e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll\" (UID: \"bffa80e8-d986-43ac-a6d5-8d828ca2a35f\") " pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll"
Oct 31 19:36:03.647129 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:03.646722    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bffa80e8-d986-43ac-a6d5-8d828ca2a35f-istiod-ca-cert\") pod \"e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll\" (UID: \"bffa80e8-d986-43ac-a6d5-8d828ca2a35f\") " pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll"
Oct 31 19:36:03.674402 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:03.647605    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bffa80e8-d986-43ac-a6d5-8d828ca2a35f-istio-podinfo\") pod \"e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll\" (UID: \"bffa80e8-d986-43ac-a6d5-8d828ca2a35f\") " pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll"
Oct 31 19:36:03.714191 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:03.688359    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw" podUID="1287db54-a587-4c11-9eb6-1606de555894" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:36:03.726349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:03.688725    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw" podUID="d504b2d3-c26d-4dcf-a7eb-efe20732ae4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 31 19:36:03.780373 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:03.713655    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bffa80e8-d986-43ac-a6d5-8d828ca2a35f-istio-envoy\") pod \"e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll\" (UID: \"bffa80e8-d986-43ac-a6d5-8d828ca2a35f\") " pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll"
Oct 31 19:36:03.780373 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:03.732103    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnr7j\" (UniqueName: \"kubernetes.io/projected/bffa80e8-d986-43ac-a6d5-8d828ca2a35f-kube-api-access-rnr7j\") pod \"e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll\" (UID: \"bffa80e8-d986-43ac-a6d5-8d828ca2a35f\") " pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll"
Oct 31 19:36:03.780373 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:03.771126    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bffa80e8-d986-43ac-a6d5-8d828ca2a35f-istio-token\") pod \"e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll\" (UID: \"bffa80e8-d986-43ac-a6d5-8d828ca2a35f\") " pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll"
Oct 31 19:36:03.780373 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:03.771208    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4" podUID="6032a997-b795-4561-805c-110cf67a9d5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 31 19:36:03.780373 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:03.771832    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:03.795996 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:03.795956    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n" podUID="dbb97168-db93-4757-9a1b-f9c5eb704fc5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:36:03.817499 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:03.804645    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll"
Oct 31 19:36:03.963837 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:03.963741    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:04.202766 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:04.202534    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669"
Oct 31 19:36:04.296821 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:04.294661    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4"
Oct 31 19:36:04.311213 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:04.307686    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw"
Oct 31 19:36:04.504550 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:04.495819    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw"
Oct 31 19:36:04.521483 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:04.521452    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:04.529729 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:04.529689    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x" podUID="a774e286-dcb8-4a2f-8541-26bf231eb08d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:36:04.538015 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:04.537993    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n"
Oct 31 19:36:04.579352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:04.571484    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l" podUID="12bea86c-61e5-465e-82fb-b4673a4d1be9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:36:04.614502 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:04.611987    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x"
Oct 31 19:36:04.631016 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:04.630965    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:36:04.631137 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:04.631070    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:36:04.631137 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:04.631114    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:36:04.637685 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:04.637664    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll"]
Oct 31 19:36:04.637889 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:04.637871    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l"
Oct 31 19:36:04.722404 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:04.719906    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:04.833551 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:04.833525    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:04.995928 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:04.995891    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:05.152421 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:05.151069    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:05.176168 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:05.176122    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-0b9ff685-openshift-default-648b65bf89-k5qjt"]
Oct 31 19:36:05.192208 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:05.176673    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-0b9ff685-openshift-default-648b65bf89-k5qjt" podUID="6ec649cc-bc05-452a-95b8-efaf6c891a4c" containerName="istio-proxy" containerID="cri-o://e52c2c269e5f6b3e4adfaab3907d07922c229c377ea17a1460ffbe26970a1b21" gracePeriod=30
Oct 31 19:36:05.518361 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:05.501949    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll" event={"ID":"bffa80e8-d986-43ac-a6d5-8d828ca2a35f","Type":"ContainerStarted","Data":"6c96f35c698866060dfbbb3dc18ffe4cc41c8ddd5e22cf2bc628827917956371"}
Oct 31 19:36:05.518361 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:05.502017    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll" event={"ID":"bffa80e8-d986-43ac-a6d5-8d828ca2a35f","Type":"ContainerStarted","Data":"62f5e7f9faaf6364a2cad595b5246dd5bd4af467ea2b063b745ed95cdd4e3ba1"}
Oct 31 19:36:05.813024 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:05.812938    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll"
Oct 31 19:36:05.813372 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:05.813347    4645 patch_prober.go:29] interesting pod/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 31 19:36:05.813562 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:05.813538    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll" podUID="bffa80e8-d986-43ac-a6d5-8d828ca2a35f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 31 19:36:05.852875 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:05.852847    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:06.312783 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:06.312695    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:06.322228 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:36:06.322197    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 19:36:06.388938 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:06.388607    4645 patch_prober.go:29] interesting pod/e2e-derive-0b9ff685-openshift-default-648b65bf89-k5qjt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Oct 31 19:36:06.388938 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:06.388667    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-0b9ff685-openshift-default-648b65bf89-k5qjt" podUID="6ec649cc-bc05-452a-95b8-efaf6c891a4c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 19:36:06.433255 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:06.433190    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:06.631061 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:06.630966    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:06.914427 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:06.911696    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll"
Oct 31 19:36:06.970791 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:06.970727    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll" podStartSLOduration=4.970707488 podStartE2EDuration="4.970707488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:36:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:36:05.573780212 +0000 UTC m=+996.147014395" watchObservedRunningTime="2026-10-31 19:36:06.970707488 +0000 UTC m=+997.543941618"
Oct 31 19:36:07.443295 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:07.443265    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:07.666364 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:07.666276    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll"
Oct 31 19:36:07.667584 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:07.667562    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll"
Oct 31 19:36:07.932990 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:07.931945    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:08.091693 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:08.091650    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:08.743387 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:08.743295    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:09.251288 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:09.251236    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:09.387823 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:09.387727    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:10.073856 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:10.073818    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:10.518645 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:10.518615    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:10.653067 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:10.653036    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:11.294740 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:11.294651    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:11.814783 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:11.814724    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:11.936389 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:11.936290    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:12.898351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:12.892789    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:13.761178 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:13.761146    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:14.546547 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:14.546379    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:14.742372 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:14.742306    4645 generic.go:356] "Generic (PLEG): container finished" podID="e71e3bcf-7168-42d4-a719-69d77cb2d362" containerID="1167df67295e61f33f0c97df63eb0765b354b2829c585eee0f0670814d5f8a0f" exitCode=0
Oct 31 19:36:14.742510 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:14.742458    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-63eab93e-openshift-default-6d4c965c9c-lwwjn" event={"ID":"e71e3bcf-7168-42d4-a719-69d77cb2d362","Type":"ContainerDied","Data":"1167df67295e61f33f0c97df63eb0765b354b2829c585eee0f0670814d5f8a0f"}
Oct 31 19:36:14.744032 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:14.744009    4645 generic.go:356] "Generic (PLEG): container finished" podID="6ec649cc-bc05-452a-95b8-efaf6c891a4c" containerID="e52c2c269e5f6b3e4adfaab3907d07922c229c377ea17a1460ffbe26970a1b21" exitCode=0
Oct 31 19:36:14.744168 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:14.744077    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0b9ff685-openshift-default-648b65bf89-k5qjt" event={"ID":"6ec649cc-bc05-452a-95b8-efaf6c891a4c","Type":"ContainerDied","Data":"e52c2c269e5f6b3e4adfaab3907d07922c229c377ea17a1460ffbe26970a1b21"}
Oct 31 19:36:14.744168 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:14.744115    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0b9ff685-openshift-default-648b65bf89-k5qjt" event={"ID":"6ec649cc-bc05-452a-95b8-efaf6c891a4c","Type":"ContainerDied","Data":"d8962c2a71c1f3eb5c5e32fd7ba0daa353e105e579e958519b2cb896d9f92a23"}
Oct 31 19:36:14.744168 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:14.744130    4645 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8962c2a71c1f3eb5c5e32fd7ba0daa353e105e579e958519b2cb896d9f92a23"
Oct 31 19:36:14.752883 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:14.752861    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-0b9ff685-openshift-default-648b65bf89-k5qjt"
Oct 31 19:36:14.774219 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:14.774201    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-63eab93e-openshift-default-6d4c965c9c-lwwjn"
Oct 31 19:36:14.850343 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:14.850301    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6ec649cc-bc05-452a-95b8-efaf6c891a4c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ec649cc-bc05-452a-95b8-efaf6c891a4c-istio-podinfo\") pod \"6ec649cc-bc05-452a-95b8-efaf6c891a4c\" (UID: \"6ec649cc-bc05-452a-95b8-efaf6c891a4c\") "
Oct 31 19:36:14.850510 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:14.850364    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e71e3bcf-7168-42d4-a719-69d77cb2d362-istio-token\" (UniqueName: \"kubernetes.io/projected/e71e3bcf-7168-42d4-a719-69d77cb2d362-istio-token\") pod \"e71e3bcf-7168-42d4-a719-69d77cb2d362\" (UID: \"e71e3bcf-7168-42d4-a719-69d77cb2d362\") "
Oct 31 19:36:14.850510 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:14.850427    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ec649cc-bc05-452a-95b8-efaf6c891a4c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ec649cc-bc05-452a-95b8-efaf6c891a4c-credential-socket\") pod \"6ec649cc-bc05-452a-95b8-efaf6c891a4c\" (UID: \"6ec649cc-bc05-452a-95b8-efaf6c891a4c\") "
Oct 31 19:36:14.850510 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:14.850457    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ec649cc-bc05-452a-95b8-efaf6c891a4c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ec649cc-bc05-452a-95b8-efaf6c891a4c-workload-certs\") pod \"6ec649cc-bc05-452a-95b8-efaf6c891a4c\" (UID: \"6ec649cc-bc05-452a-95b8-efaf6c891a4c\") "
Oct 31 19:36:14.850674 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:14.850532    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e71e3bcf-7168-42d4-a719-69d77cb2d362-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e71e3bcf-7168-42d4-a719-69d77cb2d362-istio-envoy\") pod \"e71e3bcf-7168-42d4-a719-69d77cb2d362\" (UID: \"e71e3bcf-7168-42d4-a719-69d77cb2d362\") "
Oct 31 19:36:14.850674 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:14.850571    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ec649cc-bc05-452a-95b8-efaf6c891a4c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ec649cc-bc05-452a-95b8-efaf6c891a4c-istio-envoy\") pod \"6ec649cc-bc05-452a-95b8-efaf6c891a4c\" (UID: \"6ec649cc-bc05-452a-95b8-efaf6c891a4c\") "
Oct 31 19:36:14.850674 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:14.850605    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ec649cc-bc05-452a-95b8-efaf6c891a4c-kube-api-access-929rg\" (UniqueName: \"kubernetes.io/projected/6ec649cc-bc05-452a-95b8-efaf6c891a4c-kube-api-access-929rg\") pod \"6ec649cc-bc05-452a-95b8-efaf6c891a4c\" (UID: \"6ec649cc-bc05-452a-95b8-efaf6c891a4c\") "
Oct 31 19:36:14.850832 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:14.850698    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e71e3bcf-7168-42d4-a719-69d77cb2d362-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71e3bcf-7168-42d4-a719-69d77cb2d362-workload-socket\") pod \"e71e3bcf-7168-42d4-a719-69d77cb2d362\" (UID: \"e71e3bcf-7168-42d4-a719-69d77cb2d362\") "
Oct 31 19:36:14.850832 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:14.850726    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e71e3bcf-7168-42d4-a719-69d77cb2d362-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e71e3bcf-7168-42d4-a719-69d77cb2d362-istio-data\") pod \"e71e3bcf-7168-42d4-a719-69d77cb2d362\" (UID: \"e71e3bcf-7168-42d4-a719-69d77cb2d362\") "
Oct 31 19:36:14.850832 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:14.850741    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ec649cc-bc05-452a-95b8-efaf6c891a4c-credential-socket" pod "6ec649cc-bc05-452a-95b8-efaf6c891a4c" (UID: "6ec649cc-bc05-452a-95b8-efaf6c891a4c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:36:14.850832 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:14.850758    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e71e3bcf-7168-42d4-a719-69d77cb2d362-kube-api-access-7t2fb\" (UniqueName: \"kubernetes.io/projected/e71e3bcf-7168-42d4-a719-69d77cb2d362-kube-api-access-7t2fb\") pod \"e71e3bcf-7168-42d4-a719-69d77cb2d362\" (UID: \"e71e3bcf-7168-42d4-a719-69d77cb2d362\") "
Oct 31 19:36:14.850832 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:14.850804    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ec649cc-bc05-452a-95b8-efaf6c891a4c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ec649cc-bc05-452a-95b8-efaf6c891a4c-istiod-ca-cert\") pod \"6ec649cc-bc05-452a-95b8-efaf6c891a4c\" (UID: \"6ec649cc-bc05-452a-95b8-efaf6c891a4c\") "
Oct 31 19:36:14.851106 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:14.850834    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ec649cc-bc05-452a-95b8-efaf6c891a4c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ec649cc-bc05-452a-95b8-efaf6c891a4c-workload-socket\") pod \"6ec649cc-bc05-452a-95b8-efaf6c891a4c\" (UID: \"6ec649cc-bc05-452a-95b8-efaf6c891a4c\") "
Oct 31 19:36:14.851106 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:14.850929    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ec649cc-bc05-452a-95b8-efaf6c891a4c-istio-token\" (UniqueName: \"kubernetes.io/projected/6ec649cc-bc05-452a-95b8-efaf6c891a4c-istio-token\") pod \"6ec649cc-bc05-452a-95b8-efaf6c891a4c\" (UID: \"6ec649cc-bc05-452a-95b8-efaf6c891a4c\") "
Oct 31 19:36:14.851106 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:14.850977    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ec649cc-bc05-452a-95b8-efaf6c891a4c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ec649cc-bc05-452a-95b8-efaf6c891a4c-istio-data\") pod \"6ec649cc-bc05-452a-95b8-efaf6c891a4c\" (UID: \"6ec649cc-bc05-452a-95b8-efaf6c891a4c\") "
Oct 31 19:36:14.851106 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:14.851007    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e71e3bcf-7168-42d4-a719-69d77cb2d362-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e71e3bcf-7168-42d4-a719-69d77cb2d362-istio-podinfo\") pod \"e71e3bcf-7168-42d4-a719-69d77cb2d362\" (UID: \"e71e3bcf-7168-42d4-a719-69d77cb2d362\") "
Oct 31 19:36:14.851106 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:14.851046    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e71e3bcf-7168-42d4-a719-69d77cb2d362-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71e3bcf-7168-42d4-a719-69d77cb2d362-credential-socket\") pod \"e71e3bcf-7168-42d4-a719-69d77cb2d362\" (UID: \"e71e3bcf-7168-42d4-a719-69d77cb2d362\") "
Oct 31 19:36:14.851106 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:14.851102    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e71e3bcf-7168-42d4-a719-69d77cb2d362-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e71e3bcf-7168-42d4-a719-69d77cb2d362-workload-certs\") pod \"e71e3bcf-7168-42d4-a719-69d77cb2d362\" (UID: \"e71e3bcf-7168-42d4-a719-69d77cb2d362\") "
Oct 31 19:36:14.851479 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:14.851195    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e71e3bcf-7168-42d4-a719-69d77cb2d362-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e71e3bcf-7168-42d4-a719-69d77cb2d362-istiod-ca-cert\") pod \"e71e3bcf-7168-42d4-a719-69d77cb2d362\" (UID: \"e71e3bcf-7168-42d4-a719-69d77cb2d362\") "
Oct 31 19:36:14.851479 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:14.851200    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ec649cc-bc05-452a-95b8-efaf6c891a4c-workload-certs" pod "6ec649cc-bc05-452a-95b8-efaf6c891a4c" (UID: "6ec649cc-bc05-452a-95b8-efaf6c891a4c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:36:14.851591 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:14.851482    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e71e3bcf-7168-42d4-a719-69d77cb2d362-workload-socket" pod "e71e3bcf-7168-42d4-a719-69d77cb2d362" (UID: "e71e3bcf-7168-42d4-a719-69d77cb2d362"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:36:14.851820 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:14.851793    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e71e3bcf-7168-42d4-a719-69d77cb2d362-istiod-ca-cert" pod "e71e3bcf-7168-42d4-a719-69d77cb2d362" (UID: "e71e3bcf-7168-42d4-a719-69d77cb2d362"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:36:14.852318 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:14.852064    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ec649cc-bc05-452a-95b8-efaf6c891a4c-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:36:14.852318 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:14.852105    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ec649cc-bc05-452a-95b8-efaf6c891a4c-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:36:14.852318 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:14.852119    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71e3bcf-7168-42d4-a719-69d77cb2d362-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:36:14.852318 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:14.852126    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ec649cc-bc05-452a-95b8-efaf6c891a4c-istiod-ca-cert" pod "6ec649cc-bc05-452a-95b8-efaf6c891a4c" (UID: "6ec649cc-bc05-452a-95b8-efaf6c891a4c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:36:14.852318 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:14.852134    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e71e3bcf-7168-42d4-a719-69d77cb2d362-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:36:14.852905 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:14.852715    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ec649cc-bc05-452a-95b8-efaf6c891a4c-istio-data" pod "6ec649cc-bc05-452a-95b8-efaf6c891a4c" (UID: "6ec649cc-bc05-452a-95b8-efaf6c891a4c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:36:14.853301 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:14.853037    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ec649cc-bc05-452a-95b8-efaf6c891a4c-workload-socket" pod "6ec649cc-bc05-452a-95b8-efaf6c891a4c" (UID: "6ec649cc-bc05-452a-95b8-efaf6c891a4c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:36:14.854019 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:14.853981    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ec649cc-bc05-452a-95b8-efaf6c891a4c-kube-api-access-929rg" pod "6ec649cc-bc05-452a-95b8-efaf6c891a4c" (UID: "6ec649cc-bc05-452a-95b8-efaf6c891a4c"). InnerVolumeSpecName "kube-api-access-929rg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:36:14.854019 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:14.854006    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e71e3bcf-7168-42d4-a719-69d77cb2d362-istio-token" pod "e71e3bcf-7168-42d4-a719-69d77cb2d362" (UID: "e71e3bcf-7168-42d4-a719-69d77cb2d362"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:36:14.854307 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:14.854283    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e71e3bcf-7168-42d4-a719-69d77cb2d362-credential-socket" pod "e71e3bcf-7168-42d4-a719-69d77cb2d362" (UID: "e71e3bcf-7168-42d4-a719-69d77cb2d362"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:36:14.854307 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:14.854297    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e71e3bcf-7168-42d4-a719-69d77cb2d362-istio-data" pod "e71e3bcf-7168-42d4-a719-69d77cb2d362" (UID: "e71e3bcf-7168-42d4-a719-69d77cb2d362"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:36:14.854510 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:14.854458    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e71e3bcf-7168-42d4-a719-69d77cb2d362-istio-podinfo" pod "e71e3bcf-7168-42d4-a719-69d77cb2d362" (UID: "e71e3bcf-7168-42d4-a719-69d77cb2d362"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:36:14.854580 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:14.854561    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ec649cc-bc05-452a-95b8-efaf6c891a4c-istio-podinfo" pod "6ec649cc-bc05-452a-95b8-efaf6c891a4c" (UID: "6ec649cc-bc05-452a-95b8-efaf6c891a4c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:36:14.854642 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:14.854570    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e71e3bcf-7168-42d4-a719-69d77cb2d362-kube-api-access-7t2fb" pod "e71e3bcf-7168-42d4-a719-69d77cb2d362" (UID: "e71e3bcf-7168-42d4-a719-69d77cb2d362"). InnerVolumeSpecName "kube-api-access-7t2fb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:36:14.854642 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:14.854630    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e71e3bcf-7168-42d4-a719-69d77cb2d362-workload-certs" pod "e71e3bcf-7168-42d4-a719-69d77cb2d362" (UID: "e71e3bcf-7168-42d4-a719-69d77cb2d362"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:36:14.854859 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:14.854836    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e71e3bcf-7168-42d4-a719-69d77cb2d362-istio-envoy" pod "e71e3bcf-7168-42d4-a719-69d77cb2d362" (UID: "e71e3bcf-7168-42d4-a719-69d77cb2d362"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:36:14.855153 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:14.855137    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ec649cc-bc05-452a-95b8-efaf6c891a4c-istio-envoy" pod "6ec649cc-bc05-452a-95b8-efaf6c891a4c" (UID: "6ec649cc-bc05-452a-95b8-efaf6c891a4c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:36:14.856133 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:14.856110    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ec649cc-bc05-452a-95b8-efaf6c891a4c-istio-token" pod "6ec649cc-bc05-452a-95b8-efaf6c891a4c" (UID: "6ec649cc-bc05-452a-95b8-efaf6c891a4c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:36:14.953752 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:14.953728    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ec649cc-bc05-452a-95b8-efaf6c891a4c-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:36:14.953837 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:14.953754    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e71e3bcf-7168-42d4-a719-69d77cb2d362-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:36:14.953837 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:14.953764    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e71e3bcf-7168-42d4-a719-69d77cb2d362-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:36:14.953837 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:14.953772    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ec649cc-bc05-452a-95b8-efaf6c891a4c-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:36:14.953837 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:14.953781    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-929rg\" (UniqueName: \"kubernetes.io/projected/6ec649cc-bc05-452a-95b8-efaf6c891a4c-kube-api-access-929rg\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:36:14.953837 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:14.953790    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e71e3bcf-7168-42d4-a719-69d77cb2d362-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:36:14.953837 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:14.953797    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7t2fb\" (UniqueName: \"kubernetes.io/projected/e71e3bcf-7168-42d4-a719-69d77cb2d362-kube-api-access-7t2fb\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:36:14.953837 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:14.953805    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ec649cc-bc05-452a-95b8-efaf6c891a4c-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:36:14.953837 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:14.953813    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ec649cc-bc05-452a-95b8-efaf6c891a4c-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:36:14.953837 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:14.953823    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ec649cc-bc05-452a-95b8-efaf6c891a4c-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:36:14.953837 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:14.953831    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ec649cc-bc05-452a-95b8-efaf6c891a4c-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:36:14.953837 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:14.953840    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e71e3bcf-7168-42d4-a719-69d77cb2d362-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:36:14.954203 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:14.953849    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e71e3bcf-7168-42d4-a719-69d77cb2d362-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:36:14.954203 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:14.953857    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e71e3bcf-7168-42d4-a719-69d77cb2d362-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:36:15.245048 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:15.245014    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:15.751211 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:15.751183    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-63eab93e-openshift-default-6d4c965c9c-lwwjn"
Oct 31 19:36:15.751637 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:15.751210    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-63eab93e-openshift-default-6d4c965c9c-lwwjn" event={"ID":"e71e3bcf-7168-42d4-a719-69d77cb2d362","Type":"ContainerDied","Data":"86c75b8821012291c3b4dfbab2de30050aa0994b87c6366dadfcf979aa9cba14"}
Oct 31 19:36:15.751637 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:15.751282    4645 scope.go:122] "RemoveContainer" containerID="1167df67295e61f33f0c97df63eb0765b354b2829c585eee0f0670814d5f8a0f"
Oct 31 19:36:15.751637 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:15.751519    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-0b9ff685-openshift-default-648b65bf89-k5qjt"
Oct 31 19:36:15.774406 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:15.774382    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-0b9ff685-openshift-default-648b65bf89-k5qjt"]
Oct 31 19:36:15.779069 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:15.779048    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-0b9ff685-openshift-default-648b65bf89-k5qjt"]
Oct 31 19:36:15.788396 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:15.788377    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-63eab93e-openshift-default-6d4c965c9c-lwwjn"]
Oct 31 19:36:15.792751 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:15.792730    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-63eab93e-openshift-default-6d4c965c9c-lwwjn"]
Oct 31 19:36:15.970244 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:15.970216    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:16.083703 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:16.082410    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:16.289757 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:16.289676    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:16.444498 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:16.444470    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:17.170343 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:17.170302    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:17.618608 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:17.618525    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ec649cc-bc05-452a-95b8-efaf6c891a4c" path="/var/lib/kubelet/pods/6ec649cc-bc05-452a-95b8-efaf6c891a4c/volumes"
Oct 31 19:36:17.619073 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:17.619055    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e71e3bcf-7168-42d4-a719-69d77cb2d362" path="/var/lib/kubelet/pods/e71e3bcf-7168-42d4-a719-69d77cb2d362/volumes"
Oct 31 19:36:18.079965 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:18.079927    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:18.883233 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:18.883202    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:18.997235 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:18.997202    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:19.865704 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:19.865673    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:20.667208 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:20.667176    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:21.549818 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:21.549792    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:22.370661 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:22.370630    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:23.335920 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:23.335868    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:24.081022 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:24.080990    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:24.860635 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:24.860599    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:25.914786 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:25.914749    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:26.677104 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:26.677073    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:27.448274 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:27.448236    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:28.025465 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:28.025410    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:28.169492 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:28.169456    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:28.503361 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:28.503302    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:28.961719 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:28.961684    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5"]
Oct 31 19:36:28.962373 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:28.962354    4645 cpu_manager.go:414] "containerMap: removing container" podUID="6ec649cc-bc05-452a-95b8-efaf6c891a4c" containerName="istio-proxy"
Oct 31 19:36:28.962373 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:28.962373    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ec649cc-bc05-452a-95b8-efaf6c891a4c" containerName="istio-proxy"
Oct 31 19:36:28.962530 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:28.962386    4645 cpu_manager.go:414] "containerMap: removing container" podUID="e71e3bcf-7168-42d4-a719-69d77cb2d362" containerName="istio-proxy"
Oct 31 19:36:28.962530 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:28.962391    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e71e3bcf-7168-42d4-a719-69d77cb2d362" containerName="istio-proxy"
Oct 31 19:36:28.962530 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:28.962509    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ec649cc-bc05-452a-95b8-efaf6c891a4c" containerName="istio-proxy"
Oct 31 19:36:28.962530 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:28.962518    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="e71e3bcf-7168-42d4-a719-69d77cb2d362" containerName="istio-proxy"
Oct 31 19:36:28.966381 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:28.966358    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5"
Oct 31 19:36:28.974141 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:28.974122    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-435728-openshift-default-dockercfg-sdb9g\""
Oct 31 19:36:28.990862 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:28.990827    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5"]
Oct 31 19:36:29.048530 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:29.048502    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:29.118396 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:29.109507    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/401797c4-b8ea-463e-acca-b35f26b5643f-workload-socket\") pod \"e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5\" (UID: \"401797c4-b8ea-463e-acca-b35f26b5643f\") " pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5"
Oct 31 19:36:29.170753 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:29.170609    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/401797c4-b8ea-463e-acca-b35f26b5643f-istio-envoy\") pod \"e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5\" (UID: \"401797c4-b8ea-463e-acca-b35f26b5643f\") " pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5"
Oct 31 19:36:29.171116 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:29.171090    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/401797c4-b8ea-463e-acca-b35f26b5643f-workload-certs\") pod \"e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5\" (UID: \"401797c4-b8ea-463e-acca-b35f26b5643f\") " pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5"
Oct 31 19:36:29.171850 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:29.171827    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/401797c4-b8ea-463e-acca-b35f26b5643f-istio-data\") pod \"e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5\" (UID: \"401797c4-b8ea-463e-acca-b35f26b5643f\") " pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5"
Oct 31 19:36:29.172204 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:29.172184    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/401797c4-b8ea-463e-acca-b35f26b5643f-credential-socket\") pod \"e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5\" (UID: \"401797c4-b8ea-463e-acca-b35f26b5643f\") " pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5"
Oct 31 19:36:29.172593 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:29.172575    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/401797c4-b8ea-463e-acca-b35f26b5643f-istio-token\") pod \"e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5\" (UID: \"401797c4-b8ea-463e-acca-b35f26b5643f\") " pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5"
Oct 31 19:36:29.172803 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:29.172786    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/401797c4-b8ea-463e-acca-b35f26b5643f-istio-podinfo\") pod \"e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5\" (UID: \"401797c4-b8ea-463e-acca-b35f26b5643f\") " pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5"
Oct 31 19:36:29.172985 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:29.172956    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lwfp\" (UniqueName: \"kubernetes.io/projected/401797c4-b8ea-463e-acca-b35f26b5643f-kube-api-access-5lwfp\") pod \"e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5\" (UID: \"401797c4-b8ea-463e-acca-b35f26b5643f\") " pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5"
Oct 31 19:36:29.173565 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:29.173543    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/401797c4-b8ea-463e-acca-b35f26b5643f-istiod-ca-cert\") pod \"e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5\" (UID: \"401797c4-b8ea-463e-acca-b35f26b5643f\") " pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5"
Oct 31 19:36:29.217349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:29.217206    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:29.285404 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:29.276360    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/401797c4-b8ea-463e-acca-b35f26b5643f-istiod-ca-cert\") pod \"e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5\" (UID: \"401797c4-b8ea-463e-acca-b35f26b5643f\") " pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5"
Oct 31 19:36:29.285404 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:29.276524    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/401797c4-b8ea-463e-acca-b35f26b5643f-workload-socket\") pod \"e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5\" (UID: \"401797c4-b8ea-463e-acca-b35f26b5643f\") " pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5"
Oct 31 19:36:29.285404 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:29.276592    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/401797c4-b8ea-463e-acca-b35f26b5643f-istio-envoy\") pod \"e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5\" (UID: \"401797c4-b8ea-463e-acca-b35f26b5643f\") " pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5"
Oct 31 19:36:29.285404 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:29.276669    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/401797c4-b8ea-463e-acca-b35f26b5643f-workload-certs\") pod \"e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5\" (UID: \"401797c4-b8ea-463e-acca-b35f26b5643f\") " pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5"
Oct 31 19:36:29.285404 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:29.276705    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/401797c4-b8ea-463e-acca-b35f26b5643f-istio-data\") pod \"e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5\" (UID: \"401797c4-b8ea-463e-acca-b35f26b5643f\") " pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5"
Oct 31 19:36:29.285404 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:29.276735    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/401797c4-b8ea-463e-acca-b35f26b5643f-credential-socket\") pod \"e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5\" (UID: \"401797c4-b8ea-463e-acca-b35f26b5643f\") " pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5"
Oct 31 19:36:29.285404 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:29.276836    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/401797c4-b8ea-463e-acca-b35f26b5643f-istio-token\") pod \"e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5\" (UID: \"401797c4-b8ea-463e-acca-b35f26b5643f\") " pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5"
Oct 31 19:36:29.285404 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:29.276876    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/401797c4-b8ea-463e-acca-b35f26b5643f-istio-podinfo\") pod \"e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5\" (UID: \"401797c4-b8ea-463e-acca-b35f26b5643f\") " pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5"
Oct 31 19:36:29.285404 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:29.276913    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lwfp\" (UniqueName: \"kubernetes.io/projected/401797c4-b8ea-463e-acca-b35f26b5643f-kube-api-access-5lwfp\") pod \"e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5\" (UID: \"401797c4-b8ea-463e-acca-b35f26b5643f\") " pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5"
Oct 31 19:36:29.285404 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:29.278272    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/401797c4-b8ea-463e-acca-b35f26b5643f-istiod-ca-cert\") pod \"e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5\" (UID: \"401797c4-b8ea-463e-acca-b35f26b5643f\") " pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5"
Oct 31 19:36:29.285404 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:29.278629    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/401797c4-b8ea-463e-acca-b35f26b5643f-workload-socket\") pod \"e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5\" (UID: \"401797c4-b8ea-463e-acca-b35f26b5643f\") " pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5"
Oct 31 19:36:29.295424 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:29.286756    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/401797c4-b8ea-463e-acca-b35f26b5643f-credential-socket\") pod \"e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5\" (UID: \"401797c4-b8ea-463e-acca-b35f26b5643f\") " pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5"
Oct 31 19:36:29.295688 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:29.287669    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/401797c4-b8ea-463e-acca-b35f26b5643f-workload-certs\") pod \"e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5\" (UID: \"401797c4-b8ea-463e-acca-b35f26b5643f\") " pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5"
Oct 31 19:36:29.295808 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:29.287888    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/401797c4-b8ea-463e-acca-b35f26b5643f-istio-data\") pod \"e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5\" (UID: \"401797c4-b8ea-463e-acca-b35f26b5643f\") " pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5"
Oct 31 19:36:29.312415 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:29.303941    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/401797c4-b8ea-463e-acca-b35f26b5643f-istio-podinfo\") pod \"e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5\" (UID: \"401797c4-b8ea-463e-acca-b35f26b5643f\") " pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5"
Oct 31 19:36:29.329804 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:29.329772    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lwfp\" (UniqueName: \"kubernetes.io/projected/401797c4-b8ea-463e-acca-b35f26b5643f-kube-api-access-5lwfp\") pod \"e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5\" (UID: \"401797c4-b8ea-463e-acca-b35f26b5643f\") " pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5"
Oct 31 19:36:29.332875 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:29.332821    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/401797c4-b8ea-463e-acca-b35f26b5643f-istio-token\") pod \"e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5\" (UID: \"401797c4-b8ea-463e-acca-b35f26b5643f\") " pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5"
Oct 31 19:36:29.338956 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:29.338894    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/401797c4-b8ea-463e-acca-b35f26b5643f-istio-envoy\") pod \"e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5\" (UID: \"401797c4-b8ea-463e-acca-b35f26b5643f\") " pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5"
Oct 31 19:36:29.586397 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:29.586303    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-435728-openshift-default-dockercfg-sdb9g\""
Oct 31 19:36:29.594477 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:29.594454    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5"
Oct 31 19:36:29.800111 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:29.800080    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:29.923295 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:29.923250    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:36:29.923525 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:29.923365    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:36:29.923525 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:29.923413    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:36:29.924608 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:29.924568    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5"]
Oct 31 19:36:30.338848 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:30.338808    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:30.762318 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:30.762287    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:30.872221 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:30.872192    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5" event={"ID":"401797c4-b8ea-463e-acca-b35f26b5643f","Type":"ContainerStarted","Data":"a9d11f90740c9cbd633cb93e292437b3c040d227e42a288daaa29fc5df8ff904"}
Oct 31 19:36:30.872501 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:30.872243    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5" event={"ID":"401797c4-b8ea-463e-acca-b35f26b5643f","Type":"ContainerStarted","Data":"1ab04ea69e2a70bc311473e1ab5a54bcdd1f0766c38573fd52e34cb9ec0e8983"}
Oct 31 19:36:30.908780 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:30.908715    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5" podStartSLOduration=2.908684086 podStartE2EDuration="2.908684086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:36:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:36:30.902271986 +0000 UTC m=+1021.475506115" watchObservedRunningTime="2026-10-31 19:36:30.908684086 +0000 UTC m=+1021.481918428"
Oct 31 19:36:30.924377 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:30.921606    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:31.202881 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:31.202847    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl"]
Oct 31 19:36:31.208342 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:31.208309    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl"
Oct 31 19:36:31.235597 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:31.235572    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-e1d060-openshift-default-dockercfg-w5qst\""
Oct 31 19:36:31.241141 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:31.241118    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl"]
Oct 31 19:36:31.307959 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:31.307934    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92238ea2-f330-4873-ab87-37b37bb05f9a-istio-token\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl\" (UID: \"92238ea2-f330-4873-ab87-37b37bb05f9a\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl"
Oct 31 19:36:31.308069 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:31.307972    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92238ea2-f330-4873-ab87-37b37bb05f9a-istio-envoy\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl\" (UID: \"92238ea2-f330-4873-ab87-37b37bb05f9a\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl"
Oct 31 19:36:31.308198 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:31.308181    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92238ea2-f330-4873-ab87-37b37bb05f9a-workload-socket\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl\" (UID: \"92238ea2-f330-4873-ab87-37b37bb05f9a\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl"
Oct 31 19:36:31.308242 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:31.308206    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92238ea2-f330-4873-ab87-37b37bb05f9a-istiod-ca-cert\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl\" (UID: \"92238ea2-f330-4873-ab87-37b37bb05f9a\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl"
Oct 31 19:36:31.308321 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:31.308307    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z4rl\" (UniqueName: \"kubernetes.io/projected/92238ea2-f330-4873-ab87-37b37bb05f9a-kube-api-access-9z4rl\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl\" (UID: \"92238ea2-f330-4873-ab87-37b37bb05f9a\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl"
Oct 31 19:36:31.308451 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:31.308437    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92238ea2-f330-4873-ab87-37b37bb05f9a-workload-certs\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl\" (UID: \"92238ea2-f330-4873-ab87-37b37bb05f9a\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl"
Oct 31 19:36:31.308496 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:31.308468    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92238ea2-f330-4873-ab87-37b37bb05f9a-credential-socket\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl\" (UID: \"92238ea2-f330-4873-ab87-37b37bb05f9a\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl"
Oct 31 19:36:31.308554 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:31.308541    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92238ea2-f330-4873-ab87-37b37bb05f9a-istio-data\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl\" (UID: \"92238ea2-f330-4873-ab87-37b37bb05f9a\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl"
Oct 31 19:36:31.308598 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:31.308564    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92238ea2-f330-4873-ab87-37b37bb05f9a-istio-podinfo\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl\" (UID: \"92238ea2-f330-4873-ab87-37b37bb05f9a\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl"
Oct 31 19:36:31.325129 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:31.325097    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:31.411599 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:31.411549    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92238ea2-f330-4873-ab87-37b37bb05f9a-workload-socket\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl\" (UID: \"92238ea2-f330-4873-ab87-37b37bb05f9a\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl"
Oct 31 19:36:31.434281 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:31.434247    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92238ea2-f330-4873-ab87-37b37bb05f9a-istiod-ca-cert\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl\" (UID: \"92238ea2-f330-4873-ab87-37b37bb05f9a\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl"
Oct 31 19:36:31.435537 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:31.435500    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z4rl\" (UniqueName: \"kubernetes.io/projected/92238ea2-f330-4873-ab87-37b37bb05f9a-kube-api-access-9z4rl\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl\" (UID: \"92238ea2-f330-4873-ab87-37b37bb05f9a\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl"
Oct 31 19:36:31.436207 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:31.436184    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92238ea2-f330-4873-ab87-37b37bb05f9a-workload-certs\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl\" (UID: \"92238ea2-f330-4873-ab87-37b37bb05f9a\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl"
Oct 31 19:36:31.436694 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:31.436677    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92238ea2-f330-4873-ab87-37b37bb05f9a-credential-socket\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl\" (UID: \"92238ea2-f330-4873-ab87-37b37bb05f9a\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl"
Oct 31 19:36:31.445476 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:31.445451    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92238ea2-f330-4873-ab87-37b37bb05f9a-istio-data\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl\" (UID: \"92238ea2-f330-4873-ab87-37b37bb05f9a\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl"
Oct 31 19:36:31.445916 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:31.445898    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92238ea2-f330-4873-ab87-37b37bb05f9a-istio-podinfo\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl\" (UID: \"92238ea2-f330-4873-ab87-37b37bb05f9a\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl"
Oct 31 19:36:31.446552 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:31.446533    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92238ea2-f330-4873-ab87-37b37bb05f9a-istio-token\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl\" (UID: \"92238ea2-f330-4873-ab87-37b37bb05f9a\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl"
Oct 31 19:36:31.461406 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:31.455417    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92238ea2-f330-4873-ab87-37b37bb05f9a-istio-envoy\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl\" (UID: \"92238ea2-f330-4873-ab87-37b37bb05f9a\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl"
Oct 31 19:36:31.461736 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:31.435309    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92238ea2-f330-4873-ab87-37b37bb05f9a-istiod-ca-cert\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl\" (UID: \"92238ea2-f330-4873-ab87-37b37bb05f9a\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl"
Oct 31 19:36:31.461866 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:31.445850    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92238ea2-f330-4873-ab87-37b37bb05f9a-istio-data\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl\" (UID: \"92238ea2-f330-4873-ab87-37b37bb05f9a\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl"
Oct 31 19:36:31.462057 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:31.436608    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92238ea2-f330-4873-ab87-37b37bb05f9a-workload-certs\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl\" (UID: \"92238ea2-f330-4873-ab87-37b37bb05f9a\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl"
Oct 31 19:36:31.462178 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:31.412207    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92238ea2-f330-4873-ab87-37b37bb05f9a-workload-socket\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl\" (UID: \"92238ea2-f330-4873-ab87-37b37bb05f9a\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl"
Oct 31 19:36:31.470073 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:31.463544    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92238ea2-f330-4873-ab87-37b37bb05f9a-istio-envoy\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl\" (UID: \"92238ea2-f330-4873-ab87-37b37bb05f9a\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl"
Oct 31 19:36:31.470239 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:31.437056    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92238ea2-f330-4873-ab87-37b37bb05f9a-credential-socket\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl\" (UID: \"92238ea2-f330-4873-ab87-37b37bb05f9a\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl"
Oct 31 19:36:31.479347 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:31.470757    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92238ea2-f330-4873-ab87-37b37bb05f9a-istio-podinfo\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl\" (UID: \"92238ea2-f330-4873-ab87-37b37bb05f9a\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl"
Oct 31 19:36:31.479347 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:31.478217    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92238ea2-f330-4873-ab87-37b37bb05f9a-istio-token\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl\" (UID: \"92238ea2-f330-4873-ab87-37b37bb05f9a\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl"
Oct 31 19:36:31.485347 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:31.484862    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z4rl\" (UniqueName: \"kubernetes.io/projected/92238ea2-f330-4873-ab87-37b37bb05f9a-kube-api-access-9z4rl\") pod \"e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl\" (UID: \"92238ea2-f330-4873-ab87-37b37bb05f9a\") " pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl"
Oct 31 19:36:31.550352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:31.550107    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl"
Oct 31 19:36:31.604367 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:31.597993    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5"
Oct 31 19:36:31.673863 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:31.672839    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5"
Oct 31 19:36:31.727365 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:31.727234    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:31.887878 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:31.887847    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:31.895253 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:31.895228    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5"
Oct 31 19:36:31.897905 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:31.897886    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5"
Oct 31 19:36:32.018428 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:31.996785    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl"]
Oct 31 19:36:32.056782 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:32.056742    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:36:32.056971 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:32.056821    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:36:32.056971 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:32.056852    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:36:32.519605 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:32.519571    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:32.903664 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:32.903573    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl" event={"ID":"92238ea2-f330-4873-ab87-37b37bb05f9a","Type":"ContainerStarted","Data":"809a98ee1208611517201f69b8c69c582c97c38049510312ccdfa1381982551d"}
Oct 31 19:36:32.903664 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:32.903630    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl" event={"ID":"92238ea2-f330-4873-ab87-37b37bb05f9a","Type":"ContainerStarted","Data":"4f70efcb4979e4116b22b6fbc9b45643cd61a910bc882faf71654d4fb0aa2ef4"}
Oct 31 19:36:33.389343 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:33.389299    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:33.528107 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:33.528074    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:33.551482 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:33.551459    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl"
Oct 31 19:36:33.556895 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:33.556874    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl"
Oct 31 19:36:33.586770 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:33.586708    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl" podStartSLOduration=2.586690671 podStartE2EDuration="2.586690671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:36:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:36:32.939574737 +0000 UTC m=+1023.512808869" watchObservedRunningTime="2026-10-31 19:36:33.586690671 +0000 UTC m=+1024.159924804"
Oct 31 19:36:33.857768 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:33.857738    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:33.931972 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:33.931788    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl"
Oct 31 19:36:33.933156 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:33.933126    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl"
Oct 31 19:36:34.604016 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:34.603971    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:34.972350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:34.970709    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:35.248085 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:35.248003    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:35.402369 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:35.402300    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:35.552947 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:35.552846    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:36.161707 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:36.161658    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:36.280085 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:36:36.280041    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 19:36:36.645634 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:36.645538    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:36.776249 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:36.776210    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:37.443107 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:37.443076    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:38.419233 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:38.419201    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:39.245514 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:39.245480    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:39.965812 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:39.965781    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:40.763817 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:40.763790    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:41.469493 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:41.469462    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:42.235549 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:42.235508    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:42.365590 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:42.365558    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:42.584193 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:42.584111    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:43.375088 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:43.375034    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:43.495382 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:43.495264    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:44.322202 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:44.322169    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:45.247154 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:45.247119    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:45.365564 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:45.365532    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:45.488695 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:45.488421    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:46.423469 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:46.423421    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:47.293585 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:47.293549    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:48.144215 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:48.144166    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:48.247016 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:48.246985    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:48.942744 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:48.942710    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:49.766897 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:49.766863    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:50.528350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:50.528299    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:50.646087 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:50.646054    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:51.366812 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:51.366780    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:52.163271 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:52.163240    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:52.967024 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:52.966989    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:54.139933 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:54.139902    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:55.065663 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:55.065630    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:55.419515 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:55.419487    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:55.770320 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:55.770238    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:56.983997 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:56.983956    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:58.425974 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:58.425933    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:58.603353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:58.602757    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:36:59.508353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:36:59.497466    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:00.332352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:00.331704    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l" podUID="12bea86c-61e5-465e-82fb-b4673a4d1be9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 31 19:37:00.554639 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:00.554595    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n" podUID="dbb97168-db93-4757-9a1b-f9c5eb704fc5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 19:37:00.556759 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:00.556734    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:00.557013 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:00.556995    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw" podUID="d504b2d3-c26d-4dcf-a7eb-efe20732ae4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 31 19:37:00.582637 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:00.582549    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw" podUID="1287db54-a587-4c11-9eb6-1606de555894" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:37:00.704388 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:00.703963    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:00.733667 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:00.729083    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4" podUID="6032a997-b795-4561-805c-110cf67a9d5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:37:00.771353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:00.754646    4645 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw" podUID="d504b2d3-c26d-4dcf-a7eb-efe20732ae4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:37:00.939351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:00.934643    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669" podUID="9285bee0-bf7c-4fd4-9da7-a1db021cae05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:37:01.342978 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:01.342881    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l" podUID="12bea86c-61e5-465e-82fb-b4673a4d1be9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:37:01.343834 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:01.343807    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l"
Oct 31 19:37:01.689414 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:01.658727    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n" podUID="dbb97168-db93-4757-9a1b-f9c5eb704fc5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 19:37:01.808613 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:01.710770    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw" podUID="1287db54-a587-4c11-9eb6-1606de555894" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": context deadline exceeded"
Oct 31 19:37:01.808613 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:01.713380    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n"
Oct 31 19:37:01.808613 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:01.713454    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw"
Oct 31 19:37:01.830470 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:01.830430    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4" podUID="6032a997-b795-4561-805c-110cf67a9d5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:37:02.159532 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:01.939661    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x" podUID="a774e286-dcb8-4a2f-8541-26bf231eb08d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:37:02.159532 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:01.941468    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-9th7f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:37:02.159532 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:02.133378    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9th7f" podUID="76bc1f3a-74b3-49b3-bde1-cc3b305260d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:37:02.159532 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:02.063040    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:02.183612 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:01.966589    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669" podUID="9285bee0-bf7c-4fd4-9da7-a1db021cae05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 31 19:37:02.183612 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:02.012427    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-6k6cd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:37:02.183612 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:02.175901    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6k6cd" podUID="a8fca631-527d-46c0-9197-5edd7762adb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:37:02.220555 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:01.642284    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw" podUID="d504b2d3-c26d-4dcf-a7eb-efe20732ae4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:37:02.222825 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:02.222786    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4"
Oct 31 19:37:02.223074 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:02.223057    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669"
Oct 31 19:37:02.223252 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:02.223230    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw"
Oct 31 19:37:02.240628 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:02.240603    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7026-6139d4"]
Oct 31 19:37:02.268420 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:02.268387    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7026-6139d4"
Oct 31 19:37:02.354837 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:02.343900    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l" podUID="12bea86c-61e5-465e-82fb-b4673a4d1be9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:37:02.354837 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:02.347662    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-4pmld\""
Oct 31 19:37:02.358878 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:02.358850    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-msrfk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:37:02.360120 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:02.359086    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-msrfk" podUID="48b0949c-0c89-4c80-a1dc-64d14839f620" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:37:02.376503 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:02.376461    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:02.436408 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:02.398807    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxgvn\" (UniqueName: \"kubernetes.io/projected/70d33bb1-d7b2-4713-8542-ff68ae9c7bd5-kube-api-access-pxgvn\") pod \"test-curl-7026-6139d4\" (UID: \"70d33bb1-d7b2-4713-8542-ff68ae9c7bd5\") " pod="opendatahub/test-curl-7026-6139d4"
Oct 31 19:37:02.502692 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:02.502658    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7026-6139d4"]
Oct 31 19:37:02.610362 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:02.604883    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:02.643621 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:02.640950    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxgvn\" (UniqueName: \"kubernetes.io/projected/70d33bb1-d7b2-4713-8542-ff68ae9c7bd5-kube-api-access-pxgvn\") pod \"test-curl-7026-6139d4\" (UID: \"70d33bb1-d7b2-4713-8542-ff68ae9c7bd5\") " pod="opendatahub/test-curl-7026-6139d4"
Oct 31 19:37:02.681264 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:02.681214    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n" podUID="dbb97168-db93-4757-9a1b-f9c5eb704fc5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 19:37:02.728460 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:02.723372    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw" podUID="1287db54-a587-4c11-9eb6-1606de555894" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": context deadline exceeded"
Oct 31 19:37:02.820839 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:02.820805    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxgvn\" (UniqueName: \"kubernetes.io/projected/70d33bb1-d7b2-4713-8542-ff68ae9c7bd5-kube-api-access-pxgvn\") pod \"test-curl-7026-6139d4\" (UID: \"70d33bb1-d7b2-4713-8542-ff68ae9c7bd5\") " pod="opendatahub/test-curl-7026-6139d4"
Oct 31 19:37:02.959083 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:02.959053    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw"
Oct 31 19:37:03.142044 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:03.141920    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x" podUID="a774e286-dcb8-4a2f-8541-26bf231eb08d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:37:03.143559 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:03.143533    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x"
Oct 31 19:37:03.188874 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:03.188823    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4" podUID="6032a997-b795-4561-805c-110cf67a9d5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:37:03.194740 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:03.194453    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7026-6139d4"
Oct 31 19:37:03.222009 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:03.196363    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669" podUID="9285bee0-bf7c-4fd4-9da7-a1db021cae05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 31 19:37:03.280420 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:03.267513    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:03.286152 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:03.284509    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw" podUID="d504b2d3-c26d-4dcf-a7eb-efe20732ae4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:37:03.377351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:03.345783    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l" podUID="12bea86c-61e5-465e-82fb-b4673a4d1be9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:37:03.496352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:03.479911    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:03.697467 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:03.697081    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n" podUID="dbb97168-db93-4757-9a1b-f9c5eb704fc5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 19:37:03.967413 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:03.960414    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw" podUID="1287db54-a587-4c11-9eb6-1606de555894" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:37:04.156534 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:04.156478    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x" podUID="a774e286-dcb8-4a2f-8541-26bf231eb08d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:37:04.190713 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:04.189662    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4" podUID="6032a997-b795-4561-805c-110cf67a9d5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:37:04.229532 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:04.229436    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669" podUID="9285bee0-bf7c-4fd4-9da7-a1db021cae05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:37:04.298214 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:04.298175    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw" podUID="d504b2d3-c26d-4dcf-a7eb-efe20732ae4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:37:04.374355 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:04.359451    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l" podUID="12bea86c-61e5-465e-82fb-b4673a4d1be9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 31 19:37:04.535467 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:04.535117    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:04.563149 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:04.563104    4645 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4" podUID="6032a997-b795-4561-805c-110cf67a9d5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": context deadline exceeded"
Oct 31 19:37:04.818427 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:04.790903    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n" podUID="dbb97168-db93-4757-9a1b-f9c5eb704fc5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 19:37:04.970242 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:04.960604    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw" podUID="1287db54-a587-4c11-9eb6-1606de555894" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:37:05.044570 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:05.042681    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw"
Oct 31 19:37:05.044570 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:05.042743    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n"
Oct 31 19:37:05.044570 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:05.042773    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669"
Oct 31 19:37:05.054312 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:05.052039    4645 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 19:37:05.060357 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:05.056498    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7026-6139d4"]
Oct 31 19:37:05.131129 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:05.126924    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw"
Oct 31 19:37:05.131535 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:05.131515    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l"
Oct 31 19:37:05.154262 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:05.154226    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7026-6139d4" event={"ID":"70d33bb1-d7b2-4713-8542-ff68ae9c7bd5","Type":"ContainerStarted","Data":"ee8385c773c365d435129f0b892063335475250853b6fbfa37f13c5cf5587ea1"}
Oct 31 19:37:05.165286 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:05.164414    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x" podUID="a774e286-dcb8-4a2f-8541-26bf231eb08d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 19:37:05.200127 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:05.199501    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4" podUID="6032a997-b795-4561-805c-110cf67a9d5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:37:05.218248 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:05.218124    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw"
Oct 31 19:37:05.237574 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:05.237369    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4"
Oct 31 19:37:05.239409 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:05.239389    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x"
Oct 31 19:37:05.709355 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:05.707023    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:06.254979 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:06.254934    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:06.317782 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:37:06.316030    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 19:37:06.415866 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:06.415834    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:06.769950 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:06.769861    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:07.542375 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:07.541728    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:07.631906 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:07.631878    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-67adab47-998479859-kmf5v"]
Oct 31 19:37:07.642770 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:07.642745    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-67adab47-998479859-kmf5v"]
Oct 31 19:37:07.644436 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:07.642936    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-67adab47-998479859-kmf5v"
Oct 31 19:37:07.649220 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:07.649183    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-67adab47-dockercfg-bwp87\""
Oct 31 19:37:07.654150 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:07.654085    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:07.663906 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:07.663889    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-67adab47\""
Oct 31 19:37:07.786350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:07.781191    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxhzt\" (UniqueName: \"kubernetes.io/projected/03810b83-fe09-4d95-a374-f6d7189954b4-kube-api-access-xxhzt\") pod \"payload-pre-processing-e2e-mt-67adab47-998479859-kmf5v\" (UID: \"03810b83-fe09-4d95-a374-f6d7189954b4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-67adab47-998479859-kmf5v"
Oct 31 19:37:07.786350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:07.781493    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03810b83-fe09-4d95-a374-f6d7189954b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-67adab47-998479859-kmf5v\" (UID: \"03810b83-fe09-4d95-a374-f6d7189954b4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-67adab47-998479859-kmf5v"
Oct 31 19:37:07.884872 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:07.882889    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxhzt\" (UniqueName: \"kubernetes.io/projected/03810b83-fe09-4d95-a374-f6d7189954b4-kube-api-access-xxhzt\") pod \"payload-pre-processing-e2e-mt-67adab47-998479859-kmf5v\" (UID: \"03810b83-fe09-4d95-a374-f6d7189954b4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-67adab47-998479859-kmf5v"
Oct 31 19:37:07.884872 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:07.883045    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03810b83-fe09-4d95-a374-f6d7189954b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-67adab47-998479859-kmf5v\" (UID: \"03810b83-fe09-4d95-a374-f6d7189954b4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-67adab47-998479859-kmf5v"
Oct 31 19:37:07.884872 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:07.884453    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03810b83-fe09-4d95-a374-f6d7189954b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-67adab47-998479859-kmf5v\" (UID: \"03810b83-fe09-4d95-a374-f6d7189954b4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-67adab47-998479859-kmf5v"
Oct 31 19:37:07.974039 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:07.973955    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxhzt\" (UniqueName: \"kubernetes.io/projected/03810b83-fe09-4d95-a374-f6d7189954b4-kube-api-access-xxhzt\") pod \"payload-pre-processing-e2e-mt-67adab47-998479859-kmf5v\" (UID: \"03810b83-fe09-4d95-a374-f6d7189954b4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-67adab47-998479859-kmf5v"
Oct 31 19:37:08.199314 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:08.199277    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:08.273044 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:08.272724    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-67adab47-998479859-kmf5v"
Oct 31 19:37:08.637405 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:08.637361    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-67adab47-998479859-kmf5v"]
Oct 31 19:37:08.638065 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:37:08.638036    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03810b83_fe09_4d95_a374_f6d7189954b4.slice/crio-0ce603d55dc6147dbc9a96591380fb09bb3f31d5f0b1751623acdc5bbe58e3f7 WatchSource:0}: Error finding container 0ce603d55dc6147dbc9a96591380fb09bb3f31d5f0b1751623acdc5bbe58e3f7: Status 404 returned error can't find the container with id 0ce603d55dc6147dbc9a96591380fb09bb3f31d5f0b1751623acdc5bbe58e3f7
Oct 31 19:37:08.709672 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:08.708484    4645 patch_prober.go:29] interesting pod/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:37:08.709672 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:08.708537    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll" podUID="bffa80e8-d986-43ac-a6d5-8d828ca2a35f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:37:09.158310 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:09.156986    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:09.245688 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:09.240076    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-67adab47-998479859-kmf5v" event={"ID":"03810b83-fe09-4d95-a374-f6d7189954b4","Type":"ContainerStarted","Data":"d91fbd860d0b28690210b1fdde22dc604a83c979372b15cc89537dc16416c398"}
Oct 31 19:37:09.245688 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:09.240135    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-67adab47-998479859-kmf5v" event={"ID":"03810b83-fe09-4d95-a374-f6d7189954b4","Type":"ContainerStarted","Data":"0ce603d55dc6147dbc9a96591380fb09bb3f31d5f0b1751623acdc5bbe58e3f7"}
Oct 31 19:37:09.245688 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:09.241180    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-67adab47-998479859-kmf5v"
Oct 31 19:37:09.310025 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:09.309672    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-67adab47-998479859-kmf5v" podStartSLOduration=2.309657976 podStartE2EDuration="2.309657976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:37:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:37:09.303138788 +0000 UTC m=+1059.876372916" watchObservedRunningTime="2026-10-31 19:37:09.309657976 +0000 UTC m=+1059.882892103"
Oct 31 19:37:10.163094 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:10.163062    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:10.254073 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:10.253270    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7026-6139d4" event={"ID":"70d33bb1-d7b2-4713-8542-ff68ae9c7bd5","Type":"ContainerStarted","Data":"3966b975c165e679289c53951fc4b73016fe33e1bb4cfb8d0e2d064650dc6675"}
Oct 31 19:37:10.931256 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:10.931225    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:11.057361 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:11.054592    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7026-6139d4" podStartSLOduration=5.87102363 podStartE2EDuration="10.054573942s" totalImagesPullingTime="4.183550312s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:37:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:37:10.279548791 +0000 UTC m=+1060.852782923" watchObservedRunningTime="2026-10-31 19:37:11.054573942 +0000 UTC m=+1061.627808069"
Oct 31 19:37:11.059892 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:11.059562    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7026-6139d4"]
Oct 31 19:37:11.075409 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:11.073290    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7026-6139d4"]
Oct 31 19:37:11.174253 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:11.174223    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7026-b01cc5"]
Oct 31 19:37:11.174888 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:11.174874    4645 cpu_manager.go:414] "containerMap: removing container" podUID="70d33bb1-d7b2-4713-8542-ff68ae9c7bd5" containerName="test-curl-7026-6139d4"
Oct 31 19:37:11.174937 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:11.174889    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70d33bb1-d7b2-4713-8542-ff68ae9c7bd5" containerName="test-curl-7026-6139d4"
Oct 31 19:37:11.174981 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:11.174974    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="70d33bb1-d7b2-4713-8542-ff68ae9c7bd5" containerName="test-curl-7026-6139d4"
Oct 31 19:37:11.569503 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:11.569476    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7026-b01cc5"]
Oct 31 19:37:11.569663 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:11.569571    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7026-b01cc5"
Oct 31 19:37:11.670827 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:11.670794    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2kbz\" (UniqueName: \"kubernetes.io/projected/59ae2f4a-60ab-492e-a0ff-439cd98a81a7-kube-api-access-l2kbz\") pod \"test-curl-7026-b01cc5\" (UID: \"59ae2f4a-60ab-492e-a0ff-439cd98a81a7\") " pod="opendatahub/test-curl-7026-b01cc5"
Oct 31 19:37:11.772178 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:11.772143    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2kbz\" (UniqueName: \"kubernetes.io/projected/59ae2f4a-60ab-492e-a0ff-439cd98a81a7-kube-api-access-l2kbz\") pod \"test-curl-7026-b01cc5\" (UID: \"59ae2f4a-60ab-492e-a0ff-439cd98a81a7\") " pod="opendatahub/test-curl-7026-b01cc5"
Oct 31 19:37:11.782550 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:11.782523    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:11.784619 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:11.784600    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2kbz\" (UniqueName: \"kubernetes.io/projected/59ae2f4a-60ab-492e-a0ff-439cd98a81a7-kube-api-access-l2kbz\") pod \"test-curl-7026-b01cc5\" (UID: \"59ae2f4a-60ab-492e-a0ff-439cd98a81a7\") " pod="opendatahub/test-curl-7026-b01cc5"
Oct 31 19:37:11.867900 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:11.865427    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7026-b01cc5"
Oct 31 19:37:12.199393 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:12.199360    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7026-b01cc5"]
Oct 31 19:37:12.280062 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:12.280030    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7026-b01cc5" event={"ID":"59ae2f4a-60ab-492e-a0ff-439cd98a81a7","Type":"ContainerStarted","Data":"67bcb205393db48987aa62ed5beda291cf8087a0aa879f28f4384ea5ffd001d0"}
Oct 31 19:37:12.280294 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:12.280251    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7026-6139d4" podUID="70d33bb1-d7b2-4713-8542-ff68ae9c7bd5" containerName="test-curl-7026-6139d4" containerID="cri-o://3966b975c165e679289c53951fc4b73016fe33e1bb4cfb8d0e2d064650dc6675" gracePeriod=2
Oct 31 19:37:12.502582 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:12.502494    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:13.199923 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:13.199887    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:13.290123 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:13.290029    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7026-b01cc5" event={"ID":"59ae2f4a-60ab-492e-a0ff-439cd98a81a7","Type":"ContainerStarted","Data":"43fd5c9b24f1b2e5576ff0d0ba0baa54d7db2e35fc4c9fa932b0b241b898d2aa"}
Oct 31 19:37:13.305446 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:13.305343    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7026-b01cc5" podStartSLOduration=2.305306323 podStartE2EDuration="2.305306323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:37:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:37:13.304533087 +0000 UTC m=+1063.877767217" watchObservedRunningTime="2026-10-31 19:37:13.305306323 +0000 UTC m=+1063.878540465"
Oct 31 19:37:13.961917 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:13.961888    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:14.071965 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:14.071373    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7026-b01cc5"]
Oct 31 19:37:14.073828 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:14.072502    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7026-b01cc5"]
Oct 31 19:37:14.231984 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:14.225350    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7026-c918fc"]
Oct 31 19:37:14.231984 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:14.225943    4645 cpu_manager.go:414] "containerMap: removing container" podUID="59ae2f4a-60ab-492e-a0ff-439cd98a81a7" containerName="test-curl-7026-b01cc5"
Oct 31 19:37:14.231984 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:14.225955    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59ae2f4a-60ab-492e-a0ff-439cd98a81a7" containerName="test-curl-7026-b01cc5"
Oct 31 19:37:14.231984 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:14.226059    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="59ae2f4a-60ab-492e-a0ff-439cd98a81a7" containerName="test-curl-7026-b01cc5"
Oct 31 19:37:14.326661 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:14.236074    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7026-c918fc"
Oct 31 19:37:14.326661 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:14.236927    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7026-c918fc"]
Oct 31 19:37:14.326661 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:14.318601    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7qvj\" (UniqueName: \"kubernetes.io/projected/6007cecc-95ef-4f1b-b0fd-dd5e89494e67-kube-api-access-v7qvj\") pod \"test-curl-7026-c918fc\" (UID: \"6007cecc-95ef-4f1b-b0fd-dd5e89494e67\") " pod="opendatahub/test-curl-7026-c918fc"
Oct 31 19:37:14.420505 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:14.420478    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7qvj\" (UniqueName: \"kubernetes.io/projected/6007cecc-95ef-4f1b-b0fd-dd5e89494e67-kube-api-access-v7qvj\") pod \"test-curl-7026-c918fc\" (UID: \"6007cecc-95ef-4f1b-b0fd-dd5e89494e67\") " pod="opendatahub/test-curl-7026-c918fc"
Oct 31 19:37:14.433125 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:14.433101    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7qvj\" (UniqueName: \"kubernetes.io/projected/6007cecc-95ef-4f1b-b0fd-dd5e89494e67-kube-api-access-v7qvj\") pod \"test-curl-7026-c918fc\" (UID: \"6007cecc-95ef-4f1b-b0fd-dd5e89494e67\") " pod="opendatahub/test-curl-7026-c918fc"
Oct 31 19:37:14.461056 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:14.461034    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7026-6139d4"
Oct 31 19:37:14.522173 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:14.522106    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70d33bb1-d7b2-4713-8542-ff68ae9c7bd5-kube-api-access-pxgvn\" (UniqueName: \"kubernetes.io/projected/70d33bb1-d7b2-4713-8542-ff68ae9c7bd5-kube-api-access-pxgvn\") pod \"70d33bb1-d7b2-4713-8542-ff68ae9c7bd5\" (UID: \"70d33bb1-d7b2-4713-8542-ff68ae9c7bd5\") "
Oct 31 19:37:14.524051 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:14.524022    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70d33bb1-d7b2-4713-8542-ff68ae9c7bd5-kube-api-access-pxgvn" pod "70d33bb1-d7b2-4713-8542-ff68ae9c7bd5" (UID: "70d33bb1-d7b2-4713-8542-ff68ae9c7bd5"). InnerVolumeSpecName "kube-api-access-pxgvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:37:14.723502 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:14.556984    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7026-c918fc"
Oct 31 19:37:14.723502 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:14.624132    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxgvn\" (UniqueName: \"kubernetes.io/projected/70d33bb1-d7b2-4713-8542-ff68ae9c7bd5-kube-api-access-pxgvn\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:37:14.740303 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:14.740273    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:14.875937 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:14.856427    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7026-c918fc"]
Oct 31 19:37:15.307273 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:15.307247    4645 generic.go:356] "Generic (PLEG): container finished" podID="401797c4-b8ea-463e-acca-b35f26b5643f" containerID="a9d11f90740c9cbd633cb93e292437b3c040d227e42a288daaa29fc5df8ff904" exitCode=0
Oct 31 19:37:15.307663 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:15.307339    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5" event={"ID":"401797c4-b8ea-463e-acca-b35f26b5643f","Type":"ContainerDied","Data":"a9d11f90740c9cbd633cb93e292437b3c040d227e42a288daaa29fc5df8ff904"}
Oct 31 19:37:15.307733 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:15.307715    4645 scope.go:122] "RemoveContainer" containerID="a9d11f90740c9cbd633cb93e292437b3c040d227e42a288daaa29fc5df8ff904"
Oct 31 19:37:15.309168 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:15.309145    4645 generic.go:356] "Generic (PLEG): container finished" podID="bffa80e8-d986-43ac-a6d5-8d828ca2a35f" containerID="6c96f35c698866060dfbbb3dc18ffe4cc41c8ddd5e22cf2bc628827917956371" exitCode=0
Oct 31 19:37:15.309263 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:15.309207    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll" event={"ID":"bffa80e8-d986-43ac-a6d5-8d828ca2a35f","Type":"ContainerDied","Data":"6c96f35c698866060dfbbb3dc18ffe4cc41c8ddd5e22cf2bc628827917956371"}
Oct 31 19:37:15.309263 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:15.309211    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:37:15.309387 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:15.309284    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:37:15.309387 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:15.309318    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:37:15.309615 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:15.309599    4645 scope.go:122] "RemoveContainer" containerID="6c96f35c698866060dfbbb3dc18ffe4cc41c8ddd5e22cf2bc628827917956371"
Oct 31 19:37:15.310637 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:15.310616    4645 generic.go:356] "Generic (PLEG): container finished" podID="70d33bb1-d7b2-4713-8542-ff68ae9c7bd5" containerID="3966b975c165e679289c53951fc4b73016fe33e1bb4cfb8d0e2d064650dc6675" exitCode=137
Oct 31 19:37:15.310738 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:15.310670    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7026-6139d4"
Oct 31 19:37:15.310738 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:15.310677    4645 scope.go:122] "RemoveContainer" containerID="3966b975c165e679289c53951fc4b73016fe33e1bb4cfb8d0e2d064650dc6675"
Oct 31 19:37:15.311040 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:15.311012    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:37:15.311135 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:15.311092    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:37:15.311193 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:15.311144    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:37:15.313273 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:15.313250    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7026-c918fc" event={"ID":"6007cecc-95ef-4f1b-b0fd-dd5e89494e67","Type":"ContainerStarted","Data":"f2889907d49cd563108059f04bb1787ac75b1bdd07cb29e8a558757d8e2ba52f"}
Oct 31 19:37:15.313372 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:15.313287    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7026-c918fc" event={"ID":"6007cecc-95ef-4f1b-b0fd-dd5e89494e67","Type":"ContainerStarted","Data":"2f8e3e31a034300128e6896bf0fc3f546047daca7b84f4f69bb5eef9a3d479ee"}
Oct 31 19:37:15.313372 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:15.313290    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7026-b01cc5" podUID="59ae2f4a-60ab-492e-a0ff-439cd98a81a7" containerName="test-curl-7026-b01cc5" containerID="cri-o://43fd5c9b24f1b2e5576ff0d0ba0baa54d7db2e35fc4c9fa932b0b241b898d2aa" gracePeriod=2
Oct 31 19:37:15.324141 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:15.324125    4645 scope.go:122] "RemoveContainer" containerID="3966b975c165e679289c53951fc4b73016fe33e1bb4cfb8d0e2d064650dc6675"
Oct 31 19:37:15.324761 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:37:15.324736    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3966b975c165e679289c53951fc4b73016fe33e1bb4cfb8d0e2d064650dc6675\": container with ID starting with 3966b975c165e679289c53951fc4b73016fe33e1bb4cfb8d0e2d064650dc6675 not found: ID does not exist" containerID="3966b975c165e679289c53951fc4b73016fe33e1bb4cfb8d0e2d064650dc6675"
Oct 31 19:37:15.324844 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:15.324773    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3966b975c165e679289c53951fc4b73016fe33e1bb4cfb8d0e2d064650dc6675"} err="failed to get container status \"3966b975c165e679289c53951fc4b73016fe33e1bb4cfb8d0e2d064650dc6675\": rpc error: code = NotFound desc = could not find container \"3966b975c165e679289c53951fc4b73016fe33e1bb4cfb8d0e2d064650dc6675\": container with ID starting with 3966b975c165e679289c53951fc4b73016fe33e1bb4cfb8d0e2d064650dc6675 not found: ID does not exist"
Oct 31 19:37:15.386455 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:37:15.386263    4645 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7026-b01cc5\" is forbidden: User \"system:node:ip-10-0-131-94.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-94.us-east-2.compute.internal' and this object" podUID="59ae2f4a-60ab-492e-a0ff-439cd98a81a7" pod="opendatahub/test-curl-7026-b01cc5"
Oct 31 19:37:15.387962 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:37:15.387933    4645 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7026-6139d4\" is forbidden: User \"system:node:ip-10-0-131-94.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-94.us-east-2.compute.internal' and this object" podUID="70d33bb1-d7b2-4713-8542-ff68ae9c7bd5" pod="opendatahub/test-curl-7026-6139d4"
Oct 31 19:37:15.389234 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:15.389180    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7026-c918fc" podStartSLOduration=1.3891628630000001 podStartE2EDuration="1.389162863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:37:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:37:15.38387818 +0000 UTC m=+1065.957112308" watchObservedRunningTime="2026-10-31 19:37:15.389162863 +0000 UTC m=+1065.962396992"
Oct 31 19:37:15.490349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:15.489984    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:15.622400 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:15.621759    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70d33bb1-d7b2-4713-8542-ff68ae9c7bd5" path="/var/lib/kubelet/pods/70d33bb1-d7b2-4713-8542-ff68ae9c7bd5/volumes"
Oct 31 19:37:16.009939 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:16.009840    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:16.174733 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:16.166393    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7026-c918fc"]
Oct 31 19:37:16.214357 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:16.205736    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7026-c918fc"]
Oct 31 19:37:16.342288 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:16.339306    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-67adab47-6cd87d9bdb-kwrg8"]
Oct 31 19:37:16.342288 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:16.340203    4645 cpu_manager.go:414] "containerMap: removing container" podUID="6007cecc-95ef-4f1b-b0fd-dd5e89494e67" containerName="test-curl-7026-c918fc"
Oct 31 19:37:16.342288 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:16.340224    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6007cecc-95ef-4f1b-b0fd-dd5e89494e67" containerName="test-curl-7026-c918fc"
Oct 31 19:37:16.342288 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:16.340465    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="6007cecc-95ef-4f1b-b0fd-dd5e89494e67" containerName="test-curl-7026-c918fc"
Oct 31 19:37:16.342756 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:16.342673    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:16.348613 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:16.348589    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-67adab47-6cd87d9bdb-kwrg8"]
Oct 31 19:37:16.348852 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:16.348837    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7026-c23a67"]
Oct 31 19:37:16.349320 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:16.349300    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-67adab47-6cd87d9bdb-kwrg8"
Oct 31 19:37:16.353245 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:16.353222    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-67adab47-serving-cert\""
Oct 31 19:37:16.353815 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:16.353698    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5" event={"ID":"401797c4-b8ea-463e-acca-b35f26b5643f","Type":"ContainerStarted","Data":"271ce49aa8f7f10b0663d40744450f0f5c499fd51ac960453264139275f114a8"}
Oct 31 19:37:16.353935 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:16.353829    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll" event={"ID":"bffa80e8-d986-43ac-a6d5-8d828ca2a35f","Type":"ContainerStarted","Data":"d8925cf40a6b02dee387da91b1f80712d42494f63a6af9daafb8f91075ebe447"}
Oct 31 19:37:16.354077 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:16.353771    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7026-c23a67"
Oct 31 19:37:16.354574 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:16.354552    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7026-c23a67"]
Oct 31 19:37:16.535355 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:16.535279    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe564e23-ad21-4f6f-a82f-69d549102bb9-metrics-certs\") pod \"maas-api-e2e-mt-67adab47-6cd87d9bdb-kwrg8\" (UID: \"fe564e23-ad21-4f6f-a82f-69d549102bb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-67adab47-6cd87d9bdb-kwrg8"
Oct 31 19:37:16.547569 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:16.535627    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe564e23-ad21-4f6f-a82f-69d549102bb9-maas-api-tls\") pod \"maas-api-e2e-mt-67adab47-6cd87d9bdb-kwrg8\" (UID: \"fe564e23-ad21-4f6f-a82f-69d549102bb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-67adab47-6cd87d9bdb-kwrg8"
Oct 31 19:37:16.547569 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:16.535714    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s9mv\" (UniqueName: \"kubernetes.io/projected/fe564e23-ad21-4f6f-a82f-69d549102bb9-kube-api-access-9s9mv\") pod \"maas-api-e2e-mt-67adab47-6cd87d9bdb-kwrg8\" (UID: \"fe564e23-ad21-4f6f-a82f-69d549102bb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-67adab47-6cd87d9bdb-kwrg8"
Oct 31 19:37:16.547569 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:16.535792    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe564e23-ad21-4f6f-a82f-69d549102bb9-openshift-service-ca\") pod \"maas-api-e2e-mt-67adab47-6cd87d9bdb-kwrg8\" (UID: \"fe564e23-ad21-4f6f-a82f-69d549102bb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-67adab47-6cd87d9bdb-kwrg8"
Oct 31 19:37:16.547569 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:16.535829    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdgsl\" (UniqueName: \"kubernetes.io/projected/23d0fb06-a8de-4c4b-aceb-82e6f2905831-kube-api-access-mdgsl\") pod \"test-curl-7026-c23a67\" (UID: \"23d0fb06-a8de-4c4b-aceb-82e6f2905831\") " pod="opendatahub/test-curl-7026-c23a67"
Oct 31 19:37:16.627402 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:16.623654    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5"
Oct 31 19:37:16.628789 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:16.628485    4645 patch_prober.go:29] interesting pod/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 31 19:37:16.628789 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:16.628539    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5" podUID="401797c4-b8ea-463e-acca-b35f26b5643f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 31 19:37:16.630078 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:16.630042    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:16.639625 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:16.639593    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe564e23-ad21-4f6f-a82f-69d549102bb9-metrics-certs\") pod \"maas-api-e2e-mt-67adab47-6cd87d9bdb-kwrg8\" (UID: \"fe564e23-ad21-4f6f-a82f-69d549102bb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-67adab47-6cd87d9bdb-kwrg8"
Oct 31 19:37:16.639801 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:16.639779    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe564e23-ad21-4f6f-a82f-69d549102bb9-maas-api-tls\") pod \"maas-api-e2e-mt-67adab47-6cd87d9bdb-kwrg8\" (UID: \"fe564e23-ad21-4f6f-a82f-69d549102bb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-67adab47-6cd87d9bdb-kwrg8"
Oct 31 19:37:16.639875 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:16.639860    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s9mv\" (UniqueName: \"kubernetes.io/projected/fe564e23-ad21-4f6f-a82f-69d549102bb9-kube-api-access-9s9mv\") pod \"maas-api-e2e-mt-67adab47-6cd87d9bdb-kwrg8\" (UID: \"fe564e23-ad21-4f6f-a82f-69d549102bb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-67adab47-6cd87d9bdb-kwrg8"
Oct 31 19:37:16.639934 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:16.639924    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe564e23-ad21-4f6f-a82f-69d549102bb9-openshift-service-ca\") pod \"maas-api-e2e-mt-67adab47-6cd87d9bdb-kwrg8\" (UID: \"fe564e23-ad21-4f6f-a82f-69d549102bb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-67adab47-6cd87d9bdb-kwrg8"
Oct 31 19:37:16.639991 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:16.639956    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdgsl\" (UniqueName: \"kubernetes.io/projected/23d0fb06-a8de-4c4b-aceb-82e6f2905831-kube-api-access-mdgsl\") pod \"test-curl-7026-c23a67\" (UID: \"23d0fb06-a8de-4c4b-aceb-82e6f2905831\") " pod="opendatahub/test-curl-7026-c23a67"
Oct 31 19:37:16.640469 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:37:16.640447    4645 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-67adab47-serving-cert: secret "maas-api-e2e-mt-67adab47-serving-cert" not found
Oct 31 19:37:16.640563 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:37:16.640524    4645 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe564e23-ad21-4f6f-a82f-69d549102bb9-maas-api-tls podName:fe564e23-ad21-4f6f-a82f-69d549102bb9 nodeName:}" failed. No retries permitted until 2026-10-31 19:37:17.140503625 +0000 UTC m=+1067.713737748 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fe564e23-ad21-4f6f-a82f-69d549102bb9-maas-api-tls") pod "maas-api-e2e-mt-67adab47-6cd87d9bdb-kwrg8" (UID: "fe564e23-ad21-4f6f-a82f-69d549102bb9") : secret "maas-api-e2e-mt-67adab47-serving-cert" not found
Oct 31 19:37:16.642218 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:16.642193    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe564e23-ad21-4f6f-a82f-69d549102bb9-openshift-service-ca\") pod \"maas-api-e2e-mt-67adab47-6cd87d9bdb-kwrg8\" (UID: \"fe564e23-ad21-4f6f-a82f-69d549102bb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-67adab47-6cd87d9bdb-kwrg8"
Oct 31 19:37:16.649354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:16.649001    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe564e23-ad21-4f6f-a82f-69d549102bb9-metrics-certs\") pod \"maas-api-e2e-mt-67adab47-6cd87d9bdb-kwrg8\" (UID: \"fe564e23-ad21-4f6f-a82f-69d549102bb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-67adab47-6cd87d9bdb-kwrg8"
Oct 31 19:37:16.730795 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:16.730759    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s9mv\" (UniqueName: \"kubernetes.io/projected/fe564e23-ad21-4f6f-a82f-69d549102bb9-kube-api-access-9s9mv\") pod \"maas-api-e2e-mt-67adab47-6cd87d9bdb-kwrg8\" (UID: \"fe564e23-ad21-4f6f-a82f-69d549102bb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-67adab47-6cd87d9bdb-kwrg8"
Oct 31 19:37:16.746380 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:16.746296    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdgsl\" (UniqueName: \"kubernetes.io/projected/23d0fb06-a8de-4c4b-aceb-82e6f2905831-kube-api-access-mdgsl\") pod \"test-curl-7026-c23a67\" (UID: \"23d0fb06-a8de-4c4b-aceb-82e6f2905831\") " pod="opendatahub/test-curl-7026-c23a67"
Oct 31 19:37:16.752001 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:16.751965    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:16.810467 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:16.810432    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll"
Oct 31 19:37:16.810897 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:16.810872    4645 patch_prober.go:29] interesting pod/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 31 19:37:16.811094 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:16.811054    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll" podUID="bffa80e8-d986-43ac-a6d5-8d828ca2a35f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 31 19:37:16.899414 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:16.897451    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5"
Oct 31 19:37:16.954174 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:16.954139    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7026-c23a67"
Oct 31 19:37:17.246582 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:17.239278    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe564e23-ad21-4f6f-a82f-69d549102bb9-maas-api-tls\") pod \"maas-api-e2e-mt-67adab47-6cd87d9bdb-kwrg8\" (UID: \"fe564e23-ad21-4f6f-a82f-69d549102bb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-67adab47-6cd87d9bdb-kwrg8"
Oct 31 19:37:17.300605 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:17.288854    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe564e23-ad21-4f6f-a82f-69d549102bb9-maas-api-tls\") pod \"maas-api-e2e-mt-67adab47-6cd87d9bdb-kwrg8\" (UID: \"fe564e23-ad21-4f6f-a82f-69d549102bb9\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-67adab47-6cd87d9bdb-kwrg8"
Oct 31 19:37:17.516406 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:17.512772    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:17.580106 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:17.570726    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-67adab47-6cd87d9bdb-kwrg8"
Oct 31 19:37:17.675746 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:17.674709    4645 patch_prober.go:29] interesting pod/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 31 19:37:17.675746 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:17.674785    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5" podUID="401797c4-b8ea-463e-acca-b35f26b5643f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 31 19:37:17.749378 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:17.746240    4645 generic.go:356] "Generic (PLEG): container finished" podID="59ae2f4a-60ab-492e-a0ff-439cd98a81a7" containerID="43fd5c9b24f1b2e5576ff0d0ba0baa54d7db2e35fc4c9fa932b0b241b898d2aa" exitCode=137
Oct 31 19:37:17.749378 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:17.747420    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7026-c918fc" podUID="6007cecc-95ef-4f1b-b0fd-dd5e89494e67" containerName="test-curl-7026-c918fc" containerID="cri-o://f2889907d49cd563108059f04bb1787ac75b1bdd07cb29e8a558757d8e2ba52f" gracePeriod=2
Oct 31 19:37:17.749378 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:17.748510    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7026-c23a67"]
Oct 31 19:37:17.750713 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:37:17.750688    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod23d0fb06_a8de_4c4b_aceb_82e6f2905831.slice/crio-db1ad8a0a47fcbd5389db6ddaaee63531f231e3c91279e330304066366b1e936 WatchSource:0}: Error finding container db1ad8a0a47fcbd5389db6ddaaee63531f231e3c91279e330304066366b1e936: Status 404 returned error can't find the container with id db1ad8a0a47fcbd5389db6ddaaee63531f231e3c91279e330304066366b1e936
Oct 31 19:37:17.824407 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:17.822477    4645 patch_prober.go:29] interesting pod/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 31 19:37:17.824407 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:17.822553    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll" podUID="bffa80e8-d986-43ac-a6d5-8d828ca2a35f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 31 19:37:17.831835 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:17.831799    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7026-b01cc5"
Oct 31 19:37:17.935015 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:17.934990    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59ae2f4a-60ab-492e-a0ff-439cd98a81a7-kube-api-access-l2kbz\" (UniqueName: \"kubernetes.io/projected/59ae2f4a-60ab-492e-a0ff-439cd98a81a7-kube-api-access-l2kbz\") pod \"59ae2f4a-60ab-492e-a0ff-439cd98a81a7\" (UID: \"59ae2f4a-60ab-492e-a0ff-439cd98a81a7\") "
Oct 31 19:37:17.937889 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:17.937853    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59ae2f4a-60ab-492e-a0ff-439cd98a81a7-kube-api-access-l2kbz" pod "59ae2f4a-60ab-492e-a0ff-439cd98a81a7" (UID: "59ae2f4a-60ab-492e-a0ff-439cd98a81a7"). InnerVolumeSpecName "kube-api-access-l2kbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:37:18.037829 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:18.037748    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2kbz\" (UniqueName: \"kubernetes.io/projected/59ae2f4a-60ab-492e-a0ff-439cd98a81a7-kube-api-access-l2kbz\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:37:18.105769 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:18.105724    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-67adab47-6cd87d9bdb-kwrg8"]
Oct 31 19:37:18.111108 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:37:18.111072    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe564e23_ad21_4f6f_a82f_69d549102bb9.slice/crio-6de40b38f1a7eb5ddfa0f38dabc0eb2564aada9cca39655f7e67caf192ca6162 WatchSource:0}: Error finding container 6de40b38f1a7eb5ddfa0f38dabc0eb2564aada9cca39655f7e67caf192ca6162: Status 404 returned error can't find the container with id 6de40b38f1a7eb5ddfa0f38dabc0eb2564aada9cca39655f7e67caf192ca6162
Oct 31 19:37:18.406396 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:18.406070    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:18.604405 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:18.595794    4645 patch_prober.go:29] interesting pod/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 31 19:37:18.604405 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:18.595853    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5" podUID="401797c4-b8ea-463e-acca-b35f26b5643f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 31 19:37:18.804353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:18.803659    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-67adab47-6cd87d9bdb-kwrg8" event={"ID":"fe564e23-ad21-4f6f-a82f-69d549102bb9","Type":"ContainerStarted","Data":"e454c057fc8adbc7ee5cfaf3eeb6cb5f7e1c70a35cdbddfd89d51931b19285a4"}
Oct 31 19:37:18.804353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:18.803731    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-67adab47-6cd87d9bdb-kwrg8" event={"ID":"fe564e23-ad21-4f6f-a82f-69d549102bb9","Type":"ContainerStarted","Data":"6de40b38f1a7eb5ddfa0f38dabc0eb2564aada9cca39655f7e67caf192ca6162"}
Oct 31 19:37:18.808649 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:18.804850    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-67adab47-6cd87d9bdb-kwrg8"
Oct 31 19:37:18.808649 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:18.806471    4645 patch_prober.go:29] interesting pod/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 31 19:37:18.808649 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:18.806517    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll" podUID="bffa80e8-d986-43ac-a6d5-8d828ca2a35f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 31 19:37:18.811179 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:18.810561    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7026-c23a67" event={"ID":"23d0fb06-a8de-4c4b-aceb-82e6f2905831","Type":"ContainerStarted","Data":"c9f4ce213be4b486c0fdf85aade5ca920c2e06e1aacf9bc9e6c22691d8638b29"}
Oct 31 19:37:18.811179 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:18.810592    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7026-c23a67" event={"ID":"23d0fb06-a8de-4c4b-aceb-82e6f2905831","Type":"ContainerStarted","Data":"db1ad8a0a47fcbd5389db6ddaaee63531f231e3c91279e330304066366b1e936"}
Oct 31 19:37:18.813045 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:18.813028    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7026-b01cc5"
Oct 31 19:37:18.813119 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:18.813086    4645 scope.go:122] "RemoveContainer" containerID="43fd5c9b24f1b2e5576ff0d0ba0baa54d7db2e35fc4c9fa932b0b241b898d2aa"
Oct 31 19:37:18.829422 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:18.829181    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-67adab47-6cd87d9bdb-kwrg8" podStartSLOduration=2.313688765 podStartE2EDuration="2.829165909s" totalImagesPullingTime="515.477144ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:37:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:37:18.823841704 +0000 UTC m=+1069.397075833" watchObservedRunningTime="2026-10-31 19:37:18.829165909 +0000 UTC m=+1069.402400039"
Oct 31 19:37:18.842289 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:37:18.842263    4645 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7026-b01cc5\" is forbidden: User \"system:node:ip-10-0-131-94.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-94.us-east-2.compute.internal' and this object" podUID="59ae2f4a-60ab-492e-a0ff-439cd98a81a7" pod="opendatahub/test-curl-7026-b01cc5"
Oct 31 19:37:18.844217 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:18.844173    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7026-c23a67" podStartSLOduration=2.8441614939999997 podStartE2EDuration="2.844161494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:37:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:37:18.840304457 +0000 UTC m=+1069.413538589" watchObservedRunningTime="2026-10-31 19:37:18.844161494 +0000 UTC m=+1069.417395621"
Oct 31 19:37:19.309099 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:19.309062    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:19.609454 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:19.605482    4645 patch_prober.go:29] interesting pod/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 31 19:37:19.609454 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:19.605565    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5" podUID="401797c4-b8ea-463e-acca-b35f26b5643f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 31 19:37:19.618927 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:37:19.617411    4645 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7026-b01cc5\" is forbidden: User \"system:node:ip-10-0-131-94.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-94.us-east-2.compute.internal' and this object" podUID="59ae2f4a-60ab-492e-a0ff-439cd98a81a7" pod="opendatahub/test-curl-7026-b01cc5"
Oct 31 19:37:19.625927 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:19.625902    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59ae2f4a-60ab-492e-a0ff-439cd98a81a7" path="/var/lib/kubelet/pods/59ae2f4a-60ab-492e-a0ff-439cd98a81a7/volumes"
Oct 31 19:37:19.811351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:19.804382    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7026-c23a67"]
Oct 31 19:37:19.811351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:19.809319    4645 patch_prober.go:29] interesting pod/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 31 19:37:19.811351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:19.809400    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll" podUID="bffa80e8-d986-43ac-a6d5-8d828ca2a35f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 31 19:37:19.822200 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:19.822171    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7026-c23a67"]
Oct 31 19:37:20.151843 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:20.151819    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7026-c918fc"
Oct 31 19:37:20.153762 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:37:20.153735    4645 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7026-c918fc\" is forbidden: User \"system:node:ip-10-0-131-94.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-94.us-east-2.compute.internal' and this object" podUID="6007cecc-95ef-4f1b-b0fd-dd5e89494e67" pod="opendatahub/test-curl-7026-c918fc"
Oct 31 19:37:20.230295 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:20.230269    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6007cecc-95ef-4f1b-b0fd-dd5e89494e67-kube-api-access-v7qvj\" (UniqueName: \"kubernetes.io/projected/6007cecc-95ef-4f1b-b0fd-dd5e89494e67-kube-api-access-v7qvj\") pod \"6007cecc-95ef-4f1b-b0fd-dd5e89494e67\" (UID: \"6007cecc-95ef-4f1b-b0fd-dd5e89494e67\") "
Oct 31 19:37:20.232505 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:20.232477    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6007cecc-95ef-4f1b-b0fd-dd5e89494e67-kube-api-access-v7qvj" pod "6007cecc-95ef-4f1b-b0fd-dd5e89494e67" (UID: "6007cecc-95ef-4f1b-b0fd-dd5e89494e67"). InnerVolumeSpecName "kube-api-access-v7qvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:37:20.243700 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:20.243626    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:20.337353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:20.337124    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7qvj\" (UniqueName: \"kubernetes.io/projected/6007cecc-95ef-4f1b-b0fd-dd5e89494e67-kube-api-access-v7qvj\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:37:20.599421 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:20.598750    4645 patch_prober.go:29] interesting pod/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 31 19:37:20.599421 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:20.598807    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5" podUID="401797c4-b8ea-463e-acca-b35f26b5643f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 31 19:37:20.808045 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:20.807733    4645 patch_prober.go:29] interesting pod/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 31 19:37:20.808045 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:20.807792    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll" podUID="bffa80e8-d986-43ac-a6d5-8d828ca2a35f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 31 19:37:20.835009 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:20.834953    4645 generic.go:356] "Generic (PLEG): container finished" podID="6007cecc-95ef-4f1b-b0fd-dd5e89494e67" containerID="f2889907d49cd563108059f04bb1787ac75b1bdd07cb29e8a558757d8e2ba52f" exitCode=137
Oct 31 19:37:20.835009 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:20.834996    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7026-c918fc"
Oct 31 19:37:20.835217 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:20.835038    4645 scope.go:122] "RemoveContainer" containerID="f2889907d49cd563108059f04bb1787ac75b1bdd07cb29e8a558757d8e2ba52f"
Oct 31 19:37:20.835217 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:20.835203    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7026-c23a67" podUID="23d0fb06-a8de-4c4b-aceb-82e6f2905831" containerName="test-curl-7026-c23a67" containerID="cri-o://c9f4ce213be4b486c0fdf85aade5ca920c2e06e1aacf9bc9e6c22691d8638b29" gracePeriod=2
Oct 31 19:37:20.840057 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:37:20.837755    4645 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7026-c23a67\" is forbidden: User \"system:node:ip-10-0-131-94.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-94.us-east-2.compute.internal' and this object" podUID="23d0fb06-a8de-4c4b-aceb-82e6f2905831" pod="opendatahub/test-curl-7026-c23a67"
Oct 31 19:37:20.840057 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:37:20.839282    4645 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7026-c918fc\" is forbidden: User \"system:node:ip-10-0-131-94.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-94.us-east-2.compute.internal' and this object" podUID="6007cecc-95ef-4f1b-b0fd-dd5e89494e67" pod="opendatahub/test-curl-7026-c918fc"
Oct 31 19:37:20.842289 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:37:20.841277    4645 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7026-c23a67\" is forbidden: User \"system:node:ip-10-0-131-94.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-94.us-east-2.compute.internal' and this object" podUID="23d0fb06-a8de-4c4b-aceb-82e6f2905831" pod="opendatahub/test-curl-7026-c23a67"
Oct 31 19:37:20.847674 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:37:20.847650    4645 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7026-c918fc\" is forbidden: User \"system:node:ip-10-0-131-94.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-94.us-east-2.compute.internal' and this object" podUID="6007cecc-95ef-4f1b-b0fd-dd5e89494e67" pod="opendatahub/test-curl-7026-c918fc"
Oct 31 19:37:20.850386 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:20.850322    4645 scope.go:122] "RemoveContainer" containerID="f2889907d49cd563108059f04bb1787ac75b1bdd07cb29e8a558757d8e2ba52f"
Oct 31 19:37:20.850514 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:37:20.850496    4645 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7026-c23a67\" is forbidden: User \"system:node:ip-10-0-131-94.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-94.us-east-2.compute.internal' and this object" podUID="23d0fb06-a8de-4c4b-aceb-82e6f2905831" pod="opendatahub/test-curl-7026-c23a67"
Oct 31 19:37:20.850645 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:37:20.850628    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2889907d49cd563108059f04bb1787ac75b1bdd07cb29e8a558757d8e2ba52f\": container with ID starting with f2889907d49cd563108059f04bb1787ac75b1bdd07cb29e8a558757d8e2ba52f not found: ID does not exist" containerID="f2889907d49cd563108059f04bb1787ac75b1bdd07cb29e8a558757d8e2ba52f"
Oct 31 19:37:20.850688 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:20.850654    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2889907d49cd563108059f04bb1787ac75b1bdd07cb29e8a558757d8e2ba52f"} err="failed to get container status \"f2889907d49cd563108059f04bb1787ac75b1bdd07cb29e8a558757d8e2ba52f\": rpc error: code = NotFound desc = could not find container \"f2889907d49cd563108059f04bb1787ac75b1bdd07cb29e8a558757d8e2ba52f\": container with ID starting with f2889907d49cd563108059f04bb1787ac75b1bdd07cb29e8a558757d8e2ba52f not found: ID does not exist"
Oct 31 19:37:20.852239 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:37:20.852217    4645 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7026-c918fc\" is forbidden: User \"system:node:ip-10-0-131-94.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-94.us-east-2.compute.internal' and this object" podUID="6007cecc-95ef-4f1b-b0fd-dd5e89494e67" pod="opendatahub/test-curl-7026-c918fc"
Oct 31 19:37:20.910827 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:20.910791    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:21.073063 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:21.072972    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:21.301356 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:21.276614    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:21.301356 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:21.288513    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-67adab47-998479859-kmf5v"
Oct 31 19:37:21.301356 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:37:21.295655    4645 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7026-c918fc\" is forbidden: User \"system:node:ip-10-0-131-94.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-94.us-east-2.compute.internal' and this object" podUID="6007cecc-95ef-4f1b-b0fd-dd5e89494e67" pod="opendatahub/test-curl-7026-c918fc"
Oct 31 19:37:21.375354 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:37:21.371557    4645 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7026-c23a67\" is forbidden: User \"system:node:ip-10-0-131-94.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-94.us-east-2.compute.internal' and this object" podUID="23d0fb06-a8de-4c4b-aceb-82e6f2905831" pod="opendatahub/test-curl-7026-c23a67"
Oct 31 19:37:21.605414 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:21.596549    4645 patch_prober.go:29] interesting pod/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body=
Oct 31 19:37:21.605414 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:21.596614    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5" podUID="401797c4-b8ea-463e-acca-b35f26b5643f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused"
Oct 31 19:37:21.625114 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:21.625080    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6007cecc-95ef-4f1b-b0fd-dd5e89494e67" path="/var/lib/kubelet/pods/6007cecc-95ef-4f1b-b0fd-dd5e89494e67/volumes"
Oct 31 19:37:21.807428 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:21.807387    4645 patch_prober.go:29] interesting pod/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Oct 31 19:37:21.807606 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:21.807458    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll" podUID="bffa80e8-d986-43ac-a6d5-8d828ca2a35f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Oct 31 19:37:22.082829 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:22.082794    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:22.244351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:22.243389    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:22.600260 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:22.600229    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5"
Oct 31 19:37:22.601226 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:22.601203    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5"
Oct 31 19:37:22.666750 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:22.666721    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll"
Oct 31 19:37:22.810158 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:22.810128    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll"
Oct 31 19:37:22.855530 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:22.855457    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll"
Oct 31 19:37:23.050704 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:23.050681    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7026-c23a67"
Oct 31 19:37:23.142416 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:23.142340    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23d0fb06-a8de-4c4b-aceb-82e6f2905831-kube-api-access-mdgsl\" (UniqueName: \"kubernetes.io/projected/23d0fb06-a8de-4c4b-aceb-82e6f2905831-kube-api-access-mdgsl\") pod \"23d0fb06-a8de-4c4b-aceb-82e6f2905831\" (UID: \"23d0fb06-a8de-4c4b-aceb-82e6f2905831\") "
Oct 31 19:37:23.144305 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:23.144285    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23d0fb06-a8de-4c4b-aceb-82e6f2905831-kube-api-access-mdgsl" pod "23d0fb06-a8de-4c4b-aceb-82e6f2905831" (UID: "23d0fb06-a8de-4c4b-aceb-82e6f2905831"). InnerVolumeSpecName "kube-api-access-mdgsl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:37:23.155934 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:23.155911    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:23.244537 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:23.244505    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdgsl\" (UniqueName: \"kubernetes.io/projected/23d0fb06-a8de-4c4b-aceb-82e6f2905831-kube-api-access-mdgsl\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:37:23.619168 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:23.619134    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23d0fb06-a8de-4c4b-aceb-82e6f2905831" path="/var/lib/kubelet/pods/23d0fb06-a8de-4c4b-aceb-82e6f2905831/volumes"
Oct 31 19:37:23.865460 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:23.865426    4645 generic.go:356] "Generic (PLEG): container finished" podID="23d0fb06-a8de-4c4b-aceb-82e6f2905831" containerID="c9f4ce213be4b486c0fdf85aade5ca920c2e06e1aacf9bc9e6c22691d8638b29" exitCode=137
Oct 31 19:37:23.865662 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:23.865485    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7026-c23a67"
Oct 31 19:37:23.865662 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:23.865532    4645 scope.go:122] "RemoveContainer" containerID="c9f4ce213be4b486c0fdf85aade5ca920c2e06e1aacf9bc9e6c22691d8638b29"
Oct 31 19:37:23.875242 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:23.875226    4645 scope.go:122] "RemoveContainer" containerID="c9f4ce213be4b486c0fdf85aade5ca920c2e06e1aacf9bc9e6c22691d8638b29"
Oct 31 19:37:23.875490 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:37:23.875474    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9f4ce213be4b486c0fdf85aade5ca920c2e06e1aacf9bc9e6c22691d8638b29\": container with ID starting with c9f4ce213be4b486c0fdf85aade5ca920c2e06e1aacf9bc9e6c22691d8638b29 not found: ID does not exist" containerID="c9f4ce213be4b486c0fdf85aade5ca920c2e06e1aacf9bc9e6c22691d8638b29"
Oct 31 19:37:23.875559 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:23.875499    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9f4ce213be4b486c0fdf85aade5ca920c2e06e1aacf9bc9e6c22691d8638b29"} err="failed to get container status \"c9f4ce213be4b486c0fdf85aade5ca920c2e06e1aacf9bc9e6c22691d8638b29\": rpc error: code = NotFound desc = could not find container \"c9f4ce213be4b486c0fdf85aade5ca920c2e06e1aacf9bc9e6c22691d8638b29\": container with ID starting with c9f4ce213be4b486c0fdf85aade5ca920c2e06e1aacf9bc9e6c22691d8638b29 not found: ID does not exist"
Oct 31 19:37:23.944796 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:23.944758    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:24.759241 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:24.759208    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:25.562512 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:25.562478    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:25.841635 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:25.841560    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-67adab47-6cd87d9bdb-kwrg8"
Oct 31 19:37:26.500822 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:26.500782    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:27.244469 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:27.244438    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:28.056247 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:28.056214    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:28.999145 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:28.999108    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:29.909997 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:29.909963    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:30.831336 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:30.831287    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:31.500988 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:31.500957    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:31.841822 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:31.841735    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:32.867360 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:32.867286    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-67adab47/e2e-lifecycle-model-67adab47-kserve-8f9445cd9-5rp7l"]
Oct 31 19:37:32.868407 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:32.868380    4645 cpu_manager.go:414] "containerMap: removing container" podUID="23d0fb06-a8de-4c4b-aceb-82e6f2905831" containerName="test-curl-7026-c23a67"
Oct 31 19:37:32.868407 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:32.868405    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23d0fb06-a8de-4c4b-aceb-82e6f2905831" containerName="test-curl-7026-c23a67"
Oct 31 19:37:32.868605 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:32.868591    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="23d0fb06-a8de-4c4b-aceb-82e6f2905831" containerName="test-curl-7026-c23a67"
Oct 31 19:37:32.874390 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:32.874366    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-67adab47/e2e-lifecycle-model-67adab47-kserve-8f9445cd9-5rp7l"
Oct 31 19:37:32.883510 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:32.883485    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-67adab47\"/\"kube-root-ca.crt\""
Oct 31 19:37:32.888587 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:32.888562    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-67adab47\"/\"default-dockercfg-25wf2\""
Oct 31 19:37:32.888720 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:32.888661    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-67adab47\"/\"e2e-lifecycle-model-67adab47-kserve-self-signed-certs\""
Oct 31 19:37:32.889751 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:32.889725    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-67adab47\"/\"openshift-service-ca.crt\""
Oct 31 19:37:32.891923 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:32.891894    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-67adab47/e2e-lifecycle-model-67adab47-kserve-8f9445cd9-5rp7l"]
Oct 31 19:37:32.956446 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:32.956398    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:33.036618 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:33.036581    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9153d96-5237-47a0-8bf4-1e77f69b071a-home\") pod \"e2e-lifecycle-model-67adab47-kserve-8f9445cd9-5rp7l\" (UID: \"b9153d96-5237-47a0-8bf4-1e77f69b071a\") " pod="ai-tenant-e2e-mt-67adab47/e2e-lifecycle-model-67adab47-kserve-8f9445cd9-5rp7l"
Oct 31 19:37:33.036804 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:33.036669    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9153d96-5237-47a0-8bf4-1e77f69b071a-tmp-dir\") pod \"e2e-lifecycle-model-67adab47-kserve-8f9445cd9-5rp7l\" (UID: \"b9153d96-5237-47a0-8bf4-1e77f69b071a\") " pod="ai-tenant-e2e-mt-67adab47/e2e-lifecycle-model-67adab47-kserve-8f9445cd9-5rp7l"
Oct 31 19:37:33.036804 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:33.036704    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4plcs\" (UniqueName: \"kubernetes.io/projected/b9153d96-5237-47a0-8bf4-1e77f69b071a-kube-api-access-4plcs\") pod \"e2e-lifecycle-model-67adab47-kserve-8f9445cd9-5rp7l\" (UID: \"b9153d96-5237-47a0-8bf4-1e77f69b071a\") " pod="ai-tenant-e2e-mt-67adab47/e2e-lifecycle-model-67adab47-kserve-8f9445cd9-5rp7l"
Oct 31 19:37:33.037061 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:33.037036    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9153d96-5237-47a0-8bf4-1e77f69b071a-model-cache\") pod \"e2e-lifecycle-model-67adab47-kserve-8f9445cd9-5rp7l\" (UID: \"b9153d96-5237-47a0-8bf4-1e77f69b071a\") " pod="ai-tenant-e2e-mt-67adab47/e2e-lifecycle-model-67adab47-kserve-8f9445cd9-5rp7l"
Oct 31 19:37:33.037275 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:33.037251    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9153d96-5237-47a0-8bf4-1e77f69b071a-tls-certs\") pod \"e2e-lifecycle-model-67adab47-kserve-8f9445cd9-5rp7l\" (UID: \"b9153d96-5237-47a0-8bf4-1e77f69b071a\") " pod="ai-tenant-e2e-mt-67adab47/e2e-lifecycle-model-67adab47-kserve-8f9445cd9-5rp7l"
Oct 31 19:37:33.037469 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:33.037441    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9153d96-5237-47a0-8bf4-1e77f69b071a-dshm\") pod \"e2e-lifecycle-model-67adab47-kserve-8f9445cd9-5rp7l\" (UID: \"b9153d96-5237-47a0-8bf4-1e77f69b071a\") " pod="ai-tenant-e2e-mt-67adab47/e2e-lifecycle-model-67adab47-kserve-8f9445cd9-5rp7l"
Oct 31 19:37:33.151405 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:33.148398    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9153d96-5237-47a0-8bf4-1e77f69b071a-tls-certs\") pod \"e2e-lifecycle-model-67adab47-kserve-8f9445cd9-5rp7l\" (UID: \"b9153d96-5237-47a0-8bf4-1e77f69b071a\") " pod="ai-tenant-e2e-mt-67adab47/e2e-lifecycle-model-67adab47-kserve-8f9445cd9-5rp7l"
Oct 31 19:37:33.151405 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:33.148476    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9153d96-5237-47a0-8bf4-1e77f69b071a-dshm\") pod \"e2e-lifecycle-model-67adab47-kserve-8f9445cd9-5rp7l\" (UID: \"b9153d96-5237-47a0-8bf4-1e77f69b071a\") " pod="ai-tenant-e2e-mt-67adab47/e2e-lifecycle-model-67adab47-kserve-8f9445cd9-5rp7l"
Oct 31 19:37:33.151405 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:33.148555    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9153d96-5237-47a0-8bf4-1e77f69b071a-home\") pod \"e2e-lifecycle-model-67adab47-kserve-8f9445cd9-5rp7l\" (UID: \"b9153d96-5237-47a0-8bf4-1e77f69b071a\") " pod="ai-tenant-e2e-mt-67adab47/e2e-lifecycle-model-67adab47-kserve-8f9445cd9-5rp7l"
Oct 31 19:37:33.151405 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:33.148583    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9153d96-5237-47a0-8bf4-1e77f69b071a-tmp-dir\") pod \"e2e-lifecycle-model-67adab47-kserve-8f9445cd9-5rp7l\" (UID: \"b9153d96-5237-47a0-8bf4-1e77f69b071a\") " pod="ai-tenant-e2e-mt-67adab47/e2e-lifecycle-model-67adab47-kserve-8f9445cd9-5rp7l"
Oct 31 19:37:33.151405 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:33.148606    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4plcs\" (UniqueName: \"kubernetes.io/projected/b9153d96-5237-47a0-8bf4-1e77f69b071a-kube-api-access-4plcs\") pod \"e2e-lifecycle-model-67adab47-kserve-8f9445cd9-5rp7l\" (UID: \"b9153d96-5237-47a0-8bf4-1e77f69b071a\") " pod="ai-tenant-e2e-mt-67adab47/e2e-lifecycle-model-67adab47-kserve-8f9445cd9-5rp7l"
Oct 31 19:37:33.151405 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:33.148821    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9153d96-5237-47a0-8bf4-1e77f69b071a-model-cache\") pod \"e2e-lifecycle-model-67adab47-kserve-8f9445cd9-5rp7l\" (UID: \"b9153d96-5237-47a0-8bf4-1e77f69b071a\") " pod="ai-tenant-e2e-mt-67adab47/e2e-lifecycle-model-67adab47-kserve-8f9445cd9-5rp7l"
Oct 31 19:37:33.151405 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:33.149402    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9153d96-5237-47a0-8bf4-1e77f69b071a-model-cache\") pod \"e2e-lifecycle-model-67adab47-kserve-8f9445cd9-5rp7l\" (UID: \"b9153d96-5237-47a0-8bf4-1e77f69b071a\") " pod="ai-tenant-e2e-mt-67adab47/e2e-lifecycle-model-67adab47-kserve-8f9445cd9-5rp7l"
Oct 31 19:37:33.159349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:33.153416    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9153d96-5237-47a0-8bf4-1e77f69b071a-tmp-dir\") pod \"e2e-lifecycle-model-67adab47-kserve-8f9445cd9-5rp7l\" (UID: \"b9153d96-5237-47a0-8bf4-1e77f69b071a\") " pod="ai-tenant-e2e-mt-67adab47/e2e-lifecycle-model-67adab47-kserve-8f9445cd9-5rp7l"
Oct 31 19:37:33.159349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:33.153689    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9153d96-5237-47a0-8bf4-1e77f69b071a-home\") pod \"e2e-lifecycle-model-67adab47-kserve-8f9445cd9-5rp7l\" (UID: \"b9153d96-5237-47a0-8bf4-1e77f69b071a\") " pod="ai-tenant-e2e-mt-67adab47/e2e-lifecycle-model-67adab47-kserve-8f9445cd9-5rp7l"
Oct 31 19:37:33.159349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:33.155007    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9153d96-5237-47a0-8bf4-1e77f69b071a-tls-certs\") pod \"e2e-lifecycle-model-67adab47-kserve-8f9445cd9-5rp7l\" (UID: \"b9153d96-5237-47a0-8bf4-1e77f69b071a\") " pod="ai-tenant-e2e-mt-67adab47/e2e-lifecycle-model-67adab47-kserve-8f9445cd9-5rp7l"
Oct 31 19:37:33.189351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:33.187692    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4plcs\" (UniqueName: \"kubernetes.io/projected/b9153d96-5237-47a0-8bf4-1e77f69b071a-kube-api-access-4plcs\") pod \"e2e-lifecycle-model-67adab47-kserve-8f9445cd9-5rp7l\" (UID: \"b9153d96-5237-47a0-8bf4-1e77f69b071a\") " pod="ai-tenant-e2e-mt-67adab47/e2e-lifecycle-model-67adab47-kserve-8f9445cd9-5rp7l"
Oct 31 19:37:33.205686 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:33.204887    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:33.206176 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:33.206154    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9153d96-5237-47a0-8bf4-1e77f69b071a-dshm\") pod \"e2e-lifecycle-model-67adab47-kserve-8f9445cd9-5rp7l\" (UID: \"b9153d96-5237-47a0-8bf4-1e77f69b071a\") " pod="ai-tenant-e2e-mt-67adab47/e2e-lifecycle-model-67adab47-kserve-8f9445cd9-5rp7l"
Oct 31 19:37:33.494591 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:33.494554    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-67adab47/e2e-lifecycle-model-67adab47-kserve-8f9445cd9-5rp7l"
Oct 31 19:37:33.807200 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:33.807170    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-67adab47/e2e-lifecycle-model-67adab47-kserve-8f9445cd9-5rp7l"]
Oct 31 19:37:33.810196 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:37:33.810171    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb9153d96_5237_47a0_8bf4_1e77f69b071a.slice/crio-b854245ceb75df64752cd1a544d4b434c72c0845df354750e65697e8e544014c WatchSource:0}: Error finding container b854245ceb75df64752cd1a544d4b434c72c0845df354750e65697e8e544014c: Status 404 returned error can't find the container with id b854245ceb75df64752cd1a544d4b434c72c0845df354750e65697e8e544014c
Oct 31 19:37:33.992279 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:33.992244    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-67adab47/e2e-lifecycle-model-67adab47-kserve-8f9445cd9-5rp7l" event={"ID":"b9153d96-5237-47a0-8bf4-1e77f69b071a","Type":"ContainerStarted","Data":"a61d089e43ec0882d8b96983157bbc7bd7c9721db274507a0ffc55a0a0c02e28"}
Oct 31 19:37:33.992686 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:33.992289    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-67adab47/e2e-lifecycle-model-67adab47-kserve-8f9445cd9-5rp7l" event={"ID":"b9153d96-5237-47a0-8bf4-1e77f69b071a","Type":"ContainerStarted","Data":"b854245ceb75df64752cd1a544d4b434c72c0845df354750e65697e8e544014c"}
Oct 31 19:37:34.019095 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:34.019041    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-67adab47/e2e-lifecycle-model-67adab47-kserve-8f9445cd9-5rp7l" podStartSLOduration=2.019025565 podStartE2EDuration="2.019025565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:37:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:37:34.015321697 +0000 UTC m=+1084.588555826" watchObservedRunningTime="2026-10-31 19:37:34.019025565 +0000 UTC m=+1084.592259693"
Oct 31 19:37:34.082407 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:34.079203    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:34.225353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:34.220098    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:34.495051 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:34.495008    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-67adab47/e2e-lifecycle-model-67adab47-kserve-8f9445cd9-5rp7l"
Oct 31 19:37:34.854892 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:34.854809    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:35.784940 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:35.784890    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:36.391390 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:37:36.387569    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 19:37:36.558861 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:36.558827    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:37.444558 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:37.444524    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:38.432189 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:38.432146    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:39.137439 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:39.137404    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:39.781656 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:39.781627    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:40.557513 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:40.557467    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:41.349815 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:41.349778    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:41.814955 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:41.814901    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:42.129766 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:42.129676    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:42.859823 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:42.859782    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:43.495468 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:43.495440    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-67adab47/e2e-lifecycle-model-67adab47-kserve-8f9445cd9-5rp7l"
Oct 31 19:37:43.508836 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:43.508808    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-67adab47/e2e-lifecycle-model-67adab47-kserve-8f9445cd9-5rp7l"
Oct 31 19:37:43.557708 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:43.557681    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:44.073255 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:44.073177    4645 generic.go:356] "Generic (PLEG): container finished" podID="92238ea2-f330-4873-ab87-37b37bb05f9a" containerID="809a98ee1208611517201f69b8c69c582c97c38049510312ccdfa1381982551d" exitCode=0
Oct 31 19:37:44.073624 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:44.073249    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl" event={"ID":"92238ea2-f330-4873-ab87-37b37bb05f9a","Type":"ContainerDied","Data":"809a98ee1208611517201f69b8c69c582c97c38049510312ccdfa1381982551d"}
Oct 31 19:37:44.073776 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:44.073758    4645 scope.go:122] "RemoveContainer" containerID="809a98ee1208611517201f69b8c69c582c97c38049510312ccdfa1381982551d"
Oct 31 19:37:44.075688 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:44.075654    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:37:44.075782 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:44.075742    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:37:44.075821 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:44.075786    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:37:44.085764 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:44.085746    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-67adab47/e2e-lifecycle-model-67adab47-kserve-8f9445cd9-5rp7l"
Oct 31 19:37:44.400882 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:44.400787    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:45.088052 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:45.088013    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl" event={"ID":"92238ea2-f330-4873-ab87-37b37bb05f9a","Type":"ContainerStarted","Data":"2136df6ced5bec698141e9ae7567e6f1ac864a520bf01edcbbd4c907e8805305"}
Oct 31 19:37:45.181391 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:45.181323    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:45.551594 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:45.551561    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl"
Oct 31 19:37:45.552485 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:45.552459    4645 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 31 19:37:45.552617 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:45.552515    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl" podUID="92238ea2-f330-4873-ab87-37b37bb05f9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 31 19:37:45.983824 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:45.983795    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:46.551152 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:46.551121    4645 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 31 19:37:46.551579 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:46.551187    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl" podUID="92238ea2-f330-4873-ab87-37b37bb05f9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 31 19:37:46.871372 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:46.871222    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:47.551463 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:47.551425    4645 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 31 19:37:47.551873 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:47.551494    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl" podUID="92238ea2-f330-4873-ab87-37b37bb05f9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 31 19:37:47.620389 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:47.620356    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:48.551484 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:48.551448    4645 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 31 19:37:48.551875 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:48.551521    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl" podUID="92238ea2-f330-4873-ab87-37b37bb05f9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 31 19:37:48.681007 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:48.680965    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:48.932510 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:48.932469    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl"
Oct 31 19:37:49.551606 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:49.551572    4645 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 31 19:37:49.552030 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:49.551624    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl" podUID="92238ea2-f330-4873-ab87-37b37bb05f9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 31 19:37:49.582231 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:49.582207    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:50.351305 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:50.351272    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:50.555183 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:50.555158    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl"
Oct 31 19:37:50.556226 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:50.556208    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl"
Oct 31 19:37:51.137572 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:51.137534    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:51.849216 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:51.849172    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:52.302618 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:52.302576    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:52.558808 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:52.558732    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:53.537292 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:53.537257    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:54.261973 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:54.261941    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:55.052135 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:55.052103    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:55.746362 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:55.746311    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:56.551666 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:56.551635    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:57.379856 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:57.379821    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:58.265192 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:58.265157    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:59.109391 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:59.109321    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:37:59.742188 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:37:59.742154    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:00.432842 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:00.432809    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:01.158617 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:01.158578    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:02.064112 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:02.064074    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:02.733463 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:02.733425    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:03.549823 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:03.549787    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:04.467026 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:04.466988    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:05.504771 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:05.504732    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:06.262917 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:38:06.262873    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 19:38:06.461125 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:06.461091    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:07.397520 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:07.397488    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:08.548364 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:08.548316    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:09.448344 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:09.448298    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:10.451186 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:10.451119    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:11.360802 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:11.360774    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:12.235642 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:12.235564    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:13.109253 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:13.109208    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:13.673395 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:13.673311    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:13.842509 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:13.842477    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:14.149397 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:14.149301    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:14.852241 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:14.852187    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:14.957752 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:14.957264    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:15.736043 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:15.735988    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:16.054547 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:16.054461    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:16.447512 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:16.447474    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:16.588688 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:16.588648    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:17.549623 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:17.549590    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:18.030729 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:18.030698    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:18.154157 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:18.154118    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:18.474831 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:18.474797    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:18.883982 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:18.883892    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:19.237968 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:19.237933    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl"]
Oct 31 19:38:19.247644 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:19.247611    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl"
Oct 31 19:38:19.251112 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:19.251085    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl"]
Oct 31 19:38:19.263596 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:19.263573    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-9e323f-openshift-default-dockercfg-wqm4x\""
Oct 31 19:38:19.323742 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:19.323707    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:19.369545 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:19.369512    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d505eb36-3800-4e5b-878d-c7ab58d3e901-workload-socket\") pod \"e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl\" (UID: \"d505eb36-3800-4e5b-878d-c7ab58d3e901\") " pod="openshift-ingress/e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl"
Oct 31 19:38:19.369723 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:19.369563    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d505eb36-3800-4e5b-878d-c7ab58d3e901-istio-data\") pod \"e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl\" (UID: \"d505eb36-3800-4e5b-878d-c7ab58d3e901\") " pod="openshift-ingress/e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl"
Oct 31 19:38:19.369778 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:19.369757    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d505eb36-3800-4e5b-878d-c7ab58d3e901-credential-socket\") pod \"e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl\" (UID: \"d505eb36-3800-4e5b-878d-c7ab58d3e901\") " pod="openshift-ingress/e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl"
Oct 31 19:38:19.369817 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:19.369801    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d505eb36-3800-4e5b-878d-c7ab58d3e901-istio-podinfo\") pod \"e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl\" (UID: \"d505eb36-3800-4e5b-878d-c7ab58d3e901\") " pod="openshift-ingress/e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl"
Oct 31 19:38:19.369862 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:19.369849    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d505eb36-3800-4e5b-878d-c7ab58d3e901-workload-certs\") pod \"e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl\" (UID: \"d505eb36-3800-4e5b-878d-c7ab58d3e901\") " pod="openshift-ingress/e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl"
Oct 31 19:38:19.369913 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:19.369876    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d505eb36-3800-4e5b-878d-c7ab58d3e901-istiod-ca-cert\") pod \"e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl\" (UID: \"d505eb36-3800-4e5b-878d-c7ab58d3e901\") " pod="openshift-ingress/e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl"
Oct 31 19:38:19.369913 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:19.369904    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d505eb36-3800-4e5b-878d-c7ab58d3e901-istio-envoy\") pod \"e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl\" (UID: \"d505eb36-3800-4e5b-878d-c7ab58d3e901\") " pod="openshift-ingress/e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl"
Oct 31 19:38:19.370048 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:19.370030    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hlz5\" (UniqueName: \"kubernetes.io/projected/d505eb36-3800-4e5b-878d-c7ab58d3e901-kube-api-access-2hlz5\") pod \"e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl\" (UID: \"d505eb36-3800-4e5b-878d-c7ab58d3e901\") " pod="openshift-ingress/e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl"
Oct 31 19:38:19.370098 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:19.370077    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d505eb36-3800-4e5b-878d-c7ab58d3e901-istio-token\") pod \"e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl\" (UID: \"d505eb36-3800-4e5b-878d-c7ab58d3e901\") " pod="openshift-ingress/e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl"
Oct 31 19:38:19.470023 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:19.469988    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:19.471539 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:19.471507    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d505eb36-3800-4e5b-878d-c7ab58d3e901-credential-socket\") pod \"e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl\" (UID: \"d505eb36-3800-4e5b-878d-c7ab58d3e901\") " pod="openshift-ingress/e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl"
Oct 31 19:38:19.471680 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:19.471564    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d505eb36-3800-4e5b-878d-c7ab58d3e901-istio-podinfo\") pod \"e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl\" (UID: \"d505eb36-3800-4e5b-878d-c7ab58d3e901\") " pod="openshift-ingress/e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl"
Oct 31 19:38:19.471680 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:19.471596    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d505eb36-3800-4e5b-878d-c7ab58d3e901-workload-certs\") pod \"e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl\" (UID: \"d505eb36-3800-4e5b-878d-c7ab58d3e901\") " pod="openshift-ingress/e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl"
Oct 31 19:38:19.471680 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:19.471620    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d505eb36-3800-4e5b-878d-c7ab58d3e901-istiod-ca-cert\") pod \"e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl\" (UID: \"d505eb36-3800-4e5b-878d-c7ab58d3e901\") " pod="openshift-ingress/e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl"
Oct 31 19:38:19.471680 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:19.471649    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d505eb36-3800-4e5b-878d-c7ab58d3e901-istio-envoy\") pod \"e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl\" (UID: \"d505eb36-3800-4e5b-878d-c7ab58d3e901\") " pod="openshift-ingress/e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl"
Oct 31 19:38:19.471901 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:19.471729    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hlz5\" (UniqueName: \"kubernetes.io/projected/d505eb36-3800-4e5b-878d-c7ab58d3e901-kube-api-access-2hlz5\") pod \"e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl\" (UID: \"d505eb36-3800-4e5b-878d-c7ab58d3e901\") " pod="openshift-ingress/e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl"
Oct 31 19:38:19.471901 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:19.471787    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d505eb36-3800-4e5b-878d-c7ab58d3e901-istio-token\") pod \"e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl\" (UID: \"d505eb36-3800-4e5b-878d-c7ab58d3e901\") " pod="openshift-ingress/e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl"
Oct 31 19:38:19.472011 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:19.471923    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d505eb36-3800-4e5b-878d-c7ab58d3e901-credential-socket\") pod \"e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl\" (UID: \"d505eb36-3800-4e5b-878d-c7ab58d3e901\") " pod="openshift-ingress/e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl"
Oct 31 19:38:19.472011 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:19.471989    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d505eb36-3800-4e5b-878d-c7ab58d3e901-workload-socket\") pod \"e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl\" (UID: \"d505eb36-3800-4e5b-878d-c7ab58d3e901\") " pod="openshift-ingress/e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl"
Oct 31 19:38:19.472179 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:19.472010    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d505eb36-3800-4e5b-878d-c7ab58d3e901-workload-certs\") pod \"e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl\" (UID: \"d505eb36-3800-4e5b-878d-c7ab58d3e901\") " pod="openshift-ingress/e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl"
Oct 31 19:38:19.472179 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:19.472025    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d505eb36-3800-4e5b-878d-c7ab58d3e901-istio-data\") pod \"e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl\" (UID: \"d505eb36-3800-4e5b-878d-c7ab58d3e901\") " pod="openshift-ingress/e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl"
Oct 31 19:38:19.472610 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:19.472465    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d505eb36-3800-4e5b-878d-c7ab58d3e901-istiod-ca-cert\") pod \"e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl\" (UID: \"d505eb36-3800-4e5b-878d-c7ab58d3e901\") " pod="openshift-ingress/e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl"
Oct 31 19:38:19.472610 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:19.472565    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d505eb36-3800-4e5b-878d-c7ab58d3e901-workload-socket\") pod \"e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl\" (UID: \"d505eb36-3800-4e5b-878d-c7ab58d3e901\") " pod="openshift-ingress/e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl"
Oct 31 19:38:19.472610 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:19.472601    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d505eb36-3800-4e5b-878d-c7ab58d3e901-istio-data\") pod \"e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl\" (UID: \"d505eb36-3800-4e5b-878d-c7ab58d3e901\") " pod="openshift-ingress/e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl"
Oct 31 19:38:19.474413 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:19.474390    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d505eb36-3800-4e5b-878d-c7ab58d3e901-istio-envoy\") pod \"e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl\" (UID: \"d505eb36-3800-4e5b-878d-c7ab58d3e901\") " pod="openshift-ingress/e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl"
Oct 31 19:38:19.474790 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:19.474772    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d505eb36-3800-4e5b-878d-c7ab58d3e901-istio-podinfo\") pod \"e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl\" (UID: \"d505eb36-3800-4e5b-878d-c7ab58d3e901\") " pod="openshift-ingress/e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl"
Oct 31 19:38:19.483519 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:19.483476    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d505eb36-3800-4e5b-878d-c7ab58d3e901-istio-token\") pod \"e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl\" (UID: \"d505eb36-3800-4e5b-878d-c7ab58d3e901\") " pod="openshift-ingress/e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl"
Oct 31 19:38:19.489078 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:19.489003    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hlz5\" (UniqueName: \"kubernetes.io/projected/d505eb36-3800-4e5b-878d-c7ab58d3e901-kube-api-access-2hlz5\") pod \"e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl\" (UID: \"d505eb36-3800-4e5b-878d-c7ab58d3e901\") " pod="openshift-ingress/e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl"
Oct 31 19:38:19.607357 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:19.575853    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl"
Oct 31 19:38:20.079676 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:20.079646    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl"]
Oct 31 19:38:20.080115 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:38:20.079683    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd505eb36_3800_4e5b_878d_c7ab58d3e901.slice/crio-7aeef72655eb463c200e674c8d18cd80fcb718ec52c8f38f20b2170da3feb8ad WatchSource:0}: Error finding container 7aeef72655eb463c200e674c8d18cd80fcb718ec52c8f38f20b2170da3feb8ad: Status 404 returned error can't find the container with id 7aeef72655eb463c200e674c8d18cd80fcb718ec52c8f38f20b2170da3feb8ad
Oct 31 19:38:20.082781 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:20.082749    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:38:20.082872 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:20.082811    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:38:20.082872 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:20.082843    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:38:20.469022 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:20.468981    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:20.471012 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:20.470975    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl" event={"ID":"d505eb36-3800-4e5b-878d-c7ab58d3e901","Type":"ContainerStarted","Data":"23ffc973c645e851acd83dcea2e1d7f0ebd7e78d52fece4dc3fe4714de37d73e"}
Oct 31 19:38:20.471169 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:20.471076    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl" event={"ID":"d505eb36-3800-4e5b-878d-c7ab58d3e901","Type":"ContainerStarted","Data":"7aeef72655eb463c200e674c8d18cd80fcb718ec52c8f38f20b2170da3feb8ad"}
Oct 31 19:38:20.539871 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:20.539802    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl" podStartSLOduration=1.539782573 podStartE2EDuration="1.539782573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:38:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:38:20.504530108 +0000 UTC m=+1131.077764265" watchObservedRunningTime="2026-10-31 19:38:20.539782573 +0000 UTC m=+1131.113016705"
Oct 31 19:38:20.579722 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:20.579688    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl"
Oct 31 19:38:20.686437 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:20.686397    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:21.236362 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:21.236316    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:21.367540 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:21.367501    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:21.594187 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:21.594104    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl"
Oct 31 19:38:21.614522 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:21.614484    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:22.543024 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:22.542992    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl"
Oct 31 19:38:22.543791 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:22.543772    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl"
Oct 31 19:38:22.729547 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:22.729518    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:23.049363 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:23.049294    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:23.221057 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:23.221027    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:24.002577 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:24.002536    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:24.444725 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:24.444692    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:24.602555 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:24.602518    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:25.586405 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:25.571215    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:25.709910 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:25.709861    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:27.121894 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:27.121854    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:27.331871 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:27.331837    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:27.425898 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:27.425861    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-4bac5a-78664677cc-6shsd"]
Oct 31 19:38:27.447226 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:27.447201    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4bac5a-78664677cc-6shsd"
Oct 31 19:38:27.449466 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:27.449429    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-4bac5a-78664677cc-6shsd"]
Oct 31 19:38:27.463354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:27.463307    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:27.529351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:27.525691    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-4bac5a-dockercfg-zlbv2\""
Oct 31 19:38:27.529351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:27.525787    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-4bac5a\""
Oct 31 19:38:27.535454 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:27.531060    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89xtb\" (UniqueName: \"kubernetes.io/projected/aa76773f-998f-408d-a804-31f6c4921774-kube-api-access-89xtb\") pod \"payload-processing-e2e-shared-a-w2-4bac5a-78664677cc-6shsd\" (UID: \"aa76773f-998f-408d-a804-31f6c4921774\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4bac5a-78664677cc-6shsd"
Oct 31 19:38:27.537192 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:27.537165    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa76773f-998f-408d-a804-31f6c4921774-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-4bac5a-78664677cc-6shsd\" (UID: \"aa76773f-998f-408d-a804-31f6c4921774\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4bac5a-78664677cc-6shsd"
Oct 31 19:38:27.695814 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:27.695736    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89xtb\" (UniqueName: \"kubernetes.io/projected/aa76773f-998f-408d-a804-31f6c4921774-kube-api-access-89xtb\") pod \"payload-processing-e2e-shared-a-w2-4bac5a-78664677cc-6shsd\" (UID: \"aa76773f-998f-408d-a804-31f6c4921774\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4bac5a-78664677cc-6shsd"
Oct 31 19:38:27.696191 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:27.696152    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa76773f-998f-408d-a804-31f6c4921774-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-4bac5a-78664677cc-6shsd\" (UID: \"aa76773f-998f-408d-a804-31f6c4921774\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4bac5a-78664677cc-6shsd"
Oct 31 19:38:27.697595 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:27.697559    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa76773f-998f-408d-a804-31f6c4921774-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-4bac5a-78664677cc-6shsd\" (UID: \"aa76773f-998f-408d-a804-31f6c4921774\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4bac5a-78664677cc-6shsd"
Oct 31 19:38:27.792492 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:27.792457    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89xtb\" (UniqueName: \"kubernetes.io/projected/aa76773f-998f-408d-a804-31f6c4921774-kube-api-access-89xtb\") pod \"payload-processing-e2e-shared-a-w2-4bac5a-78664677cc-6shsd\" (UID: \"aa76773f-998f-408d-a804-31f6c4921774\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4bac5a-78664677cc-6shsd"
Oct 31 19:38:28.068248 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:28.068162    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4bac5a-78664677cc-6shsd"
Oct 31 19:38:28.356468 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:38:28.356434    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa76773f_998f_408d_a804_31f6c4921774.slice/crio-c657e70ab77bb8468a78a3e1946667bafea487d69899a3a1f473ed0119c424eb WatchSource:0}: Error finding container c657e70ab77bb8468a78a3e1946667bafea487d69899a3a1f473ed0119c424eb: Status 404 returned error can't find the container with id c657e70ab77bb8468a78a3e1946667bafea487d69899a3a1f473ed0119c424eb
Oct 31 19:38:28.356865 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:28.356583    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-4bac5a-78664677cc-6shsd"]
Oct 31 19:38:28.461702 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:28.461674    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:28.609685 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:28.609599    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:28.822853 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:28.822819    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4bac5a-78664677cc-6shsd" event={"ID":"aa76773f-998f-408d-a804-31f6c4921774","Type":"ContainerStarted","Data":"e88fd7cd9d0e1fc554eb036fb0aa37effec4b66817bff1b01a519b8a943229d2"}
Oct 31 19:38:28.823021 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:28.822869    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4bac5a-78664677cc-6shsd" event={"ID":"aa76773f-998f-408d-a804-31f6c4921774","Type":"ContainerStarted","Data":"c657e70ab77bb8468a78a3e1946667bafea487d69899a3a1f473ed0119c424eb"}
Oct 31 19:38:28.823021 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:28.822951    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4bac5a-78664677cc-6shsd"
Oct 31 19:38:28.857296 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:28.857238    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4bac5a-78664677cc-6shsd" podStartSLOduration=1.8572237870000001 podStartE2EDuration="1.857223787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:38:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:38:28.855495717 +0000 UTC m=+1139.428729845" watchObservedRunningTime="2026-10-31 19:38:28.857223787 +0000 UTC m=+1139.430457916"
Oct 31 19:38:29.867390 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:29.867351    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:30.677922 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:30.677880    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:31.507928 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:31.507890    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:32.282972 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:32.282939    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:32.423088 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:32.423029    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:33.512989 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:33.512952    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:34.210180 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:34.210142    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:35.056842 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:35.056806    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:35.840803 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:35.840762    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:35.962349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:35.955842    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:36.249619 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:38:36.249578    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 19:38:36.669591 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:36.669553    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:37.546487 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:37.546434    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:37.666815 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:37.666769    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:37.787990 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:37.787954    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:38.645669 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:38.645638    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:39.517516 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:39.517476    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:39.834696 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:39.834621    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4bac5a-78664677cc-6shsd"
Oct 31 19:38:40.379628 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:40.379597    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:40.523957 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:40.523921    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:41.962742 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:41.962707    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:43.049499 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:43.046121    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:43.871129 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:43.871096    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7011-1dcfd8"]
Oct 31 19:38:43.875932 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:43.875912    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7011-1dcfd8"
Oct 31 19:38:43.881867 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:43.881841    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-4pmld\""
Oct 31 19:38:43.896677 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:43.896650    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7011-1dcfd8"]
Oct 31 19:38:43.928227 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:43.928199    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:43.964600 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:43.959915    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2swx\" (UniqueName: \"kubernetes.io/projected/154e68d0-2cec-45ed-95f6-26a6e8f4056a-kube-api-access-f2swx\") pod \"test-curl-7011-1dcfd8\" (UID: \"154e68d0-2cec-45ed-95f6-26a6e8f4056a\") " pod="opendatahub/test-curl-7011-1dcfd8"
Oct 31 19:38:44.073683 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:44.073643    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2swx\" (UniqueName: \"kubernetes.io/projected/154e68d0-2cec-45ed-95f6-26a6e8f4056a-kube-api-access-f2swx\") pod \"test-curl-7011-1dcfd8\" (UID: \"154e68d0-2cec-45ed-95f6-26a6e8f4056a\") " pod="opendatahub/test-curl-7011-1dcfd8"
Oct 31 19:38:44.086144 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:44.086120    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2swx\" (UniqueName: \"kubernetes.io/projected/154e68d0-2cec-45ed-95f6-26a6e8f4056a-kube-api-access-f2swx\") pod \"test-curl-7011-1dcfd8\" (UID: \"154e68d0-2cec-45ed-95f6-26a6e8f4056a\") " pod="opendatahub/test-curl-7011-1dcfd8"
Oct 31 19:38:44.199098 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:44.199076    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7011-1dcfd8"
Oct 31 19:38:44.440125 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:44.440097    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:44.477213 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:38:44.477185    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod154e68d0_2cec_45ed_95f6_26a6e8f4056a.slice/crio-2d38ae6b7ccbabd45adb06598317776edcbcfae66930247d7dea5605c368e911 WatchSource:0}: Error finding container 2d38ae6b7ccbabd45adb06598317776edcbcfae66930247d7dea5605c368e911: Status 404 returned error can't find the container with id 2d38ae6b7ccbabd45adb06598317776edcbcfae66930247d7dea5605c368e911
Oct 31 19:38:44.477864 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:44.477843    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7011-1dcfd8"]
Oct 31 19:38:44.748429 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:44.748317    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:44.941160 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:44.941120    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7011-1dcfd8" event={"ID":"154e68d0-2cec-45ed-95f6-26a6e8f4056a","Type":"ContainerStarted","Data":"77fbafcef66bb696b603f842d8c9ea61cb9b23fd6a110242ee9972e6d3695d95"}
Oct 31 19:38:44.941363 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:44.941171    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7011-1dcfd8" event={"ID":"154e68d0-2cec-45ed-95f6-26a6e8f4056a","Type":"ContainerStarted","Data":"2d38ae6b7ccbabd45adb06598317776edcbcfae66930247d7dea5605c368e911"}
Oct 31 19:38:44.958585 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:44.958537    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7011-1dcfd8" podStartSLOduration=1.958523571 podStartE2EDuration="1.958523571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:38:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:38:44.955931828 +0000 UTC m=+1155.529165957" watchObservedRunningTime="2026-10-31 19:38:44.958523571 +0000 UTC m=+1155.531757701"
Oct 31 19:38:45.554404 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:45.554313    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:45.666506 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:45.666472    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:45.702171 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:45.702139    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7011-1dcfd8"]
Oct 31 19:38:45.708183 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:45.708149    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7011-1dcfd8"]
Oct 31 19:38:46.393394 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:46.393287    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:46.954800 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:46.954765    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7011-1dcfd8" podUID="154e68d0-2cec-45ed-95f6-26a6e8f4056a" containerName="test-curl-7011-1dcfd8" containerID="cri-o://77fbafcef66bb696b603f842d8c9ea61cb9b23fd6a110242ee9972e6d3695d95" gracePeriod=2
Oct 31 19:38:46.958012 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:38:46.957982    4645 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7011-1dcfd8\" is forbidden: User \"system:node:ip-10-0-131-94.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-94.us-east-2.compute.internal' and this object" podUID="154e68d0-2cec-45ed-95f6-26a6e8f4056a" pod="opendatahub/test-curl-7011-1dcfd8"
Oct 31 19:38:47.389172 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:47.389067    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:48.639120 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:48.639077    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:49.179745 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:49.179722    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7011-1dcfd8"
Oct 31 19:38:49.231375 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:49.231308    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/154e68d0-2cec-45ed-95f6-26a6e8f4056a-kube-api-access-f2swx\" (UniqueName: \"kubernetes.io/projected/154e68d0-2cec-45ed-95f6-26a6e8f4056a-kube-api-access-f2swx\") pod \"154e68d0-2cec-45ed-95f6-26a6e8f4056a\" (UID: \"154e68d0-2cec-45ed-95f6-26a6e8f4056a\") "
Oct 31 19:38:49.233372 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:49.233340    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/154e68d0-2cec-45ed-95f6-26a6e8f4056a-kube-api-access-f2swx" pod "154e68d0-2cec-45ed-95f6-26a6e8f4056a" (UID: "154e68d0-2cec-45ed-95f6-26a6e8f4056a"). InnerVolumeSpecName "kube-api-access-f2swx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:38:49.333077 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:49.333054    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2swx\" (UniqueName: \"kubernetes.io/projected/154e68d0-2cec-45ed-95f6-26a6e8f4056a-kube-api-access-f2swx\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:38:49.688378 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:49.688349    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="154e68d0-2cec-45ed-95f6-26a6e8f4056a" path="/var/lib/kubelet/pods/154e68d0-2cec-45ed-95f6-26a6e8f4056a/volumes"
Oct 31 19:38:50.054403 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:50.054281    4645 generic.go:356] "Generic (PLEG): container finished" podID="154e68d0-2cec-45ed-95f6-26a6e8f4056a" containerID="77fbafcef66bb696b603f842d8c9ea61cb9b23fd6a110242ee9972e6d3695d95" exitCode=137
Oct 31 19:38:50.054729 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:50.054708    4645 scope.go:122] "RemoveContainer" containerID="77fbafcef66bb696b603f842d8c9ea61cb9b23fd6a110242ee9972e6d3695d95"
Oct 31 19:38:50.055048 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:50.055026    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7011-1dcfd8"
Oct 31 19:38:50.207549 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:50.203858    4645 scope.go:122] "RemoveContainer" containerID="77fbafcef66bb696b603f842d8c9ea61cb9b23fd6a110242ee9972e6d3695d95"
Oct 31 19:38:50.232689 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:38:50.231427    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77fbafcef66bb696b603f842d8c9ea61cb9b23fd6a110242ee9972e6d3695d95\": container with ID starting with 77fbafcef66bb696b603f842d8c9ea61cb9b23fd6a110242ee9972e6d3695d95 not found: ID does not exist" containerID="77fbafcef66bb696b603f842d8c9ea61cb9b23fd6a110242ee9972e6d3695d95"
Oct 31 19:38:50.232689 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:50.231485    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77fbafcef66bb696b603f842d8c9ea61cb9b23fd6a110242ee9972e6d3695d95"} err="failed to get container status \"77fbafcef66bb696b603f842d8c9ea61cb9b23fd6a110242ee9972e6d3695d95\": rpc error: code = NotFound desc = could not find container \"77fbafcef66bb696b603f842d8c9ea61cb9b23fd6a110242ee9972e6d3695d95\": container with ID starting with 77fbafcef66bb696b603f842d8c9ea61cb9b23fd6a110242ee9972e6d3695d95 not found: ID does not exist"
Oct 31 19:38:50.533134 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:50.533099    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:51.563354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:51.562947    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:52.467353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:52.464347    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:53.137353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:53.132594    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw" podUID="1287db54-a587-4c11-9eb6-1606de555894" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:38:53.171350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:53.170769    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x" podUID="a774e286-dcb8-4a2f-8541-26bf231eb08d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 19:38:53.212350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:53.211537    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4" podUID="6032a997-b795-4561-805c-110cf67a9d5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:38:53.226351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:53.225391    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw" podUID="d504b2d3-c26d-4dcf-a7eb-efe20732ae4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:38:53.241855 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:53.238393    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669" podUID="9285bee0-bf7c-4fd4-9da7-a1db021cae05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:38:53.345349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:53.344940    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l" podUID="12bea86c-61e5-465e-82fb-b4673a4d1be9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:38:53.467354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:53.463349    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:53.629922 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:53.629884    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:54.289113 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:54.289073    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:54.587555 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:54.587424    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:54.781221 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:54.781176    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:55.277812 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:55.277772    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:55.544513 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:55.544422    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:56.531605 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:56.531552    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-67adab47-998479859-kmf5v"]
Oct 31 19:38:56.720432 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:56.720393    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:56.847636 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:56.847547    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:56.971782 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:56.971745    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:57.879823 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:57.879784    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:58.552218 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:58.552179    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:59.076800 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:59.076763    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:59.294430 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:59.294394    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:59.703984 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:59.703941    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw"]
Oct 31 19:38:59.709208 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:59.709166    4645 cpu_manager.go:414] "containerMap: removing container" podUID="154e68d0-2cec-45ed-95f6-26a6e8f4056a" containerName="test-curl-7011-1dcfd8"
Oct 31 19:38:59.709208 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:59.709200    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="154e68d0-2cec-45ed-95f6-26a6e8f4056a" containerName="test-curl-7011-1dcfd8"
Oct 31 19:38:59.714506 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:59.714479    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="154e68d0-2cec-45ed-95f6-26a6e8f4056a" containerName="test-curl-7011-1dcfd8"
Oct 31 19:38:59.728439 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:59.728394    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw"
Oct 31 19:38:59.733283 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:59.733254    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw"]
Oct 31 19:38:59.738574 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:59.738543    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-d4b21c-openshift-default-dockercfg-k8zbt\""
Oct 31 19:38:59.774389 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:59.774353    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:59.868181 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:59.850707    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qjv2\" (UniqueName: \"kubernetes.io/projected/585024d3-5768-463e-acdc-81151a7c0461-kube-api-access-6qjv2\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw\" (UID: \"585024d3-5768-463e-acdc-81151a7c0461\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw"
Oct 31 19:38:59.868652 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:59.868625    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/585024d3-5768-463e-acdc-81151a7c0461-istio-token\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw\" (UID: \"585024d3-5768-463e-acdc-81151a7c0461\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw"
Oct 31 19:38:59.868912 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:59.868891    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/585024d3-5768-463e-acdc-81151a7c0461-workload-certs\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw\" (UID: \"585024d3-5768-463e-acdc-81151a7c0461\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw"
Oct 31 19:38:59.869092 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:59.869079    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/585024d3-5768-463e-acdc-81151a7c0461-istiod-ca-cert\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw\" (UID: \"585024d3-5768-463e-acdc-81151a7c0461\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw"
Oct 31 19:38:59.869428 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:59.869402    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/585024d3-5768-463e-acdc-81151a7c0461-credential-socket\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw\" (UID: \"585024d3-5768-463e-acdc-81151a7c0461\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw"
Oct 31 19:38:59.869712 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:59.869690    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/585024d3-5768-463e-acdc-81151a7c0461-istio-envoy\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw\" (UID: \"585024d3-5768-463e-acdc-81151a7c0461\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw"
Oct 31 19:38:59.869919 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:59.869903    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/585024d3-5768-463e-acdc-81151a7c0461-workload-socket\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw\" (UID: \"585024d3-5768-463e-acdc-81151a7c0461\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw"
Oct 31 19:38:59.870257 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:59.870081    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/585024d3-5768-463e-acdc-81151a7c0461-istio-data\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw\" (UID: \"585024d3-5768-463e-acdc-81151a7c0461\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw"
Oct 31 19:38:59.870542 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:59.870517    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/585024d3-5768-463e-acdc-81151a7c0461-istio-podinfo\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw\" (UID: \"585024d3-5768-463e-acdc-81151a7c0461\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw"
Oct 31 19:38:59.898929 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:59.898895    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:38:59.984446 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:59.983594    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/585024d3-5768-463e-acdc-81151a7c0461-istio-data\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw\" (UID: \"585024d3-5768-463e-acdc-81151a7c0461\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw"
Oct 31 19:38:59.984446 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:59.983657    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/585024d3-5768-463e-acdc-81151a7c0461-istio-podinfo\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw\" (UID: \"585024d3-5768-463e-acdc-81151a7c0461\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw"
Oct 31 19:38:59.984446 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:59.983717    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qjv2\" (UniqueName: \"kubernetes.io/projected/585024d3-5768-463e-acdc-81151a7c0461-kube-api-access-6qjv2\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw\" (UID: \"585024d3-5768-463e-acdc-81151a7c0461\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw"
Oct 31 19:38:59.984446 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:59.983846    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/585024d3-5768-463e-acdc-81151a7c0461-istio-token\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw\" (UID: \"585024d3-5768-463e-acdc-81151a7c0461\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw"
Oct 31 19:38:59.984446 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:59.983913    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/585024d3-5768-463e-acdc-81151a7c0461-workload-certs\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw\" (UID: \"585024d3-5768-463e-acdc-81151a7c0461\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw"
Oct 31 19:38:59.984446 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:59.983966    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/585024d3-5768-463e-acdc-81151a7c0461-istiod-ca-cert\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw\" (UID: \"585024d3-5768-463e-acdc-81151a7c0461\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw"
Oct 31 19:38:59.984446 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:59.984125    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/585024d3-5768-463e-acdc-81151a7c0461-credential-socket\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw\" (UID: \"585024d3-5768-463e-acdc-81151a7c0461\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw"
Oct 31 19:38:59.984446 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:59.984213    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/585024d3-5768-463e-acdc-81151a7c0461-istio-envoy\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw\" (UID: \"585024d3-5768-463e-acdc-81151a7c0461\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw"
Oct 31 19:38:59.984446 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:59.984285    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/585024d3-5768-463e-acdc-81151a7c0461-workload-socket\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw\" (UID: \"585024d3-5768-463e-acdc-81151a7c0461\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw"
Oct 31 19:38:59.985242 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:59.985218    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/585024d3-5768-463e-acdc-81151a7c0461-istio-data\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw\" (UID: \"585024d3-5768-463e-acdc-81151a7c0461\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw"
Oct 31 19:38:59.986231 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:59.986211    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/585024d3-5768-463e-acdc-81151a7c0461-workload-socket\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw\" (UID: \"585024d3-5768-463e-acdc-81151a7c0461\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw"
Oct 31 19:38:59.987211 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:59.987189    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/585024d3-5768-463e-acdc-81151a7c0461-istiod-ca-cert\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw\" (UID: \"585024d3-5768-463e-acdc-81151a7c0461\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw"
Oct 31 19:38:59.996091 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:59.996049    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/585024d3-5768-463e-acdc-81151a7c0461-workload-certs\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw\" (UID: \"585024d3-5768-463e-acdc-81151a7c0461\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw"
Oct 31 19:38:59.996673 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:38:59.996638    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/585024d3-5768-463e-acdc-81151a7c0461-credential-socket\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw\" (UID: \"585024d3-5768-463e-acdc-81151a7c0461\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw"
Oct 31 19:39:00.012423 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:00.004001    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/585024d3-5768-463e-acdc-81151a7c0461-istio-envoy\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw\" (UID: \"585024d3-5768-463e-acdc-81151a7c0461\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw"
Oct 31 19:39:00.013130 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:00.013097    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/585024d3-5768-463e-acdc-81151a7c0461-istio-token\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw\" (UID: \"585024d3-5768-463e-acdc-81151a7c0461\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw"
Oct 31 19:39:00.017356 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:00.016573    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/585024d3-5768-463e-acdc-81151a7c0461-istio-podinfo\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw\" (UID: \"585024d3-5768-463e-acdc-81151a7c0461\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw"
Oct 31 19:39:00.046355 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:00.043086    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qjv2\" (UniqueName: \"kubernetes.io/projected/585024d3-5768-463e-acdc-81151a7c0461-kube-api-access-6qjv2\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw\" (UID: \"585024d3-5768-463e-acdc-81151a7c0461\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw"
Oct 31 19:39:00.052407 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:00.050006    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw"
Oct 31 19:39:00.434990 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:00.434960    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw"]
Oct 31 19:39:00.436105 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:39:00.436081    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod585024d3_5768_463e_acdc_81151a7c0461.slice/crio-36b1612862d08cd8192297d61592f0bb2ca942e01474d1a472abb551f8ecc7a8 WatchSource:0}: Error finding container 36b1612862d08cd8192297d61592f0bb2ca942e01474d1a472abb551f8ecc7a8: Status 404 returned error can't find the container with id 36b1612862d08cd8192297d61592f0bb2ca942e01474d1a472abb551f8ecc7a8
Oct 31 19:39:00.437731 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:00.437704    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:00.438974 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:00.438943    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:39:00.439057 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:00.439015    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:39:00.439057 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:00.439047    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:39:00.982189 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:00.982159    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:01.243059 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:01.242974    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:01.265818 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:01.265777    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw" event={"ID":"585024d3-5768-463e-acdc-81151a7c0461","Type":"ContainerStarted","Data":"a6e1edb71ab92a73b91ed16fb9102cc4540988ed9ded70c1dff5c5400f189265"}
Oct 31 19:39:01.266005 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:01.265846    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw" event={"ID":"585024d3-5768-463e-acdc-81151a7c0461","Type":"ContainerStarted","Data":"36b1612862d08cd8192297d61592f0bb2ca942e01474d1a472abb551f8ecc7a8"}
Oct 31 19:39:01.309832 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:01.309765    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw" podStartSLOduration=2.309747615 podStartE2EDuration="2.309747615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:38:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:39:01.304956697 +0000 UTC m=+1171.878190857" watchObservedRunningTime="2026-10-31 19:39:01.309747615 +0000 UTC m=+1171.882981743"
Oct 31 19:39:01.496753 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:01.496660    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:01.532172 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:01.532118    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-67adab47-998479859-kmf5v" podUID="03810b83-fe09-4d95-a374-f6d7189954b4" containerName="payload-pre-processing" containerID="cri-o://d91fbd860d0b28690210b1fdde22dc604a83c979372b15cc89537dc16416c398" gracePeriod=25
Oct 31 19:39:01.955135 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:01.955102    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:02.067705 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:02.067676    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw"
Oct 31 19:39:02.100424 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:02.099648    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:02.170350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:02.153689    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw"
Oct 31 19:39:02.225409 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:02.224971    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:02.310353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:02.310144    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw"
Oct 31 19:39:02.336349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:02.331646    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw"
Oct 31 19:39:02.978720 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:02.978674    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:03.360450 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:03.360370    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:03.680060 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:03.680026    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:03.815007 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:03.814960    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:04.762490 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:04.762454    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:05.065841 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:05.065760    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d4b21c-6bf64967d7-rj87s"]
Oct 31 19:39:05.070453 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:05.070431    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d4b21c-6bf64967d7-rj87s"
Oct 31 19:39:05.073983 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:05.073959    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d4b21c-764698f6f-dt2q6"]
Oct 31 19:39:05.078530 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:05.078508    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d4b21c-764698f6f-dt2q6"
Oct 31 19:39:05.079116 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:05.079093    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-d4b21c\""
Oct 31 19:39:05.079366 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:05.079206    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-d4b21c-dockercfg-xpbbd\""
Oct 31 19:39:05.084295 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:05.084264    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d4b21c-6bf64967d7-rj87s"]
Oct 31 19:39:05.090323 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:05.090298    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d4b21c-764698f6f-dt2q6"]
Oct 31 19:39:05.217433 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:05.202918    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef5a2c9c-2e89-4945-a70d-8eea47d0b517-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d4b21c-6bf64967d7-rj87s\" (UID: \"ef5a2c9c-2e89-4945-a70d-8eea47d0b517\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d4b21c-6bf64967d7-rj87s"
Oct 31 19:39:05.229461 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:05.217765    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pr8c\" (UniqueName: \"kubernetes.io/projected/49ea0b50-2dfa-435c-9940-98e57cbb6aa4-kube-api-access-5pr8c\") pod \"payload-processing-e2e-api-b-d4b21c-764698f6f-dt2q6\" (UID: \"49ea0b50-2dfa-435c-9940-98e57cbb6aa4\") " pod="openshift-ingress/payload-processing-e2e-api-b-d4b21c-764698f6f-dt2q6"
Oct 31 19:39:05.230023 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:05.229989    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgdk8\" (UniqueName: \"kubernetes.io/projected/ef5a2c9c-2e89-4945-a70d-8eea47d0b517-kube-api-access-mgdk8\") pod \"payload-pre-processing-e2e-api-b-d4b21c-6bf64967d7-rj87s\" (UID: \"ef5a2c9c-2e89-4945-a70d-8eea47d0b517\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d4b21c-6bf64967d7-rj87s"
Oct 31 19:39:05.233348 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:05.230202    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49ea0b50-2dfa-435c-9940-98e57cbb6aa4-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d4b21c-764698f6f-dt2q6\" (UID: \"49ea0b50-2dfa-435c-9940-98e57cbb6aa4\") " pod="openshift-ingress/payload-processing-e2e-api-b-d4b21c-764698f6f-dt2q6"
Oct 31 19:39:05.356576 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:05.356482    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49ea0b50-2dfa-435c-9940-98e57cbb6aa4-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d4b21c-764698f6f-dt2q6\" (UID: \"49ea0b50-2dfa-435c-9940-98e57cbb6aa4\") " pod="openshift-ingress/payload-processing-e2e-api-b-d4b21c-764698f6f-dt2q6"
Oct 31 19:39:05.367353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:05.357888    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef5a2c9c-2e89-4945-a70d-8eea47d0b517-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d4b21c-6bf64967d7-rj87s\" (UID: \"ef5a2c9c-2e89-4945-a70d-8eea47d0b517\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d4b21c-6bf64967d7-rj87s"
Oct 31 19:39:05.369376 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:05.369347    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pr8c\" (UniqueName: \"kubernetes.io/projected/49ea0b50-2dfa-435c-9940-98e57cbb6aa4-kube-api-access-5pr8c\") pod \"payload-processing-e2e-api-b-d4b21c-764698f6f-dt2q6\" (UID: \"49ea0b50-2dfa-435c-9940-98e57cbb6aa4\") " pod="openshift-ingress/payload-processing-e2e-api-b-d4b21c-764698f6f-dt2q6"
Oct 31 19:39:05.384878 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:05.369833    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgdk8\" (UniqueName: \"kubernetes.io/projected/ef5a2c9c-2e89-4945-a70d-8eea47d0b517-kube-api-access-mgdk8\") pod \"payload-pre-processing-e2e-api-b-d4b21c-6bf64967d7-rj87s\" (UID: \"ef5a2c9c-2e89-4945-a70d-8eea47d0b517\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d4b21c-6bf64967d7-rj87s"
Oct 31 19:39:05.386155 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:05.357638    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49ea0b50-2dfa-435c-9940-98e57cbb6aa4-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d4b21c-764698f6f-dt2q6\" (UID: \"49ea0b50-2dfa-435c-9940-98e57cbb6aa4\") " pod="openshift-ingress/payload-processing-e2e-api-b-d4b21c-764698f6f-dt2q6"
Oct 31 19:39:05.386346 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:05.361400    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef5a2c9c-2e89-4945-a70d-8eea47d0b517-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d4b21c-6bf64967d7-rj87s\" (UID: \"ef5a2c9c-2e89-4945-a70d-8eea47d0b517\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d4b21c-6bf64967d7-rj87s"
Oct 31 19:39:05.439211 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:05.439174    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pr8c\" (UniqueName: \"kubernetes.io/projected/49ea0b50-2dfa-435c-9940-98e57cbb6aa4-kube-api-access-5pr8c\") pod \"payload-processing-e2e-api-b-d4b21c-764698f6f-dt2q6\" (UID: \"49ea0b50-2dfa-435c-9940-98e57cbb6aa4\") " pod="openshift-ingress/payload-processing-e2e-api-b-d4b21c-764698f6f-dt2q6"
Oct 31 19:39:05.440091 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:05.440016    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgdk8\" (UniqueName: \"kubernetes.io/projected/ef5a2c9c-2e89-4945-a70d-8eea47d0b517-kube-api-access-mgdk8\") pod \"payload-pre-processing-e2e-api-b-d4b21c-6bf64967d7-rj87s\" (UID: \"ef5a2c9c-2e89-4945-a70d-8eea47d0b517\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d4b21c-6bf64967d7-rj87s"
Oct 31 19:39:05.607063 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:05.606981    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:05.694684 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:05.694653    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d4b21c-6bf64967d7-rj87s"
Oct 31 19:39:05.711054 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:05.711017    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d4b21c-764698f6f-dt2q6"
Oct 31 19:39:05.986569 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:39:05.986534    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef5a2c9c_2e89_4945_a70d_8eea47d0b517.slice/crio-42265bdca8cce7c2b337598ae8ffb6a056d12b343a7bffc366ce1c2dcd02e528 WatchSource:0}: Error finding container 42265bdca8cce7c2b337598ae8ffb6a056d12b343a7bffc366ce1c2dcd02e528: Status 404 returned error can't find the container with id 42265bdca8cce7c2b337598ae8ffb6a056d12b343a7bffc366ce1c2dcd02e528
Oct 31 19:39:05.987161 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:05.987131    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d4b21c-6bf64967d7-rj87s"]
Oct 31 19:39:06.011096 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:06.011073    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d4b21c-764698f6f-dt2q6"]
Oct 31 19:39:06.011230 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:39:06.011204    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49ea0b50_2dfa_435c_9940_98e57cbb6aa4.slice/crio-3b487d82dfcdd62b92838b747716f5242c1ef1d98f4ef2275b3d672865079daa WatchSource:0}: Error finding container 3b487d82dfcdd62b92838b747716f5242c1ef1d98f4ef2275b3d672865079daa: Status 404 returned error can't find the container with id 3b487d82dfcdd62b92838b747716f5242c1ef1d98f4ef2275b3d672865079daa
Oct 31 19:39:06.106650 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:06.106627    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:06.277642 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:39:06.277561    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 19:39:06.369456 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:06.369427    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:06.386653 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:06.386615    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d4b21c-764698f6f-dt2q6" event={"ID":"49ea0b50-2dfa-435c-9940-98e57cbb6aa4","Type":"ContainerStarted","Data":"2b40fe03181750a4a1389e65a8e88682a3e083c5d4955a733b603f9e2d954778"}
Oct 31 19:39:06.386814 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:06.386677    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-d4b21c-764698f6f-dt2q6"
Oct 31 19:39:06.386814 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:06.386706    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d4b21c-764698f6f-dt2q6" event={"ID":"49ea0b50-2dfa-435c-9940-98e57cbb6aa4","Type":"ContainerStarted","Data":"3b487d82dfcdd62b92838b747716f5242c1ef1d98f4ef2275b3d672865079daa"}
Oct 31 19:39:06.388201 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:06.388175    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d4b21c-6bf64967d7-rj87s" event={"ID":"ef5a2c9c-2e89-4945-a70d-8eea47d0b517","Type":"ContainerStarted","Data":"90ec39e391aaa91fe1831b72bfa7f80b0edf99e521d255b8a67bd6c94f1d9be6"}
Oct 31 19:39:06.388258 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:06.388231    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d4b21c-6bf64967d7-rj87s" event={"ID":"ef5a2c9c-2e89-4945-a70d-8eea47d0b517","Type":"ContainerStarted","Data":"42265bdca8cce7c2b337598ae8ffb6a056d12b343a7bffc366ce1c2dcd02e528"}
Oct 31 19:39:06.388301 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:06.388262    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d4b21c-6bf64967d7-rj87s"
Oct 31 19:39:06.409285 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:06.409243    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-d4b21c-764698f6f-dt2q6" podStartSLOduration=1.40922614 podStartE2EDuration="1.40922614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:39:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:39:06.407290654 +0000 UTC m=+1176.980524784" watchObservedRunningTime="2026-10-31 19:39:06.40922614 +0000 UTC m=+1176.982460314"
Oct 31 19:39:06.432202 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:06.432151    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d4b21c-6bf64967d7-rj87s" podStartSLOduration=1.432139748 podStartE2EDuration="1.432139748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:39:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:39:06.430486136 +0000 UTC m=+1177.003720264" watchObservedRunningTime="2026-10-31 19:39:06.432139748 +0000 UTC m=+1177.005373877"
Oct 31 19:39:06.498944 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:06.498913    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:07.660892 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:07.660860    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:07.804381 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:07.799637    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:07.939004 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:07.938922    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:09.034013 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:09.033982    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:09.822360 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:09.822271    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:10.681498 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:10.681279    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:11.262823 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:11.262786    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-67adab47-998479859-kmf5v" podUID="03810b83-fe09-4d95-a374-f6d7189954b4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:39:11.513230 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:11.513141    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:12.257491 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:12.257452    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:13.052456 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:13.052417    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:13.798343 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:13.798185    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:14.405369 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:14.405317    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-67adab47-6cd87d9bdb-kwrg8"]
Oct 31 19:39:14.409348 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:14.406187    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-67adab47-6cd87d9bdb-kwrg8" podUID="fe564e23-ad21-4f6f-a82f-69d549102bb9" containerName="maas-api" containerID="cri-o://e454c057fc8adbc7ee5cfaf3eeb6cb5f7e1c70a35cdbddfd89d51931b19285a4" gracePeriod=30
Oct 31 19:39:14.455751 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:14.455720    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:14.869396 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:14.855239    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:14.999363 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:14.989456    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:15.067275 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:15.067238    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-67adab47-6cd87d9bdb-kwrg8"
Oct 31 19:39:15.114370 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:15.114296    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:15.220023 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:15.219986    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe564e23-ad21-4f6f-a82f-69d549102bb9-kube-api-access-9s9mv\" (UniqueName: \"kubernetes.io/projected/fe564e23-ad21-4f6f-a82f-69d549102bb9-kube-api-access-9s9mv\") pod \"fe564e23-ad21-4f6f-a82f-69d549102bb9\" (UID: \"fe564e23-ad21-4f6f-a82f-69d549102bb9\") "
Oct 31 19:39:15.220306 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:15.220270    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe564e23-ad21-4f6f-a82f-69d549102bb9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe564e23-ad21-4f6f-a82f-69d549102bb9-openshift-service-ca\") pod \"fe564e23-ad21-4f6f-a82f-69d549102bb9\" (UID: \"fe564e23-ad21-4f6f-a82f-69d549102bb9\") "
Oct 31 19:39:15.220466 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:15.220375    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe564e23-ad21-4f6f-a82f-69d549102bb9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe564e23-ad21-4f6f-a82f-69d549102bb9-metrics-certs\") pod \"fe564e23-ad21-4f6f-a82f-69d549102bb9\" (UID: \"fe564e23-ad21-4f6f-a82f-69d549102bb9\") "
Oct 31 19:39:15.221826 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:15.220765    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe564e23-ad21-4f6f-a82f-69d549102bb9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe564e23-ad21-4f6f-a82f-69d549102bb9-maas-api-tls\") pod \"fe564e23-ad21-4f6f-a82f-69d549102bb9\" (UID: \"fe564e23-ad21-4f6f-a82f-69d549102bb9\") "
Oct 31 19:39:15.222751 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:15.222723    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe564e23-ad21-4f6f-a82f-69d549102bb9-openshift-service-ca" pod "fe564e23-ad21-4f6f-a82f-69d549102bb9" (UID: "fe564e23-ad21-4f6f-a82f-69d549102bb9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:39:15.234847 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:15.234815    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe564e23-ad21-4f6f-a82f-69d549102bb9-metrics-certs" pod "fe564e23-ad21-4f6f-a82f-69d549102bb9" (UID: "fe564e23-ad21-4f6f-a82f-69d549102bb9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:39:15.234847 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:15.234825    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe564e23-ad21-4f6f-a82f-69d549102bb9-kube-api-access-9s9mv" pod "fe564e23-ad21-4f6f-a82f-69d549102bb9" (UID: "fe564e23-ad21-4f6f-a82f-69d549102bb9"). InnerVolumeSpecName "kube-api-access-9s9mv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:39:15.235089 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:15.235074    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe564e23-ad21-4f6f-a82f-69d549102bb9-maas-api-tls" pod "fe564e23-ad21-4f6f-a82f-69d549102bb9" (UID: "fe564e23-ad21-4f6f-a82f-69d549102bb9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:39:15.323674 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:15.323640    4645 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe564e23-ad21-4f6f-a82f-69d549102bb9-openshift-service-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:39:15.323818 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:15.323681    4645 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe564e23-ad21-4f6f-a82f-69d549102bb9-metrics-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:39:15.323818 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:15.323695    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe564e23-ad21-4f6f-a82f-69d549102bb9-maas-api-tls\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:39:15.323818 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:15.323709    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9s9mv\" (UniqueName: \"kubernetes.io/projected/fe564e23-ad21-4f6f-a82f-69d549102bb9-kube-api-access-9s9mv\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:39:15.457156 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:15.457121    4645 generic.go:356] "Generic (PLEG): container finished" podID="fe564e23-ad21-4f6f-a82f-69d549102bb9" containerID="e454c057fc8adbc7ee5cfaf3eeb6cb5f7e1c70a35cdbddfd89d51931b19285a4" exitCode=0
Oct 31 19:39:15.457368 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:15.457173    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-67adab47-6cd87d9bdb-kwrg8"
Oct 31 19:39:15.457368 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:15.457204    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-67adab47-6cd87d9bdb-kwrg8" event={"ID":"fe564e23-ad21-4f6f-a82f-69d549102bb9","Type":"ContainerDied","Data":"e454c057fc8adbc7ee5cfaf3eeb6cb5f7e1c70a35cdbddfd89d51931b19285a4"}
Oct 31 19:39:15.457368 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:15.457256    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-67adab47-6cd87d9bdb-kwrg8" event={"ID":"fe564e23-ad21-4f6f-a82f-69d549102bb9","Type":"ContainerDied","Data":"6de40b38f1a7eb5ddfa0f38dabc0eb2564aada9cca39655f7e67caf192ca6162"}
Oct 31 19:39:15.457368 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:15.457278    4645 scope.go:122] "RemoveContainer" containerID="e454c057fc8adbc7ee5cfaf3eeb6cb5f7e1c70a35cdbddfd89d51931b19285a4"
Oct 31 19:39:15.468599 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:15.468581    4645 scope.go:122] "RemoveContainer" containerID="e454c057fc8adbc7ee5cfaf3eeb6cb5f7e1c70a35cdbddfd89d51931b19285a4"
Oct 31 19:39:15.468868 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:39:15.468850    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e454c057fc8adbc7ee5cfaf3eeb6cb5f7e1c70a35cdbddfd89d51931b19285a4\": container with ID starting with e454c057fc8adbc7ee5cfaf3eeb6cb5f7e1c70a35cdbddfd89d51931b19285a4 not found: ID does not exist" containerID="e454c057fc8adbc7ee5cfaf3eeb6cb5f7e1c70a35cdbddfd89d51931b19285a4"
Oct 31 19:39:15.468916 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:15.468879    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e454c057fc8adbc7ee5cfaf3eeb6cb5f7e1c70a35cdbddfd89d51931b19285a4"} err="failed to get container status \"e454c057fc8adbc7ee5cfaf3eeb6cb5f7e1c70a35cdbddfd89d51931b19285a4\": rpc error: code = NotFound desc = could not find container \"e454c057fc8adbc7ee5cfaf3eeb6cb5f7e1c70a35cdbddfd89d51931b19285a4\": container with ID starting with e454c057fc8adbc7ee5cfaf3eeb6cb5f7e1c70a35cdbddfd89d51931b19285a4 not found: ID does not exist"
Oct 31 19:39:15.484481 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:15.484422    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-67adab47-6cd87d9bdb-kwrg8"]
Oct 31 19:39:15.491064 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:15.491032    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-67adab47-6cd87d9bdb-kwrg8"]
Oct 31 19:39:15.619692 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:15.619661    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe564e23-ad21-4f6f-a82f-69d549102bb9" path="/var/lib/kubelet/pods/fe564e23-ad21-4f6f-a82f-69d549102bb9/volumes"
Oct 31 19:39:16.111059 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:16.111023    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:16.242028 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:16.240347    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:16.985515 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:16.985481    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:17.397896 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:17.397804    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d4b21c-6bf64967d7-rj87s"
Oct 31 19:39:17.398391 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:17.397998    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-d4b21c-764698f6f-dt2q6"
Oct 31 19:39:17.520645 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:17.520605    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt"]
Oct 31 19:39:17.521372 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:17.521351    4645 cpu_manager.go:414] "containerMap: removing container" podUID="fe564e23-ad21-4f6f-a82f-69d549102bb9" containerName="maas-api"
Oct 31 19:39:17.521372 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:17.521370    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe564e23-ad21-4f6f-a82f-69d549102bb9" containerName="maas-api"
Oct 31 19:39:17.521571 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:17.521557    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe564e23-ad21-4f6f-a82f-69d549102bb9" containerName="maas-api"
Oct 31 19:39:17.525057 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:17.525035    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt"
Oct 31 19:39:17.546025 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:17.545187    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt"]
Oct 31 19:39:17.577174 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:17.576353    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-67adab47-openshift-default-dockercfg-bbdfj\""
Oct 31 19:39:17.583934 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:17.583903    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4abbcb7a-0390-4ed4-8d3b-468921489dc9-istio-podinfo\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt\" (UID: \"4abbcb7a-0390-4ed4-8d3b-468921489dc9\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt"
Oct 31 19:39:17.584094 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:17.584018    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n667j\" (UniqueName: \"kubernetes.io/projected/4abbcb7a-0390-4ed4-8d3b-468921489dc9-kube-api-access-n667j\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt\" (UID: \"4abbcb7a-0390-4ed4-8d3b-468921489dc9\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt"
Oct 31 19:39:17.584197 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:17.584173    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4abbcb7a-0390-4ed4-8d3b-468921489dc9-istiod-ca-cert\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt\" (UID: \"4abbcb7a-0390-4ed4-8d3b-468921489dc9\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt"
Oct 31 19:39:17.584256 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:17.584228    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4abbcb7a-0390-4ed4-8d3b-468921489dc9-workload-socket\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt\" (UID: \"4abbcb7a-0390-4ed4-8d3b-468921489dc9\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt"
Oct 31 19:39:17.584402 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:17.584387    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4abbcb7a-0390-4ed4-8d3b-468921489dc9-istio-token\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt\" (UID: \"4abbcb7a-0390-4ed4-8d3b-468921489dc9\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt"
Oct 31 19:39:17.584527 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:17.584508    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4abbcb7a-0390-4ed4-8d3b-468921489dc9-workload-certs\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt\" (UID: \"4abbcb7a-0390-4ed4-8d3b-468921489dc9\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt"
Oct 31 19:39:17.584587 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:17.584538    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4abbcb7a-0390-4ed4-8d3b-468921489dc9-credential-socket\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt\" (UID: \"4abbcb7a-0390-4ed4-8d3b-468921489dc9\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt"
Oct 31 19:39:17.584587 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:17.584571    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4abbcb7a-0390-4ed4-8d3b-468921489dc9-istio-envoy\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt\" (UID: \"4abbcb7a-0390-4ed4-8d3b-468921489dc9\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt"
Oct 31 19:39:17.584707 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:17.584605    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4abbcb7a-0390-4ed4-8d3b-468921489dc9-istio-data\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt\" (UID: \"4abbcb7a-0390-4ed4-8d3b-468921489dc9\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt"
Oct 31 19:39:17.685995 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:17.685954    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4abbcb7a-0390-4ed4-8d3b-468921489dc9-istio-podinfo\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt\" (UID: \"4abbcb7a-0390-4ed4-8d3b-468921489dc9\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt"
Oct 31 19:39:17.686206 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:17.686055    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n667j\" (UniqueName: \"kubernetes.io/projected/4abbcb7a-0390-4ed4-8d3b-468921489dc9-kube-api-access-n667j\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt\" (UID: \"4abbcb7a-0390-4ed4-8d3b-468921489dc9\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt"
Oct 31 19:39:17.686206 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:17.686180    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4abbcb7a-0390-4ed4-8d3b-468921489dc9-istiod-ca-cert\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt\" (UID: \"4abbcb7a-0390-4ed4-8d3b-468921489dc9\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt"
Oct 31 19:39:17.686348 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:17.686234    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4abbcb7a-0390-4ed4-8d3b-468921489dc9-workload-socket\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt\" (UID: \"4abbcb7a-0390-4ed4-8d3b-468921489dc9\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt"
Oct 31 19:39:17.686348 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:17.686319    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4abbcb7a-0390-4ed4-8d3b-468921489dc9-istio-token\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt\" (UID: \"4abbcb7a-0390-4ed4-8d3b-468921489dc9\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt"
Oct 31 19:39:17.686486 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:17.686421    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4abbcb7a-0390-4ed4-8d3b-468921489dc9-workload-certs\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt\" (UID: \"4abbcb7a-0390-4ed4-8d3b-468921489dc9\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt"
Oct 31 19:39:17.686486 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:17.686453    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4abbcb7a-0390-4ed4-8d3b-468921489dc9-credential-socket\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt\" (UID: \"4abbcb7a-0390-4ed4-8d3b-468921489dc9\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt"
Oct 31 19:39:17.686607 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:17.686494    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4abbcb7a-0390-4ed4-8d3b-468921489dc9-istio-envoy\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt\" (UID: \"4abbcb7a-0390-4ed4-8d3b-468921489dc9\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt"
Oct 31 19:39:17.686607 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:17.686546    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4abbcb7a-0390-4ed4-8d3b-468921489dc9-istio-data\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt\" (UID: \"4abbcb7a-0390-4ed4-8d3b-468921489dc9\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt"
Oct 31 19:39:17.686797 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:17.686770    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4abbcb7a-0390-4ed4-8d3b-468921489dc9-workload-socket\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt\" (UID: \"4abbcb7a-0390-4ed4-8d3b-468921489dc9\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt"
Oct 31 19:39:17.686962 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:17.686943    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4abbcb7a-0390-4ed4-8d3b-468921489dc9-istio-data\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt\" (UID: \"4abbcb7a-0390-4ed4-8d3b-468921489dc9\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt"
Oct 31 19:39:17.687128 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:17.687108    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4abbcb7a-0390-4ed4-8d3b-468921489dc9-workload-certs\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt\" (UID: \"4abbcb7a-0390-4ed4-8d3b-468921489dc9\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt"
Oct 31 19:39:17.687193 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:17.687129    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4abbcb7a-0390-4ed4-8d3b-468921489dc9-credential-socket\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt\" (UID: \"4abbcb7a-0390-4ed4-8d3b-468921489dc9\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt"
Oct 31 19:39:17.687193 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:17.687134    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4abbcb7a-0390-4ed4-8d3b-468921489dc9-istiod-ca-cert\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt\" (UID: \"4abbcb7a-0390-4ed4-8d3b-468921489dc9\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt"
Oct 31 19:39:17.688725 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:17.688696    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4abbcb7a-0390-4ed4-8d3b-468921489dc9-istio-podinfo\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt\" (UID: \"4abbcb7a-0390-4ed4-8d3b-468921489dc9\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt"
Oct 31 19:39:17.689531 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:17.689510    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4abbcb7a-0390-4ed4-8d3b-468921489dc9-istio-envoy\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt\" (UID: \"4abbcb7a-0390-4ed4-8d3b-468921489dc9\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt"
Oct 31 19:39:17.699868 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:17.699844    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4abbcb7a-0390-4ed4-8d3b-468921489dc9-istio-token\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt\" (UID: \"4abbcb7a-0390-4ed4-8d3b-468921489dc9\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt"
Oct 31 19:39:17.700197 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:17.700177    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n667j\" (UniqueName: \"kubernetes.io/projected/4abbcb7a-0390-4ed4-8d3b-468921489dc9-kube-api-access-n667j\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt\" (UID: \"4abbcb7a-0390-4ed4-8d3b-468921489dc9\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt"
Oct 31 19:39:17.839127 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:17.839086    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt"
Oct 31 19:39:17.969689 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:17.969604    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:18.129466 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:39:18.128892    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4abbcb7a_0390_4ed4_8d3b_468921489dc9.slice/crio-89d9d84e923b555cff4c606b0390e61062463eaa04c31fe62bb720eb57ca1776 WatchSource:0}: Error finding container 89d9d84e923b555cff4c606b0390e61062463eaa04c31fe62bb720eb57ca1776: Status 404 returned error can't find the container with id 89d9d84e923b555cff4c606b0390e61062463eaa04c31fe62bb720eb57ca1776
Oct 31 19:39:18.129466 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:18.129315    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt"]
Oct 31 19:39:18.132172 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:18.132130    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:39:18.132298 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:18.132211    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:39:18.132298 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:18.132252    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:39:18.488373 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:18.488321    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt" event={"ID":"4abbcb7a-0390-4ed4-8d3b-468921489dc9","Type":"ContainerStarted","Data":"d62594b322266a964f7e928ec70564d883404edd2ae582b721e271ffbc01f14f"}
Oct 31 19:39:18.488808 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:18.488411    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt" event={"ID":"4abbcb7a-0390-4ed4-8d3b-468921489dc9","Type":"ContainerStarted","Data":"89d9d84e923b555cff4c606b0390e61062463eaa04c31fe62bb720eb57ca1776"}
Oct 31 19:39:18.514900 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:18.514850    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt" podStartSLOduration=1.514836056 podStartE2EDuration="1.514836056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:39:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:39:18.511418161 +0000 UTC m=+1189.084652291" watchObservedRunningTime="2026-10-31 19:39:18.514836056 +0000 UTC m=+1189.088070245"
Oct 31 19:39:18.839873 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:18.839786    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt"
Oct 31 19:39:18.985667 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:18.983186    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:19.096394 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:19.096277    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:19.221352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:19.216638    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:19.858364 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:19.855654    4645 patch_prober.go:29] interesting pod/e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 31 19:39:19.858364 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:19.855813    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt" podUID="4abbcb7a-0390-4ed4-8d3b-468921489dc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 31 19:39:20.128912 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:20.128746    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:20.839834 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:20.839797    4645 patch_prober.go:29] interesting pod/e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 31 19:39:20.840039 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:20.839860    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt" podUID="4abbcb7a-0390-4ed4-8d3b-468921489dc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 31 19:39:21.054850 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:21.054802    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:21.284723 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:21.284688    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-67adab47-998479859-kmf5v" podUID="03810b83-fe09-4d95-a374-f6d7189954b4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:39:21.839511 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:21.839429    4645 patch_prober.go:29] interesting pod/e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 31 19:39:21.839728 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:21.839698    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt" podUID="4abbcb7a-0390-4ed4-8d3b-468921489dc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 31 19:39:21.905036 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:21.905007    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-67adab47-998479859-kmf5v"
Oct 31 19:39:22.080985 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:22.080953    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03810b83-fe09-4d95-a374-f6d7189954b4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03810b83-fe09-4d95-a374-f6d7189954b4-plugins-config-volume\") pod \"03810b83-fe09-4d95-a374-f6d7189954b4\" (UID: \"03810b83-fe09-4d95-a374-f6d7189954b4\") "
Oct 31 19:39:22.081497 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:22.081197    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03810b83-fe09-4d95-a374-f6d7189954b4-kube-api-access-xxhzt\" (UniqueName: \"kubernetes.io/projected/03810b83-fe09-4d95-a374-f6d7189954b4-kube-api-access-xxhzt\") pod \"03810b83-fe09-4d95-a374-f6d7189954b4\" (UID: \"03810b83-fe09-4d95-a374-f6d7189954b4\") "
Oct 31 19:39:22.081497 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:22.081421    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03810b83-fe09-4d95-a374-f6d7189954b4-plugins-config-volume" pod "03810b83-fe09-4d95-a374-f6d7189954b4" (UID: "03810b83-fe09-4d95-a374-f6d7189954b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:39:22.084782 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:22.084754    4645 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03810b83-fe09-4d95-a374-f6d7189954b4-plugins-config-volume\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:39:22.085067 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:22.085038    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03810b83-fe09-4d95-a374-f6d7189954b4-kube-api-access-xxhzt" pod "03810b83-fe09-4d95-a374-f6d7189954b4" (UID: "03810b83-fe09-4d95-a374-f6d7189954b4"). InnerVolumeSpecName "kube-api-access-xxhzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:39:22.188931 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:22.188894    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxhzt\" (UniqueName: \"kubernetes.io/projected/03810b83-fe09-4d95-a374-f6d7189954b4-kube-api-access-xxhzt\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:39:22.243383 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:22.243347    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:22.404880 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:22.404788    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:22.521772 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:22.521736    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:22.522056 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:22.522024    4645 generic.go:356] "Generic (PLEG): container finished" podID="03810b83-fe09-4d95-a374-f6d7189954b4" containerID="d91fbd860d0b28690210b1fdde22dc604a83c979372b15cc89537dc16416c398" exitCode=0
Oct 31 19:39:22.522196 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:22.522110    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-67adab47-998479859-kmf5v"
Oct 31 19:39:22.522296 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:22.522276    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-67adab47-998479859-kmf5v" event={"ID":"03810b83-fe09-4d95-a374-f6d7189954b4","Type":"ContainerDied","Data":"d91fbd860d0b28690210b1fdde22dc604a83c979372b15cc89537dc16416c398"}
Oct 31 19:39:22.522417 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:22.522398    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-67adab47-998479859-kmf5v" event={"ID":"03810b83-fe09-4d95-a374-f6d7189954b4","Type":"ContainerDied","Data":"0ce603d55dc6147dbc9a96591380fb09bb3f31d5f0b1751623acdc5bbe58e3f7"}
Oct 31 19:39:22.522484 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:22.522442    4645 scope.go:122] "RemoveContainer" containerID="d91fbd860d0b28690210b1fdde22dc604a83c979372b15cc89537dc16416c398"
Oct 31 19:39:22.533498 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:22.533474    4645 scope.go:122] "RemoveContainer" containerID="d91fbd860d0b28690210b1fdde22dc604a83c979372b15cc89537dc16416c398"
Oct 31 19:39:22.533773 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:39:22.533756    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d91fbd860d0b28690210b1fdde22dc604a83c979372b15cc89537dc16416c398\": container with ID starting with d91fbd860d0b28690210b1fdde22dc604a83c979372b15cc89537dc16416c398 not found: ID does not exist" containerID="d91fbd860d0b28690210b1fdde22dc604a83c979372b15cc89537dc16416c398"
Oct 31 19:39:22.533843 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:22.533780    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d91fbd860d0b28690210b1fdde22dc604a83c979372b15cc89537dc16416c398"} err="failed to get container status \"d91fbd860d0b28690210b1fdde22dc604a83c979372b15cc89537dc16416c398\": rpc error: code = NotFound desc = could not find container \"d91fbd860d0b28690210b1fdde22dc604a83c979372b15cc89537dc16416c398\": container with ID starting with d91fbd860d0b28690210b1fdde22dc604a83c979372b15cc89537dc16416c398 not found: ID does not exist"
Oct 31 19:39:22.546787 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:22.546760    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-67adab47-998479859-kmf5v"]
Oct 31 19:39:22.553061 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:22.553037    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-67adab47-998479859-kmf5v"]
Oct 31 19:39:22.633790 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:22.633757    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:22.839531 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:22.839501    4645 patch_prober.go:29] interesting pod/e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 31 19:39:22.839693 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:22.839586    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt" podUID="4abbcb7a-0390-4ed4-8d3b-468921489dc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 31 19:39:22.893308 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:22.893255    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:22.942383 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:22.942353    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-67adab47-998479859-c8zq5"]
Oct 31 19:39:22.943034 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:22.943020    4645 cpu_manager.go:414] "containerMap: removing container" podUID="03810b83-fe09-4d95-a374-f6d7189954b4" containerName="payload-pre-processing"
Oct 31 19:39:22.943080 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:22.943035    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03810b83-fe09-4d95-a374-f6d7189954b4" containerName="payload-pre-processing"
Oct 31 19:39:22.943152 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:22.943143    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="03810b83-fe09-4d95-a374-f6d7189954b4" containerName="payload-pre-processing"
Oct 31 19:39:22.947554 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:22.947532    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-67adab47-998479859-c8zq5"
Oct 31 19:39:22.950849 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:22.950827    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-67adab47-dockercfg-zjwrc\""
Oct 31 19:39:22.982712 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:22.982689    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-67adab47\""
Oct 31 19:39:22.990292 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:22.990271    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-67adab47-998479859-c8zq5"]
Oct 31 19:39:23.012798 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:23.012768    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cgq2\" (UniqueName: \"kubernetes.io/projected/cff005f5-fa01-4cc4-a22f-c0c884db435a-kube-api-access-8cgq2\") pod \"payload-pre-processing-e2e-mt-67adab47-998479859-c8zq5\" (UID: \"cff005f5-fa01-4cc4-a22f-c0c884db435a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-67adab47-998479859-c8zq5"
Oct 31 19:39:23.013001 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:23.012977    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cff005f5-fa01-4cc4-a22f-c0c884db435a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-67adab47-998479859-c8zq5\" (UID: \"cff005f5-fa01-4cc4-a22f-c0c884db435a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-67adab47-998479859-c8zq5"
Oct 31 19:39:23.024030 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:23.024007    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-67adab47-7ddbcc466c-n2bkz"]
Oct 31 19:39:23.028338 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:23.028310    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-67adab47-7ddbcc466c-n2bkz"
Oct 31 19:39:23.148409 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:23.120557    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cgq2\" (UniqueName: \"kubernetes.io/projected/cff005f5-fa01-4cc4-a22f-c0c884db435a-kube-api-access-8cgq2\") pod \"payload-pre-processing-e2e-mt-67adab47-998479859-c8zq5\" (UID: \"cff005f5-fa01-4cc4-a22f-c0c884db435a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-67adab47-998479859-c8zq5"
Oct 31 19:39:23.167696 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:23.167652    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cff005f5-fa01-4cc4-a22f-c0c884db435a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-67adab47-998479859-c8zq5\" (UID: \"cff005f5-fa01-4cc4-a22f-c0c884db435a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-67adab47-998479859-c8zq5"
Oct 31 19:39:23.169070 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:23.169042    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cff005f5-fa01-4cc4-a22f-c0c884db435a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-67adab47-998479859-c8zq5\" (UID: \"cff005f5-fa01-4cc4-a22f-c0c884db435a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-67adab47-998479859-c8zq5"
Oct 31 19:39:23.170894 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:23.170869    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:23.309349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:23.296182    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be80a9a3-7971-49e3-9ab3-92fec946a71f-plugins-config-volume\") pod \"payload-processing-e2e-mt-67adab47-7ddbcc466c-n2bkz\" (UID: \"be80a9a3-7971-49e3-9ab3-92fec946a71f\") " pod="openshift-ingress/payload-processing-e2e-mt-67adab47-7ddbcc466c-n2bkz"
Oct 31 19:39:23.309615 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:23.309586    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cc6d\" (UniqueName: \"kubernetes.io/projected/be80a9a3-7971-49e3-9ab3-92fec946a71f-kube-api-access-5cc6d\") pod \"payload-processing-e2e-mt-67adab47-7ddbcc466c-n2bkz\" (UID: \"be80a9a3-7971-49e3-9ab3-92fec946a71f\") " pod="openshift-ingress/payload-processing-e2e-mt-67adab47-7ddbcc466c-n2bkz"
Oct 31 19:39:23.331353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:23.310349    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-67adab47-7ddbcc466c-n2bkz"]
Oct 31 19:39:23.434349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:23.429931    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be80a9a3-7971-49e3-9ab3-92fec946a71f-plugins-config-volume\") pod \"payload-processing-e2e-mt-67adab47-7ddbcc466c-n2bkz\" (UID: \"be80a9a3-7971-49e3-9ab3-92fec946a71f\") " pod="openshift-ingress/payload-processing-e2e-mt-67adab47-7ddbcc466c-n2bkz"
Oct 31 19:39:23.434349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:23.429988    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cc6d\" (UniqueName: \"kubernetes.io/projected/be80a9a3-7971-49e3-9ab3-92fec946a71f-kube-api-access-5cc6d\") pod \"payload-processing-e2e-mt-67adab47-7ddbcc466c-n2bkz\" (UID: \"be80a9a3-7971-49e3-9ab3-92fec946a71f\") " pod="openshift-ingress/payload-processing-e2e-mt-67adab47-7ddbcc466c-n2bkz"
Oct 31 19:39:23.434349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:23.431458    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be80a9a3-7971-49e3-9ab3-92fec946a71f-plugins-config-volume\") pod \"payload-processing-e2e-mt-67adab47-7ddbcc466c-n2bkz\" (UID: \"be80a9a3-7971-49e3-9ab3-92fec946a71f\") " pod="openshift-ingress/payload-processing-e2e-mt-67adab47-7ddbcc466c-n2bkz"
Oct 31 19:39:23.461649 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:23.461578    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cc6d\" (UniqueName: \"kubernetes.io/projected/be80a9a3-7971-49e3-9ab3-92fec946a71f-kube-api-access-5cc6d\") pod \"payload-processing-e2e-mt-67adab47-7ddbcc466c-n2bkz\" (UID: \"be80a9a3-7971-49e3-9ab3-92fec946a71f\") " pod="openshift-ingress/payload-processing-e2e-mt-67adab47-7ddbcc466c-n2bkz"
Oct 31 19:39:23.473309 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:23.473239    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cgq2\" (UniqueName: \"kubernetes.io/projected/cff005f5-fa01-4cc4-a22f-c0c884db435a-kube-api-access-8cgq2\") pod \"payload-pre-processing-e2e-mt-67adab47-998479859-c8zq5\" (UID: \"cff005f5-fa01-4cc4-a22f-c0c884db435a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-67adab47-998479859-c8zq5"
Oct 31 19:39:23.597711 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:23.597665    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-67adab47-998479859-c8zq5"
Oct 31 19:39:23.626366 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:23.626319    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03810b83-fe09-4d95-a374-f6d7189954b4" path="/var/lib/kubelet/pods/03810b83-fe09-4d95-a374-f6d7189954b4/volumes"
Oct 31 19:39:23.647759 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:23.647728    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-67adab47-7ddbcc466c-n2bkz"
Oct 31 19:39:23.840070 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:23.840039    4645 patch_prober.go:29] interesting pod/e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 31 19:39:23.840217 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:23.840091    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt" podUID="4abbcb7a-0390-4ed4-8d3b-468921489dc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 31 19:39:23.889294 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:23.889259    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:24.014351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:23.990615    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-67adab47-998479859-c8zq5"]
Oct 31 19:39:24.253918 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:24.253837    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-67adab47-7ddbcc466c-n2bkz"]
Oct 31 19:39:24.581960 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:24.581859    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-67adab47-998479859-c8zq5" event={"ID":"cff005f5-fa01-4cc4-a22f-c0c884db435a","Type":"ContainerStarted","Data":"59e61b64adc0cf959c9976c7f29285dd6b09c8d132e33e9bede83a0165fa8ce8"}
Oct 31 19:39:24.581960 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:24.581947    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-67adab47-998479859-c8zq5" event={"ID":"cff005f5-fa01-4cc4-a22f-c0c884db435a","Type":"ContainerStarted","Data":"7632121fd5753600e1e72dcd901443a11d31cbb953ebfbeed697410ba0d192cf"}
Oct 31 19:39:24.582191 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:24.582084    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-67adab47-998479859-c8zq5"
Oct 31 19:39:24.583805 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:24.583775    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-67adab47-7ddbcc466c-n2bkz" event={"ID":"be80a9a3-7971-49e3-9ab3-92fec946a71f","Type":"ContainerStarted","Data":"045435328da86f1f3a68a82b74e6a62fd2a50bafff4574781e6d04746ff48cf0"}
Oct 31 19:39:24.583911 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:24.583822    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-67adab47-7ddbcc466c-n2bkz" event={"ID":"be80a9a3-7971-49e3-9ab3-92fec946a71f","Type":"ContainerStarted","Data":"520f079a70c6e132369b5ebbc6827797f958c46681126da477d8c47917846217"}
Oct 31 19:39:24.583911 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:24.583844    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-67adab47-7ddbcc466c-n2bkz"
Oct 31 19:39:24.603543 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:24.603504    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-67adab47-998479859-c8zq5" podStartSLOduration=2.603490778 podStartE2EDuration="2.603490778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:39:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:39:24.602428921 +0000 UTC m=+1195.175663052" watchObservedRunningTime="2026-10-31 19:39:24.603490778 +0000 UTC m=+1195.176724906"
Oct 31 19:39:24.736352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:24.727811    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-67adab47-7ddbcc466c-n2bkz" podStartSLOduration=2.72779088 podStartE2EDuration="2.72779088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:39:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:39:24.643767345 +0000 UTC m=+1195.217001477" watchObservedRunningTime="2026-10-31 19:39:24.72779088 +0000 UTC m=+1195.301025011"
Oct 31 19:39:24.840478 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:24.840378    4645 patch_prober.go:29] interesting pod/e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Oct 31 19:39:24.840773 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:24.840719    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt" podUID="4abbcb7a-0390-4ed4-8d3b-468921489dc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Oct 31 19:39:24.899874 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:24.899840    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:25.091926 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:25.091833    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:25.844804 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:25.844764    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt"
Oct 31 19:39:25.845266 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:25.845245    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt"
Oct 31 19:39:25.846204 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:25.846185    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt"
Oct 31 19:39:26.279114 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:26.279078    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:27.118480 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:27.118444    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:27.910353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:27.905989    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:28.738410 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:28.738375    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:29.563708 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:29.563672    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:30.388035 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:30.387998    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:31.210001 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:31.209968    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:32.218961 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:32.218906    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-67adab47-6cd87d9bdb-r5w6j"]
Oct 31 19:39:32.235616 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:32.235587    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-67adab47-6cd87d9bdb-r5w6j"
Oct 31 19:39:32.237516 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:32.237489    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-67adab47-6cd87d9bdb-r5w6j"]
Oct 31 19:39:32.251240 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:32.251214    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-67adab47-serving-cert\""
Oct 31 19:39:32.292311 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:32.292271    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:32.407398 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:32.407321    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8npm\" (UniqueName: \"kubernetes.io/projected/68fd2c11-7939-43c0-96b2-7e3f0021a71b-kube-api-access-d8npm\") pod \"maas-api-e2e-mt-67adab47-6cd87d9bdb-r5w6j\" (UID: \"68fd2c11-7939-43c0-96b2-7e3f0021a71b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-67adab47-6cd87d9bdb-r5w6j"
Oct 31 19:39:32.407576 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:32.407455    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68fd2c11-7939-43c0-96b2-7e3f0021a71b-maas-api-tls\") pod \"maas-api-e2e-mt-67adab47-6cd87d9bdb-r5w6j\" (UID: \"68fd2c11-7939-43c0-96b2-7e3f0021a71b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-67adab47-6cd87d9bdb-r5w6j"
Oct 31 19:39:32.407576 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:32.407522    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68fd2c11-7939-43c0-96b2-7e3f0021a71b-metrics-certs\") pod \"maas-api-e2e-mt-67adab47-6cd87d9bdb-r5w6j\" (UID: \"68fd2c11-7939-43c0-96b2-7e3f0021a71b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-67adab47-6cd87d9bdb-r5w6j"
Oct 31 19:39:32.407657 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:32.407635    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68fd2c11-7939-43c0-96b2-7e3f0021a71b-openshift-service-ca\") pod \"maas-api-e2e-mt-67adab47-6cd87d9bdb-r5w6j\" (UID: \"68fd2c11-7939-43c0-96b2-7e3f0021a71b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-67adab47-6cd87d9bdb-r5w6j"
Oct 31 19:39:32.425544 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:32.425511    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:32.513413 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:32.512771    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8npm\" (UniqueName: \"kubernetes.io/projected/68fd2c11-7939-43c0-96b2-7e3f0021a71b-kube-api-access-d8npm\") pod \"maas-api-e2e-mt-67adab47-6cd87d9bdb-r5w6j\" (UID: \"68fd2c11-7939-43c0-96b2-7e3f0021a71b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-67adab47-6cd87d9bdb-r5w6j"
Oct 31 19:39:32.513413 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:32.512892    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68fd2c11-7939-43c0-96b2-7e3f0021a71b-maas-api-tls\") pod \"maas-api-e2e-mt-67adab47-6cd87d9bdb-r5w6j\" (UID: \"68fd2c11-7939-43c0-96b2-7e3f0021a71b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-67adab47-6cd87d9bdb-r5w6j"
Oct 31 19:39:32.513413 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:32.512978    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68fd2c11-7939-43c0-96b2-7e3f0021a71b-metrics-certs\") pod \"maas-api-e2e-mt-67adab47-6cd87d9bdb-r5w6j\" (UID: \"68fd2c11-7939-43c0-96b2-7e3f0021a71b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-67adab47-6cd87d9bdb-r5w6j"
Oct 31 19:39:32.513413 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:32.513143    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68fd2c11-7939-43c0-96b2-7e3f0021a71b-openshift-service-ca\") pod \"maas-api-e2e-mt-67adab47-6cd87d9bdb-r5w6j\" (UID: \"68fd2c11-7939-43c0-96b2-7e3f0021a71b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-67adab47-6cd87d9bdb-r5w6j"
Oct 31 19:39:32.517125 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:32.514399    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68fd2c11-7939-43c0-96b2-7e3f0021a71b-openshift-service-ca\") pod \"maas-api-e2e-mt-67adab47-6cd87d9bdb-r5w6j\" (UID: \"68fd2c11-7939-43c0-96b2-7e3f0021a71b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-67adab47-6cd87d9bdb-r5w6j"
Oct 31 19:39:32.556350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:32.547041    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8npm\" (UniqueName: \"kubernetes.io/projected/68fd2c11-7939-43c0-96b2-7e3f0021a71b-kube-api-access-d8npm\") pod \"maas-api-e2e-mt-67adab47-6cd87d9bdb-r5w6j\" (UID: \"68fd2c11-7939-43c0-96b2-7e3f0021a71b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-67adab47-6cd87d9bdb-r5w6j"
Oct 31 19:39:32.556582 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:32.555955    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68fd2c11-7939-43c0-96b2-7e3f0021a71b-maas-api-tls\") pod \"maas-api-e2e-mt-67adab47-6cd87d9bdb-r5w6j\" (UID: \"68fd2c11-7939-43c0-96b2-7e3f0021a71b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-67adab47-6cd87d9bdb-r5w6j"
Oct 31 19:39:32.607351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:32.606855    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68fd2c11-7939-43c0-96b2-7e3f0021a71b-metrics-certs\") pod \"maas-api-e2e-mt-67adab47-6cd87d9bdb-r5w6j\" (UID: \"68fd2c11-7939-43c0-96b2-7e3f0021a71b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-67adab47-6cd87d9bdb-r5w6j"
Oct 31 19:39:32.870396 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:32.866917    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-67adab47-6cd87d9bdb-r5w6j"
Oct 31 19:39:33.853871 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:33.853811    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-67adab47-6cd87d9bdb-r5w6j"]
Oct 31 19:39:34.093418 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:34.084779    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:34.940606 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:34.940567    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-67adab47-6cd87d9bdb-r5w6j" event={"ID":"68fd2c11-7939-43c0-96b2-7e3f0021a71b","Type":"ContainerStarted","Data":"ca2e29e684aaf5d70b3651e66887b4578bc805f6260e3f912aa3b187eef8e6e5"}
Oct 31 19:39:35.084398 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:35.079410    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-67adab47/e2e-lifecycle-model-67adab47-kserve-8f9445cd9-5rp7l" podUID="b9153d96-5237-47a0-8bf4-1e77f69b071a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:39:35.181400 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:35.180421    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw" podUID="1287db54-a587-4c11-9eb6-1606de555894" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:39:35.181400 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:35.180972    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x" podUID="a774e286-dcb8-4a2f-8541-26bf231eb08d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 19:39:35.199440 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:35.198538    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4" podUID="6032a997-b795-4561-805c-110cf67a9d5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:39:35.246406 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:35.234364    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669" podUID="9285bee0-bf7c-4fd4-9da7-a1db021cae05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:39:35.246406 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:35.240978    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw" podUID="d504b2d3-c26d-4dcf-a7eb-efe20732ae4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:39:35.351349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:35.350373    4645 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l" podUID="12bea86c-61e5-465e-82fb-b4673a4d1be9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:39:35.374350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:35.362396    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l" podUID="12bea86c-61e5-465e-82fb-b4673a4d1be9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 31 19:39:35.506517 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:35.504153    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:35.620353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:35.608364    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-67adab47-7ddbcc466c-n2bkz"
Oct 31 19:39:35.746909 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:35.746861    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n" podUID="dbb97168-db93-4757-9a1b-f9c5eb704fc5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:39:35.879417 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:35.879299    4645 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw" podUID="1287db54-a587-4c11-9eb6-1606de555894" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:39:36.254814 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:36.254662    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-67adab47-6cd87d9bdb-r5w6j" event={"ID":"68fd2c11-7939-43c0-96b2-7e3f0021a71b","Type":"ContainerStarted","Data":"8d5c600b49f55f78939130fc72c4357be3cefccc64a8e3289037469136eff697"}
Oct 31 19:39:36.259366 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:36.259340    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-67adab47-998479859-c8zq5"
Oct 31 19:39:36.259784 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:36.259609    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-67adab47-6cd87d9bdb-r5w6j"
Oct 31 19:39:36.305357 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:36.302718    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-67adab47-6cd87d9bdb-r5w6j" podStartSLOduration=3.575250769 podStartE2EDuration="4.302698075s" totalImagesPullingTime="727.447306ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:39:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:39:36.285033607 +0000 UTC m=+1206.858267739" watchObservedRunningTime="2026-10-31 19:39:36.302698075 +0000 UTC m=+1206.875932205"
Oct 31 19:39:36.396086 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:39:36.396044    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 19:39:36.733510 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:36.733473    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:37.821800 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:37.821764    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:38.747207 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:38.747175    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:39.859378 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:39.856620    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:40.928953 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:40.928911    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:41.625538 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:41.625500    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:42.773324 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:42.773291    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:43.225455 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:43.225425    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-67adab47-6cd87d9bdb-r5w6j"
Oct 31 19:39:43.656638 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:43.656539    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:44.447975 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:44.447941    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:44.580691 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:44.580659    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:45.238840 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:45.238809    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:46.098313 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:46.098273    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:46.924145 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:46.924112    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:47.271422 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:47.269058    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:48.209065 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:48.209029    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:48.361195 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:48.361167    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:48.715666 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:48.715635    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:49.009417 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:49.004803    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:49.124732 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:49.124652    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84"]
Oct 31 19:39:49.125633 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:49.125593    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84" podUID="1fdda8be-d1cd-4235-a995-a6f5af5b7fff" containerName="istio-proxy" containerID="cri-o://663cfae531af8a2ac21c01f2a5fcbaa95a17b99955660cf249732bace82e8a79" gracePeriod=30
Oct 31 19:39:49.152493 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:49.152431    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd"]
Oct 31 19:39:49.152962 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:49.152898    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd" podUID="2907ee98-8c2b-456b-9308-16f5eb074e37" containerName="istio-proxy" containerID="cri-o://51b7d601382c1887a13615212f8695d89b91b011e9e9d7db175cacfb8671b5c0" gracePeriod=30
Oct 31 19:39:49.154740 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:49.154719    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd"]
Oct 31 19:39:49.155129 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:49.155058    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd" podUID="b36d1635-5796-4ff5-83e3-7d63ff424c78" containerName="istio-proxy" containerID="cri-o://be1177ce28e491d0d1df526c206ecfffdd7c102f847f8a052b306f944a2dcc4a" gracePeriod=30
Oct 31 19:39:49.796026 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:49.795990    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:50.553407 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:50.553370    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:51.482529 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:51.482489    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:52.753131 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:52.753089    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:53.032688 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:53.032580    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:53.160194 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:53.160147    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:53.956370 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:53.956288    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:54.341641 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:54.341508    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:54.384659 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:54.384604    4645 generic.go:356] "Generic (PLEG): container finished" podID="2907ee98-8c2b-456b-9308-16f5eb074e37" containerID="51b7d601382c1887a13615212f8695d89b91b011e9e9d7db175cacfb8671b5c0" exitCode=0
Oct 31 19:39:54.384860 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:54.384738    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd" event={"ID":"2907ee98-8c2b-456b-9308-16f5eb074e37","Type":"ContainerDied","Data":"51b7d601382c1887a13615212f8695d89b91b011e9e9d7db175cacfb8671b5c0"}
Oct 31 19:39:54.387111 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:54.387083    4645 generic.go:356] "Generic (PLEG): container finished" podID="1fdda8be-d1cd-4235-a995-a6f5af5b7fff" containerID="663cfae531af8a2ac21c01f2a5fcbaa95a17b99955660cf249732bace82e8a79" exitCode=0
Oct 31 19:39:54.387421 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:54.387272    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84" event={"ID":"1fdda8be-d1cd-4235-a995-a6f5af5b7fff","Type":"ContainerDied","Data":"663cfae531af8a2ac21c01f2a5fcbaa95a17b99955660cf249732bace82e8a79"}
Oct 31 19:39:54.389753 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:54.389394    4645 generic.go:356] "Generic (PLEG): container finished" podID="b36d1635-5796-4ff5-83e3-7d63ff424c78" containerID="be1177ce28e491d0d1df526c206ecfffdd7c102f847f8a052b306f944a2dcc4a" exitCode=0
Oct 31 19:39:54.391386 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:54.389604    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd" event={"ID":"b36d1635-5796-4ff5-83e3-7d63ff424c78","Type":"ContainerDied","Data":"be1177ce28e491d0d1df526c206ecfffdd7c102f847f8a052b306f944a2dcc4a"}
Oct 31 19:39:54.437614 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:54.437583    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e73256-57b5c8849f-n9qzl"]
Oct 31 19:39:54.444777 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:54.444743    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e73256-57b5c8849f-n9qzl"
Oct 31 19:39:54.463215 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:54.463171    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e73256-57b5c8849f-n9qzl"]
Oct 31 19:39:54.465093 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:54.464996    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:54.538243 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:54.509576    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-e73256\""
Oct 31 19:39:54.637946 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:54.601554    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-e73256-dockercfg-jd5zc\""
Oct 31 19:39:54.637946 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:54.603435    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9857546-78a4-42de-8e5d-896fa0f91d42-plugins-config-volume\") pod \"payload-processing-e2e-api-a-e73256-57b5c8849f-n9qzl\" (UID: \"f9857546-78a4-42de-8e5d-896fa0f91d42\") " pod="openshift-ingress/payload-processing-e2e-api-a-e73256-57b5c8849f-n9qzl"
Oct 31 19:39:54.637946 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:54.603507    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2vvq\" (UniqueName: \"kubernetes.io/projected/f9857546-78a4-42de-8e5d-896fa0f91d42-kube-api-access-h2vvq\") pod \"payload-processing-e2e-api-a-e73256-57b5c8849f-n9qzl\" (UID: \"f9857546-78a4-42de-8e5d-896fa0f91d42\") " pod="openshift-ingress/payload-processing-e2e-api-a-e73256-57b5c8849f-n9qzl"
Oct 31 19:39:54.721354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:54.718158    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9857546-78a4-42de-8e5d-896fa0f91d42-plugins-config-volume\") pod \"payload-processing-e2e-api-a-e73256-57b5c8849f-n9qzl\" (UID: \"f9857546-78a4-42de-8e5d-896fa0f91d42\") " pod="openshift-ingress/payload-processing-e2e-api-a-e73256-57b5c8849f-n9qzl"
Oct 31 19:39:54.721354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:54.718232    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2vvq\" (UniqueName: \"kubernetes.io/projected/f9857546-78a4-42de-8e5d-896fa0f91d42-kube-api-access-h2vvq\") pod \"payload-processing-e2e-api-a-e73256-57b5c8849f-n9qzl\" (UID: \"f9857546-78a4-42de-8e5d-896fa0f91d42\") " pod="openshift-ingress/payload-processing-e2e-api-a-e73256-57b5c8849f-n9qzl"
Oct 31 19:39:54.721354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:54.719639    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9857546-78a4-42de-8e5d-896fa0f91d42-plugins-config-volume\") pod \"payload-processing-e2e-api-a-e73256-57b5c8849f-n9qzl\" (UID: \"f9857546-78a4-42de-8e5d-896fa0f91d42\") " pod="openshift-ingress/payload-processing-e2e-api-a-e73256-57b5c8849f-n9qzl"
Oct 31 19:39:54.810411 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:54.795666    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd"
Oct 31 19:39:54.838363 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:54.801616    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2vvq\" (UniqueName: \"kubernetes.io/projected/f9857546-78a4-42de-8e5d-896fa0f91d42-kube-api-access-h2vvq\") pod \"payload-processing-e2e-api-a-e73256-57b5c8849f-n9qzl\" (UID: \"f9857546-78a4-42de-8e5d-896fa0f91d42\") " pod="openshift-ingress/payload-processing-e2e-api-a-e73256-57b5c8849f-n9qzl"
Oct 31 19:39:54.922712 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:54.922673    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2907ee98-8c2b-456b-9308-16f5eb074e37-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2907ee98-8c2b-456b-9308-16f5eb074e37-istio-data\") pod \"2907ee98-8c2b-456b-9308-16f5eb074e37\" (UID: \"2907ee98-8c2b-456b-9308-16f5eb074e37\") "
Oct 31 19:39:54.922923 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:54.922753    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2907ee98-8c2b-456b-9308-16f5eb074e37-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2907ee98-8c2b-456b-9308-16f5eb074e37-workload-socket\") pod \"2907ee98-8c2b-456b-9308-16f5eb074e37\" (UID: \"2907ee98-8c2b-456b-9308-16f5eb074e37\") "
Oct 31 19:39:54.923015 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:54.922921    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2907ee98-8c2b-456b-9308-16f5eb074e37-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2907ee98-8c2b-456b-9308-16f5eb074e37-istiod-ca-cert\") pod \"2907ee98-8c2b-456b-9308-16f5eb074e37\" (UID: \"2907ee98-8c2b-456b-9308-16f5eb074e37\") "
Oct 31 19:39:54.923215 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:54.923087    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2907ee98-8c2b-456b-9308-16f5eb074e37-kube-api-access-pwzl9\" (UniqueName: \"kubernetes.io/projected/2907ee98-8c2b-456b-9308-16f5eb074e37-kube-api-access-pwzl9\") pod \"2907ee98-8c2b-456b-9308-16f5eb074e37\" (UID: \"2907ee98-8c2b-456b-9308-16f5eb074e37\") "
Oct 31 19:39:54.923215 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:54.923124    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2907ee98-8c2b-456b-9308-16f5eb074e37-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2907ee98-8c2b-456b-9308-16f5eb074e37-istio-envoy\") pod \"2907ee98-8c2b-456b-9308-16f5eb074e37\" (UID: \"2907ee98-8c2b-456b-9308-16f5eb074e37\") "
Oct 31 19:39:54.924137 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:54.923216    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2907ee98-8c2b-456b-9308-16f5eb074e37-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2907ee98-8c2b-456b-9308-16f5eb074e37-istio-podinfo\") pod \"2907ee98-8c2b-456b-9308-16f5eb074e37\" (UID: \"2907ee98-8c2b-456b-9308-16f5eb074e37\") "
Oct 31 19:39:54.924137 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:54.923296    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2907ee98-8c2b-456b-9308-16f5eb074e37-istio-token\" (UniqueName: \"kubernetes.io/projected/2907ee98-8c2b-456b-9308-16f5eb074e37-istio-token\") pod \"2907ee98-8c2b-456b-9308-16f5eb074e37\" (UID: \"2907ee98-8c2b-456b-9308-16f5eb074e37\") "
Oct 31 19:39:54.924137 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:54.923406    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2907ee98-8c2b-456b-9308-16f5eb074e37-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2907ee98-8c2b-456b-9308-16f5eb074e37-workload-certs\") pod \"2907ee98-8c2b-456b-9308-16f5eb074e37\" (UID: \"2907ee98-8c2b-456b-9308-16f5eb074e37\") "
Oct 31 19:39:54.924137 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:54.923439    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2907ee98-8c2b-456b-9308-16f5eb074e37-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2907ee98-8c2b-456b-9308-16f5eb074e37-credential-socket\") pod \"2907ee98-8c2b-456b-9308-16f5eb074e37\" (UID: \"2907ee98-8c2b-456b-9308-16f5eb074e37\") "
Oct 31 19:39:54.924137 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:54.923897    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2907ee98-8c2b-456b-9308-16f5eb074e37-istiod-ca-cert" pod "2907ee98-8c2b-456b-9308-16f5eb074e37" (UID: "2907ee98-8c2b-456b-9308-16f5eb074e37"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:39:54.924775 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:54.924421    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2907ee98-8c2b-456b-9308-16f5eb074e37-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:39:54.925158 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:54.925127    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2907ee98-8c2b-456b-9308-16f5eb074e37-workload-socket" pod "2907ee98-8c2b-456b-9308-16f5eb074e37" (UID: "2907ee98-8c2b-456b-9308-16f5eb074e37"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:39:54.925553 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:54.925527    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2907ee98-8c2b-456b-9308-16f5eb074e37-istio-data" pod "2907ee98-8c2b-456b-9308-16f5eb074e37" (UID: "2907ee98-8c2b-456b-9308-16f5eb074e37"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:39:54.925793 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:54.925769    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2907ee98-8c2b-456b-9308-16f5eb074e37-workload-certs" pod "2907ee98-8c2b-456b-9308-16f5eb074e37" (UID: "2907ee98-8c2b-456b-9308-16f5eb074e37"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:39:54.926996 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:54.926969    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2907ee98-8c2b-456b-9308-16f5eb074e37-credential-socket" pod "2907ee98-8c2b-456b-9308-16f5eb074e37" (UID: "2907ee98-8c2b-456b-9308-16f5eb074e37"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:39:54.934509 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:54.934447    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2907ee98-8c2b-456b-9308-16f5eb074e37-istio-podinfo" pod "2907ee98-8c2b-456b-9308-16f5eb074e37" (UID: "2907ee98-8c2b-456b-9308-16f5eb074e37"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:39:54.935391 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:54.935013    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2907ee98-8c2b-456b-9308-16f5eb074e37-istio-envoy" pod "2907ee98-8c2b-456b-9308-16f5eb074e37" (UID: "2907ee98-8c2b-456b-9308-16f5eb074e37"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:39:54.939892 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:54.939855    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2907ee98-8c2b-456b-9308-16f5eb074e37-kube-api-access-pwzl9" pod "2907ee98-8c2b-456b-9308-16f5eb074e37" (UID: "2907ee98-8c2b-456b-9308-16f5eb074e37"). InnerVolumeSpecName "kube-api-access-pwzl9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:39:54.940703 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:54.940674    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2907ee98-8c2b-456b-9308-16f5eb074e37-istio-token" pod "2907ee98-8c2b-456b-9308-16f5eb074e37" (UID: "2907ee98-8c2b-456b-9308-16f5eb074e37"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:39:54.943508 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:54.943481    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84"
Oct 31 19:39:54.977790 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:54.977760    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd"
Oct 31 19:39:55.026172 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.026120    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwzl9\" (UniqueName: \"kubernetes.io/projected/2907ee98-8c2b-456b-9308-16f5eb074e37-kube-api-access-pwzl9\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:39:55.026172 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.026165    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2907ee98-8c2b-456b-9308-16f5eb074e37-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:39:55.026418 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.026181    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2907ee98-8c2b-456b-9308-16f5eb074e37-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:39:55.026418 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.026197    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2907ee98-8c2b-456b-9308-16f5eb074e37-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:39:55.026418 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.026211    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2907ee98-8c2b-456b-9308-16f5eb074e37-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:39:55.026418 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.026224    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2907ee98-8c2b-456b-9308-16f5eb074e37-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:39:55.026418 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.026242    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2907ee98-8c2b-456b-9308-16f5eb074e37-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:39:55.026418 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.026255    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2907ee98-8c2b-456b-9308-16f5eb074e37-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:39:55.086075 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.085981    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e73256-57b5c8849f-n9qzl"
Oct 31 19:39:55.127529 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.127495    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fdda8be-d1cd-4235-a995-a6f5af5b7fff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fdda8be-d1cd-4235-a995-a6f5af5b7fff-workload-certs\") pod \"1fdda8be-d1cd-4235-a995-a6f5af5b7fff\" (UID: \"1fdda8be-d1cd-4235-a995-a6f5af5b7fff\") "
Oct 31 19:39:55.127765 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.127547    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fdda8be-d1cd-4235-a995-a6f5af5b7fff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fdda8be-d1cd-4235-a995-a6f5af5b7fff-credential-socket\") pod \"1fdda8be-d1cd-4235-a995-a6f5af5b7fff\" (UID: \"1fdda8be-d1cd-4235-a995-a6f5af5b7fff\") "
Oct 31 19:39:55.127765 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.127614    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fdda8be-d1cd-4235-a995-a6f5af5b7fff-istio-token\" (UniqueName: \"kubernetes.io/projected/1fdda8be-d1cd-4235-a995-a6f5af5b7fff-istio-token\") pod \"1fdda8be-d1cd-4235-a995-a6f5af5b7fff\" (UID: \"1fdda8be-d1cd-4235-a995-a6f5af5b7fff\") "
Oct 31 19:39:55.127765 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.127644    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fdda8be-d1cd-4235-a995-a6f5af5b7fff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fdda8be-d1cd-4235-a995-a6f5af5b7fff-workload-socket\") pod \"1fdda8be-d1cd-4235-a995-a6f5af5b7fff\" (UID: \"1fdda8be-d1cd-4235-a995-a6f5af5b7fff\") "
Oct 31 19:39:55.127765 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.127693    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b36d1635-5796-4ff5-83e3-7d63ff424c78-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b36d1635-5796-4ff5-83e3-7d63ff424c78-istio-envoy\") pod \"b36d1635-5796-4ff5-83e3-7d63ff424c78\" (UID: \"b36d1635-5796-4ff5-83e3-7d63ff424c78\") "
Oct 31 19:39:55.127765 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.127746    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b36d1635-5796-4ff5-83e3-7d63ff424c78-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b36d1635-5796-4ff5-83e3-7d63ff424c78-workload-certs\") pod \"b36d1635-5796-4ff5-83e3-7d63ff424c78\" (UID: \"b36d1635-5796-4ff5-83e3-7d63ff424c78\") "
Oct 31 19:39:55.128030 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.127783    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fdda8be-d1cd-4235-a995-a6f5af5b7fff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fdda8be-d1cd-4235-a995-a6f5af5b7fff-istiod-ca-cert\") pod \"1fdda8be-d1cd-4235-a995-a6f5af5b7fff\" (UID: \"1fdda8be-d1cd-4235-a995-a6f5af5b7fff\") "
Oct 31 19:39:55.128030 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.127825    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b36d1635-5796-4ff5-83e3-7d63ff424c78-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b36d1635-5796-4ff5-83e3-7d63ff424c78-istio-data\") pod \"b36d1635-5796-4ff5-83e3-7d63ff424c78\" (UID: \"b36d1635-5796-4ff5-83e3-7d63ff424c78\") "
Oct 31 19:39:55.128030 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.127920    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b36d1635-5796-4ff5-83e3-7d63ff424c78-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b36d1635-5796-4ff5-83e3-7d63ff424c78-istio-podinfo\") pod \"b36d1635-5796-4ff5-83e3-7d63ff424c78\" (UID: \"b36d1635-5796-4ff5-83e3-7d63ff424c78\") "
Oct 31 19:39:55.128030 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.127978    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fdda8be-d1cd-4235-a995-a6f5af5b7fff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fdda8be-d1cd-4235-a995-a6f5af5b7fff-istio-data\") pod \"1fdda8be-d1cd-4235-a995-a6f5af5b7fff\" (UID: \"1fdda8be-d1cd-4235-a995-a6f5af5b7fff\") "
Oct 31 19:39:55.128257 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.128036    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1fdda8be-d1cd-4235-a995-a6f5af5b7fff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fdda8be-d1cd-4235-a995-a6f5af5b7fff-istio-podinfo\") pod \"1fdda8be-d1cd-4235-a995-a6f5af5b7fff\" (UID: \"1fdda8be-d1cd-4235-a995-a6f5af5b7fff\") "
Oct 31 19:39:55.128257 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.128071    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b36d1635-5796-4ff5-83e3-7d63ff424c78-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b36d1635-5796-4ff5-83e3-7d63ff424c78-istiod-ca-cert\") pod \"b36d1635-5796-4ff5-83e3-7d63ff424c78\" (UID: \"b36d1635-5796-4ff5-83e3-7d63ff424c78\") "
Oct 31 19:39:55.128257 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.128104    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b36d1635-5796-4ff5-83e3-7d63ff424c78-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b36d1635-5796-4ff5-83e3-7d63ff424c78-workload-socket\") pod \"b36d1635-5796-4ff5-83e3-7d63ff424c78\" (UID: \"b36d1635-5796-4ff5-83e3-7d63ff424c78\") "
Oct 31 19:39:55.128257 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.128183    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b36d1635-5796-4ff5-83e3-7d63ff424c78-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b36d1635-5796-4ff5-83e3-7d63ff424c78-credential-socket\") pod \"b36d1635-5796-4ff5-83e3-7d63ff424c78\" (UID: \"b36d1635-5796-4ff5-83e3-7d63ff424c78\") "
Oct 31 19:39:55.128522 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.128295    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fdda8be-d1cd-4235-a995-a6f5af5b7fff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fdda8be-d1cd-4235-a995-a6f5af5b7fff-istio-envoy\") pod \"1fdda8be-d1cd-4235-a995-a6f5af5b7fff\" (UID: \"1fdda8be-d1cd-4235-a995-a6f5af5b7fff\") "
Oct 31 19:39:55.128522 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.128384    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b36d1635-5796-4ff5-83e3-7d63ff424c78-istio-token\" (UniqueName: \"kubernetes.io/projected/b36d1635-5796-4ff5-83e3-7d63ff424c78-istio-token\") pod \"b36d1635-5796-4ff5-83e3-7d63ff424c78\" (UID: \"b36d1635-5796-4ff5-83e3-7d63ff424c78\") "
Oct 31 19:39:55.128522 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.128459    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b36d1635-5796-4ff5-83e3-7d63ff424c78-kube-api-access-hsxh4\" (UniqueName: \"kubernetes.io/projected/b36d1635-5796-4ff5-83e3-7d63ff424c78-kube-api-access-hsxh4\") pod \"b36d1635-5796-4ff5-83e3-7d63ff424c78\" (UID: \"b36d1635-5796-4ff5-83e3-7d63ff424c78\") "
Oct 31 19:39:55.128522 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.128514    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fdda8be-d1cd-4235-a995-a6f5af5b7fff-kube-api-access-msqpb\" (UniqueName: \"kubernetes.io/projected/1fdda8be-d1cd-4235-a995-a6f5af5b7fff-kube-api-access-msqpb\") pod \"1fdda8be-d1cd-4235-a995-a6f5af5b7fff\" (UID: \"1fdda8be-d1cd-4235-a995-a6f5af5b7fff\") "
Oct 31 19:39:55.128718 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.128675    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b36d1635-5796-4ff5-83e3-7d63ff424c78-istio-data" pod "b36d1635-5796-4ff5-83e3-7d63ff424c78" (UID: "b36d1635-5796-4ff5-83e3-7d63ff424c78"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:39:55.128904 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.128884    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fdda8be-d1cd-4235-a995-a6f5af5b7fff-workload-certs" pod "1fdda8be-d1cd-4235-a995-a6f5af5b7fff" (UID: "1fdda8be-d1cd-4235-a995-a6f5af5b7fff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:39:55.129112 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.129092    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fdda8be-d1cd-4235-a995-a6f5af5b7fff-credential-socket" pod "1fdda8be-d1cd-4235-a995-a6f5af5b7fff" (UID: "1fdda8be-d1cd-4235-a995-a6f5af5b7fff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:39:55.129664 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.129636    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fdda8be-d1cd-4235-a995-a6f5af5b7fff-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:39:55.129792 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.129667    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fdda8be-d1cd-4235-a995-a6f5af5b7fff-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:39:55.129792 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.129682    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b36d1635-5796-4ff5-83e3-7d63ff424c78-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:39:55.130753 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.130707    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b36d1635-5796-4ff5-83e3-7d63ff424c78-workload-certs" pod "b36d1635-5796-4ff5-83e3-7d63ff424c78" (UID: "b36d1635-5796-4ff5-83e3-7d63ff424c78"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:39:55.131273 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.131226    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b36d1635-5796-4ff5-83e3-7d63ff424c78-credential-socket" pod "b36d1635-5796-4ff5-83e3-7d63ff424c78" (UID: "b36d1635-5796-4ff5-83e3-7d63ff424c78"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:39:55.131726 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.131676    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fdda8be-d1cd-4235-a995-a6f5af5b7fff-istio-data" pod "1fdda8be-d1cd-4235-a995-a6f5af5b7fff" (UID: "1fdda8be-d1cd-4235-a995-a6f5af5b7fff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:39:55.132113 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.131943    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fdda8be-d1cd-4235-a995-a6f5af5b7fff-istio-token" pod "1fdda8be-d1cd-4235-a995-a6f5af5b7fff" (UID: "1fdda8be-d1cd-4235-a995-a6f5af5b7fff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:39:55.132351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.132276    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b36d1635-5796-4ff5-83e3-7d63ff424c78-istio-podinfo" pod "b36d1635-5796-4ff5-83e3-7d63ff424c78" (UID: "b36d1635-5796-4ff5-83e3-7d63ff424c78"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:39:55.132513 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.132471    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fdda8be-d1cd-4235-a995-a6f5af5b7fff-istiod-ca-cert" pod "1fdda8be-d1cd-4235-a995-a6f5af5b7fff" (UID: "1fdda8be-d1cd-4235-a995-a6f5af5b7fff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:39:55.133463 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.132929    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b36d1635-5796-4ff5-83e3-7d63ff424c78-istio-envoy" pod "b36d1635-5796-4ff5-83e3-7d63ff424c78" (UID: "b36d1635-5796-4ff5-83e3-7d63ff424c78"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:39:55.133463 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.133170    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fdda8be-d1cd-4235-a995-a6f5af5b7fff-workload-socket" pod "1fdda8be-d1cd-4235-a995-a6f5af5b7fff" (UID: "1fdda8be-d1cd-4235-a995-a6f5af5b7fff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:39:55.134800 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.134489    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1fdda8be-d1cd-4235-a995-a6f5af5b7fff-istio-podinfo" pod "1fdda8be-d1cd-4235-a995-a6f5af5b7fff" (UID: "1fdda8be-d1cd-4235-a995-a6f5af5b7fff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:39:55.134800 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.134719    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b36d1635-5796-4ff5-83e3-7d63ff424c78-istiod-ca-cert" pod "b36d1635-5796-4ff5-83e3-7d63ff424c78" (UID: "b36d1635-5796-4ff5-83e3-7d63ff424c78"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:39:55.134977 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.134841    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b36d1635-5796-4ff5-83e3-7d63ff424c78-kube-api-access-hsxh4" pod "b36d1635-5796-4ff5-83e3-7d63ff424c78" (UID: "b36d1635-5796-4ff5-83e3-7d63ff424c78"). InnerVolumeSpecName "kube-api-access-hsxh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:39:55.135804 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.135771    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fdda8be-d1cd-4235-a995-a6f5af5b7fff-istio-envoy" pod "1fdda8be-d1cd-4235-a995-a6f5af5b7fff" (UID: "1fdda8be-d1cd-4235-a995-a6f5af5b7fff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:39:55.135916 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.135853    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b36d1635-5796-4ff5-83e3-7d63ff424c78-istio-token" pod "b36d1635-5796-4ff5-83e3-7d63ff424c78" (UID: "b36d1635-5796-4ff5-83e3-7d63ff424c78"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:39:55.136474 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.136453    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fdda8be-d1cd-4235-a995-a6f5af5b7fff-kube-api-access-msqpb" pod "1fdda8be-d1cd-4235-a995-a6f5af5b7fff" (UID: "1fdda8be-d1cd-4235-a995-a6f5af5b7fff"). InnerVolumeSpecName "kube-api-access-msqpb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:39:55.136566 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.136480    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b36d1635-5796-4ff5-83e3-7d63ff424c78-workload-socket" pod "b36d1635-5796-4ff5-83e3-7d63ff424c78" (UID: "b36d1635-5796-4ff5-83e3-7d63ff424c78"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:39:55.153506 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.153465    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:55.266997 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.260134    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fdda8be-d1cd-4235-a995-a6f5af5b7fff-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:39:55.266997 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.260188    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fdda8be-d1cd-4235-a995-a6f5af5b7fff-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:39:55.266997 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.260208    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b36d1635-5796-4ff5-83e3-7d63ff424c78-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:39:55.266997 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.260227    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b36d1635-5796-4ff5-83e3-7d63ff424c78-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:39:55.266997 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.260243    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b36d1635-5796-4ff5-83e3-7d63ff424c78-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:39:55.266997 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.260258    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fdda8be-d1cd-4235-a995-a6f5af5b7fff-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:39:55.266997 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.260272    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b36d1635-5796-4ff5-83e3-7d63ff424c78-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:39:55.266997 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.260285    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsxh4\" (UniqueName: \"kubernetes.io/projected/b36d1635-5796-4ff5-83e3-7d63ff424c78-kube-api-access-hsxh4\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:39:55.266997 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.260298    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msqpb\" (UniqueName: \"kubernetes.io/projected/1fdda8be-d1cd-4235-a995-a6f5af5b7fff-kube-api-access-msqpb\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:39:55.266997 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.260313    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fdda8be-d1cd-4235-a995-a6f5af5b7fff-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:39:55.266997 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.260347    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fdda8be-d1cd-4235-a995-a6f5af5b7fff-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:39:55.266997 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.260366    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b36d1635-5796-4ff5-83e3-7d63ff424c78-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:39:55.266997 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.260385    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b36d1635-5796-4ff5-83e3-7d63ff424c78-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:39:55.266997 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.260402    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fdda8be-d1cd-4235-a995-a6f5af5b7fff-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:39:55.266997 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.260422    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b36d1635-5796-4ff5-83e3-7d63ff424c78-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:39:55.390952 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.390918    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e73256-57b5c8849f-n9qzl"]
Oct 31 19:39:55.391242 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:39:55.391213    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9857546_78a4_42de_8e5d_896fa0f91d42.slice/crio-db7787792fe6fb7fbe49cf5370e03045a0a2c0bd9c38e659b82b92fa3342d9c8 WatchSource:0}: Error finding container db7787792fe6fb7fbe49cf5370e03045a0a2c0bd9c38e659b82b92fa3342d9c8: Status 404 returned error can't find the container with id db7787792fe6fb7fbe49cf5370e03045a0a2c0bd9c38e659b82b92fa3342d9c8
Oct 31 19:39:55.402019 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.401992    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd"
Oct 31 19:39:55.402160 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.401989    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd" event={"ID":"2907ee98-8c2b-456b-9308-16f5eb074e37","Type":"ContainerDied","Data":"86e9debbdf5f8cc152586cf64431f6404ee245caddd5f32137510fb730e7ae85"}
Oct 31 19:39:55.402160 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.402120    4645 scope.go:122] "RemoveContainer" containerID="51b7d601382c1887a13615212f8695d89b91b011e9e9d7db175cacfb8671b5c0"
Oct 31 19:39:55.403584 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.403553    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84" event={"ID":"1fdda8be-d1cd-4235-a995-a6f5af5b7fff","Type":"ContainerDied","Data":"93f2b6958c98229184f836e500ea312400ddf584daf9592962d5d300e0af389a"}
Oct 31 19:39:55.403717 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.403674    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84"
Oct 31 19:39:55.409231 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.408853    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e73256-57b5c8849f-n9qzl" event={"ID":"f9857546-78a4-42de-8e5d-896fa0f91d42","Type":"ContainerStarted","Data":"db7787792fe6fb7fbe49cf5370e03045a0a2c0bd9c38e659b82b92fa3342d9c8"}
Oct 31 19:39:55.410971 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.410793    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd"
Oct 31 19:39:55.410971 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.410794    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd" event={"ID":"b36d1635-5796-4ff5-83e3-7d63ff424c78","Type":"ContainerDied","Data":"3d8363b34ad0e3628026dce6fdd30305241213411799e24cb4243cec5f32e8df"}
Oct 31 19:39:55.416522 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.416497    4645 scope.go:122] "RemoveContainer" containerID="663cfae531af8a2ac21c01f2a5fcbaa95a17b99955660cf249732bace82e8a79"
Oct 31 19:39:55.439915 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.439890    4645 scope.go:122] "RemoveContainer" containerID="be1177ce28e491d0d1df526c206ecfffdd7c102f847f8a052b306f944a2dcc4a"
Oct 31 19:39:55.452446 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.452412    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd"]
Oct 31 19:39:55.459055 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.459027    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-ncxxd"]
Oct 31 19:39:55.468015 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.467979    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd"]
Oct 31 19:39:55.475546 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.475515    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-9zkgd"]
Oct 31 19:39:55.489459 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.489416    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84"]
Oct 31 19:39:55.496002 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.495969    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-qzm84"]
Oct 31 19:39:55.620465 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.620418    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fdda8be-d1cd-4235-a995-a6f5af5b7fff" path="/var/lib/kubelet/pods/1fdda8be-d1cd-4235-a995-a6f5af5b7fff/volumes"
Oct 31 19:39:55.621101 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.621078    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2907ee98-8c2b-456b-9308-16f5eb074e37" path="/var/lib/kubelet/pods/2907ee98-8c2b-456b-9308-16f5eb074e37/volumes"
Oct 31 19:39:55.621694 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:55.621674    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b36d1635-5796-4ff5-83e3-7d63ff424c78" path="/var/lib/kubelet/pods/b36d1635-5796-4ff5-83e3-7d63ff424c78/volumes"
Oct 31 19:39:56.126632 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:56.126597    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:56.420974 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:56.420938    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e73256-57b5c8849f-n9qzl" event={"ID":"f9857546-78a4-42de-8e5d-896fa0f91d42","Type":"ContainerStarted","Data":"909d0db5de2c0c28812f7e8bc73adbab82c8efd7c25605962cc26a0232c8c023"}
Oct 31 19:39:56.421122 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:56.421001    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-e73256-57b5c8849f-n9qzl"
Oct 31 19:39:56.441079 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:56.441036    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-e73256-57b5c8849f-n9qzl" podStartSLOduration=2.441024053 podStartE2EDuration="2.441024053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:39:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:39:56.437928987 +0000 UTC m=+1227.011163116" watchObservedRunningTime="2026-10-31 19:39:56.441024053 +0000 UTC m=+1227.014258182"
Oct 31 19:39:56.948376 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:56.946427    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:57.053660 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:57.053631    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:58.032018 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:58.031984    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:58.749388 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:58.749349    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:59.056132 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:59.056043    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:59.524627 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:59.524588    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c"]
Oct 31 19:39:59.527639 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:59.527604    4645 cpu_manager.go:414] "containerMap: removing container" podUID="1fdda8be-d1cd-4235-a995-a6f5af5b7fff" containerName="istio-proxy"
Oct 31 19:39:59.527782 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:59.527642    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fdda8be-d1cd-4235-a995-a6f5af5b7fff" containerName="istio-proxy"
Oct 31 19:39:59.527782 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:59.527689    4645 cpu_manager.go:414] "containerMap: removing container" podUID="2907ee98-8c2b-456b-9308-16f5eb074e37" containerName="istio-proxy"
Oct 31 19:39:59.527782 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:59.527700    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2907ee98-8c2b-456b-9308-16f5eb074e37" containerName="istio-proxy"
Oct 31 19:39:59.527960 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:59.527920    4645 cpu_manager.go:414] "containerMap: removing container" podUID="b36d1635-5796-4ff5-83e3-7d63ff424c78" containerName="istio-proxy"
Oct 31 19:39:59.527960 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:59.527930    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b36d1635-5796-4ff5-83e3-7d63ff424c78" containerName="istio-proxy"
Oct 31 19:39:59.528895 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:59.528863    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fdda8be-d1cd-4235-a995-a6f5af5b7fff" containerName="istio-proxy"
Oct 31 19:39:59.529054 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:59.528914    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="2907ee98-8c2b-456b-9308-16f5eb074e37" containerName="istio-proxy"
Oct 31 19:39:59.529054 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:59.528942    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="b36d1635-5796-4ff5-83e3-7d63ff424c78" containerName="istio-proxy"
Oct 31 19:39:59.544993 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:59.544962    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c"]
Oct 31 19:39:59.545153 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:59.545062    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c"
Oct 31 19:39:59.580352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:59.563005    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-fd1c91-openshift-default-dockercfg-hm6l2\""
Oct 31 19:39:59.585007 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:59.584970    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:59.591542 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:59.591507    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29a9754a-f686-404c-b428-a3b20ba0aa75-istiod-ca-cert\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c\" (UID: \"29a9754a-f686-404c-b428-a3b20ba0aa75\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c"
Oct 31 19:39:59.591667 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:59.591579    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29a9754a-f686-404c-b428-a3b20ba0aa75-workload-socket\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c\" (UID: \"29a9754a-f686-404c-b428-a3b20ba0aa75\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c"
Oct 31 19:39:59.591794 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:59.591767    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs9sw\" (UniqueName: \"kubernetes.io/projected/29a9754a-f686-404c-b428-a3b20ba0aa75-kube-api-access-fs9sw\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c\" (UID: \"29a9754a-f686-404c-b428-a3b20ba0aa75\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c"
Oct 31 19:39:59.591940 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:59.591920    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29a9754a-f686-404c-b428-a3b20ba0aa75-istio-token\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c\" (UID: \"29a9754a-f686-404c-b428-a3b20ba0aa75\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c"
Oct 31 19:39:59.592009 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:59.591960    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29a9754a-f686-404c-b428-a3b20ba0aa75-credential-socket\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c\" (UID: \"29a9754a-f686-404c-b428-a3b20ba0aa75\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c"
Oct 31 19:39:59.592009 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:59.591997    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29a9754a-f686-404c-b428-a3b20ba0aa75-istio-podinfo\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c\" (UID: \"29a9754a-f686-404c-b428-a3b20ba0aa75\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c"
Oct 31 19:39:59.592157 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:59.592135    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29a9754a-f686-404c-b428-a3b20ba0aa75-istio-envoy\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c\" (UID: \"29a9754a-f686-404c-b428-a3b20ba0aa75\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c"
Oct 31 19:39:59.592280 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:59.592264    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29a9754a-f686-404c-b428-a3b20ba0aa75-workload-certs\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c\" (UID: \"29a9754a-f686-404c-b428-a3b20ba0aa75\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c"
Oct 31 19:39:59.592409 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:59.592384    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29a9754a-f686-404c-b428-a3b20ba0aa75-istio-data\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c\" (UID: \"29a9754a-f686-404c-b428-a3b20ba0aa75\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c"
Oct 31 19:39:59.723354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:59.706944    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs9sw\" (UniqueName: \"kubernetes.io/projected/29a9754a-f686-404c-b428-a3b20ba0aa75-kube-api-access-fs9sw\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c\" (UID: \"29a9754a-f686-404c-b428-a3b20ba0aa75\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c"
Oct 31 19:39:59.723354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:59.707081    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29a9754a-f686-404c-b428-a3b20ba0aa75-istio-token\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c\" (UID: \"29a9754a-f686-404c-b428-a3b20ba0aa75\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c"
Oct 31 19:39:59.723354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:59.707115    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29a9754a-f686-404c-b428-a3b20ba0aa75-credential-socket\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c\" (UID: \"29a9754a-f686-404c-b428-a3b20ba0aa75\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c"
Oct 31 19:39:59.723354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:59.707151    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29a9754a-f686-404c-b428-a3b20ba0aa75-istio-podinfo\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c\" (UID: \"29a9754a-f686-404c-b428-a3b20ba0aa75\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c"
Oct 31 19:39:59.723354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:59.707242    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29a9754a-f686-404c-b428-a3b20ba0aa75-istio-envoy\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c\" (UID: \"29a9754a-f686-404c-b428-a3b20ba0aa75\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c"
Oct 31 19:39:59.723354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:59.707359    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29a9754a-f686-404c-b428-a3b20ba0aa75-workload-certs\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c\" (UID: \"29a9754a-f686-404c-b428-a3b20ba0aa75\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c"
Oct 31 19:39:59.723354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:59.707443    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29a9754a-f686-404c-b428-a3b20ba0aa75-istio-data\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c\" (UID: \"29a9754a-f686-404c-b428-a3b20ba0aa75\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c"
Oct 31 19:39:59.723354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:59.707508    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29a9754a-f686-404c-b428-a3b20ba0aa75-istiod-ca-cert\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c\" (UID: \"29a9754a-f686-404c-b428-a3b20ba0aa75\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c"
Oct 31 19:39:59.723354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:59.707566    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29a9754a-f686-404c-b428-a3b20ba0aa75-workload-socket\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c\" (UID: \"29a9754a-f686-404c-b428-a3b20ba0aa75\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c"
Oct 31 19:39:59.723354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:59.708300    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29a9754a-f686-404c-b428-a3b20ba0aa75-workload-socket\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c\" (UID: \"29a9754a-f686-404c-b428-a3b20ba0aa75\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c"
Oct 31 19:39:59.728899 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:59.727048    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29a9754a-f686-404c-b428-a3b20ba0aa75-workload-certs\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c\" (UID: \"29a9754a-f686-404c-b428-a3b20ba0aa75\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c"
Oct 31 19:39:59.739094 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:59.738271    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:39:59.739094 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:59.738883    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29a9754a-f686-404c-b428-a3b20ba0aa75-istio-data\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c\" (UID: \"29a9754a-f686-404c-b428-a3b20ba0aa75\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c"
Oct 31 19:39:59.767439 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:59.739242    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29a9754a-f686-404c-b428-a3b20ba0aa75-credential-socket\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c\" (UID: \"29a9754a-f686-404c-b428-a3b20ba0aa75\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c"
Oct 31 19:39:59.767439 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:59.739847    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29a9754a-f686-404c-b428-a3b20ba0aa75-istiod-ca-cert\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c\" (UID: \"29a9754a-f686-404c-b428-a3b20ba0aa75\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c"
Oct 31 19:39:59.801842 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:59.801726    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29a9754a-f686-404c-b428-a3b20ba0aa75-istio-token\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c\" (UID: \"29a9754a-f686-404c-b428-a3b20ba0aa75\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c"
Oct 31 19:39:59.802342 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:59.802275    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29a9754a-f686-404c-b428-a3b20ba0aa75-istio-envoy\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c\" (UID: \"29a9754a-f686-404c-b428-a3b20ba0aa75\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c"
Oct 31 19:39:59.834353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:59.802978    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29a9754a-f686-404c-b428-a3b20ba0aa75-istio-podinfo\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c\" (UID: \"29a9754a-f686-404c-b428-a3b20ba0aa75\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c"
Oct 31 19:39:59.904753 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:59.895045    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs9sw\" (UniqueName: \"kubernetes.io/projected/29a9754a-f686-404c-b428-a3b20ba0aa75-kube-api-access-fs9sw\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c\" (UID: \"29a9754a-f686-404c-b428-a3b20ba0aa75\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c"
Oct 31 19:39:59.904753 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:39:59.895501    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c"
Oct 31 19:40:00.390409 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:00.390380    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c"]
Oct 31 19:40:00.391193 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:40:00.391167    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29a9754a_f686_404c_b428_a3b20ba0aa75.slice/crio-270c644810a2b9e45180888280b0578ba49a024a0535f7f9ff152af78469e8f1 WatchSource:0}: Error finding container 270c644810a2b9e45180888280b0578ba49a024a0535f7f9ff152af78469e8f1: Status 404 returned error can't find the container with id 270c644810a2b9e45180888280b0578ba49a024a0535f7f9ff152af78469e8f1
Oct 31 19:40:00.394584 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:00.394547    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:40:00.394675 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:00.394619    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:40:00.394675 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:00.394648    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:40:00.547633 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:00.547598    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:00.570376 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:00.565482    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c" event={"ID":"29a9754a-f686-404c-b428-a3b20ba0aa75","Type":"ContainerStarted","Data":"f64fb7efbcea841caf806db1831039c30ba96412296c4726fdb2790508abea98"}
Oct 31 19:40:00.570376 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:00.565593    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c" event={"ID":"29a9754a-f686-404c-b428-a3b20ba0aa75","Type":"ContainerStarted","Data":"270c644810a2b9e45180888280b0578ba49a024a0535f7f9ff152af78469e8f1"}
Oct 31 19:40:00.618452 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:00.618419    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7011-e54d70"]
Oct 31 19:40:00.620818 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:00.620761    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c" podStartSLOduration=1.6207420099999998 podStartE2EDuration="1.62074201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:39:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:40:00.595803296 +0000 UTC m=+1231.169037431" watchObservedRunningTime="2026-10-31 19:40:00.62074201 +0000 UTC m=+1231.193976140"
Oct 31 19:40:00.630484 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:00.629385    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7011-e54d70"]
Oct 31 19:40:00.630484 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:00.629472    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7011-e54d70"
Oct 31 19:40:00.635571 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:00.634600    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-4pmld\""
Oct 31 19:40:00.812409 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:00.808270    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn4nr\" (UniqueName: \"kubernetes.io/projected/79002f30-65de-4817-9c86-79b2b3e12643-kube-api-access-dn4nr\") pod \"test-curl-7011-e54d70\" (UID: \"79002f30-65de-4817-9c86-79b2b3e12643\") " pod="opendatahub/test-curl-7011-e54d70"
Oct 31 19:40:00.909742 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:00.909641    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c"
Oct 31 19:40:00.913925 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:00.913893    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn4nr\" (UniqueName: \"kubernetes.io/projected/79002f30-65de-4817-9c86-79b2b3e12643-kube-api-access-dn4nr\") pod \"test-curl-7011-e54d70\" (UID: \"79002f30-65de-4817-9c86-79b2b3e12643\") " pod="opendatahub/test-curl-7011-e54d70"
Oct 31 19:40:00.932293 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:00.932261    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn4nr\" (UniqueName: \"kubernetes.io/projected/79002f30-65de-4817-9c86-79b2b3e12643-kube-api-access-dn4nr\") pod \"test-curl-7011-e54d70\" (UID: \"79002f30-65de-4817-9c86-79b2b3e12643\") " pod="opendatahub/test-curl-7011-e54d70"
Oct 31 19:40:00.951200 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:00.951164    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7011-e54d70"
Oct 31 19:40:01.304653 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:01.304622    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7011-e54d70"]
Oct 31 19:40:01.387758 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:01.387635    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:01.589404 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:01.589292    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7011-e54d70" event={"ID":"79002f30-65de-4817-9c86-79b2b3e12643","Type":"ContainerStarted","Data":"08d4c6a0f480f6487f63625988c856497665ffbbc6fe48f7e21bbed88e03550c"}
Oct 31 19:40:01.589404 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:01.589358    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7011-e54d70" event={"ID":"79002f30-65de-4817-9c86-79b2b3e12643","Type":"ContainerStarted","Data":"0de476d356ed9f1ed7ebeae1fa800da99955719d7aea867a9f803d6e20d2f5ab"}
Oct 31 19:40:01.610722 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:01.610671    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7011-e54d70" podStartSLOduration=1.61065748 podStartE2EDuration="1.61065748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:40:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:40:01.606209868 +0000 UTC m=+1232.179443996" watchObservedRunningTime="2026-10-31 19:40:01.61065748 +0000 UTC m=+1232.183891609"
Oct 31 19:40:01.902407 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:01.902322    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c"
Oct 31 19:40:02.406789 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:02.406755    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7011-e54d70"]
Oct 31 19:40:02.411679 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:02.411650    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7011-e54d70"]
Oct 31 19:40:02.426683 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:02.426658    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:02.617795 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:02.617764    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c"
Oct 31 19:40:02.618861 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:02.618839    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c"
Oct 31 19:40:02.738143 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:02.738104    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:03.279812 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:03.279767    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:03.435126 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:03.435094    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:03.622307 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:03.622225    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7011-e54d70" podUID="79002f30-65de-4817-9c86-79b2b3e12643" containerName="test-curl-7011-e54d70" containerID="cri-o://08d4c6a0f480f6487f63625988c856497665ffbbc6fe48f7e21bbed88e03550c" gracePeriod=2
Oct 31 19:40:03.787163 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:03.787121    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:04.486577 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:04.486541    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:04.547928 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:04.547892    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-fd1c91-5fb9c8fcf9-xvrkd"]
Oct 31 19:40:04.548875 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:04.548853    4645 cpu_manager.go:414] "containerMap: removing container" podUID="79002f30-65de-4817-9c86-79b2b3e12643" containerName="test-curl-7011-e54d70"
Oct 31 19:40:04.548971 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:04.548875    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79002f30-65de-4817-9c86-79b2b3e12643" containerName="test-curl-7011-e54d70"
Oct 31 19:40:04.549093 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:04.549080    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="79002f30-65de-4817-9c86-79b2b3e12643" containerName="test-curl-7011-e54d70"
Oct 31 19:40:04.553507 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:04.553484    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-fd1c91-5fb9c8fcf9-xvrkd"
Oct 31 19:40:04.562253 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:04.562227    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-fd1c91-5fb9c8fcf9-xvrkd"]
Oct 31 19:40:04.564586 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:04.564564    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-fd1c91\""
Oct 31 19:40:04.564711 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:04.564677    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-fd1c91-dockercfg-znk9h\""
Oct 31 19:40:04.603208 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:04.603169    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-fd1c91-84dcbc9465-5r662"]
Oct 31 19:40:04.609040 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:04.609008    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-fd1c91-84dcbc9465-5r662"
Oct 31 19:40:04.622890 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:04.622806    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-fd1c91-84dcbc9465-5r662"]
Oct 31 19:40:04.786020 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:04.785713    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pckj7\" (UniqueName: \"kubernetes.io/projected/600fee45-c980-4705-96d9-fc084f4bea96-kube-api-access-pckj7\") pod \"payload-pre-processing-e2e-shared-a-w0-fd1c91-5fb9c8fcf9-xvrkd\" (UID: \"600fee45-c980-4705-96d9-fc084f4bea96\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-fd1c91-5fb9c8fcf9-xvrkd"
Oct 31 19:40:04.786020 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:04.785776    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/600fee45-c980-4705-96d9-fc084f4bea96-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-fd1c91-5fb9c8fcf9-xvrkd\" (UID: \"600fee45-c980-4705-96d9-fc084f4bea96\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-fd1c91-5fb9c8fcf9-xvrkd"
Oct 31 19:40:04.786020 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:04.785894    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7bm6\" (UniqueName: \"kubernetes.io/projected/af2f7267-2d2f-46cf-b8e2-101e6a328b68-kube-api-access-x7bm6\") pod \"payload-processing-e2e-shared-a-w0-fd1c91-84dcbc9465-5r662\" (UID: \"af2f7267-2d2f-46cf-b8e2-101e6a328b68\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-fd1c91-84dcbc9465-5r662"
Oct 31 19:40:04.786020 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:04.785956    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af2f7267-2d2f-46cf-b8e2-101e6a328b68-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-fd1c91-84dcbc9465-5r662\" (UID: \"af2f7267-2d2f-46cf-b8e2-101e6a328b68\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-fd1c91-84dcbc9465-5r662"
Oct 31 19:40:04.948824 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:04.860103    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:04.948824 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:04.888430    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7bm6\" (UniqueName: \"kubernetes.io/projected/af2f7267-2d2f-46cf-b8e2-101e6a328b68-kube-api-access-x7bm6\") pod \"payload-processing-e2e-shared-a-w0-fd1c91-84dcbc9465-5r662\" (UID: \"af2f7267-2d2f-46cf-b8e2-101e6a328b68\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-fd1c91-84dcbc9465-5r662"
Oct 31 19:40:04.948824 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:04.888507    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af2f7267-2d2f-46cf-b8e2-101e6a328b68-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-fd1c91-84dcbc9465-5r662\" (UID: \"af2f7267-2d2f-46cf-b8e2-101e6a328b68\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-fd1c91-84dcbc9465-5r662"
Oct 31 19:40:04.948824 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:04.888810    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pckj7\" (UniqueName: \"kubernetes.io/projected/600fee45-c980-4705-96d9-fc084f4bea96-kube-api-access-pckj7\") pod \"payload-pre-processing-e2e-shared-a-w0-fd1c91-5fb9c8fcf9-xvrkd\" (UID: \"600fee45-c980-4705-96d9-fc084f4bea96\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-fd1c91-5fb9c8fcf9-xvrkd"
Oct 31 19:40:04.948824 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:04.888850    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/600fee45-c980-4705-96d9-fc084f4bea96-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-fd1c91-5fb9c8fcf9-xvrkd\" (UID: \"600fee45-c980-4705-96d9-fc084f4bea96\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-fd1c91-5fb9c8fcf9-xvrkd"
Oct 31 19:40:04.948824 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:04.910896    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/600fee45-c980-4705-96d9-fc084f4bea96-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-fd1c91-5fb9c8fcf9-xvrkd\" (UID: \"600fee45-c980-4705-96d9-fc084f4bea96\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-fd1c91-5fb9c8fcf9-xvrkd"
Oct 31 19:40:04.948824 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:04.911816    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af2f7267-2d2f-46cf-b8e2-101e6a328b68-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-fd1c91-84dcbc9465-5r662\" (UID: \"af2f7267-2d2f-46cf-b8e2-101e6a328b68\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-fd1c91-84dcbc9465-5r662"
Oct 31 19:40:04.991360 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:04.988046    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pckj7\" (UniqueName: \"kubernetes.io/projected/600fee45-c980-4705-96d9-fc084f4bea96-kube-api-access-pckj7\") pod \"payload-pre-processing-e2e-shared-a-w0-fd1c91-5fb9c8fcf9-xvrkd\" (UID: \"600fee45-c980-4705-96d9-fc084f4bea96\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-fd1c91-5fb9c8fcf9-xvrkd"
Oct 31 19:40:05.015837 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:05.015283    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7bm6\" (UniqueName: \"kubernetes.io/projected/af2f7267-2d2f-46cf-b8e2-101e6a328b68-kube-api-access-x7bm6\") pod \"payload-processing-e2e-shared-a-w0-fd1c91-84dcbc9465-5r662\" (UID: \"af2f7267-2d2f-46cf-b8e2-101e6a328b68\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-fd1c91-84dcbc9465-5r662"
Oct 31 19:40:05.172586 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:05.172547    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-fd1c91-5fb9c8fcf9-xvrkd"
Oct 31 19:40:05.233210 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:05.233181    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:05.238056 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:05.237246    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-fd1c91-84dcbc9465-5r662"
Oct 31 19:40:05.472882 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:40:05.472845    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod600fee45_c980_4705_96d9_fc084f4bea96.slice/crio-f32e481bf5131eb27288879e456bb047437e92b720e73d500241d43ccbb9dd40 WatchSource:0}: Error finding container f32e481bf5131eb27288879e456bb047437e92b720e73d500241d43ccbb9dd40: Status 404 returned error can't find the container with id f32e481bf5131eb27288879e456bb047437e92b720e73d500241d43ccbb9dd40
Oct 31 19:40:05.473813 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:05.473787    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-fd1c91-5fb9c8fcf9-xvrkd"]
Oct 31 19:40:05.644313 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:05.643898    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-fd1c91-84dcbc9465-5r662"]
Oct 31 19:40:05.658773 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:05.657390    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-fd1c91-5fb9c8fcf9-xvrkd" event={"ID":"600fee45-c980-4705-96d9-fc084f4bea96","Type":"ContainerStarted","Data":"60fa6d013b43f2246d859b83c10f7cd8a182fe5e0d67b6ce03f08ddb863f7e7a"}
Oct 31 19:40:05.658773 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:05.657457    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-fd1c91-5fb9c8fcf9-xvrkd" event={"ID":"600fee45-c980-4705-96d9-fc084f4bea96","Type":"ContainerStarted","Data":"f32e481bf5131eb27288879e456bb047437e92b720e73d500241d43ccbb9dd40"}
Oct 31 19:40:05.658773 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:05.657628    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-fd1c91-5fb9c8fcf9-xvrkd"
Oct 31 19:40:05.674359 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:40:05.674308    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf2f7267_2d2f_46cf_b8e2_101e6a328b68.slice/crio-39863d5ca8bf99d815b2bbfe79224ba12e1df4a1ed80b7ffe3a6a588e22b4d55 WatchSource:0}: Error finding container 39863d5ca8bf99d815b2bbfe79224ba12e1df4a1ed80b7ffe3a6a588e22b4d55: Status 404 returned error can't find the container with id 39863d5ca8bf99d815b2bbfe79224ba12e1df4a1ed80b7ffe3a6a588e22b4d55
Oct 31 19:40:05.678984 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:05.677954    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-fd1c91-5fb9c8fcf9-xvrkd" podStartSLOduration=1.677934595 podStartE2EDuration="1.677934595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:40:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:40:05.674184952 +0000 UTC m=+1236.247419080" watchObservedRunningTime="2026-10-31 19:40:05.677934595 +0000 UTC m=+1236.251168726"
Oct 31 19:40:05.987811 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:05.987783    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7011-e54d70"
Oct 31 19:40:06.175572 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:06.175540    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79002f30-65de-4817-9c86-79b2b3e12643-kube-api-access-dn4nr\" (UniqueName: \"kubernetes.io/projected/79002f30-65de-4817-9c86-79b2b3e12643-kube-api-access-dn4nr\") pod \"79002f30-65de-4817-9c86-79b2b3e12643\" (UID: \"79002f30-65de-4817-9c86-79b2b3e12643\") "
Oct 31 19:40:06.177743 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:06.177716    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79002f30-65de-4817-9c86-79b2b3e12643-kube-api-access-dn4nr" pod "79002f30-65de-4817-9c86-79b2b3e12643" (UID: "79002f30-65de-4817-9c86-79b2b3e12643"). InnerVolumeSpecName "kube-api-access-dn4nr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:40:06.249653 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:40:06.249576    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 19:40:06.265816 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:06.265785    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:06.277626 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:06.277604    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dn4nr\" (UniqueName: \"kubernetes.io/projected/79002f30-65de-4817-9c86-79b2b3e12643-kube-api-access-dn4nr\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:06.667055 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:06.667017    4645 generic.go:356] "Generic (PLEG): container finished" podID="79002f30-65de-4817-9c86-79b2b3e12643" containerID="08d4c6a0f480f6487f63625988c856497665ffbbc6fe48f7e21bbed88e03550c" exitCode=137
Oct 31 19:40:06.667570 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:06.667075    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7011-e54d70"
Oct 31 19:40:06.667570 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:06.667125    4645 scope.go:122] "RemoveContainer" containerID="08d4c6a0f480f6487f63625988c856497665ffbbc6fe48f7e21bbed88e03550c"
Oct 31 19:40:06.669227 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:06.669184    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-fd1c91-84dcbc9465-5r662" event={"ID":"af2f7267-2d2f-46cf-b8e2-101e6a328b68","Type":"ContainerStarted","Data":"0074c9f1f0aa3e09ff2e12f2d179a38bd914178c99b2596828a33eef2412f8c2"}
Oct 31 19:40:06.669375 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:06.669248    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-fd1c91-84dcbc9465-5r662" event={"ID":"af2f7267-2d2f-46cf-b8e2-101e6a328b68","Type":"ContainerStarted","Data":"39863d5ca8bf99d815b2bbfe79224ba12e1df4a1ed80b7ffe3a6a588e22b4d55"}
Oct 31 19:40:06.669510 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:06.669490    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-fd1c91-84dcbc9465-5r662"
Oct 31 19:40:06.681405 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:06.681384    4645 scope.go:122] "RemoveContainer" containerID="08d4c6a0f480f6487f63625988c856497665ffbbc6fe48f7e21bbed88e03550c"
Oct 31 19:40:06.681753 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:40:06.681726    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08d4c6a0f480f6487f63625988c856497665ffbbc6fe48f7e21bbed88e03550c\": container with ID starting with 08d4c6a0f480f6487f63625988c856497665ffbbc6fe48f7e21bbed88e03550c not found: ID does not exist" containerID="08d4c6a0f480f6487f63625988c856497665ffbbc6fe48f7e21bbed88e03550c"
Oct 31 19:40:06.681910 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:06.681768    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08d4c6a0f480f6487f63625988c856497665ffbbc6fe48f7e21bbed88e03550c"} err="failed to get container status \"08d4c6a0f480f6487f63625988c856497665ffbbc6fe48f7e21bbed88e03550c\": rpc error: code = NotFound desc = could not find container \"08d4c6a0f480f6487f63625988c856497665ffbbc6fe48f7e21bbed88e03550c\": container with ID starting with 08d4c6a0f480f6487f63625988c856497665ffbbc6fe48f7e21bbed88e03550c not found: ID does not exist"
Oct 31 19:40:06.691818 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:40:06.691790    4645 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7011-e54d70\" is forbidden: User \"system:node:ip-10-0-131-94.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-94.us-east-2.compute.internal' and this object" podUID="79002f30-65de-4817-9c86-79b2b3e12643" pod="opendatahub/test-curl-7011-e54d70"
Oct 31 19:40:06.696381 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:06.696339    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-fd1c91-84dcbc9465-5r662" podStartSLOduration=2.696314591 podStartE2EDuration="2.696314591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:40:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:40:06.689673876 +0000 UTC m=+1237.262908005" watchObservedRunningTime="2026-10-31 19:40:06.696314591 +0000 UTC m=+1237.269548720"
Oct 31 19:40:06.777836 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:06.777635    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:07.152662 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:07.152584    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:07.396353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:07.359797    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:07.496344 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:07.487960    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:07.519349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:07.514513    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-e73256-57b5c8849f-n9qzl"
Oct 31 19:40:07.581629 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:40:07.581592    4645 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7011-e54d70\" is forbidden: User \"system:node:ip-10-0-131-94.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-94.us-east-2.compute.internal' and this object" podUID="79002f30-65de-4817-9c86-79b2b3e12643" pod="opendatahub/test-curl-7011-e54d70"
Oct 31 19:40:07.722272 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:07.722237    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79002f30-65de-4817-9c86-79b2b3e12643" path="/var/lib/kubelet/pods/79002f30-65de-4817-9c86-79b2b3e12643/volumes"
Oct 31 19:40:08.177835 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:08.177802    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:08.520129 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:08.518516    4645 patch_prober.go:29] interesting pod/e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": read tcp 10.133.0.2:45998->10.133.0.94:15021: read: connection reset by peer" start-of-body=
Oct 31 19:40:08.520129 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:08.518580    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl" podUID="d505eb36-3800-4e5b-878d-c7ab58d3e901" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": read tcp 10.133.0.2:45998->10.133.0.94:15021: read: connection reset by peer"
Oct 31 19:40:08.756751 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:08.756723    4645 generic.go:356] "Generic (PLEG): container finished" podID="d505eb36-3800-4e5b-878d-c7ab58d3e901" containerID="23ffc973c645e851acd83dcea2e1d7f0ebd7e78d52fece4dc3fe4714de37d73e" exitCode=0
Oct 31 19:40:08.757150 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:08.756796    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl" event={"ID":"d505eb36-3800-4e5b-878d-c7ab58d3e901","Type":"ContainerDied","Data":"23ffc973c645e851acd83dcea2e1d7f0ebd7e78d52fece4dc3fe4714de37d73e"}
Oct 31 19:40:08.757211 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:08.757153    4645 scope.go:122] "RemoveContainer" containerID="23ffc973c645e851acd83dcea2e1d7f0ebd7e78d52fece4dc3fe4714de37d73e"
Oct 31 19:40:08.758672 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:08.758644    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:40:08.758734 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:08.758710    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:40:08.758770 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:08.758742    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:40:09.175096 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:09.175062    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:09.786590 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:09.786557    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl" event={"ID":"d505eb36-3800-4e5b-878d-c7ab58d3e901","Type":"ContainerStarted","Data":"4093fde47b02bdbc38bddfd04586bf689533e63c3b9210b5b21434ddd1862921"}
Oct 31 19:40:09.895038 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:09.895005    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e73256-98c5999-c2vqh"]
Oct 31 19:40:09.900720 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:09.900686    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e73256-98c5999-c2vqh"
Oct 31 19:40:09.913827 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:09.913791    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e73256-98c5999-c2vqh"]
Oct 31 19:40:09.914076 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:09.914053    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-e73256-serving-cert\""
Oct 31 19:40:09.946848 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:09.946812    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:09.985322 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:09.985266    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3bd9ab0-338b-4718-8489-beb3c99992f1-maas-api-tls\") pod \"maas-api-e2e-api-a-e73256-98c5999-c2vqh\" (UID: \"f3bd9ab0-338b-4718-8489-beb3c99992f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e73256-98c5999-c2vqh"
Oct 31 19:40:09.985565 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:09.985417    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3bd9ab0-338b-4718-8489-beb3c99992f1-openshift-service-ca\") pod \"maas-api-e2e-api-a-e73256-98c5999-c2vqh\" (UID: \"f3bd9ab0-338b-4718-8489-beb3c99992f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e73256-98c5999-c2vqh"
Oct 31 19:40:09.985565 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:09.985488    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpqfm\" (UniqueName: \"kubernetes.io/projected/f3bd9ab0-338b-4718-8489-beb3c99992f1-kube-api-access-lpqfm\") pod \"maas-api-e2e-api-a-e73256-98c5999-c2vqh\" (UID: \"f3bd9ab0-338b-4718-8489-beb3c99992f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e73256-98c5999-c2vqh"
Oct 31 19:40:09.985565 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:09.985541    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3bd9ab0-338b-4718-8489-beb3c99992f1-metrics-certs\") pod \"maas-api-e2e-api-a-e73256-98c5999-c2vqh\" (UID: \"f3bd9ab0-338b-4718-8489-beb3c99992f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e73256-98c5999-c2vqh"
Oct 31 19:40:10.083415 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:10.079268    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:10.096433 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:10.091188    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3bd9ab0-338b-4718-8489-beb3c99992f1-openshift-service-ca\") pod \"maas-api-e2e-api-a-e73256-98c5999-c2vqh\" (UID: \"f3bd9ab0-338b-4718-8489-beb3c99992f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e73256-98c5999-c2vqh"
Oct 31 19:40:10.096433 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:10.091370    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpqfm\" (UniqueName: \"kubernetes.io/projected/f3bd9ab0-338b-4718-8489-beb3c99992f1-kube-api-access-lpqfm\") pod \"maas-api-e2e-api-a-e73256-98c5999-c2vqh\" (UID: \"f3bd9ab0-338b-4718-8489-beb3c99992f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e73256-98c5999-c2vqh"
Oct 31 19:40:10.096433 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:10.091474    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3bd9ab0-338b-4718-8489-beb3c99992f1-metrics-certs\") pod \"maas-api-e2e-api-a-e73256-98c5999-c2vqh\" (UID: \"f3bd9ab0-338b-4718-8489-beb3c99992f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e73256-98c5999-c2vqh"
Oct 31 19:40:10.096433 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:10.091594    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3bd9ab0-338b-4718-8489-beb3c99992f1-maas-api-tls\") pod \"maas-api-e2e-api-a-e73256-98c5999-c2vqh\" (UID: \"f3bd9ab0-338b-4718-8489-beb3c99992f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e73256-98c5999-c2vqh"
Oct 31 19:40:10.096433 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:40:10.091823    4645 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-e73256-serving-cert: secret "maas-api-e2e-api-a-e73256-serving-cert" not found
Oct 31 19:40:10.096433 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:40:10.091901    4645 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3bd9ab0-338b-4718-8489-beb3c99992f1-maas-api-tls podName:f3bd9ab0-338b-4718-8489-beb3c99992f1 nodeName:}" failed. No retries permitted until 2026-10-31 19:40:10.591879301 +0000 UTC m=+1241.165113426 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f3bd9ab0-338b-4718-8489-beb3c99992f1-maas-api-tls") pod "maas-api-e2e-api-a-e73256-98c5999-c2vqh" (UID: "f3bd9ab0-338b-4718-8489-beb3c99992f1") : secret "maas-api-e2e-api-a-e73256-serving-cert" not found
Oct 31 19:40:10.100356 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:10.099696    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3bd9ab0-338b-4718-8489-beb3c99992f1-openshift-service-ca\") pod \"maas-api-e2e-api-a-e73256-98c5999-c2vqh\" (UID: \"f3bd9ab0-338b-4718-8489-beb3c99992f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e73256-98c5999-c2vqh"
Oct 31 19:40:10.153982 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:10.147468    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpqfm\" (UniqueName: \"kubernetes.io/projected/f3bd9ab0-338b-4718-8489-beb3c99992f1-kube-api-access-lpqfm\") pod \"maas-api-e2e-api-a-e73256-98c5999-c2vqh\" (UID: \"f3bd9ab0-338b-4718-8489-beb3c99992f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e73256-98c5999-c2vqh"
Oct 31 19:40:10.154269 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:10.154241    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3bd9ab0-338b-4718-8489-beb3c99992f1-metrics-certs\") pod \"maas-api-e2e-api-a-e73256-98c5999-c2vqh\" (UID: \"f3bd9ab0-338b-4718-8489-beb3c99992f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e73256-98c5999-c2vqh"
Oct 31 19:40:10.313964 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:10.313927    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:10.589352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:10.588613    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl"
Oct 31 19:40:10.589352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:10.589082    4645 patch_prober.go:29] interesting pod/e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 31 19:40:10.589352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:10.589147    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl" podUID="d505eb36-3800-4e5b-878d-c7ab58d3e901" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 31 19:40:10.624359 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:10.612755    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3bd9ab0-338b-4718-8489-beb3c99992f1-maas-api-tls\") pod \"maas-api-e2e-api-a-e73256-98c5999-c2vqh\" (UID: \"f3bd9ab0-338b-4718-8489-beb3c99992f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e73256-98c5999-c2vqh"
Oct 31 19:40:10.639361 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:10.629566    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3bd9ab0-338b-4718-8489-beb3c99992f1-maas-api-tls\") pod \"maas-api-e2e-api-a-e73256-98c5999-c2vqh\" (UID: \"f3bd9ab0-338b-4718-8489-beb3c99992f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e73256-98c5999-c2vqh"
Oct 31 19:40:10.831753 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:10.831721    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e73256-98c5999-c2vqh"
Oct 31 19:40:11.187753 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:11.187717    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e73256-98c5999-c2vqh"]
Oct 31 19:40:11.188489 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:40:11.188461    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3bd9ab0_338b_4718_8489_beb3c99992f1.slice/crio-a1485bc914b42d67c2a1ef74e11e0a262026e73bbd9c731d7a0044ce8107139b WatchSource:0}: Error finding container a1485bc914b42d67c2a1ef74e11e0a262026e73bbd9c731d7a0044ce8107139b: Status 404 returned error can't find the container with id a1485bc914b42d67c2a1ef74e11e0a262026e73bbd9c731d7a0044ce8107139b
Oct 31 19:40:11.213604 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:11.213564    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:11.516794 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:11.516705    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:11.580698 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:11.580594    4645 patch_prober.go:29] interesting pod/e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 31 19:40:11.580698 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:11.580653    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl" podUID="d505eb36-3800-4e5b-878d-c7ab58d3e901" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 31 19:40:11.713156 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:11.713120    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:11.814186 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:11.814159    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e73256-98c5999-c2vqh" event={"ID":"f3bd9ab0-338b-4718-8489-beb3c99992f1","Type":"ContainerStarted","Data":"a1485bc914b42d67c2a1ef74e11e0a262026e73bbd9c731d7a0044ce8107139b"}
Oct 31 19:40:12.376352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:12.369844    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:12.584545 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:12.584501    4645 patch_prober.go:29] interesting pod/e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 31 19:40:12.584860 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:12.584822    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl" podUID="d505eb36-3800-4e5b-878d-c7ab58d3e901" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 31 19:40:12.787949 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:12.787914    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:12.822580 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:12.822535    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e73256-98c5999-c2vqh" event={"ID":"f3bd9ab0-338b-4718-8489-beb3c99992f1","Type":"ContainerStarted","Data":"6c0bb2586938193ce81a5c469f9e3eea69cc86e93ba4f57e0c138d34ef9b6235"}
Oct 31 19:40:12.822754 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:12.822644    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e73256-98c5999-c2vqh"
Oct 31 19:40:12.855166 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:12.855119    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e73256-98c5999-c2vqh" podStartSLOduration=3.242478774 podStartE2EDuration="3.855106264s" totalImagesPullingTime="612.62749ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:40:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:40:12.853692847 +0000 UTC m=+1243.426926975" watchObservedRunningTime="2026-10-31 19:40:12.855106264 +0000 UTC m=+1243.428340392"
Oct 31 19:40:12.934180 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:12.934147    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:13.483351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:13.482462    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-67adab47-7ddbcc466c-n2bkz"]
Oct 31 19:40:13.489519 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:13.489342    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-67adab47-998479859-c8zq5"]
Oct 31 19:40:13.549914 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:13.549881    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:13.585936 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:13.585897    4645 patch_prober.go:29] interesting pod/e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 31 19:40:13.586094 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:13.586000    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl" podUID="d505eb36-3800-4e5b-878d-c7ab58d3e901" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 31 19:40:13.653909 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:13.653882    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:13.781892 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:13.781801    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:14.580833 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:14.580796    4645 patch_prober.go:29] interesting pod/e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Oct 31 19:40:14.581405 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:14.580856    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl" podUID="d505eb36-3800-4e5b-878d-c7ab58d3e901" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Oct 31 19:40:14.775810 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:14.775775    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:15.643761 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:15.643724    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl"
Oct 31 19:40:15.644204 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:15.644004    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl"
Oct 31 19:40:15.644962 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:15.644929    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl"
Oct 31 19:40:15.755417 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:15.755386    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:16.160815 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:16.160781    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:16.575729 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:16.575643    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:16.674546 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:16.674517    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-fd1c91-5fb9c8fcf9-xvrkd"
Oct 31 19:40:17.370995 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:17.370960    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:17.751117 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:17.751090    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-fd1c91-84dcbc9465-5r662"
Oct 31 19:40:18.220357 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:18.215777    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:18.483813 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:18.483723    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-67adab47-7ddbcc466c-n2bkz" podUID="be80a9a3-7971-49e3-9ab3-92fec946a71f" containerName="payload-processing" containerID="cri-o://045435328da86f1f3a68a82b74e6a62fd2a50bafff4574781e6d04746ff48cf0" gracePeriod=25
Oct 31 19:40:18.490278 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:18.490234    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-67adab47-998479859-c8zq5" podUID="cff005f5-fa01-4cc4-a22f-c0c884db435a" containerName="payload-pre-processing" containerID="cri-o://59e61b64adc0cf959c9976c7f29285dd6b09c8d132e33e9bede83a0165fa8ce8" gracePeriod=25
Oct 31 19:40:18.814045 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:18.814015    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-67adab47-7ddbcc466c-n2bkz"
Oct 31 19:40:18.862678 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:18.862645    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e73256-98c5999-c2vqh"
Oct 31 19:40:18.872901 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:18.872871    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:18.961352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:18.932067    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be80a9a3-7971-49e3-9ab3-92fec946a71f-kube-api-access-5cc6d\" (UniqueName: \"kubernetes.io/projected/be80a9a3-7971-49e3-9ab3-92fec946a71f-kube-api-access-5cc6d\") pod \"be80a9a3-7971-49e3-9ab3-92fec946a71f\" (UID: \"be80a9a3-7971-49e3-9ab3-92fec946a71f\") "
Oct 31 19:40:19.003435 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:18.974582    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be80a9a3-7971-49e3-9ab3-92fec946a71f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be80a9a3-7971-49e3-9ab3-92fec946a71f-plugins-config-volume\") pod \"be80a9a3-7971-49e3-9ab3-92fec946a71f\" (UID: \"be80a9a3-7971-49e3-9ab3-92fec946a71f\") "
Oct 31 19:40:19.003435 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:18.998965    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be80a9a3-7971-49e3-9ab3-92fec946a71f-kube-api-access-5cc6d" pod "be80a9a3-7971-49e3-9ab3-92fec946a71f" (UID: "be80a9a3-7971-49e3-9ab3-92fec946a71f"). InnerVolumeSpecName "kube-api-access-5cc6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:40:19.003435 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:18.999374    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be80a9a3-7971-49e3-9ab3-92fec946a71f-plugins-config-volume" pod "be80a9a3-7971-49e3-9ab3-92fec946a71f" (UID: "be80a9a3-7971-49e3-9ab3-92fec946a71f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:40:19.120947 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:19.120799    4645 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be80a9a3-7971-49e3-9ab3-92fec946a71f-plugins-config-volume\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:19.120947 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:19.120860    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cc6d\" (UniqueName: \"kubernetes.io/projected/be80a9a3-7971-49e3-9ab3-92fec946a71f-kube-api-access-5cc6d\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:19.298273 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:19.298240    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-67adab47-998479859-c8zq5"
Oct 31 19:40:19.302193 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:19.301599    4645 generic.go:356] "Generic (PLEG): container finished" podID="cff005f5-fa01-4cc4-a22f-c0c884db435a" containerID="59e61b64adc0cf959c9976c7f29285dd6b09c8d132e33e9bede83a0165fa8ce8" exitCode=0
Oct 31 19:40:19.302193 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:19.301762    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-67adab47-998479859-c8zq5" event={"ID":"cff005f5-fa01-4cc4-a22f-c0c884db435a","Type":"ContainerDied","Data":"59e61b64adc0cf959c9976c7f29285dd6b09c8d132e33e9bede83a0165fa8ce8"}
Oct 31 19:40:19.302193 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:19.301816    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-67adab47-998479859-c8zq5" event={"ID":"cff005f5-fa01-4cc4-a22f-c0c884db435a","Type":"ContainerDied","Data":"7632121fd5753600e1e72dcd901443a11d31cbb953ebfbeed697410ba0d192cf"}
Oct 31 19:40:19.302193 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:19.301847    4645 scope.go:122] "RemoveContainer" containerID="59e61b64adc0cf959c9976c7f29285dd6b09c8d132e33e9bede83a0165fa8ce8"
Oct 31 19:40:19.305249 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:19.305226    4645 generic.go:356] "Generic (PLEG): container finished" podID="be80a9a3-7971-49e3-9ab3-92fec946a71f" containerID="045435328da86f1f3a68a82b74e6a62fd2a50bafff4574781e6d04746ff48cf0" exitCode=0
Oct 31 19:40:19.305492 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:19.305473    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-67adab47-7ddbcc466c-n2bkz" event={"ID":"be80a9a3-7971-49e3-9ab3-92fec946a71f","Type":"ContainerDied","Data":"045435328da86f1f3a68a82b74e6a62fd2a50bafff4574781e6d04746ff48cf0"}
Oct 31 19:40:19.305587 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:19.305518    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-67adab47-7ddbcc466c-n2bkz" event={"ID":"be80a9a3-7971-49e3-9ab3-92fec946a71f","Type":"ContainerDied","Data":"520f079a70c6e132369b5ebbc6827797f958c46681126da477d8c47917846217"}
Oct 31 19:40:19.305587 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:19.305581    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-67adab47-7ddbcc466c-n2bkz"
Oct 31 19:40:19.328391 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:19.325127    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cff005f5-fa01-4cc4-a22f-c0c884db435a-kube-api-access-8cgq2\" (UniqueName: \"kubernetes.io/projected/cff005f5-fa01-4cc4-a22f-c0c884db435a-kube-api-access-8cgq2\") pod \"cff005f5-fa01-4cc4-a22f-c0c884db435a\" (UID: \"cff005f5-fa01-4cc4-a22f-c0c884db435a\") "
Oct 31 19:40:19.328391 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:19.325679    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cff005f5-fa01-4cc4-a22f-c0c884db435a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cff005f5-fa01-4cc4-a22f-c0c884db435a-plugins-config-volume\") pod \"cff005f5-fa01-4cc4-a22f-c0c884db435a\" (UID: \"cff005f5-fa01-4cc4-a22f-c0c884db435a\") "
Oct 31 19:40:19.328391 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:19.327672    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cff005f5-fa01-4cc4-a22f-c0c884db435a-plugins-config-volume" pod "cff005f5-fa01-4cc4-a22f-c0c884db435a" (UID: "cff005f5-fa01-4cc4-a22f-c0c884db435a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:40:19.328726 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:19.328598    4645 scope.go:122] "RemoveContainer" containerID="59e61b64adc0cf959c9976c7f29285dd6b09c8d132e33e9bede83a0165fa8ce8"
Oct 31 19:40:19.329731 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:40:19.329633    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59e61b64adc0cf959c9976c7f29285dd6b09c8d132e33e9bede83a0165fa8ce8\": container with ID starting with 59e61b64adc0cf959c9976c7f29285dd6b09c8d132e33e9bede83a0165fa8ce8 not found: ID does not exist" containerID="59e61b64adc0cf959c9976c7f29285dd6b09c8d132e33e9bede83a0165fa8ce8"
Oct 31 19:40:19.329731 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:19.329692    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59e61b64adc0cf959c9976c7f29285dd6b09c8d132e33e9bede83a0165fa8ce8"} err="failed to get container status \"59e61b64adc0cf959c9976c7f29285dd6b09c8d132e33e9bede83a0165fa8ce8\": rpc error: code = NotFound desc = could not find container \"59e61b64adc0cf959c9976c7f29285dd6b09c8d132e33e9bede83a0165fa8ce8\": container with ID starting with 59e61b64adc0cf959c9976c7f29285dd6b09c8d132e33e9bede83a0165fa8ce8 not found: ID does not exist"
Oct 31 19:40:19.329731 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:19.329716    4645 scope.go:122] "RemoveContainer" containerID="045435328da86f1f3a68a82b74e6a62fd2a50bafff4574781e6d04746ff48cf0"
Oct 31 19:40:19.336399 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:19.336375    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cff005f5-fa01-4cc4-a22f-c0c884db435a-kube-api-access-8cgq2" pod "cff005f5-fa01-4cc4-a22f-c0c884db435a" (UID: "cff005f5-fa01-4cc4-a22f-c0c884db435a"). InnerVolumeSpecName "kube-api-access-8cgq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:40:19.346807 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:19.346719    4645 scope.go:122] "RemoveContainer" containerID="045435328da86f1f3a68a82b74e6a62fd2a50bafff4574781e6d04746ff48cf0"
Oct 31 19:40:19.347094 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:40:19.347058    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"045435328da86f1f3a68a82b74e6a62fd2a50bafff4574781e6d04746ff48cf0\": container with ID starting with 045435328da86f1f3a68a82b74e6a62fd2a50bafff4574781e6d04746ff48cf0 not found: ID does not exist" containerID="045435328da86f1f3a68a82b74e6a62fd2a50bafff4574781e6d04746ff48cf0"
Oct 31 19:40:19.347167 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:19.347105    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"045435328da86f1f3a68a82b74e6a62fd2a50bafff4574781e6d04746ff48cf0"} err="failed to get container status \"045435328da86f1f3a68a82b74e6a62fd2a50bafff4574781e6d04746ff48cf0\": rpc error: code = NotFound desc = could not find container \"045435328da86f1f3a68a82b74e6a62fd2a50bafff4574781e6d04746ff48cf0\": container with ID starting with 045435328da86f1f3a68a82b74e6a62fd2a50bafff4574781e6d04746ff48cf0 not found: ID does not exist"
Oct 31 19:40:19.353522 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:19.353186    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-67adab47-7ddbcc466c-n2bkz"]
Oct 31 19:40:19.359269 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:19.359236    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-67adab47-7ddbcc466c-n2bkz"]
Oct 31 19:40:19.433872 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:19.433835    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cgq2\" (UniqueName: \"kubernetes.io/projected/cff005f5-fa01-4cc4-a22f-c0c884db435a-kube-api-access-8cgq2\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:19.434191 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:19.434106    4645 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cff005f5-fa01-4cc4-a22f-c0c884db435a-plugins-config-volume\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:19.623060 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:19.623022    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be80a9a3-7971-49e3-9ab3-92fec946a71f" path="/var/lib/kubelet/pods/be80a9a3-7971-49e3-9ab3-92fec946a71f/volumes"
Oct 31 19:40:19.874464 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:19.874370    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:20.356435 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:20.355144    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-67adab47-998479859-c8zq5"
Oct 31 19:40:20.430452 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:20.430416    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-67adab47-998479859-c8zq5"]
Oct 31 19:40:20.444352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:20.441382    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-67adab47-998479859-c8zq5"]
Oct 31 19:40:20.853835 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:20.853795    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:20.965356 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:20.962911    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:21.083351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:21.077430    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:21.723349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:21.720148    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cff005f5-fa01-4cc4-a22f-c0c884db435a" path="/var/lib/kubelet/pods/cff005f5-fa01-4cc4-a22f-c0c884db435a/volumes"
Oct 31 19:40:21.979405 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:21.978972    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:22.988290 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:22.988256    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:23.797724 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:23.797687    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:24.451106 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:24.451079    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:24.668907 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:24.668874    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:24.867912 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:24.867824    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:25.128991 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:25.128907    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:25.444903 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:25.444867    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:26.228190 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:26.228150    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:27.246407 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:27.246373    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:28.003737 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:28.003653    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-67adab47-6cd87d9bdb-r5w6j"]
Oct 31 19:40:28.003955 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:28.003932    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-67adab47-6cd87d9bdb-r5w6j" podUID="68fd2c11-7939-43c0-96b2-7e3f0021a71b" containerName="maas-api" containerID="cri-o://8d5c600b49f55f78939130fc72c4357be3cefccc64a8e3289037469136eff697" gracePeriod=30
Oct 31 19:40:28.101587 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:28.101553    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:28.467361 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:28.422999    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:28.646407 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:28.646306    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:28.661545 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:28.660057    4645 generic.go:356] "Generic (PLEG): container finished" podID="68fd2c11-7939-43c0-96b2-7e3f0021a71b" containerID="8d5c600b49f55f78939130fc72c4357be3cefccc64a8e3289037469136eff697" exitCode=0
Oct 31 19:40:28.661545 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:28.660270    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-67adab47-6cd87d9bdb-r5w6j" event={"ID":"68fd2c11-7939-43c0-96b2-7e3f0021a71b","Type":"ContainerDied","Data":"8d5c600b49f55f78939130fc72c4357be3cefccc64a8e3289037469136eff697"}
Oct 31 19:40:28.721288 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:28.721200    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-67adab47-6cd87d9bdb-r5w6j"
Oct 31 19:40:28.775724 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:28.775694    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:28.812148 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:28.812119    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68fd2c11-7939-43c0-96b2-7e3f0021a71b-kube-api-access-d8npm\" (UniqueName: \"kubernetes.io/projected/68fd2c11-7939-43c0-96b2-7e3f0021a71b-kube-api-access-d8npm\") pod \"68fd2c11-7939-43c0-96b2-7e3f0021a71b\" (UID: \"68fd2c11-7939-43c0-96b2-7e3f0021a71b\") "
Oct 31 19:40:28.812368 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:28.812208    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68fd2c11-7939-43c0-96b2-7e3f0021a71b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/68fd2c11-7939-43c0-96b2-7e3f0021a71b-metrics-certs\") pod \"68fd2c11-7939-43c0-96b2-7e3f0021a71b\" (UID: \"68fd2c11-7939-43c0-96b2-7e3f0021a71b\") "
Oct 31 19:40:28.812368 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:28.812263    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68fd2c11-7939-43c0-96b2-7e3f0021a71b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68fd2c11-7939-43c0-96b2-7e3f0021a71b-openshift-service-ca\") pod \"68fd2c11-7939-43c0-96b2-7e3f0021a71b\" (UID: \"68fd2c11-7939-43c0-96b2-7e3f0021a71b\") "
Oct 31 19:40:28.812511 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:28.812410    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68fd2c11-7939-43c0-96b2-7e3f0021a71b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68fd2c11-7939-43c0-96b2-7e3f0021a71b-maas-api-tls\") pod \"68fd2c11-7939-43c0-96b2-7e3f0021a71b\" (UID: \"68fd2c11-7939-43c0-96b2-7e3f0021a71b\") "
Oct 31 19:40:28.812702 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:28.812673    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68fd2c11-7939-43c0-96b2-7e3f0021a71b-openshift-service-ca" pod "68fd2c11-7939-43c0-96b2-7e3f0021a71b" (UID: "68fd2c11-7939-43c0-96b2-7e3f0021a71b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:40:28.813134 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:28.813107    4645 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68fd2c11-7939-43c0-96b2-7e3f0021a71b-openshift-service-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:28.814542 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:28.814518    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68fd2c11-7939-43c0-96b2-7e3f0021a71b-kube-api-access-d8npm" pod "68fd2c11-7939-43c0-96b2-7e3f0021a71b" (UID: "68fd2c11-7939-43c0-96b2-7e3f0021a71b"). InnerVolumeSpecName "kube-api-access-d8npm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:40:28.814898 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:28.814870    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68fd2c11-7939-43c0-96b2-7e3f0021a71b-maas-api-tls" pod "68fd2c11-7939-43c0-96b2-7e3f0021a71b" (UID: "68fd2c11-7939-43c0-96b2-7e3f0021a71b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:40:28.814898 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:28.814893    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68fd2c11-7939-43c0-96b2-7e3f0021a71b-metrics-certs" pod "68fd2c11-7939-43c0-96b2-7e3f0021a71b" (UID: "68fd2c11-7939-43c0-96b2-7e3f0021a71b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:40:28.920028 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:28.919985    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8npm\" (UniqueName: \"kubernetes.io/projected/68fd2c11-7939-43c0-96b2-7e3f0021a71b-kube-api-access-d8npm\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:28.920028 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:28.920030    4645 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68fd2c11-7939-43c0-96b2-7e3f0021a71b-metrics-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:28.920028 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:28.920045    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68fd2c11-7939-43c0-96b2-7e3f0021a71b-maas-api-tls\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:29.158196 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:29.158113    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:29.668420 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:29.668149    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-67adab47-6cd87d9bdb-r5w6j"
Oct 31 19:40:29.668420 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:29.668187    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-67adab47-6cd87d9bdb-r5w6j" event={"ID":"68fd2c11-7939-43c0-96b2-7e3f0021a71b","Type":"ContainerDied","Data":"ca2e29e684aaf5d70b3651e66887b4578bc805f6260e3f912aa3b187eef8e6e5"}
Oct 31 19:40:29.668420 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:29.668252    4645 scope.go:122] "RemoveContainer" containerID="8d5c600b49f55f78939130fc72c4357be3cefccc64a8e3289037469136eff697"
Oct 31 19:40:29.874098 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:29.866766    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-67adab47-6cd87d9bdb-r5w6j"]
Oct 31 19:40:29.970016 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:29.968370    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-67adab47-6cd87d9bdb-r5w6j"]
Oct 31 19:40:30.036105 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:30.035918    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc"]
Oct 31 19:40:30.037390 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:30.037366    4645 cpu_manager.go:414] "containerMap: removing container" podUID="be80a9a3-7971-49e3-9ab3-92fec946a71f" containerName="payload-processing"
Oct 31 19:40:30.037519 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:30.037505    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be80a9a3-7971-49e3-9ab3-92fec946a71f" containerName="payload-processing"
Oct 31 19:40:30.037700 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:30.037688    4645 cpu_manager.go:414] "containerMap: removing container" podUID="68fd2c11-7939-43c0-96b2-7e3f0021a71b" containerName="maas-api"
Oct 31 19:40:30.037848 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:30.037825    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68fd2c11-7939-43c0-96b2-7e3f0021a71b" containerName="maas-api"
Oct 31 19:40:30.037983 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:30.037970    4645 cpu_manager.go:414] "containerMap: removing container" podUID="cff005f5-fa01-4cc4-a22f-c0c884db435a" containerName="payload-pre-processing"
Oct 31 19:40:30.038057 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:30.038046    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cff005f5-fa01-4cc4-a22f-c0c884db435a" containerName="payload-pre-processing"
Oct 31 19:40:30.038322 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:30.038307    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="68fd2c11-7939-43c0-96b2-7e3f0021a71b" containerName="maas-api"
Oct 31 19:40:30.038489 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:30.038476    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="cff005f5-fa01-4cc4-a22f-c0c884db435a" containerName="payload-pre-processing"
Oct 31 19:40:30.038580 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:30.038569    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="be80a9a3-7971-49e3-9ab3-92fec946a71f" containerName="payload-processing"
Oct 31 19:40:30.080182 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:30.080141    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc"]
Oct 31 19:40:30.080311 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:30.080279    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc"
Oct 31 19:40:30.171644 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:30.171611    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8844ebc7-a83d-4b83-859e-2eb4d1e816c2-istio-podinfo\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc\" (UID: \"8844ebc7-a83d-4b83-859e-2eb4d1e816c2\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc"
Oct 31 19:40:30.171836 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:30.171686    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xk4v\" (UniqueName: \"kubernetes.io/projected/8844ebc7-a83d-4b83-859e-2eb4d1e816c2-kube-api-access-9xk4v\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc\" (UID: \"8844ebc7-a83d-4b83-859e-2eb4d1e816c2\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc"
Oct 31 19:40:30.171836 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:30.171720    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8844ebc7-a83d-4b83-859e-2eb4d1e816c2-workload-socket\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc\" (UID: \"8844ebc7-a83d-4b83-859e-2eb4d1e816c2\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc"
Oct 31 19:40:30.171836 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:30.171736    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8844ebc7-a83d-4b83-859e-2eb4d1e816c2-workload-certs\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc\" (UID: \"8844ebc7-a83d-4b83-859e-2eb4d1e816c2\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc"
Oct 31 19:40:30.171998 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:30.171886    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8844ebc7-a83d-4b83-859e-2eb4d1e816c2-istio-envoy\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc\" (UID: \"8844ebc7-a83d-4b83-859e-2eb4d1e816c2\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc"
Oct 31 19:40:30.171998 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:30.171991    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8844ebc7-a83d-4b83-859e-2eb4d1e816c2-istio-data\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc\" (UID: \"8844ebc7-a83d-4b83-859e-2eb4d1e816c2\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc"
Oct 31 19:40:30.172103 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:30.172090    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8844ebc7-a83d-4b83-859e-2eb4d1e816c2-istiod-ca-cert\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc\" (UID: \"8844ebc7-a83d-4b83-859e-2eb4d1e816c2\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc"
Oct 31 19:40:30.172183 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:30.172157    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8844ebc7-a83d-4b83-859e-2eb4d1e816c2-istio-token\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc\" (UID: \"8844ebc7-a83d-4b83-859e-2eb4d1e816c2\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc"
Oct 31 19:40:30.172248 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:30.172233    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8844ebc7-a83d-4b83-859e-2eb4d1e816c2-credential-socket\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc\" (UID: \"8844ebc7-a83d-4b83-859e-2eb4d1e816c2\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc"
Oct 31 19:40:30.195940 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:30.195910    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-03dfee-7c45f6ff5c-swqcn"]
Oct 31 19:40:30.237121 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:30.233186    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-03dfee-7c45f6ff5c-swqcn"]
Oct 31 19:40:30.237121 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:30.233307    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-03dfee-7c45f6ff5c-swqcn"
Oct 31 19:40:30.239617 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:30.239591    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-03dfee-serving-cert\""
Oct 31 19:40:30.275358 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:30.275309    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nv87\" (UniqueName: \"kubernetes.io/projected/bf7da3cd-1911-44f0-9e13-b102f000b7d5-kube-api-access-4nv87\") pod \"maas-api-e2e-inf-a-03dfee-7c45f6ff5c-swqcn\" (UID: \"bf7da3cd-1911-44f0-9e13-b102f000b7d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-03dfee-7c45f6ff5c-swqcn"
Oct 31 19:40:30.275527 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:30.275403    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8844ebc7-a83d-4b83-859e-2eb4d1e816c2-istio-envoy\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc\" (UID: \"8844ebc7-a83d-4b83-859e-2eb4d1e816c2\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc"
Oct 31 19:40:30.275527 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:30.275437    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf7da3cd-1911-44f0-9e13-b102f000b7d5-metrics-certs\") pod \"maas-api-e2e-inf-a-03dfee-7c45f6ff5c-swqcn\" (UID: \"bf7da3cd-1911-44f0-9e13-b102f000b7d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-03dfee-7c45f6ff5c-swqcn"
Oct 31 19:40:30.275646 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:30.275556    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8844ebc7-a83d-4b83-859e-2eb4d1e816c2-istio-data\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc\" (UID: \"8844ebc7-a83d-4b83-859e-2eb4d1e816c2\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc"
Oct 31 19:40:30.275734 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:30.275719    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8844ebc7-a83d-4b83-859e-2eb4d1e816c2-istiod-ca-cert\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc\" (UID: \"8844ebc7-a83d-4b83-859e-2eb4d1e816c2\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc"
Oct 31 19:40:30.275804 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:30.275790    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8844ebc7-a83d-4b83-859e-2eb4d1e816c2-istio-token\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc\" (UID: \"8844ebc7-a83d-4b83-859e-2eb4d1e816c2\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc"
Oct 31 19:40:30.275871 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:30.275847    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8844ebc7-a83d-4b83-859e-2eb4d1e816c2-istio-data\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc\" (UID: \"8844ebc7-a83d-4b83-859e-2eb4d1e816c2\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc"
Oct 31 19:40:30.275997 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:30.275853    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8844ebc7-a83d-4b83-859e-2eb4d1e816c2-credential-socket\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc\" (UID: \"8844ebc7-a83d-4b83-859e-2eb4d1e816c2\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc"
Oct 31 19:40:30.276354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:30.276312    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8844ebc7-a83d-4b83-859e-2eb4d1e816c2-istio-podinfo\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc\" (UID: \"8844ebc7-a83d-4b83-859e-2eb4d1e816c2\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc"
Oct 31 19:40:30.276449 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:30.276367    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf7da3cd-1911-44f0-9e13-b102f000b7d5-openshift-service-ca\") pod \"maas-api-e2e-inf-a-03dfee-7c45f6ff5c-swqcn\" (UID: \"bf7da3cd-1911-44f0-9e13-b102f000b7d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-03dfee-7c45f6ff5c-swqcn"
Oct 31 19:40:30.276449 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:30.276410    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8844ebc7-a83d-4b83-859e-2eb4d1e816c2-istiod-ca-cert\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc\" (UID: \"8844ebc7-a83d-4b83-859e-2eb4d1e816c2\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc"
Oct 31 19:40:30.276449 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:30.276434    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xk4v\" (UniqueName: \"kubernetes.io/projected/8844ebc7-a83d-4b83-859e-2eb4d1e816c2-kube-api-access-9xk4v\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc\" (UID: \"8844ebc7-a83d-4b83-859e-2eb4d1e816c2\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc"
Oct 31 19:40:30.276625 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:30.276479    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8844ebc7-a83d-4b83-859e-2eb4d1e816c2-workload-socket\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc\" (UID: \"8844ebc7-a83d-4b83-859e-2eb4d1e816c2\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc"
Oct 31 19:40:30.276625 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:30.276505    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8844ebc7-a83d-4b83-859e-2eb4d1e816c2-workload-certs\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc\" (UID: \"8844ebc7-a83d-4b83-859e-2eb4d1e816c2\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc"
Oct 31 19:40:30.276625 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:30.276543    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf7da3cd-1911-44f0-9e13-b102f000b7d5-maas-api-tls\") pod \"maas-api-e2e-inf-a-03dfee-7c45f6ff5c-swqcn\" (UID: \"bf7da3cd-1911-44f0-9e13-b102f000b7d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-03dfee-7c45f6ff5c-swqcn"
Oct 31 19:40:30.276781 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:30.276130    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8844ebc7-a83d-4b83-859e-2eb4d1e816c2-credential-socket\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc\" (UID: \"8844ebc7-a83d-4b83-859e-2eb4d1e816c2\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc"
Oct 31 19:40:30.277042 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:30.277014    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8844ebc7-a83d-4b83-859e-2eb4d1e816c2-workload-socket\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc\" (UID: \"8844ebc7-a83d-4b83-859e-2eb4d1e816c2\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc"
Oct 31 19:40:30.277148 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:30.277073    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8844ebc7-a83d-4b83-859e-2eb4d1e816c2-workload-certs\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc\" (UID: \"8844ebc7-a83d-4b83-859e-2eb4d1e816c2\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc"
Oct 31 19:40:30.278526 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:30.278501    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8844ebc7-a83d-4b83-859e-2eb4d1e816c2-istio-envoy\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc\" (UID: \"8844ebc7-a83d-4b83-859e-2eb4d1e816c2\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc"
Oct 31 19:40:30.279559 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:30.279537    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8844ebc7-a83d-4b83-859e-2eb4d1e816c2-istio-podinfo\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc\" (UID: \"8844ebc7-a83d-4b83-859e-2eb4d1e816c2\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc"
Oct 31 19:40:30.295654 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:30.295629    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8844ebc7-a83d-4b83-859e-2eb4d1e816c2-istio-token\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc\" (UID: \"8844ebc7-a83d-4b83-859e-2eb4d1e816c2\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc"
Oct 31 19:40:30.295756 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:30.295658    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xk4v\" (UniqueName: \"kubernetes.io/projected/8844ebc7-a83d-4b83-859e-2eb4d1e816c2-kube-api-access-9xk4v\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc\" (UID: \"8844ebc7-a83d-4b83-859e-2eb4d1e816c2\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc"
Oct 31 19:40:30.401312 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:30.400485    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf7da3cd-1911-44f0-9e13-b102f000b7d5-openshift-service-ca\") pod \"maas-api-e2e-inf-a-03dfee-7c45f6ff5c-swqcn\" (UID: \"bf7da3cd-1911-44f0-9e13-b102f000b7d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-03dfee-7c45f6ff5c-swqcn"
Oct 31 19:40:30.401312 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:30.400601    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf7da3cd-1911-44f0-9e13-b102f000b7d5-maas-api-tls\") pod \"maas-api-e2e-inf-a-03dfee-7c45f6ff5c-swqcn\" (UID: \"bf7da3cd-1911-44f0-9e13-b102f000b7d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-03dfee-7c45f6ff5c-swqcn"
Oct 31 19:40:30.401312 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:30.400682    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nv87\" (UniqueName: \"kubernetes.io/projected/bf7da3cd-1911-44f0-9e13-b102f000b7d5-kube-api-access-4nv87\") pod \"maas-api-e2e-inf-a-03dfee-7c45f6ff5c-swqcn\" (UID: \"bf7da3cd-1911-44f0-9e13-b102f000b7d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-03dfee-7c45f6ff5c-swqcn"
Oct 31 19:40:30.401312 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:30.400711    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf7da3cd-1911-44f0-9e13-b102f000b7d5-metrics-certs\") pod \"maas-api-e2e-inf-a-03dfee-7c45f6ff5c-swqcn\" (UID: \"bf7da3cd-1911-44f0-9e13-b102f000b7d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-03dfee-7c45f6ff5c-swqcn"
Oct 31 19:40:30.411390 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:40:30.401781    4645 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-03dfee-serving-cert: secret "maas-api-e2e-inf-a-03dfee-serving-cert" not found
Oct 31 19:40:30.411390 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:40:30.401852    4645 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf7da3cd-1911-44f0-9e13-b102f000b7d5-maas-api-tls podName:bf7da3cd-1911-44f0-9e13-b102f000b7d5 nodeName:}" failed. No retries permitted until 2026-10-31 19:40:30.901832295 +0000 UTC m=+1261.475066406 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bf7da3cd-1911-44f0-9e13-b102f000b7d5-maas-api-tls") pod "maas-api-e2e-inf-a-03dfee-7c45f6ff5c-swqcn" (UID: "bf7da3cd-1911-44f0-9e13-b102f000b7d5") : secret "maas-api-e2e-inf-a-03dfee-serving-cert" not found
Oct 31 19:40:30.411390 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:30.402879    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf7da3cd-1911-44f0-9e13-b102f000b7d5-openshift-service-ca\") pod \"maas-api-e2e-inf-a-03dfee-7c45f6ff5c-swqcn\" (UID: \"bf7da3cd-1911-44f0-9e13-b102f000b7d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-03dfee-7c45f6ff5c-swqcn"
Oct 31 19:40:30.420369 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:30.414884    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf7da3cd-1911-44f0-9e13-b102f000b7d5-metrics-certs\") pod \"maas-api-e2e-inf-a-03dfee-7c45f6ff5c-swqcn\" (UID: \"bf7da3cd-1911-44f0-9e13-b102f000b7d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-03dfee-7c45f6ff5c-swqcn"
Oct 31 19:40:30.437238 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:30.436019    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc"
Oct 31 19:40:30.437238 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:30.436422    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nv87\" (UniqueName: \"kubernetes.io/projected/bf7da3cd-1911-44f0-9e13-b102f000b7d5-kube-api-access-4nv87\") pod \"maas-api-e2e-inf-a-03dfee-7c45f6ff5c-swqcn\" (UID: \"bf7da3cd-1911-44f0-9e13-b102f000b7d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-03dfee-7c45f6ff5c-swqcn"
Oct 31 19:40:30.495518 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:30.492552    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:30.604076 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:30.604045    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:30.775853 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:40:30.775814    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8844ebc7_a83d_4b83_859e_2eb4d1e816c2.slice/crio-43f8d43cb61a5a2576d7fd746acf91ef1c4652682154d3a048b10b8aa2d3bf04 WatchSource:0}: Error finding container 43f8d43cb61a5a2576d7fd746acf91ef1c4652682154d3a048b10b8aa2d3bf04: Status 404 returned error can't find the container with id 43f8d43cb61a5a2576d7fd746acf91ef1c4652682154d3a048b10b8aa2d3bf04
Oct 31 19:40:30.776991 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:30.776914    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc"]
Oct 31 19:40:30.779468 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:30.779430    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:40:30.779580 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:30.779500    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:40:30.779580 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:30.779537    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:40:30.909571 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:30.909537    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf7da3cd-1911-44f0-9e13-b102f000b7d5-maas-api-tls\") pod \"maas-api-e2e-inf-a-03dfee-7c45f6ff5c-swqcn\" (UID: \"bf7da3cd-1911-44f0-9e13-b102f000b7d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-03dfee-7c45f6ff5c-swqcn"
Oct 31 19:40:30.911843 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:30.911821    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf7da3cd-1911-44f0-9e13-b102f000b7d5-maas-api-tls\") pod \"maas-api-e2e-inf-a-03dfee-7c45f6ff5c-swqcn\" (UID: \"bf7da3cd-1911-44f0-9e13-b102f000b7d5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-03dfee-7c45f6ff5c-swqcn"
Oct 31 19:40:31.169960 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:31.168526    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-03dfee-7c45f6ff5c-swqcn"
Oct 31 19:40:31.228378 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:31.224867    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-67adab47-6cd87d9bdb-r5w6j" podUID="68fd2c11-7939-43c0-96b2-7e3f0021a71b" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.103:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:40:31.564862 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:31.564833    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-03dfee-7c45f6ff5c-swqcn"]
Oct 31 19:40:31.565306 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:40:31.565281    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf7da3cd_1911_44f0_9e13_b102f000b7d5.slice/crio-062a512e51c355c70b02bd7f000338b254582d93864f3dfd6cc564c5f1a2dfa6 WatchSource:0}: Error finding container 062a512e51c355c70b02bd7f000338b254582d93864f3dfd6cc564c5f1a2dfa6: Status 404 returned error can't find the container with id 062a512e51c355c70b02bd7f000338b254582d93864f3dfd6cc564c5f1a2dfa6
Oct 31 19:40:31.619242 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:31.619214    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68fd2c11-7939-43c0-96b2-7e3f0021a71b" path="/var/lib/kubelet/pods/68fd2c11-7939-43c0-96b2-7e3f0021a71b/volumes"
Oct 31 19:40:31.685579 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:31.685542    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-03dfee-7c45f6ff5c-swqcn" event={"ID":"bf7da3cd-1911-44f0-9e13-b102f000b7d5","Type":"ContainerStarted","Data":"062a512e51c355c70b02bd7f000338b254582d93864f3dfd6cc564c5f1a2dfa6"}
Oct 31 19:40:31.687405 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:31.687378    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc" event={"ID":"8844ebc7-a83d-4b83-859e-2eb4d1e816c2","Type":"ContainerStarted","Data":"74805777beab59124da8de48f2182a3636ead44b1a52773b0670ac65012c59d1"}
Oct 31 19:40:31.687541 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:31.687426    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc" event={"ID":"8844ebc7-a83d-4b83-859e-2eb4d1e816c2","Type":"ContainerStarted","Data":"43f8d43cb61a5a2576d7fd746acf91ef1c4652682154d3a048b10b8aa2d3bf04"}
Oct 31 19:40:31.715172 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:31.715022    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc" podStartSLOduration=1.715006951 podStartE2EDuration="1.715006951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:40:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:40:31.712718413 +0000 UTC m=+1262.285952542" watchObservedRunningTime="2026-10-31 19:40:31.715006951 +0000 UTC m=+1262.288241082"
Oct 31 19:40:31.715957 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:31.715934    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:32.436829 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:32.436794    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc"
Oct 31 19:40:32.438034 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:32.438013    4645 patch_prober.go:29] interesting pod/e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Oct 31 19:40:32.438103 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:32.438059    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc" podUID="8844ebc7-a83d-4b83-859e-2eb4d1e816c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Oct 31 19:40:32.700145 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:32.700064    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7"]
Oct 31 19:40:32.942754 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:32.709726    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-03dfee-7c45f6ff5c-swqcn" event={"ID":"bf7da3cd-1911-44f0-9e13-b102f000b7d5","Type":"ContainerStarted","Data":"99c23c35bc4e7974b79555b922bdfac8457a10f289ce5beb8827a730140350d5"}
Oct 31 19:40:32.942754 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:32.709826    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7"
Oct 31 19:40:32.942754 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:32.715761    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7"]
Oct 31 19:40:32.942754 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:32.733931    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-03dfee-7c45f6ff5c-swqcn" podStartSLOduration=2.163460569 podStartE2EDuration="2.733917567s" totalImagesPullingTime="570.456998ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:40:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:40:32.732428862 +0000 UTC m=+1263.305662985" watchObservedRunningTime="2026-10-31 19:40:32.733917567 +0000 UTC m=+1263.307151696"
Oct 31 19:40:32.942754 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:32.783900    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:32.942754 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:32.808657    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd915c45-10cb-4d0f-a519-44bbe338e43d-istio-token\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7\" (UID: \"fd915c45-10cb-4d0f-a519-44bbe338e43d\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7"
Oct 31 19:40:32.942754 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:32.808716    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd915c45-10cb-4d0f-a519-44bbe338e43d-istio-envoy\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7\" (UID: \"fd915c45-10cb-4d0f-a519-44bbe338e43d\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7"
Oct 31 19:40:32.942754 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:32.808879    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd915c45-10cb-4d0f-a519-44bbe338e43d-istio-podinfo\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7\" (UID: \"fd915c45-10cb-4d0f-a519-44bbe338e43d\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7"
Oct 31 19:40:32.942754 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:32.808956    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd915c45-10cb-4d0f-a519-44bbe338e43d-istio-data\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7\" (UID: \"fd915c45-10cb-4d0f-a519-44bbe338e43d\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7"
Oct 31 19:40:32.942754 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:32.808975    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zctzl\" (UniqueName: \"kubernetes.io/projected/fd915c45-10cb-4d0f-a519-44bbe338e43d-kube-api-access-zctzl\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7\" (UID: \"fd915c45-10cb-4d0f-a519-44bbe338e43d\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7"
Oct 31 19:40:32.942754 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:32.809002    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd915c45-10cb-4d0f-a519-44bbe338e43d-workload-socket\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7\" (UID: \"fd915c45-10cb-4d0f-a519-44bbe338e43d\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7"
Oct 31 19:40:32.942754 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:32.809107    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd915c45-10cb-4d0f-a519-44bbe338e43d-credential-socket\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7\" (UID: \"fd915c45-10cb-4d0f-a519-44bbe338e43d\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7"
Oct 31 19:40:32.942754 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:32.809144    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd915c45-10cb-4d0f-a519-44bbe338e43d-istiod-ca-cert\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7\" (UID: \"fd915c45-10cb-4d0f-a519-44bbe338e43d\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7"
Oct 31 19:40:32.942754 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:32.809313    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd915c45-10cb-4d0f-a519-44bbe338e43d-workload-certs\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7\" (UID: \"fd915c45-10cb-4d0f-a519-44bbe338e43d\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7"
Oct 31 19:40:32.942754 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:32.919913    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd915c45-10cb-4d0f-a519-44bbe338e43d-istio-token\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7\" (UID: \"fd915c45-10cb-4d0f-a519-44bbe338e43d\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7"
Oct 31 19:40:32.942754 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:32.919986    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd915c45-10cb-4d0f-a519-44bbe338e43d-istio-envoy\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7\" (UID: \"fd915c45-10cb-4d0f-a519-44bbe338e43d\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7"
Oct 31 19:40:32.942754 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:32.920066    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd915c45-10cb-4d0f-a519-44bbe338e43d-istio-podinfo\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7\" (UID: \"fd915c45-10cb-4d0f-a519-44bbe338e43d\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7"
Oct 31 19:40:32.942754 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:32.920139    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd915c45-10cb-4d0f-a519-44bbe338e43d-istio-data\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7\" (UID: \"fd915c45-10cb-4d0f-a519-44bbe338e43d\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7"
Oct 31 19:40:32.942754 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:32.920167    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zctzl\" (UniqueName: \"kubernetes.io/projected/fd915c45-10cb-4d0f-a519-44bbe338e43d-kube-api-access-zctzl\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7\" (UID: \"fd915c45-10cb-4d0f-a519-44bbe338e43d\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7"
Oct 31 19:40:32.942754 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:32.920209    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd915c45-10cb-4d0f-a519-44bbe338e43d-workload-socket\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7\" (UID: \"fd915c45-10cb-4d0f-a519-44bbe338e43d\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7"
Oct 31 19:40:32.942754 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:32.920260    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd915c45-10cb-4d0f-a519-44bbe338e43d-credential-socket\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7\" (UID: \"fd915c45-10cb-4d0f-a519-44bbe338e43d\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7"
Oct 31 19:40:32.942754 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:32.920303    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd915c45-10cb-4d0f-a519-44bbe338e43d-istiod-ca-cert\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7\" (UID: \"fd915c45-10cb-4d0f-a519-44bbe338e43d\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7"
Oct 31 19:40:32.942754 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:32.920441    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd915c45-10cb-4d0f-a519-44bbe338e43d-workload-certs\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7\" (UID: \"fd915c45-10cb-4d0f-a519-44bbe338e43d\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7"
Oct 31 19:40:32.942754 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:32.921048    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd915c45-10cb-4d0f-a519-44bbe338e43d-workload-certs\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7\" (UID: \"fd915c45-10cb-4d0f-a519-44bbe338e43d\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7"
Oct 31 19:40:32.942754 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:32.922212    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd915c45-10cb-4d0f-a519-44bbe338e43d-credential-socket\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7\" (UID: \"fd915c45-10cb-4d0f-a519-44bbe338e43d\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7"
Oct 31 19:40:32.942754 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:32.922509    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd915c45-10cb-4d0f-a519-44bbe338e43d-workload-socket\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7\" (UID: \"fd915c45-10cb-4d0f-a519-44bbe338e43d\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7"
Oct 31 19:40:32.942754 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:32.923135    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd915c45-10cb-4d0f-a519-44bbe338e43d-istiod-ca-cert\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7\" (UID: \"fd915c45-10cb-4d0f-a519-44bbe338e43d\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7"
Oct 31 19:40:32.942754 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:32.928187    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd915c45-10cb-4d0f-a519-44bbe338e43d-istio-data\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7\" (UID: \"fd915c45-10cb-4d0f-a519-44bbe338e43d\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7"
Oct 31 19:40:32.942754 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:32.931003    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd915c45-10cb-4d0f-a519-44bbe338e43d-istio-podinfo\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7\" (UID: \"fd915c45-10cb-4d0f-a519-44bbe338e43d\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7"
Oct 31 19:40:32.942754 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:32.931892    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd915c45-10cb-4d0f-a519-44bbe338e43d-istio-envoy\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7\" (UID: \"fd915c45-10cb-4d0f-a519-44bbe338e43d\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7"
Oct 31 19:40:32.942754 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:32.934378    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd915c45-10cb-4d0f-a519-44bbe338e43d-istio-token\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7\" (UID: \"fd915c45-10cb-4d0f-a519-44bbe338e43d\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7"
Oct 31 19:40:32.942754 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:32.939303    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zctzl\" (UniqueName: \"kubernetes.io/projected/fd915c45-10cb-4d0f-a519-44bbe338e43d-kube-api-access-zctzl\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7\" (UID: \"fd915c45-10cb-4d0f-a519-44bbe338e43d\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7"
Oct 31 19:40:33.047014 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:33.046906    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7"
Oct 31 19:40:33.051830 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:33.051609    4645 patch_prober.go:29] interesting pod/e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": read tcp 10.133.0.2:39484->10.133.0.97:15021: read: connection reset by peer" start-of-body=
Oct 31 19:40:33.051830 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:33.051660    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw" podUID="585024d3-5768-463e-acdc-81151a7c0461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": read tcp 10.133.0.2:39484->10.133.0.97:15021: read: connection reset by peer"
Oct 31 19:40:33.386893 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:40:33.386864    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd915c45_10cb_4d0f_a519_44bbe338e43d.slice/crio-9785b05bcc6df7ce3915597ea14ce88f5d04b6bdeb3254207677c09f9a9176c3 WatchSource:0}: Error finding container 9785b05bcc6df7ce3915597ea14ce88f5d04b6bdeb3254207677c09f9a9176c3: Status 404 returned error can't find the container with id 9785b05bcc6df7ce3915597ea14ce88f5d04b6bdeb3254207677c09f9a9176c3
Oct 31 19:40:33.387968 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:33.387947    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7"]
Oct 31 19:40:33.390315 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:33.390282    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:40:33.390436 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:33.390372    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:40:33.390436 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:33.390418    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:40:33.427736 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:33.427711    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:33.436857 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:33.436836    4645 patch_prober.go:29] interesting pod/e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Oct 31 19:40:33.437135 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:33.436875    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc" podUID="8844ebc7-a83d-4b83-859e-2eb4d1e816c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Oct 31 19:40:33.541553 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:33.541453    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:33.722366 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:33.719628    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:33.814676 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:33.814589    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl"]
Oct 31 19:40:33.814878 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:33.814814    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl" podUID="d505eb36-3800-4e5b-878d-c7ab58d3e901" containerName="istio-proxy" containerID="cri-o://4093fde47b02bdbc38bddfd04586bf689533e63c3b9210b5b21434ddd1862921" gracePeriod=30
Oct 31 19:40:33.840470 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:33.840432    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7" event={"ID":"fd915c45-10cb-4d0f-a519-44bbe338e43d","Type":"ContainerStarted","Data":"738c6800661a462c22ac22e82d1ec79ffa27edcd8faf80becbf78ae8e49c0200"}
Oct 31 19:40:33.840726 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:33.840699    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7" event={"ID":"fd915c45-10cb-4d0f-a519-44bbe338e43d","Type":"ContainerStarted","Data":"9785b05bcc6df7ce3915597ea14ce88f5d04b6bdeb3254207677c09f9a9176c3"}
Oct 31 19:40:33.844007 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:33.843983    4645 generic.go:356] "Generic (PLEG): container finished" podID="585024d3-5768-463e-acdc-81151a7c0461" containerID="a6e1edb71ab92a73b91ed16fb9102cc4540988ed9ded70c1dff5c5400f189265" exitCode=0
Oct 31 19:40:33.845099 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:33.845060    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw" event={"ID":"585024d3-5768-463e-acdc-81151a7c0461","Type":"ContainerDied","Data":"a6e1edb71ab92a73b91ed16fb9102cc4540988ed9ded70c1dff5c5400f189265"}
Oct 31 19:40:33.848653 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:33.848624    4645 scope.go:122] "RemoveContainer" containerID="a6e1edb71ab92a73b91ed16fb9102cc4540988ed9ded70c1dff5c5400f189265"
Oct 31 19:40:33.849312 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:33.849294    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-03dfee-7c45f6ff5c-swqcn"
Oct 31 19:40:33.904887 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:33.887272    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:40:33.980126 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:33.978547    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:40:33.980126 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:33.978652    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:40:34.090354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:34.053917    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7"
Oct 31 19:40:34.099502 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:34.061742    4645 patch_prober.go:29] interesting pod/e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Oct 31 19:40:34.112348 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:34.106434    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7" podUID="fd915c45-10cb-4d0f-a519-44bbe338e43d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Oct 31 19:40:34.112348 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:34.101864    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:34.112348 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:34.101389    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7" podStartSLOduration=2.10136787 podStartE2EDuration="2.10136787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:40:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:40:34.08185943 +0000 UTC m=+1264.655093571" watchObservedRunningTime="2026-10-31 19:40:34.10136787 +0000 UTC m=+1264.674602000"
Oct 31 19:40:34.331014 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:34.323630    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-67adab47/e2e-lifecycle-model-67adab47-kserve-8f9445cd9-5rp7l"]
Oct 31 19:40:34.448660 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:34.447608    4645 patch_prober.go:29] interesting pod/e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Oct 31 19:40:34.448660 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:34.447681    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc" podUID="8844ebc7-a83d-4b83-859e-2eb4d1e816c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Oct 31 19:40:34.475434 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:34.466708    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc"]
Oct 31 19:40:34.475434 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:34.466941    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc" podUID="a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7" containerName="istio-proxy" containerID="cri-o://fa9954d53d791144c7a3b2dfdfe4bfa44b08b62e95fb0fcb5d1835f950c47e74" gracePeriod=30
Oct 31 19:40:34.475434 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:34.472374    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2"]
Oct 31 19:40:34.475434 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:34.472585    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2" podUID="8332946b-f51b-45e0-befe-8d8ce8a46c7d" containerName="istio-proxy" containerID="cri-o://ab576a191d5cace8de5e3988efae6e1f50c60884e16cab8ffa6e1a2c5a9591aa" gracePeriod=30
Oct 31 19:40:34.495407 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:34.487746    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4"]
Oct 31 19:40:34.495407 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:34.487972    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4" podUID="adb9b9a5-acbc-4967-b115-ed4e68a84b7e" containerName="istio-proxy" containerID="cri-o://8687de1652c92cd6304e37f64d2cf017bdec67320d27f72c072cf9a6162a3879" gracePeriod=30
Oct 31 19:40:34.542169 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:34.542128    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:34.702105 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:34.700776    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:34.922459 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:34.918832    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw" event={"ID":"585024d3-5768-463e-acdc-81151a7c0461","Type":"ContainerStarted","Data":"69ec18e93e369eb6f8cac06277a33448631ff62e4d2f446fecdc5e4899fdd649"}
Oct 31 19:40:35.052351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:35.052255    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw"
Oct 31 19:40:35.066350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:35.052208    4645 patch_prober.go:29] interesting pod/e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Oct 31 19:40:35.066350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:35.052918    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7" podUID="fd915c45-10cb-4d0f-a519-44bbe338e43d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Oct 31 19:40:35.066350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:35.053424    4645 patch_prober.go:29] interesting pod/e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 31 19:40:35.066350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:35.053468    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw" podUID="585024d3-5768-463e-acdc-81151a7c0461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 31 19:40:35.353812 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:35.353733    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96"]
Oct 31 19:40:35.424591 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:35.420893    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:35.447742 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:35.447706    4645 patch_prober.go:29] interesting pod/e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Oct 31 19:40:35.448000 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:35.447969    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc" podUID="8844ebc7-a83d-4b83-859e-2eb4d1e816c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Oct 31 19:40:35.467380 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:35.462551    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96"]
Oct 31 19:40:35.467380 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:35.462685    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96"
Oct 31 19:40:35.492750 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:35.492721    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-1c0c1d-openshift-default-dockercfg-nxkzt\""
Oct 31 19:40:35.752378 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:35.692859    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e00a761e-b818-439d-b61a-a3ea1c580278-istio-podinfo\") pod \"e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96\" (UID: \"e00a761e-b818-439d-b61a-a3ea1c580278\") " pod="openshift-ingress/e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96"
Oct 31 19:40:35.752378 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:35.714534    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e00a761e-b818-439d-b61a-a3ea1c580278-istio-data\") pod \"e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96\" (UID: \"e00a761e-b818-439d-b61a-a3ea1c580278\") " pod="openshift-ingress/e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96"
Oct 31 19:40:35.752378 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:35.714699    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e00a761e-b818-439d-b61a-a3ea1c580278-istio-envoy\") pod \"e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96\" (UID: \"e00a761e-b818-439d-b61a-a3ea1c580278\") " pod="openshift-ingress/e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96"
Oct 31 19:40:35.752378 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:35.714836    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e00a761e-b818-439d-b61a-a3ea1c580278-workload-socket\") pod \"e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96\" (UID: \"e00a761e-b818-439d-b61a-a3ea1c580278\") " pod="openshift-ingress/e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96"
Oct 31 19:40:35.752378 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:35.714870    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsbrq\" (UniqueName: \"kubernetes.io/projected/e00a761e-b818-439d-b61a-a3ea1c580278-kube-api-access-zsbrq\") pod \"e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96\" (UID: \"e00a761e-b818-439d-b61a-a3ea1c580278\") " pod="openshift-ingress/e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96"
Oct 31 19:40:35.752378 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:35.714949    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e00a761e-b818-439d-b61a-a3ea1c580278-istiod-ca-cert\") pod \"e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96\" (UID: \"e00a761e-b818-439d-b61a-a3ea1c580278\") " pod="openshift-ingress/e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96"
Oct 31 19:40:35.752378 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:35.715041    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e00a761e-b818-439d-b61a-a3ea1c580278-istio-token\") pod \"e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96\" (UID: \"e00a761e-b818-439d-b61a-a3ea1c580278\") " pod="openshift-ingress/e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96"
Oct 31 19:40:35.752378 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:35.715145    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e00a761e-b818-439d-b61a-a3ea1c580278-credential-socket\") pod \"e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96\" (UID: \"e00a761e-b818-439d-b61a-a3ea1c580278\") " pod="openshift-ingress/e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96"
Oct 31 19:40:35.752378 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:35.715178    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e00a761e-b818-439d-b61a-a3ea1c580278-workload-certs\") pod \"e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96\" (UID: \"e00a761e-b818-439d-b61a-a3ea1c580278\") " pod="openshift-ingress/e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96"
Oct 31 19:40:35.846784 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:35.844277    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:35.885352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:35.870557    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e00a761e-b818-439d-b61a-a3ea1c580278-workload-socket\") pod \"e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96\" (UID: \"e00a761e-b818-439d-b61a-a3ea1c580278\") " pod="openshift-ingress/e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96"
Oct 31 19:40:35.885352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:35.870610    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsbrq\" (UniqueName: \"kubernetes.io/projected/e00a761e-b818-439d-b61a-a3ea1c580278-kube-api-access-zsbrq\") pod \"e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96\" (UID: \"e00a761e-b818-439d-b61a-a3ea1c580278\") " pod="openshift-ingress/e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96"
Oct 31 19:40:35.885352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:35.870701    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e00a761e-b818-439d-b61a-a3ea1c580278-istiod-ca-cert\") pod \"e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96\" (UID: \"e00a761e-b818-439d-b61a-a3ea1c580278\") " pod="openshift-ingress/e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96"
Oct 31 19:40:35.885352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:35.870790    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e00a761e-b818-439d-b61a-a3ea1c580278-istio-token\") pod \"e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96\" (UID: \"e00a761e-b818-439d-b61a-a3ea1c580278\") " pod="openshift-ingress/e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96"
Oct 31 19:40:35.885352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:35.870873    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e00a761e-b818-439d-b61a-a3ea1c580278-credential-socket\") pod \"e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96\" (UID: \"e00a761e-b818-439d-b61a-a3ea1c580278\") " pod="openshift-ingress/e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96"
Oct 31 19:40:35.885352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:35.870904    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e00a761e-b818-439d-b61a-a3ea1c580278-workload-certs\") pod \"e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96\" (UID: \"e00a761e-b818-439d-b61a-a3ea1c580278\") " pod="openshift-ingress/e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96"
Oct 31 19:40:35.885352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:35.878532    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e00a761e-b818-439d-b61a-a3ea1c580278-workload-socket\") pod \"e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96\" (UID: \"e00a761e-b818-439d-b61a-a3ea1c580278\") " pod="openshift-ingress/e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96"
Oct 31 19:40:35.885352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:35.879716    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e00a761e-b818-439d-b61a-a3ea1c580278-istiod-ca-cert\") pod \"e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96\" (UID: \"e00a761e-b818-439d-b61a-a3ea1c580278\") " pod="openshift-ingress/e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96"
Oct 31 19:40:35.885352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:35.880025    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e00a761e-b818-439d-b61a-a3ea1c580278-credential-socket\") pod \"e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96\" (UID: \"e00a761e-b818-439d-b61a-a3ea1c580278\") " pod="openshift-ingress/e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96"
Oct 31 19:40:35.885352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:35.880417    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e00a761e-b818-439d-b61a-a3ea1c580278-workload-certs\") pod \"e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96\" (UID: \"e00a761e-b818-439d-b61a-a3ea1c580278\") " pod="openshift-ingress/e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96"
Oct 31 19:40:35.921350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:35.894479    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e00a761e-b818-439d-b61a-a3ea1c580278-istio-podinfo\") pod \"e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96\" (UID: \"e00a761e-b818-439d-b61a-a3ea1c580278\") " pod="openshift-ingress/e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96"
Oct 31 19:40:35.921350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:35.894595    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e00a761e-b818-439d-b61a-a3ea1c580278-istio-data\") pod \"e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96\" (UID: \"e00a761e-b818-439d-b61a-a3ea1c580278\") " pod="openshift-ingress/e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96"
Oct 31 19:40:35.921350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:35.894731    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e00a761e-b818-439d-b61a-a3ea1c580278-istio-envoy\") pod \"e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96\" (UID: \"e00a761e-b818-439d-b61a-a3ea1c580278\") " pod="openshift-ingress/e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96"
Oct 31 19:40:35.921350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:35.904214    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e00a761e-b818-439d-b61a-a3ea1c580278-istio-data\") pod \"e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96\" (UID: \"e00a761e-b818-439d-b61a-a3ea1c580278\") " pod="openshift-ingress/e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96"
Oct 31 19:40:36.050441 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:36.008762    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e00a761e-b818-439d-b61a-a3ea1c580278-istio-envoy\") pod \"e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96\" (UID: \"e00a761e-b818-439d-b61a-a3ea1c580278\") " pod="openshift-ingress/e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96"
Oct 31 19:40:36.083365 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:36.064756    4645 patch_prober.go:29] interesting pod/e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Oct 31 19:40:36.083365 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:36.064819    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7" podUID="fd915c45-10cb-4d0f-a519-44bbe338e43d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Oct 31 19:40:36.083365 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:36.078863    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-67adab47/e2e-lifecycle-model-67adab47-kserve-8f9445cd9-5rp7l" podUID="b9153d96-5237-47a0-8bf4-1e77f69b071a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:40:36.100361 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:36.095430    4645 patch_prober.go:29] interesting pod/e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 31 19:40:36.100361 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:36.095517    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw" podUID="585024d3-5768-463e-acdc-81151a7c0461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 31 19:40:36.100607 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:36.100440    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e00a761e-b818-439d-b61a-a3ea1c580278-istio-podinfo\") pod \"e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96\" (UID: \"e00a761e-b818-439d-b61a-a3ea1c580278\") " pod="openshift-ingress/e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96"
Oct 31 19:40:36.122378 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:36.115234    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsbrq\" (UniqueName: \"kubernetes.io/projected/e00a761e-b818-439d-b61a-a3ea1c580278-kube-api-access-zsbrq\") pod \"e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96\" (UID: \"e00a761e-b818-439d-b61a-a3ea1c580278\") " pod="openshift-ingress/e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96"
Oct 31 19:40:36.126349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:36.126215    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e00a761e-b818-439d-b61a-a3ea1c580278-istio-token\") pod \"e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96\" (UID: \"e00a761e-b818-439d-b61a-a3ea1c580278\") " pod="openshift-ingress/e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96"
Oct 31 19:40:36.127040 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:36.127010    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:36.163566 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:36.163524    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw" podUID="1287db54-a587-4c11-9eb6-1606de555894" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:40:36.235891 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:36.235850    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4" podUID="6032a997-b795-4561-805c-110cf67a9d5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:40:36.396836 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:36.393635    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96"
Oct 31 19:40:36.443788 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:36.443749    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:36.634982 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:36.634930    4645 patch_prober.go:29] interesting pod/e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Oct 31 19:40:36.635692 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:36.635592    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc" podUID="8844ebc7-a83d-4b83-859e-2eb4d1e816c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Oct 31 19:40:36.636580 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:40:36.636548    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 19:40:36.637031 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:36.637005    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:36.749651 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:36.749413    4645 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n" podUID="dbb97168-db93-4757-9a1b-f9c5eb704fc5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:40:37.061413 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:37.060582    4645 patch_prober.go:29] interesting pod/e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Oct 31 19:40:37.061413 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:37.060642    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7" podUID="fd915c45-10cb-4d0f-a519-44bbe338e43d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Oct 31 19:40:37.061413 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:37.061230    4645 patch_prober.go:29] interesting pod/e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 31 19:40:37.061413 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:37.061272    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw" podUID="585024d3-5768-463e-acdc-81151a7c0461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 31 19:40:37.088532 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:37.088489    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-67adab47/e2e-lifecycle-model-67adab47-kserve-8f9445cd9-5rp7l" podUID="b9153d96-5237-47a0-8bf4-1e77f69b071a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:40:37.114383 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:37.089108    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-67adab47/e2e-lifecycle-model-67adab47-kserve-8f9445cd9-5rp7l"
Oct 31 19:40:37.278749 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:37.185972    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x" podUID="a774e286-dcb8-4a2f-8541-26bf231eb08d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 19:40:37.278749 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:37.199000    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw" podUID="1287db54-a587-4c11-9eb6-1606de555894" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:40:37.278749 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:37.202448    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw"
Oct 31 19:40:37.278749 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:37.229407    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw" podUID="d504b2d3-c26d-4dcf-a7eb-efe20732ae4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:40:37.278749 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:37.242206    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669" podUID="9285bee0-bf7c-4fd4-9da7-a1db021cae05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 31 19:40:37.278749 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:37.244739    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4" podUID="6032a997-b795-4561-805c-110cf67a9d5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 31 19:40:37.278749 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:37.245089    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4"
Oct 31 19:40:37.315604 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:37.312380    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-67adab47/e2e-lifecycle-model-67adab47-kserve-8f9445cd9-5rp7l"
Oct 31 19:40:37.415461 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:37.415412    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l" podUID="12bea86c-61e5-465e-82fb-b4673a4d1be9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 31 19:40:37.509388 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:37.480234    4645 patch_prober.go:29] interesting pod/e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Oct 31 19:40:37.509388 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:37.480292    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc" podUID="8844ebc7-a83d-4b83-859e-2eb4d1e816c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Oct 31 19:40:37.509388 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:37.484721    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:37.821597 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:37.716844    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n" podUID="dbb97168-db93-4757-9a1b-f9c5eb704fc5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 19:40:38.148826 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:38.148773    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:40:38.149824 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:38.149789    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:40:38.150077 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:38.150049    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:40:38.184469 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:38.157027    4645 patch_prober.go:29] interesting pod/e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 31 19:40:38.184469 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:38.157087    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw" podUID="585024d3-5768-463e-acdc-81151a7c0461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 31 19:40:38.300352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:38.259718    4645 patch_prober.go:29] interesting pod/e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Oct 31 19:40:38.300352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:38.288495    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7" podUID="fd915c45-10cb-4d0f-a519-44bbe338e43d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Oct 31 19:40:38.347963 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:38.347881    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw"
Oct 31 19:40:38.348429 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:38.348411    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4"
Oct 31 19:40:38.349017 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:38.349000    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96"]
Oct 31 19:40:38.448399 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:38.435373    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:38.455015 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:38.454980    4645 patch_prober.go:29] interesting pod/e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Oct 31 19:40:38.455288 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:38.455256    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc" podUID="8844ebc7-a83d-4b83-859e-2eb4d1e816c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Oct 31 19:40:38.829369 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:38.811451    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96" event={"ID":"e00a761e-b818-439d-b61a-a3ea1c580278","Type":"ContainerStarted","Data":"50800259c1dffc6adfe5f7f539e4b48f1189a5e8b7b07457addb071afaf8d90f"}
Oct 31 19:40:39.047760 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:39.047338    4645 patch_prober.go:29] interesting pod/e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Oct 31 19:40:39.047760 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:39.047387    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7" podUID="fd915c45-10cb-4d0f-a519-44bbe338e43d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Oct 31 19:40:39.051634 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:39.051602    4645 patch_prober.go:29] interesting pod/e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 31 19:40:39.051961 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:39.051934    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw" podUID="585024d3-5768-463e-acdc-81151a7c0461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 31 19:40:39.438380 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:39.437613    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:39.456579 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:39.454850    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc"
Oct 31 19:40:39.478355 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:39.469732    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc"
Oct 31 19:40:39.495359 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:39.483656    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc"
Oct 31 19:40:39.833883 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:39.772917    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:40.141383 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:40.094089    4645 patch_prober.go:29] interesting pod/e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Oct 31 19:40:40.141383 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:40.094161    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7" podUID="fd915c45-10cb-4d0f-a519-44bbe338e43d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Oct 31 19:40:40.141383 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:40.095530    4645 patch_prober.go:29] interesting pod/e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 31 19:40:40.141383 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:40.095572    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw" podUID="585024d3-5768-463e-acdc-81151a7c0461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 31 19:40:40.141383 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:40.096019    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:40.197830 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:40.196526    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-03dfee-7c45f6ff5c-swqcn"
Oct 31 19:40:40.254599 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:40.253803    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96" event={"ID":"e00a761e-b818-439d-b61a-a3ea1c580278","Type":"ContainerStarted","Data":"1d529541c816d13d49dae9bc8cddbdecd570b75a58bcb8147fb30e2c7899cd92"}
Oct 31 19:40:40.318624 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:40.318223    4645 generic.go:356] "Generic (PLEG): container finished" podID="d505eb36-3800-4e5b-878d-c7ab58d3e901" containerID="4093fde47b02bdbc38bddfd04586bf689533e63c3b9210b5b21434ddd1862921" exitCode=0
Oct 31 19:40:40.320061 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:40.320038    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl" event={"ID":"d505eb36-3800-4e5b-878d-c7ab58d3e901","Type":"ContainerDied","Data":"4093fde47b02bdbc38bddfd04586bf689533e63c3b9210b5b21434ddd1862921"}
Oct 31 19:40:40.320283 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:40.320242    4645 scope.go:122] "RemoveContainer" containerID="23ffc973c645e851acd83dcea2e1d7f0ebd7e78d52fece4dc3fe4714de37d73e"
Oct 31 19:40:40.345921 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:40.345552    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl"
Oct 31 19:40:40.378354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:40.374060    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96" podStartSLOduration=5.374039053 podStartE2EDuration="5.374039053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:40:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:40:40.340199546 +0000 UTC m=+1270.913433676" watchObservedRunningTime="2026-10-31 19:40:40.374039053 +0000 UTC m=+1270.947273184"
Oct 31 19:40:40.432057 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:40.431983    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:40.432283 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:40.432246    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96"
Oct 31 19:40:40.443255 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:40.438516    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d505eb36-3800-4e5b-878d-c7ab58d3e901-istio-token\" (UniqueName: \"kubernetes.io/projected/d505eb36-3800-4e5b-878d-c7ab58d3e901-istio-token\") pod \"d505eb36-3800-4e5b-878d-c7ab58d3e901\" (UID: \"d505eb36-3800-4e5b-878d-c7ab58d3e901\") "
Oct 31 19:40:40.443255 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:40.438564    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d505eb36-3800-4e5b-878d-c7ab58d3e901-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d505eb36-3800-4e5b-878d-c7ab58d3e901-workload-socket\") pod \"d505eb36-3800-4e5b-878d-c7ab58d3e901\" (UID: \"d505eb36-3800-4e5b-878d-c7ab58d3e901\") "
Oct 31 19:40:40.443255 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:40.438658    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d505eb36-3800-4e5b-878d-c7ab58d3e901-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d505eb36-3800-4e5b-878d-c7ab58d3e901-workload-certs\") pod \"d505eb36-3800-4e5b-878d-c7ab58d3e901\" (UID: \"d505eb36-3800-4e5b-878d-c7ab58d3e901\") "
Oct 31 19:40:40.443255 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:40.438773    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d505eb36-3800-4e5b-878d-c7ab58d3e901-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d505eb36-3800-4e5b-878d-c7ab58d3e901-istiod-ca-cert\") pod \"d505eb36-3800-4e5b-878d-c7ab58d3e901\" (UID: \"d505eb36-3800-4e5b-878d-c7ab58d3e901\") "
Oct 31 19:40:40.443255 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:40.438964    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d505eb36-3800-4e5b-878d-c7ab58d3e901-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d505eb36-3800-4e5b-878d-c7ab58d3e901-istio-podinfo\") pod \"d505eb36-3800-4e5b-878d-c7ab58d3e901\" (UID: \"d505eb36-3800-4e5b-878d-c7ab58d3e901\") "
Oct 31 19:40:40.443255 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:40.439016    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d505eb36-3800-4e5b-878d-c7ab58d3e901-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d505eb36-3800-4e5b-878d-c7ab58d3e901-istio-envoy\") pod \"d505eb36-3800-4e5b-878d-c7ab58d3e901\" (UID: \"d505eb36-3800-4e5b-878d-c7ab58d3e901\") "
Oct 31 19:40:40.443255 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:40.439058    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d505eb36-3800-4e5b-878d-c7ab58d3e901-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d505eb36-3800-4e5b-878d-c7ab58d3e901-istio-data\") pod \"d505eb36-3800-4e5b-878d-c7ab58d3e901\" (UID: \"d505eb36-3800-4e5b-878d-c7ab58d3e901\") "
Oct 31 19:40:40.443255 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:40.440208    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d505eb36-3800-4e5b-878d-c7ab58d3e901-workload-certs" pod "d505eb36-3800-4e5b-878d-c7ab58d3e901" (UID: "d505eb36-3800-4e5b-878d-c7ab58d3e901"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:40.443778 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:40.443708    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d505eb36-3800-4e5b-878d-c7ab58d3e901-kube-api-access-2hlz5\" (UniqueName: \"kubernetes.io/projected/d505eb36-3800-4e5b-878d-c7ab58d3e901-kube-api-access-2hlz5\") pod \"d505eb36-3800-4e5b-878d-c7ab58d3e901\" (UID: \"d505eb36-3800-4e5b-878d-c7ab58d3e901\") "
Oct 31 19:40:40.443838 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:40.443777    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d505eb36-3800-4e5b-878d-c7ab58d3e901-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d505eb36-3800-4e5b-878d-c7ab58d3e901-credential-socket\") pod \"d505eb36-3800-4e5b-878d-c7ab58d3e901\" (UID: \"d505eb36-3800-4e5b-878d-c7ab58d3e901\") "
Oct 31 19:40:40.522069 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:40.445130    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d505eb36-3800-4e5b-878d-c7ab58d3e901-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:40.522069 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:40.445719    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d505eb36-3800-4e5b-878d-c7ab58d3e901-istiod-ca-cert" pod "d505eb36-3800-4e5b-878d-c7ab58d3e901" (UID: "d505eb36-3800-4e5b-878d-c7ab58d3e901"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:40:40.522069 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:40.446222    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96"
Oct 31 19:40:40.522069 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:40.446823    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d505eb36-3800-4e5b-878d-c7ab58d3e901-workload-socket" pod "d505eb36-3800-4e5b-878d-c7ab58d3e901" (UID: "d505eb36-3800-4e5b-878d-c7ab58d3e901"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:40.522069 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:40.447776    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d505eb36-3800-4e5b-878d-c7ab58d3e901-istio-token" pod "d505eb36-3800-4e5b-878d-c7ab58d3e901" (UID: "d505eb36-3800-4e5b-878d-c7ab58d3e901"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:40:40.522069 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:40.448081    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d505eb36-3800-4e5b-878d-c7ab58d3e901-credential-socket" pod "d505eb36-3800-4e5b-878d-c7ab58d3e901" (UID: "d505eb36-3800-4e5b-878d-c7ab58d3e901"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:40.522069 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:40.454268    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d505eb36-3800-4e5b-878d-c7ab58d3e901-istio-data" pod "d505eb36-3800-4e5b-878d-c7ab58d3e901" (UID: "d505eb36-3800-4e5b-878d-c7ab58d3e901"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:40.522069 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:40.521950    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d505eb36-3800-4e5b-878d-c7ab58d3e901-kube-api-access-2hlz5" pod "d505eb36-3800-4e5b-878d-c7ab58d3e901" (UID: "d505eb36-3800-4e5b-878d-c7ab58d3e901"). InnerVolumeSpecName "kube-api-access-2hlz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:40:40.526371 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:40.524837    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d505eb36-3800-4e5b-878d-c7ab58d3e901-istio-podinfo" pod "d505eb36-3800-4e5b-878d-c7ab58d3e901" (UID: "d505eb36-3800-4e5b-878d-c7ab58d3e901"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:40:40.527712 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:40.527685    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d505eb36-3800-4e5b-878d-c7ab58d3e901-istio-envoy" pod "d505eb36-3800-4e5b-878d-c7ab58d3e901" (UID: "d505eb36-3800-4e5b-878d-c7ab58d3e901"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:40.564357 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:40.549205    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hlz5\" (UniqueName: \"kubernetes.io/projected/d505eb36-3800-4e5b-878d-c7ab58d3e901-kube-api-access-2hlz5\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:40.564623 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:40.564596    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d505eb36-3800-4e5b-878d-c7ab58d3e901-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:40.572418 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:40.572389    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d505eb36-3800-4e5b-878d-c7ab58d3e901-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:40.572626 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:40.572609    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d505eb36-3800-4e5b-878d-c7ab58d3e901-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:40.572742 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:40.572729    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d505eb36-3800-4e5b-878d-c7ab58d3e901-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:40.597374 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:40.591849    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d505eb36-3800-4e5b-878d-c7ab58d3e901-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:40.597374 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:40.591885    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d505eb36-3800-4e5b-878d-c7ab58d3e901-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:40.597374 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:40.591901    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d505eb36-3800-4e5b-878d-c7ab58d3e901-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:40.597374 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:40.570525    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:40.947088 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:40.947059    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4"
Oct 31 19:40:41.030380 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.028636    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adb9b9a5-acbc-4967-b115-ed4e68a84b7e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb9b9a5-acbc-4967-b115-ed4e68a84b7e-credential-socket\") pod \"adb9b9a5-acbc-4967-b115-ed4e68a84b7e\" (UID: \"adb9b9a5-acbc-4967-b115-ed4e68a84b7e\") "
Oct 31 19:40:41.030380 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.028723    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adb9b9a5-acbc-4967-b115-ed4e68a84b7e-kube-api-access-rb9wg\" (UniqueName: \"kubernetes.io/projected/adb9b9a5-acbc-4967-b115-ed4e68a84b7e-kube-api-access-rb9wg\") pod \"adb9b9a5-acbc-4967-b115-ed4e68a84b7e\" (UID: \"adb9b9a5-acbc-4967-b115-ed4e68a84b7e\") "
Oct 31 19:40:41.030380 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.028781    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adb9b9a5-acbc-4967-b115-ed4e68a84b7e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb9b9a5-acbc-4967-b115-ed4e68a84b7e-workload-socket\") pod \"adb9b9a5-acbc-4967-b115-ed4e68a84b7e\" (UID: \"adb9b9a5-acbc-4967-b115-ed4e68a84b7e\") "
Oct 31 19:40:41.030380 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.028808    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adb9b9a5-acbc-4967-b115-ed4e68a84b7e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adb9b9a5-acbc-4967-b115-ed4e68a84b7e-istio-data\") pod \"adb9b9a5-acbc-4967-b115-ed4e68a84b7e\" (UID: \"adb9b9a5-acbc-4967-b115-ed4e68a84b7e\") "
Oct 31 19:40:41.030380 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.028869    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adb9b9a5-acbc-4967-b115-ed4e68a84b7e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adb9b9a5-acbc-4967-b115-ed4e68a84b7e-istiod-ca-cert\") pod \"adb9b9a5-acbc-4967-b115-ed4e68a84b7e\" (UID: \"adb9b9a5-acbc-4967-b115-ed4e68a84b7e\") "
Oct 31 19:40:41.030380 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.029005    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/adb9b9a5-acbc-4967-b115-ed4e68a84b7e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adb9b9a5-acbc-4967-b115-ed4e68a84b7e-istio-podinfo\") pod \"adb9b9a5-acbc-4967-b115-ed4e68a84b7e\" (UID: \"adb9b9a5-acbc-4967-b115-ed4e68a84b7e\") "
Oct 31 19:40:41.030380 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.029055    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adb9b9a5-acbc-4967-b115-ed4e68a84b7e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adb9b9a5-acbc-4967-b115-ed4e68a84b7e-workload-certs\") pod \"adb9b9a5-acbc-4967-b115-ed4e68a84b7e\" (UID: \"adb9b9a5-acbc-4967-b115-ed4e68a84b7e\") "
Oct 31 19:40:41.030380 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.029082    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adb9b9a5-acbc-4967-b115-ed4e68a84b7e-istio-token\" (UniqueName: \"kubernetes.io/projected/adb9b9a5-acbc-4967-b115-ed4e68a84b7e-istio-token\") pod \"adb9b9a5-acbc-4967-b115-ed4e68a84b7e\" (UID: \"adb9b9a5-acbc-4967-b115-ed4e68a84b7e\") "
Oct 31 19:40:41.030380 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.029137    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adb9b9a5-acbc-4967-b115-ed4e68a84b7e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adb9b9a5-acbc-4967-b115-ed4e68a84b7e-istio-envoy\") pod \"adb9b9a5-acbc-4967-b115-ed4e68a84b7e\" (UID: \"adb9b9a5-acbc-4967-b115-ed4e68a84b7e\") "
Oct 31 19:40:41.030380 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.030370    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adb9b9a5-acbc-4967-b115-ed4e68a84b7e-istio-data" pod "adb9b9a5-acbc-4967-b115-ed4e68a84b7e" (UID: "adb9b9a5-acbc-4967-b115-ed4e68a84b7e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:41.031219 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.030620    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adb9b9a5-acbc-4967-b115-ed4e68a84b7e-credential-socket" pod "adb9b9a5-acbc-4967-b115-ed4e68a84b7e" (UID: "adb9b9a5-acbc-4967-b115-ed4e68a84b7e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:41.031559 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.031536    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adb9b9a5-acbc-4967-b115-ed4e68a84b7e-workload-socket" pod "adb9b9a5-acbc-4967-b115-ed4e68a84b7e" (UID: "adb9b9a5-acbc-4967-b115-ed4e68a84b7e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:41.031910 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.031852    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adb9b9a5-acbc-4967-b115-ed4e68a84b7e-workload-certs" pod "adb9b9a5-acbc-4967-b115-ed4e68a84b7e" (UID: "adb9b9a5-acbc-4967-b115-ed4e68a84b7e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:41.032396 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.032364    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adb9b9a5-acbc-4967-b115-ed4e68a84b7e-istiod-ca-cert" pod "adb9b9a5-acbc-4967-b115-ed4e68a84b7e" (UID: "adb9b9a5-acbc-4967-b115-ed4e68a84b7e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:40:41.034472 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.034449    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/adb9b9a5-acbc-4967-b115-ed4e68a84b7e-istio-podinfo" pod "adb9b9a5-acbc-4967-b115-ed4e68a84b7e" (UID: "adb9b9a5-acbc-4967-b115-ed4e68a84b7e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:40:41.035839 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.035817    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adb9b9a5-acbc-4967-b115-ed4e68a84b7e-kube-api-access-rb9wg" pod "adb9b9a5-acbc-4967-b115-ed4e68a84b7e" (UID: "adb9b9a5-acbc-4967-b115-ed4e68a84b7e"). InnerVolumeSpecName "kube-api-access-rb9wg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:40:41.036757 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.036736    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adb9b9a5-acbc-4967-b115-ed4e68a84b7e-istio-token" pod "adb9b9a5-acbc-4967-b115-ed4e68a84b7e" (UID: "adb9b9a5-acbc-4967-b115-ed4e68a84b7e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:40:41.039348 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.037616    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adb9b9a5-acbc-4967-b115-ed4e68a84b7e-istio-envoy" pod "adb9b9a5-acbc-4967-b115-ed4e68a84b7e" (UID: "adb9b9a5-acbc-4967-b115-ed4e68a84b7e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:41.049239 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.049213    4645 patch_prober.go:29] interesting pod/e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Oct 31 19:40:41.049365 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.049264    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7" podUID="fd915c45-10cb-4d0f-a519-44bbe338e43d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Oct 31 19:40:41.053100 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.052619    4645 patch_prober.go:29] interesting pod/e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 31 19:40:41.053100 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.052664    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw" podUID="585024d3-5768-463e-acdc-81151a7c0461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 31 19:40:41.136836 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.136161    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/adb9b9a5-acbc-4967-b115-ed4e68a84b7e-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:41.136836 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.136206    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/adb9b9a5-acbc-4967-b115-ed4e68a84b7e-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:41.136836 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.136221    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/adb9b9a5-acbc-4967-b115-ed4e68a84b7e-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:41.136836 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.136236    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/adb9b9a5-acbc-4967-b115-ed4e68a84b7e-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:41.136836 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.136252    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb9b9a5-acbc-4967-b115-ed4e68a84b7e-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:41.136836 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.136266    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb9wg\" (UniqueName: \"kubernetes.io/projected/adb9b9a5-acbc-4967-b115-ed4e68a84b7e-kube-api-access-rb9wg\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:41.136836 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.136281    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/adb9b9a5-acbc-4967-b115-ed4e68a84b7e-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:41.136836 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.136295    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/adb9b9a5-acbc-4967-b115-ed4e68a84b7e-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:41.136836 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.136310    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/adb9b9a5-acbc-4967-b115-ed4e68a84b7e-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:41.151314 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.151288    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2"
Oct 31 19:40:41.153478 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.153458    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc"
Oct 31 19:40:41.244364 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.238028    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8332946b-f51b-45e0-befe-8d8ce8a46c7d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8332946b-f51b-45e0-befe-8d8ce8a46c7d-istio-envoy\") pod \"8332946b-f51b-45e0-befe-8d8ce8a46c7d\" (UID: \"8332946b-f51b-45e0-befe-8d8ce8a46c7d\") "
Oct 31 19:40:41.244364 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.238075    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7-istio-envoy\") pod \"a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7\" (UID: \"a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7\") "
Oct 31 19:40:41.244364 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.238125    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7-istio-token\" (UniqueName: \"kubernetes.io/projected/a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7-istio-token\") pod \"a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7\" (UID: \"a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7\") "
Oct 31 19:40:41.244364 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.238267    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8332946b-f51b-45e0-befe-8d8ce8a46c7d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8332946b-f51b-45e0-befe-8d8ce8a46c7d-istiod-ca-cert\") pod \"8332946b-f51b-45e0-befe-8d8ce8a46c7d\" (UID: \"8332946b-f51b-45e0-befe-8d8ce8a46c7d\") "
Oct 31 19:40:41.244364 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.238378    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8332946b-f51b-45e0-befe-8d8ce8a46c7d-kube-api-access-9xlt4\" (UniqueName: \"kubernetes.io/projected/8332946b-f51b-45e0-befe-8d8ce8a46c7d-kube-api-access-9xlt4\") pod \"8332946b-f51b-45e0-befe-8d8ce8a46c7d\" (UID: \"8332946b-f51b-45e0-befe-8d8ce8a46c7d\") "
Oct 31 19:40:41.244364 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.238438    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8332946b-f51b-45e0-befe-8d8ce8a46c7d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8332946b-f51b-45e0-befe-8d8ce8a46c7d-workload-socket\") pod \"8332946b-f51b-45e0-befe-8d8ce8a46c7d\" (UID: \"8332946b-f51b-45e0-befe-8d8ce8a46c7d\") "
Oct 31 19:40:41.244364 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.238527    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8332946b-f51b-45e0-befe-8d8ce8a46c7d-istio-token\" (UniqueName: \"kubernetes.io/projected/8332946b-f51b-45e0-befe-8d8ce8a46c7d-istio-token\") pod \"8332946b-f51b-45e0-befe-8d8ce8a46c7d\" (UID: \"8332946b-f51b-45e0-befe-8d8ce8a46c7d\") "
Oct 31 19:40:41.244364 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.238624    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7-istiod-ca-cert\") pod \"a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7\" (UID: \"a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7\") "
Oct 31 19:40:41.244364 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.238694    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7-workload-socket\") pod \"a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7\" (UID: \"a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7\") "
Oct 31 19:40:41.244364 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.238752    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8332946b-f51b-45e0-befe-8d8ce8a46c7d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8332946b-f51b-45e0-befe-8d8ce8a46c7d-istio-podinfo\") pod \"8332946b-f51b-45e0-befe-8d8ce8a46c7d\" (UID: \"8332946b-f51b-45e0-befe-8d8ce8a46c7d\") "
Oct 31 19:40:41.244364 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.238792    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7-istio-podinfo\") pod \"a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7\" (UID: \"a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7\") "
Oct 31 19:40:41.244364 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.238863    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7-kube-api-access-jzpd6\" (UniqueName: \"kubernetes.io/projected/a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7-kube-api-access-jzpd6\") pod \"a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7\" (UID: \"a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7\") "
Oct 31 19:40:41.244364 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.238916    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7-workload-certs\") pod \"a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7\" (UID: \"a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7\") "
Oct 31 19:40:41.244364 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.238946    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7-credential-socket\") pod \"a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7\" (UID: \"a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7\") "
Oct 31 19:40:41.244364 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.238990    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8332946b-f51b-45e0-befe-8d8ce8a46c7d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8332946b-f51b-45e0-befe-8d8ce8a46c7d-istio-data\") pod \"8332946b-f51b-45e0-befe-8d8ce8a46c7d\" (UID: \"8332946b-f51b-45e0-befe-8d8ce8a46c7d\") "
Oct 31 19:40:41.244364 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.239022    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8332946b-f51b-45e0-befe-8d8ce8a46c7d-workload-socket" pod "8332946b-f51b-45e0-befe-8d8ce8a46c7d" (UID: "8332946b-f51b-45e0-befe-8d8ce8a46c7d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:41.244364 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.239035    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8332946b-f51b-45e0-befe-8d8ce8a46c7d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8332946b-f51b-45e0-befe-8d8ce8a46c7d-credential-socket\") pod \"8332946b-f51b-45e0-befe-8d8ce8a46c7d\" (UID: \"8332946b-f51b-45e0-befe-8d8ce8a46c7d\") "
Oct 31 19:40:41.244364 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.239113    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7-istio-data\") pod \"a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7\" (UID: \"a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7\") "
Oct 31 19:40:41.244364 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.239149    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7-istiod-ca-cert" pod "a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7" (UID: "a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:40:41.244364 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.239237    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8332946b-f51b-45e0-befe-8d8ce8a46c7d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8332946b-f51b-45e0-befe-8d8ce8a46c7d-workload-certs\") pod \"8332946b-f51b-45e0-befe-8d8ce8a46c7d\" (UID: \"8332946b-f51b-45e0-befe-8d8ce8a46c7d\") "
Oct 31 19:40:41.244364 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.240248    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8332946b-f51b-45e0-befe-8d8ce8a46c7d-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:41.244364 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.240279    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:41.244364 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.240581    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8332946b-f51b-45e0-befe-8d8ce8a46c7d-workload-certs" pod "8332946b-f51b-45e0-befe-8d8ce8a46c7d" (UID: "8332946b-f51b-45e0-befe-8d8ce8a46c7d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:41.244364 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.240776    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7-workload-certs" pod "a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7" (UID: "a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:41.244364 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.240953    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7-credential-socket" pod "a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7" (UID: "a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:41.244364 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.241125    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8332946b-f51b-45e0-befe-8d8ce8a46c7d-istio-data" pod "8332946b-f51b-45e0-befe-8d8ce8a46c7d" (UID: "8332946b-f51b-45e0-befe-8d8ce8a46c7d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:41.244364 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.241298    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8332946b-f51b-45e0-befe-8d8ce8a46c7d-credential-socket" pod "8332946b-f51b-45e0-befe-8d8ce8a46c7d" (UID: "8332946b-f51b-45e0-befe-8d8ce8a46c7d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:41.244364 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.241529    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7-istio-data" pod "a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7" (UID: "a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:41.244364 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.244136    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7-istio-envoy" pod "a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7" (UID: "a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:41.246065 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.244831    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8332946b-f51b-45e0-befe-8d8ce8a46c7d-istio-token" pod "8332946b-f51b-45e0-befe-8d8ce8a46c7d" (UID: "8332946b-f51b-45e0-befe-8d8ce8a46c7d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:40:41.246065 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.245103    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7-workload-socket" pod "a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7" (UID: "a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:41.246065 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.245942    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8332946b-f51b-45e0-befe-8d8ce8a46c7d-istiod-ca-cert" pod "8332946b-f51b-45e0-befe-8d8ce8a46c7d" (UID: "8332946b-f51b-45e0-befe-8d8ce8a46c7d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:40:41.247760 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.247207    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8332946b-f51b-45e0-befe-8d8ce8a46c7d-istio-podinfo" pod "8332946b-f51b-45e0-befe-8d8ce8a46c7d" (UID: "8332946b-f51b-45e0-befe-8d8ce8a46c7d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:40:41.247760 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.247723    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7-istio-token" pod "a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7" (UID: "a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:40:41.247760 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.247753    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7-kube-api-access-jzpd6" pod "a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7" (UID: "a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7"). InnerVolumeSpecName "kube-api-access-jzpd6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:40:41.248005 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.247833    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8332946b-f51b-45e0-befe-8d8ce8a46c7d-istio-envoy" pod "8332946b-f51b-45e0-befe-8d8ce8a46c7d" (UID: "8332946b-f51b-45e0-befe-8d8ce8a46c7d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:41.248506 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.248112    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8332946b-f51b-45e0-befe-8d8ce8a46c7d-kube-api-access-9xlt4" pod "8332946b-f51b-45e0-befe-8d8ce8a46c7d" (UID: "8332946b-f51b-45e0-befe-8d8ce8a46c7d"). InnerVolumeSpecName "kube-api-access-9xlt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:40:41.251658 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.251624    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7-istio-podinfo" pod "a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7" (UID: "a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:40:41.330921 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.330887    4645 generic.go:356] "Generic (PLEG): container finished" podID="8332946b-f51b-45e0-befe-8d8ce8a46c7d" containerID="ab576a191d5cace8de5e3988efae6e1f50c60884e16cab8ffa6e1a2c5a9591aa" exitCode=0
Oct 31 19:40:41.331107 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.331092    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2" event={"ID":"8332946b-f51b-45e0-befe-8d8ce8a46c7d","Type":"ContainerDied","Data":"ab576a191d5cace8de5e3988efae6e1f50c60884e16cab8ffa6e1a2c5a9591aa"}
Oct 31 19:40:41.331157 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.331135    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2" event={"ID":"8332946b-f51b-45e0-befe-8d8ce8a46c7d","Type":"ContainerDied","Data":"8a8dc77b3a1a0f71c972669b797cef8f86d1d4767638bb6557a65a40411118ae"}
Oct 31 19:40:41.331157 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.331155    4645 scope.go:122] "RemoveContainer" containerID="ab576a191d5cace8de5e3988efae6e1f50c60884e16cab8ffa6e1a2c5a9591aa"
Oct 31 19:40:41.331246 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.331236    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2"
Oct 31 19:40:41.341018 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.336537    4645 generic.go:356] "Generic (PLEG): container finished" podID="a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7" containerID="fa9954d53d791144c7a3b2dfdfe4bfa44b08b62e95fb0fcb5d1835f950c47e74" exitCode=0
Oct 31 19:40:41.341018 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.336637    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc"
Oct 31 19:40:41.341018 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.336639    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc" event={"ID":"a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7","Type":"ContainerDied","Data":"fa9954d53d791144c7a3b2dfdfe4bfa44b08b62e95fb0fcb5d1835f950c47e74"}
Oct 31 19:40:41.341018 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.336694    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc" event={"ID":"a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7","Type":"ContainerDied","Data":"fdf15a3d116dc80e9753e35af90abe589ba5ea49c5d8d33cbf9e00375d6fddbe"}
Oct 31 19:40:41.341018 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.339220    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl" event={"ID":"d505eb36-3800-4e5b-878d-c7ab58d3e901","Type":"ContainerDied","Data":"7aeef72655eb463c200e674c8d18cd80fcb718ec52c8f38f20b2170da3feb8ad"}
Oct 31 19:40:41.341018 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.339309    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl"
Oct 31 19:40:41.342802 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.342617    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8332946b-f51b-45e0-befe-8d8ce8a46c7d-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:41.342802 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.342637    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8332946b-f51b-45e0-befe-8d8ce8a46c7d-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:41.342802 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.342649    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:41.342802 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.342663    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8332946b-f51b-45e0-befe-8d8ce8a46c7d-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:41.342802 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.342675    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8332946b-f51b-45e0-befe-8d8ce8a46c7d-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:41.342802 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.342683    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:41.342802 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.342691    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:41.342802 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.342701    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8332946b-f51b-45e0-befe-8d8ce8a46c7d-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:41.342802 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.342709    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xlt4\" (UniqueName: \"kubernetes.io/projected/8332946b-f51b-45e0-befe-8d8ce8a46c7d-kube-api-access-9xlt4\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:41.342802 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.342717    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8332946b-f51b-45e0-befe-8d8ce8a46c7d-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:41.342802 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.342726    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:41.342802 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.342734    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8332946b-f51b-45e0-befe-8d8ce8a46c7d-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:41.342802 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.342742    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:41.342802 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.342755    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzpd6\" (UniqueName: \"kubernetes.io/projected/a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7-kube-api-access-jzpd6\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:41.342802 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.342769    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:41.342802 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.342780    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:41.348370 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.343661    4645 generic.go:356] "Generic (PLEG): container finished" podID="adb9b9a5-acbc-4967-b115-ed4e68a84b7e" containerID="8687de1652c92cd6304e37f64d2cf017bdec67320d27f72c072cf9a6162a3879" exitCode=0
Oct 31 19:40:41.348370 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.344794    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4"
Oct 31 19:40:41.386367 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.363882    4645 scope.go:122] "RemoveContainer" containerID="ab576a191d5cace8de5e3988efae6e1f50c60884e16cab8ffa6e1a2c5a9591aa"
Oct 31 19:40:41.386367 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:40:41.377595    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab576a191d5cace8de5e3988efae6e1f50c60884e16cab8ffa6e1a2c5a9591aa\": container with ID starting with ab576a191d5cace8de5e3988efae6e1f50c60884e16cab8ffa6e1a2c5a9591aa not found: ID does not exist" containerID="ab576a191d5cace8de5e3988efae6e1f50c60884e16cab8ffa6e1a2c5a9591aa"
Oct 31 19:40:41.386367 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.377640    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab576a191d5cace8de5e3988efae6e1f50c60884e16cab8ffa6e1a2c5a9591aa"} err="failed to get container status \"ab576a191d5cace8de5e3988efae6e1f50c60884e16cab8ffa6e1a2c5a9591aa\": rpc error: code = NotFound desc = could not find container \"ab576a191d5cace8de5e3988efae6e1f50c60884e16cab8ffa6e1a2c5a9591aa\": container with ID starting with ab576a191d5cace8de5e3988efae6e1f50c60884e16cab8ffa6e1a2c5a9591aa not found: ID does not exist"
Oct 31 19:40:41.386367 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.377672    4645 scope.go:122] "RemoveContainer" containerID="fa9954d53d791144c7a3b2dfdfe4bfa44b08b62e95fb0fcb5d1835f950c47e74"
Oct 31 19:40:41.386367 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.377796    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4" event={"ID":"adb9b9a5-acbc-4967-b115-ed4e68a84b7e","Type":"ContainerDied","Data":"8687de1652c92cd6304e37f64d2cf017bdec67320d27f72c072cf9a6162a3879"}
Oct 31 19:40:41.442468 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.433409    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96"
Oct 31 19:40:41.442468 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.433931    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4" event={"ID":"adb9b9a5-acbc-4967-b115-ed4e68a84b7e","Type":"ContainerDied","Data":"d29eb8b9e3aeb80456687f5b4b7617f0d678c6f780a8193f4db3995a0603a642"}
Oct 31 19:40:41.442468 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.439231    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96"
Oct 31 19:40:41.452885 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.452866    4645 scope.go:122] "RemoveContainer" containerID="fa9954d53d791144c7a3b2dfdfe4bfa44b08b62e95fb0fcb5d1835f950c47e74"
Oct 31 19:40:41.453153 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:40:41.453136    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa9954d53d791144c7a3b2dfdfe4bfa44b08b62e95fb0fcb5d1835f950c47e74\": container with ID starting with fa9954d53d791144c7a3b2dfdfe4bfa44b08b62e95fb0fcb5d1835f950c47e74 not found: ID does not exist" containerID="fa9954d53d791144c7a3b2dfdfe4bfa44b08b62e95fb0fcb5d1835f950c47e74"
Oct 31 19:40:41.453232 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.453170    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa9954d53d791144c7a3b2dfdfe4bfa44b08b62e95fb0fcb5d1835f950c47e74"} err="failed to get container status \"fa9954d53d791144c7a3b2dfdfe4bfa44b08b62e95fb0fcb5d1835f950c47e74\": rpc error: code = NotFound desc = could not find container \"fa9954d53d791144c7a3b2dfdfe4bfa44b08b62e95fb0fcb5d1835f950c47e74\": container with ID starting with fa9954d53d791144c7a3b2dfdfe4bfa44b08b62e95fb0fcb5d1835f950c47e74 not found: ID does not exist"
Oct 31 19:40:41.453232 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.453185    4645 scope.go:122] "RemoveContainer" containerID="4093fde47b02bdbc38bddfd04586bf689533e63c3b9210b5b21434ddd1862921"
Oct 31 19:40:41.528932 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.528891    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:41.710435 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.697277    4645 scope.go:122] "RemoveContainer" containerID="8687de1652c92cd6304e37f64d2cf017bdec67320d27f72c072cf9a6162a3879"
Oct 31 19:40:41.817363 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.817317    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl"]
Oct 31 19:40:41.827966 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.827939    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-9e323f-openshift-default-75fcf97f65-bcfzl"]
Oct 31 19:40:41.837270 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.837244    4645 scope.go:122] "RemoveContainer" containerID="8687de1652c92cd6304e37f64d2cf017bdec67320d27f72c072cf9a6162a3879"
Oct 31 19:40:41.837774 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:40:41.837733    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8687de1652c92cd6304e37f64d2cf017bdec67320d27f72c072cf9a6162a3879\": container with ID starting with 8687de1652c92cd6304e37f64d2cf017bdec67320d27f72c072cf9a6162a3879 not found: ID does not exist" containerID="8687de1652c92cd6304e37f64d2cf017bdec67320d27f72c072cf9a6162a3879"
Oct 31 19:40:41.838737 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.837812    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8687de1652c92cd6304e37f64d2cf017bdec67320d27f72c072cf9a6162a3879"} err="failed to get container status \"8687de1652c92cd6304e37f64d2cf017bdec67320d27f72c072cf9a6162a3879\": rpc error: code = NotFound desc = could not find container \"8687de1652c92cd6304e37f64d2cf017bdec67320d27f72c072cf9a6162a3879\": container with ID starting with 8687de1652c92cd6304e37f64d2cf017bdec67320d27f72c072cf9a6162a3879 not found: ID does not exist"
Oct 31 19:40:41.838853 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.838830    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4"]
Oct 31 19:40:41.850554 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.850532    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-drnb4"]
Oct 31 19:40:41.884407 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.884377    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc"]
Oct 31 19:40:41.889194 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.889117    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-b6fwc"]
Oct 31 19:40:41.895849 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.895831    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2"]
Oct 31 19:40:41.903379 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:41.902142    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-7qhq2"]
Oct 31 19:40:42.047618 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:42.047534    4645 patch_prober.go:29] interesting pod/e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body=
Oct 31 19:40:42.047970 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:42.047611    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7" podUID="fd915c45-10cb-4d0f-a519-44bbe338e43d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused"
Oct 31 19:40:42.050856 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:42.050833    4645 patch_prober.go:29] interesting pod/e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body=
Oct 31 19:40:42.050965 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:42.050877    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw" podUID="585024d3-5768-463e-acdc-81151a7c0461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused"
Oct 31 19:40:42.634559 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:42.634521    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:42.921921 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:42.921895    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:43.043220 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:43.043190    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:43.053187 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:43.053164    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7"
Oct 31 19:40:43.053623 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:43.053451    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7"
Oct 31 19:40:43.054303 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:43.054284    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7"
Oct 31 19:40:43.056118 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:43.056099    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw"
Oct 31 19:40:43.056369 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:43.056352    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw"
Oct 31 19:40:43.056997 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:43.056980    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw"
Oct 31 19:40:43.544466 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:43.544435    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:43.622535 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:43.622499    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8332946b-f51b-45e0-befe-8d8ce8a46c7d" path="/var/lib/kubelet/pods/8332946b-f51b-45e0-befe-8d8ce8a46c7d/volumes"
Oct 31 19:40:43.623123 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:43.623108    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7" path="/var/lib/kubelet/pods/a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7/volumes"
Oct 31 19:40:43.623689 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:43.623673    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adb9b9a5-acbc-4967-b115-ed4e68a84b7e" path="/var/lib/kubelet/pods/adb9b9a5-acbc-4967-b115-ed4e68a84b7e/volumes"
Oct 31 19:40:43.624542 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:43.624521    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d505eb36-3800-4e5b-878d-c7ab58d3e901" path="/var/lib/kubelet/pods/d505eb36-3800-4e5b-878d-c7ab58d3e901/volumes"
Oct 31 19:40:44.120316 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:44.120279    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:44.175434 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:44.175403    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1c0c1d-65fc757fd7-c79ts"]
Oct 31 19:40:44.176484 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:44.176462    4645 cpu_manager.go:414] "containerMap: removing container" podUID="d505eb36-3800-4e5b-878d-c7ab58d3e901" containerName="istio-proxy"
Oct 31 19:40:44.176593 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:44.176486    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d505eb36-3800-4e5b-878d-c7ab58d3e901" containerName="istio-proxy"
Oct 31 19:40:44.176593 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:44.176551    4645 cpu_manager.go:414] "containerMap: removing container" podUID="adb9b9a5-acbc-4967-b115-ed4e68a84b7e" containerName="istio-proxy"
Oct 31 19:40:44.176593 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:44.176562    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adb9b9a5-acbc-4967-b115-ed4e68a84b7e" containerName="istio-proxy"
Oct 31 19:40:44.176593 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:44.176592    4645 cpu_manager.go:414] "containerMap: removing container" podUID="a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7" containerName="istio-proxy"
Oct 31 19:40:44.176746 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:44.176601    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7" containerName="istio-proxy"
Oct 31 19:40:44.176746 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:44.176615    4645 cpu_manager.go:414] "containerMap: removing container" podUID="8332946b-f51b-45e0-befe-8d8ce8a46c7d" containerName="istio-proxy"
Oct 31 19:40:44.176746 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:44.176623    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8332946b-f51b-45e0-befe-8d8ce8a46c7d" containerName="istio-proxy"
Oct 31 19:40:44.176863 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:44.176792    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0477b6d-15d1-4f1e-a42c-8fbc54c3c8c7" containerName="istio-proxy"
Oct 31 19:40:44.176863 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:44.176808    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="8332946b-f51b-45e0-befe-8d8ce8a46c7d" containerName="istio-proxy"
Oct 31 19:40:44.176863 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:44.176821    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="d505eb36-3800-4e5b-878d-c7ab58d3e901" containerName="istio-proxy"
Oct 31 19:40:44.176863 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:44.176831    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="d505eb36-3800-4e5b-878d-c7ab58d3e901" containerName="istio-proxy"
Oct 31 19:40:44.176863 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:44.176847    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="adb9b9a5-acbc-4967-b115-ed4e68a84b7e" containerName="istio-proxy"
Oct 31 19:40:44.182890 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:44.182870    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1c0c1d-65fc757fd7-c79ts"
Oct 31 19:40:44.193264 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:44.193243    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1c0c1d-65fc757fd7-c79ts"]
Oct 31 19:40:44.200170 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:44.197554    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-1c0c1d-dockercfg-tgcxf\""
Oct 31 19:40:44.244662 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:44.244632    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc2c8b81-5e05-4a66-b7b0-23f993cf4f7c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-1c0c1d-65fc757fd7-c79ts\" (UID: \"bc2c8b81-5e05-4a66-b7b0-23f993cf4f7c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1c0c1d-65fc757fd7-c79ts"
Oct 31 19:40:44.245022 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:44.244996    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:44.245194 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:44.245124    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbsgh\" (UniqueName: \"kubernetes.io/projected/bc2c8b81-5e05-4a66-b7b0-23f993cf4f7c-kube-api-access-lbsgh\") pod \"payload-pre-processing-e2e-api-b-1c0c1d-65fc757fd7-c79ts\" (UID: \"bc2c8b81-5e05-4a66-b7b0-23f993cf4f7c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1c0c1d-65fc757fd7-c79ts"
Oct 31 19:40:44.275081 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:44.275062    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-1c0c1d\""
Oct 31 19:40:44.346314 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:44.346285    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbsgh\" (UniqueName: \"kubernetes.io/projected/bc2c8b81-5e05-4a66-b7b0-23f993cf4f7c-kube-api-access-lbsgh\") pod \"payload-pre-processing-e2e-api-b-1c0c1d-65fc757fd7-c79ts\" (UID: \"bc2c8b81-5e05-4a66-b7b0-23f993cf4f7c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1c0c1d-65fc757fd7-c79ts"
Oct 31 19:40:44.346649 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:44.346625    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc2c8b81-5e05-4a66-b7b0-23f993cf4f7c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-1c0c1d-65fc757fd7-c79ts\" (UID: \"bc2c8b81-5e05-4a66-b7b0-23f993cf4f7c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1c0c1d-65fc757fd7-c79ts"
Oct 31 19:40:44.347228 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:44.347207    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc2c8b81-5e05-4a66-b7b0-23f993cf4f7c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-1c0c1d-65fc757fd7-c79ts\" (UID: \"bc2c8b81-5e05-4a66-b7b0-23f993cf4f7c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1c0c1d-65fc757fd7-c79ts"
Oct 31 19:40:44.413404 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:44.401306    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbsgh\" (UniqueName: \"kubernetes.io/projected/bc2c8b81-5e05-4a66-b7b0-23f993cf4f7c-kube-api-access-lbsgh\") pod \"payload-pre-processing-e2e-api-b-1c0c1d-65fc757fd7-c79ts\" (UID: \"bc2c8b81-5e05-4a66-b7b0-23f993cf4f7c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-1c0c1d-65fc757fd7-c79ts"
Oct 31 19:40:44.504350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:44.503645    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1c0c1d-65fc757fd7-c79ts"
Oct 31 19:40:44.853619 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:44.853590    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:44.956499 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:44.956470    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1c0c1d-65fc757fd7-c79ts"]
Oct 31 19:40:44.956777 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:40:44.956749    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc2c8b81_5e05_4a66_b7b0_23f993cf4f7c.slice/crio-600c4fd605affcc8b27d87af12c86a53e2afb2ea7f7be0ae59eb6edc17005a12 WatchSource:0}: Error finding container 600c4fd605affcc8b27d87af12c86a53e2afb2ea7f7be0ae59eb6edc17005a12: Status 404 returned error can't find the container with id 600c4fd605affcc8b27d87af12c86a53e2afb2ea7f7be0ae59eb6edc17005a12
Oct 31 19:40:45.264681 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:45.264601    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:45.325443 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:45.325414    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d4b21c-764698f6f-dt2q6"]
Oct 31 19:40:45.341653 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:45.341630    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d4b21c-6bf64967d7-rj87s"]
Oct 31 19:40:45.398569 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:45.398542    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:45.460991 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:45.460953    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1c0c1d-65fc757fd7-c79ts" event={"ID":"bc2c8b81-5e05-4a66-b7b0-23f993cf4f7c","Type":"ContainerStarted","Data":"2d00d04edbf5bd0d000fb9f6bee310837ae5ae214a802d4f79ccea80a42fc416"}
Oct 31 19:40:45.461151 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:45.461021    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1c0c1d-65fc757fd7-c79ts" event={"ID":"bc2c8b81-5e05-4a66-b7b0-23f993cf4f7c","Type":"ContainerStarted","Data":"600c4fd605affcc8b27d87af12c86a53e2afb2ea7f7be0ae59eb6edc17005a12"}
Oct 31 19:40:45.461239 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:45.461214    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1c0c1d-65fc757fd7-c79ts"
Oct 31 19:40:45.489693 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:45.489649    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1c0c1d-65fc757fd7-c79ts" podStartSLOduration=1.4896348289999999 podStartE2EDuration="1.489634829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:40:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:40:45.486452373 +0000 UTC m=+1276.059686514" watchObservedRunningTime="2026-10-31 19:40:45.489634829 +0000 UTC m=+1276.062868959"
Oct 31 19:40:45.594700 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:45.594641    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:46.260211 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:46.260167    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:47.151888 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:47.151855    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:47.751829 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:47.751798    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg"]
Oct 31 19:40:47.753250 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:47.753221    4645 cpu_manager.go:414] "containerMap: removing container" podUID="d505eb36-3800-4e5b-878d-c7ab58d3e901" containerName="istio-proxy"
Oct 31 19:40:47.753250 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:47.753249    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d505eb36-3800-4e5b-878d-c7ab58d3e901" containerName="istio-proxy"
Oct 31 19:40:47.758749 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:47.758729    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg"
Oct 31 19:40:47.769424 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:47.769397    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg"]
Oct 31 19:40:47.864100 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:47.864071    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bacc59a-7abc-43b3-81dd-a8829710ad35-workload-certs\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg\" (UID: \"9bacc59a-7abc-43b3-81dd-a8829710ad35\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg"
Oct 31 19:40:47.864254 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:47.864134    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bacc59a-7abc-43b3-81dd-a8829710ad35-istio-data\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg\" (UID: \"9bacc59a-7abc-43b3-81dd-a8829710ad35\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg"
Oct 31 19:40:47.864254 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:47.864232    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bacc59a-7abc-43b3-81dd-a8829710ad35-istio-envoy\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg\" (UID: \"9bacc59a-7abc-43b3-81dd-a8829710ad35\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg"
Oct 31 19:40:47.864364 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:47.864287    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bacc59a-7abc-43b3-81dd-a8829710ad35-istio-token\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg\" (UID: \"9bacc59a-7abc-43b3-81dd-a8829710ad35\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg"
Oct 31 19:40:47.864364 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:47.864310    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bacc59a-7abc-43b3-81dd-a8829710ad35-istiod-ca-cert\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg\" (UID: \"9bacc59a-7abc-43b3-81dd-a8829710ad35\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg"
Oct 31 19:40:47.864511 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:47.864491    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wt4fl\" (UniqueName: \"kubernetes.io/projected/9bacc59a-7abc-43b3-81dd-a8829710ad35-kube-api-access-wt4fl\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg\" (UID: \"9bacc59a-7abc-43b3-81dd-a8829710ad35\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg"
Oct 31 19:40:47.864562 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:47.864521    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bacc59a-7abc-43b3-81dd-a8829710ad35-credential-socket\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg\" (UID: \"9bacc59a-7abc-43b3-81dd-a8829710ad35\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg"
Oct 31 19:40:47.864659 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:47.864639    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bacc59a-7abc-43b3-81dd-a8829710ad35-istio-podinfo\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg\" (UID: \"9bacc59a-7abc-43b3-81dd-a8829710ad35\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg"
Oct 31 19:40:47.864713 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:47.864682    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bacc59a-7abc-43b3-81dd-a8829710ad35-workload-socket\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg\" (UID: \"9bacc59a-7abc-43b3-81dd-a8829710ad35\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg"
Oct 31 19:40:47.965724 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:47.965698    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wt4fl\" (UniqueName: \"kubernetes.io/projected/9bacc59a-7abc-43b3-81dd-a8829710ad35-kube-api-access-wt4fl\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg\" (UID: \"9bacc59a-7abc-43b3-81dd-a8829710ad35\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg"
Oct 31 19:40:47.965839 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:47.965730    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bacc59a-7abc-43b3-81dd-a8829710ad35-credential-socket\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg\" (UID: \"9bacc59a-7abc-43b3-81dd-a8829710ad35\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg"
Oct 31 19:40:47.965839 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:47.965774    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bacc59a-7abc-43b3-81dd-a8829710ad35-istio-podinfo\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg\" (UID: \"9bacc59a-7abc-43b3-81dd-a8829710ad35\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg"
Oct 31 19:40:47.965839 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:47.965795    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bacc59a-7abc-43b3-81dd-a8829710ad35-workload-socket\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg\" (UID: \"9bacc59a-7abc-43b3-81dd-a8829710ad35\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg"
Oct 31 19:40:47.965839 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:47.965823    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bacc59a-7abc-43b3-81dd-a8829710ad35-workload-certs\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg\" (UID: \"9bacc59a-7abc-43b3-81dd-a8829710ad35\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg"
Oct 31 19:40:47.966046 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:47.965878    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bacc59a-7abc-43b3-81dd-a8829710ad35-istio-data\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg\" (UID: \"9bacc59a-7abc-43b3-81dd-a8829710ad35\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg"
Oct 31 19:40:47.966046 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:47.965905    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bacc59a-7abc-43b3-81dd-a8829710ad35-istio-envoy\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg\" (UID: \"9bacc59a-7abc-43b3-81dd-a8829710ad35\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg"
Oct 31 19:40:47.966046 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:47.965977    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bacc59a-7abc-43b3-81dd-a8829710ad35-istio-token\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg\" (UID: \"9bacc59a-7abc-43b3-81dd-a8829710ad35\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg"
Oct 31 19:40:47.966046 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:47.966014    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bacc59a-7abc-43b3-81dd-a8829710ad35-istiod-ca-cert\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg\" (UID: \"9bacc59a-7abc-43b3-81dd-a8829710ad35\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg"
Oct 31 19:40:47.966264 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:47.966171    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bacc59a-7abc-43b3-81dd-a8829710ad35-workload-socket\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg\" (UID: \"9bacc59a-7abc-43b3-81dd-a8829710ad35\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg"
Oct 31 19:40:47.966264 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:47.966240    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bacc59a-7abc-43b3-81dd-a8829710ad35-credential-socket\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg\" (UID: \"9bacc59a-7abc-43b3-81dd-a8829710ad35\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg"
Oct 31 19:40:47.966444 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:47.966424    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bacc59a-7abc-43b3-81dd-a8829710ad35-workload-certs\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg\" (UID: \"9bacc59a-7abc-43b3-81dd-a8829710ad35\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg"
Oct 31 19:40:47.966507 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:47.966487    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bacc59a-7abc-43b3-81dd-a8829710ad35-istio-data\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg\" (UID: \"9bacc59a-7abc-43b3-81dd-a8829710ad35\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg"
Oct 31 19:40:47.966764 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:47.966744    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bacc59a-7abc-43b3-81dd-a8829710ad35-istiod-ca-cert\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg\" (UID: \"9bacc59a-7abc-43b3-81dd-a8829710ad35\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg"
Oct 31 19:40:47.968361 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:47.968310    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bacc59a-7abc-43b3-81dd-a8829710ad35-istio-envoy\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg\" (UID: \"9bacc59a-7abc-43b3-81dd-a8829710ad35\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg"
Oct 31 19:40:47.968487 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:47.968465    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bacc59a-7abc-43b3-81dd-a8829710ad35-istio-podinfo\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg\" (UID: \"9bacc59a-7abc-43b3-81dd-a8829710ad35\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg"
Oct 31 19:40:47.985701 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:47.985679    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bacc59a-7abc-43b3-81dd-a8829710ad35-istio-token\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg\" (UID: \"9bacc59a-7abc-43b3-81dd-a8829710ad35\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg"
Oct 31 19:40:47.985914 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:47.985894    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wt4fl\" (UniqueName: \"kubernetes.io/projected/9bacc59a-7abc-43b3-81dd-a8829710ad35-kube-api-access-wt4fl\") pod \"e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg\" (UID: \"9bacc59a-7abc-43b3-81dd-a8829710ad35\") " pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg"
Oct 31 19:40:48.028779 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:48.028730    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:48.081350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:48.077780    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg"
Oct 31 19:40:48.435571 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:48.435544    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg"]
Oct 31 19:40:48.437391 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:40:48.437368    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bacc59a_7abc_43b3_81dd_a8829710ad35.slice/crio-5f3ddaf49dfbe7170868440577e352df5a231fcbed10844c66e99954fbca55c8 WatchSource:0}: Error finding container 5f3ddaf49dfbe7170868440577e352df5a231fcbed10844c66e99954fbca55c8: Status 404 returned error can't find the container with id 5f3ddaf49dfbe7170868440577e352df5a231fcbed10844c66e99954fbca55c8
Oct 31 19:40:48.440387 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:48.440308    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:40:48.440486 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:48.440431    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:40:48.440486 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:48.440467    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:40:48.482062 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:48.482038    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg" event={"ID":"9bacc59a-7abc-43b3-81dd-a8829710ad35","Type":"ContainerStarted","Data":"5f3ddaf49dfbe7170868440577e352df5a231fcbed10844c66e99954fbca55c8"}
Oct 31 19:40:48.777948 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:48.777865    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:48.882761 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:48.882728    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:49.356443 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:49.356407    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm"]
Oct 31 19:40:49.356652 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:49.356625    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm" podUID="0f482c94-43ba-4570-9b2a-ef4886627127" containerName="istio-proxy" containerID="cri-o://185cdf71b9e21f8f49d2323c2f6de1ed0c6a9f1d6544be04ca5065dd4c992af1" gracePeriod=30
Oct 31 19:40:49.361135 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:49.361099    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf"]
Oct 31 19:40:49.361498 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:49.361470    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf" podUID="b2a68f87-af40-4d21-9ef0-620f397e13be" containerName="istio-proxy" containerID="cri-o://aaa54faf936afb8ee998be89fa312e75432531e53d0905dcca6e7f2fe700885c" gracePeriod=30
Oct 31 19:40:49.366236 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:49.366213    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:49.367553 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:49.367528    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl"]
Oct 31 19:40:49.367739 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:49.367714    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl" podUID="619ab3b5-8258-4aea-90fe-a351097d91cd" containerName="istio-proxy" containerID="cri-o://91eaff5584a21d4367366fb3491e9729eeaad0a898f0406a181141ad0977f780" gracePeriod=30
Oct 31 19:40:49.491573 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:49.491528    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:49.526141 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:49.503189    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg" event={"ID":"9bacc59a-7abc-43b3-81dd-a8829710ad35","Type":"ContainerStarted","Data":"b8dce505a1fc1201cc5c6775c022b0b334366af36acbbff66fd883f23b33be5f"}
Oct 31 19:40:49.532720 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:49.529823    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-67adab47/e2e-lifecycle-model-67adab47-kserve-8f9445cd9-5rp7l" podUID="b9153d96-5237-47a0-8bf4-1e77f69b071a" containerName="main" containerID="cri-o://a61d089e43ec0882d8b96983157bbc7bd7c9721db274507a0ffc55a0a0c02e28" gracePeriod=45
Oct 31 19:40:49.562412 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:49.561729    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg" podStartSLOduration=2.561705054 podStartE2EDuration="2.561705054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:40:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:40:49.533244215 +0000 UTC m=+1280.106478343" watchObservedRunningTime="2026-10-31 19:40:49.561705054 +0000 UTC m=+1280.134939185"
Oct 31 19:40:49.839873 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:49.839839    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:49.923567 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:49.919161    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2"]
Oct 31 19:40:49.924969 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:49.924945    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2"
Oct 31 19:40:49.936599 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:49.936570    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-f5c1ab-openshift-default-dockercfg-r9nbk\""
Oct 31 19:40:49.991353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:49.989843    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2"]
Oct 31 19:40:49.991626 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:49.991599    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:50.012380 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:50.003222    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-67adab47/e2e-lifecycle-model-67adab47-kserve-8f9445cd9-5rp7l"
Oct 31 19:40:50.043356 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:50.014917    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/813ae008-e079-4406-83f7-6eae7acc3ce1-istio-envoy\") pod \"e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2\" (UID: \"813ae008-e079-4406-83f7-6eae7acc3ce1\") " pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2"
Oct 31 19:40:50.043356 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:50.015013    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/813ae008-e079-4406-83f7-6eae7acc3ce1-workload-socket\") pod \"e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2\" (UID: \"813ae008-e079-4406-83f7-6eae7acc3ce1\") " pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2"
Oct 31 19:40:50.043356 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:50.015049    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/813ae008-e079-4406-83f7-6eae7acc3ce1-workload-certs\") pod \"e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2\" (UID: \"813ae008-e079-4406-83f7-6eae7acc3ce1\") " pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2"
Oct 31 19:40:50.043356 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:50.015096    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/813ae008-e079-4406-83f7-6eae7acc3ce1-istio-data\") pod \"e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2\" (UID: \"813ae008-e079-4406-83f7-6eae7acc3ce1\") " pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2"
Oct 31 19:40:50.043356 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:50.015145    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/813ae008-e079-4406-83f7-6eae7acc3ce1-istio-podinfo\") pod \"e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2\" (UID: \"813ae008-e079-4406-83f7-6eae7acc3ce1\") " pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2"
Oct 31 19:40:50.043356 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:50.015178    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9vgc\" (UniqueName: \"kubernetes.io/projected/813ae008-e079-4406-83f7-6eae7acc3ce1-kube-api-access-k9vgc\") pod \"e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2\" (UID: \"813ae008-e079-4406-83f7-6eae7acc3ce1\") " pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2"
Oct 31 19:40:50.043356 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:50.015256    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/813ae008-e079-4406-83f7-6eae7acc3ce1-istiod-ca-cert\") pod \"e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2\" (UID: \"813ae008-e079-4406-83f7-6eae7acc3ce1\") " pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2"
Oct 31 19:40:50.043356 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:50.015285    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/813ae008-e079-4406-83f7-6eae7acc3ce1-credential-socket\") pod \"e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2\" (UID: \"813ae008-e079-4406-83f7-6eae7acc3ce1\") " pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2"
Oct 31 19:40:50.043356 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:50.015360    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/813ae008-e079-4406-83f7-6eae7acc3ce1-istio-token\") pod \"e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2\" (UID: \"813ae008-e079-4406-83f7-6eae7acc3ce1\") " pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2"
Oct 31 19:40:50.094216 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:50.093000    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg"
Oct 31 19:40:50.133362 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:50.125541    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9153d96-5237-47a0-8bf4-1e77f69b071a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9153d96-5237-47a0-8bf4-1e77f69b071a-tmp-dir\") pod \"b9153d96-5237-47a0-8bf4-1e77f69b071a\" (UID: \"b9153d96-5237-47a0-8bf4-1e77f69b071a\") "
Oct 31 19:40:50.133362 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:50.125673    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9153d96-5237-47a0-8bf4-1e77f69b071a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9153d96-5237-47a0-8bf4-1e77f69b071a-dshm\") pod \"b9153d96-5237-47a0-8bf4-1e77f69b071a\" (UID: \"b9153d96-5237-47a0-8bf4-1e77f69b071a\") "
Oct 31 19:40:50.133362 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:50.125733    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9153d96-5237-47a0-8bf4-1e77f69b071a-home\" (UniqueName: \"kubernetes.io/empty-dir/b9153d96-5237-47a0-8bf4-1e77f69b071a-home\") pod \"b9153d96-5237-47a0-8bf4-1e77f69b071a\" (UID: \"b9153d96-5237-47a0-8bf4-1e77f69b071a\") "
Oct 31 19:40:50.133362 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:50.125807    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9153d96-5237-47a0-8bf4-1e77f69b071a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9153d96-5237-47a0-8bf4-1e77f69b071a-model-cache\") pod \"b9153d96-5237-47a0-8bf4-1e77f69b071a\" (UID: \"b9153d96-5237-47a0-8bf4-1e77f69b071a\") "
Oct 31 19:40:50.133362 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:50.125845    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9153d96-5237-47a0-8bf4-1e77f69b071a-kube-api-access-4plcs\" (UniqueName: \"kubernetes.io/projected/b9153d96-5237-47a0-8bf4-1e77f69b071a-kube-api-access-4plcs\") pod \"b9153d96-5237-47a0-8bf4-1e77f69b071a\" (UID: \"b9153d96-5237-47a0-8bf4-1e77f69b071a\") "
Oct 31 19:40:50.133362 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:50.125886    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9153d96-5237-47a0-8bf4-1e77f69b071a-tls-certs\" (UniqueName: \"kubernetes.io/secret/b9153d96-5237-47a0-8bf4-1e77f69b071a-tls-certs\") pod \"b9153d96-5237-47a0-8bf4-1e77f69b071a\" (UID: \"b9153d96-5237-47a0-8bf4-1e77f69b071a\") "
Oct 31 19:40:50.133362 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:50.132655    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9153d96-5237-47a0-8bf4-1e77f69b071a-home" pod "b9153d96-5237-47a0-8bf4-1e77f69b071a" (UID: "b9153d96-5237-47a0-8bf4-1e77f69b071a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:50.133362 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:50.132841    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9153d96-5237-47a0-8bf4-1e77f69b071a-tmp-dir" pod "b9153d96-5237-47a0-8bf4-1e77f69b071a" (UID: "b9153d96-5237-47a0-8bf4-1e77f69b071a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:50.141958 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:50.136417    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9153d96-5237-47a0-8bf4-1e77f69b071a-model-cache" pod "b9153d96-5237-47a0-8bf4-1e77f69b071a" (UID: "b9153d96-5237-47a0-8bf4-1e77f69b071a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:50.180642 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:50.150653    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/813ae008-e079-4406-83f7-6eae7acc3ce1-istio-envoy\") pod \"e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2\" (UID: \"813ae008-e079-4406-83f7-6eae7acc3ce1\") " pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2"
Oct 31 19:40:50.180642 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:50.150760    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/813ae008-e079-4406-83f7-6eae7acc3ce1-workload-socket\") pod \"e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2\" (UID: \"813ae008-e079-4406-83f7-6eae7acc3ce1\") " pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2"
Oct 31 19:40:50.180642 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:50.150796    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/813ae008-e079-4406-83f7-6eae7acc3ce1-workload-certs\") pod \"e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2\" (UID: \"813ae008-e079-4406-83f7-6eae7acc3ce1\") " pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2"
Oct 31 19:40:50.180642 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:50.150849    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/813ae008-e079-4406-83f7-6eae7acc3ce1-istio-data\") pod \"e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2\" (UID: \"813ae008-e079-4406-83f7-6eae7acc3ce1\") " pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2"
Oct 31 19:40:50.180642 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:50.150903    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/813ae008-e079-4406-83f7-6eae7acc3ce1-istio-podinfo\") pod \"e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2\" (UID: \"813ae008-e079-4406-83f7-6eae7acc3ce1\") " pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2"
Oct 31 19:40:50.180642 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:50.150936    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9vgc\" (UniqueName: \"kubernetes.io/projected/813ae008-e079-4406-83f7-6eae7acc3ce1-kube-api-access-k9vgc\") pod \"e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2\" (UID: \"813ae008-e079-4406-83f7-6eae7acc3ce1\") " pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2"
Oct 31 19:40:50.180642 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:50.151029    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/813ae008-e079-4406-83f7-6eae7acc3ce1-istiod-ca-cert\") pod \"e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2\" (UID: \"813ae008-e079-4406-83f7-6eae7acc3ce1\") " pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2"
Oct 31 19:40:50.180642 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:50.151058    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/813ae008-e079-4406-83f7-6eae7acc3ce1-credential-socket\") pod \"e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2\" (UID: \"813ae008-e079-4406-83f7-6eae7acc3ce1\") " pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2"
Oct 31 19:40:50.180642 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:50.151126    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/813ae008-e079-4406-83f7-6eae7acc3ce1-istio-token\") pod \"e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2\" (UID: \"813ae008-e079-4406-83f7-6eae7acc3ce1\") " pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2"
Oct 31 19:40:50.180642 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:50.169784    4645 patch_prober.go:29] interesting pod/e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 31 19:40:50.180642 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:50.169839    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg" podUID="9bacc59a-7abc-43b3-81dd-a8829710ad35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 31 19:40:50.180642 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:50.173609    4645 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9153d96-5237-47a0-8bf4-1e77f69b071a-home\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:50.180642 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:50.173638    4645 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9153d96-5237-47a0-8bf4-1e77f69b071a-model-cache\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:50.180642 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:50.173657    4645 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9153d96-5237-47a0-8bf4-1e77f69b071a-tmp-dir\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:50.183741 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:50.183713    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:50.201355 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:50.199759    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/813ae008-e079-4406-83f7-6eae7acc3ce1-workload-certs\") pod \"e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2\" (UID: \"813ae008-e079-4406-83f7-6eae7acc3ce1\") " pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2"
Oct 31 19:40:50.201355 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:50.200014    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/813ae008-e079-4406-83f7-6eae7acc3ce1-workload-socket\") pod \"e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2\" (UID: \"813ae008-e079-4406-83f7-6eae7acc3ce1\") " pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2"
Oct 31 19:40:50.201355 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:50.200653    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/813ae008-e079-4406-83f7-6eae7acc3ce1-istiod-ca-cert\") pod \"e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2\" (UID: \"813ae008-e079-4406-83f7-6eae7acc3ce1\") " pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2"
Oct 31 19:40:50.201355 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:50.201172    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/813ae008-e079-4406-83f7-6eae7acc3ce1-istio-data\") pod \"e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2\" (UID: \"813ae008-e079-4406-83f7-6eae7acc3ce1\") " pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2"
Oct 31 19:40:50.201724 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:50.201505    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/813ae008-e079-4406-83f7-6eae7acc3ce1-credential-socket\") pod \"e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2\" (UID: \"813ae008-e079-4406-83f7-6eae7acc3ce1\") " pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2"
Oct 31 19:40:50.373235 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:50.344422    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d4b21c-6bf64967d7-rj87s" podUID="ef5a2c9c-2e89-4945-a70d-8eea47d0b517" containerName="payload-pre-processing" containerID="cri-o://90ec39e391aaa91fe1831b72bfa7f80b0edf99e521d255b8a67bd6c94f1d9be6" gracePeriod=25
Oct 31 19:40:50.373235 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:50.344638    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-d4b21c-764698f6f-dt2q6" podUID="49ea0b50-2dfa-435c-9940-98e57cbb6aa4" containerName="payload-processing" containerID="cri-o://2b40fe03181750a4a1389e65a8e88682a3e083c5d4955a733b603f9e2d954778" gracePeriod=25
Oct 31 19:40:50.402349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:50.398690    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/813ae008-e079-4406-83f7-6eae7acc3ce1-istio-token\") pod \"e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2\" (UID: \"813ae008-e079-4406-83f7-6eae7acc3ce1\") " pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2"
Oct 31 19:40:50.455632 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:50.455549    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9153d96-5237-47a0-8bf4-1e77f69b071a-kube-api-access-4plcs" pod "b9153d96-5237-47a0-8bf4-1e77f69b071a" (UID: "b9153d96-5237-47a0-8bf4-1e77f69b071a"). InnerVolumeSpecName "kube-api-access-4plcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:40:50.509721 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:50.507902    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4plcs\" (UniqueName: \"kubernetes.io/projected/b9153d96-5237-47a0-8bf4-1e77f69b071a-kube-api-access-4plcs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:50.519168 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:50.517547    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9153d96-5237-47a0-8bf4-1e77f69b071a-dshm" pod "b9153d96-5237-47a0-8bf4-1e77f69b071a" (UID: "b9153d96-5237-47a0-8bf4-1e77f69b071a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:50.519168 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:50.518926    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9153d96-5237-47a0-8bf4-1e77f69b071a-tls-certs" pod "b9153d96-5237-47a0-8bf4-1e77f69b071a" (UID: "b9153d96-5237-47a0-8bf4-1e77f69b071a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:40:50.520064 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:50.519986    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/813ae008-e079-4406-83f7-6eae7acc3ce1-istio-envoy\") pod \"e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2\" (UID: \"813ae008-e079-4406-83f7-6eae7acc3ce1\") " pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2"
Oct 31 19:40:50.521115 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:50.521088    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/813ae008-e079-4406-83f7-6eae7acc3ce1-istio-podinfo\") pod \"e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2\" (UID: \"813ae008-e079-4406-83f7-6eae7acc3ce1\") " pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2"
Oct 31 19:40:50.527718 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:50.527691    4645 generic.go:356] "Generic (PLEG): container finished" podID="b9153d96-5237-47a0-8bf4-1e77f69b071a" containerID="a61d089e43ec0882d8b96983157bbc7bd7c9721db274507a0ffc55a0a0c02e28" exitCode=0
Oct 31 19:40:50.528559 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:50.527936    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-67adab47/e2e-lifecycle-model-67adab47-kserve-8f9445cd9-5rp7l" event={"ID":"b9153d96-5237-47a0-8bf4-1e77f69b071a","Type":"ContainerDied","Data":"a61d089e43ec0882d8b96983157bbc7bd7c9721db274507a0ffc55a0a0c02e28"}
Oct 31 19:40:50.528559 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:50.527963    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-67adab47/e2e-lifecycle-model-67adab47-kserve-8f9445cd9-5rp7l"
Oct 31 19:40:50.528559 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:50.527989    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-67adab47/e2e-lifecycle-model-67adab47-kserve-8f9445cd9-5rp7l" event={"ID":"b9153d96-5237-47a0-8bf4-1e77f69b071a","Type":"ContainerDied","Data":"b854245ceb75df64752cd1a544d4b434c72c0845df354750e65697e8e544014c"}
Oct 31 19:40:50.528559 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:50.528014    4645 scope.go:122] "RemoveContainer" containerID="a61d089e43ec0882d8b96983157bbc7bd7c9721db274507a0ffc55a0a0c02e28"
Oct 31 19:40:50.528559 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:50.528070    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9vgc\" (UniqueName: \"kubernetes.io/projected/813ae008-e079-4406-83f7-6eae7acc3ce1-kube-api-access-k9vgc\") pod \"e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2\" (UID: \"813ae008-e079-4406-83f7-6eae7acc3ce1\") " pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2"
Oct 31 19:40:50.601190 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:50.600867    4645 scope.go:122] "RemoveContainer" containerID="a61d089e43ec0882d8b96983157bbc7bd7c9721db274507a0ffc55a0a0c02e28"
Oct 31 19:40:50.601312 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:40:50.601228    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a61d089e43ec0882d8b96983157bbc7bd7c9721db274507a0ffc55a0a0c02e28\": container with ID starting with a61d089e43ec0882d8b96983157bbc7bd7c9721db274507a0ffc55a0a0c02e28 not found: ID does not exist" containerID="a61d089e43ec0882d8b96983157bbc7bd7c9721db274507a0ffc55a0a0c02e28"
Oct 31 19:40:50.601312 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:50.601265    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a61d089e43ec0882d8b96983157bbc7bd7c9721db274507a0ffc55a0a0c02e28"} err="failed to get container status \"a61d089e43ec0882d8b96983157bbc7bd7c9721db274507a0ffc55a0a0c02e28\": rpc error: code = NotFound desc = could not find container \"a61d089e43ec0882d8b96983157bbc7bd7c9721db274507a0ffc55a0a0c02e28\": container with ID starting with a61d089e43ec0882d8b96983157bbc7bd7c9721db274507a0ffc55a0a0c02e28 not found: ID does not exist"
Oct 31 19:40:50.610034 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:50.609636    4645 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9153d96-5237-47a0-8bf4-1e77f69b071a-dshm\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:50.610034 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:50.609662    4645 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9153d96-5237-47a0-8bf4-1e77f69b071a-tls-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:50.614806 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:50.614782    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-67adab47/e2e-lifecycle-model-67adab47-kserve-8f9445cd9-5rp7l"]
Oct 31 19:40:50.622957 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:50.622929    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-67adab47/e2e-lifecycle-model-67adab47-kserve-8f9445cd9-5rp7l"]
Oct 31 19:40:50.684148 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:50.683971    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:50.777120 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:50.770749    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2"
Oct 31 19:40:50.777120 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:50.776016    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d4b21c-6bf64967d7-rj87s"
Oct 31 19:40:50.833702 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:50.818976    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef5a2c9c-2e89-4945-a70d-8eea47d0b517-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef5a2c9c-2e89-4945-a70d-8eea47d0b517-plugins-config-volume\") pod \"ef5a2c9c-2e89-4945-a70d-8eea47d0b517\" (UID: \"ef5a2c9c-2e89-4945-a70d-8eea47d0b517\") "
Oct 31 19:40:50.833702 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:50.819036    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef5a2c9c-2e89-4945-a70d-8eea47d0b517-kube-api-access-mgdk8\" (UniqueName: \"kubernetes.io/projected/ef5a2c9c-2e89-4945-a70d-8eea47d0b517-kube-api-access-mgdk8\") pod \"ef5a2c9c-2e89-4945-a70d-8eea47d0b517\" (UID: \"ef5a2c9c-2e89-4945-a70d-8eea47d0b517\") "
Oct 31 19:40:50.833702 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:50.826869    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:50.833702 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:50.827282    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef5a2c9c-2e89-4945-a70d-8eea47d0b517-plugins-config-volume" pod "ef5a2c9c-2e89-4945-a70d-8eea47d0b517" (UID: "ef5a2c9c-2e89-4945-a70d-8eea47d0b517"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:40:50.833702 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:50.828658    4645 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef5a2c9c-2e89-4945-a70d-8eea47d0b517-plugins-config-volume\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:50.837400 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:50.835921    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef5a2c9c-2e89-4945-a70d-8eea47d0b517-kube-api-access-mgdk8" pod "ef5a2c9c-2e89-4945-a70d-8eea47d0b517" (UID: "ef5a2c9c-2e89-4945-a70d-8eea47d0b517"). InnerVolumeSpecName "kube-api-access-mgdk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:40:50.852110 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:50.851698    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d4b21c-764698f6f-dt2q6"
Oct 31 19:40:50.929878 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:50.929854    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49ea0b50-2dfa-435c-9940-98e57cbb6aa4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49ea0b50-2dfa-435c-9940-98e57cbb6aa4-plugins-config-volume\") pod \"49ea0b50-2dfa-435c-9940-98e57cbb6aa4\" (UID: \"49ea0b50-2dfa-435c-9940-98e57cbb6aa4\") "
Oct 31 19:40:50.930494 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:50.930467    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49ea0b50-2dfa-435c-9940-98e57cbb6aa4-plugins-config-volume" pod "49ea0b50-2dfa-435c-9940-98e57cbb6aa4" (UID: "49ea0b50-2dfa-435c-9940-98e57cbb6aa4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:40:50.930620 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:50.930605    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49ea0b50-2dfa-435c-9940-98e57cbb6aa4-kube-api-access-5pr8c\" (UniqueName: \"kubernetes.io/projected/49ea0b50-2dfa-435c-9940-98e57cbb6aa4-kube-api-access-5pr8c\") pod \"49ea0b50-2dfa-435c-9940-98e57cbb6aa4\" (UID: \"49ea0b50-2dfa-435c-9940-98e57cbb6aa4\") "
Oct 31 19:40:50.931736 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:50.931716    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgdk8\" (UniqueName: \"kubernetes.io/projected/ef5a2c9c-2e89-4945-a70d-8eea47d0b517-kube-api-access-mgdk8\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:50.931868 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:50.931853    4645 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49ea0b50-2dfa-435c-9940-98e57cbb6aa4-plugins-config-volume\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:50.933648 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:50.933624    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49ea0b50-2dfa-435c-9940-98e57cbb6aa4-kube-api-access-5pr8c" pod "49ea0b50-2dfa-435c-9940-98e57cbb6aa4" (UID: "49ea0b50-2dfa-435c-9940-98e57cbb6aa4"). InnerVolumeSpecName "kube-api-access-5pr8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:40:51.033839 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:51.033762    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pr8c\" (UniqueName: \"kubernetes.io/projected/49ea0b50-2dfa-435c-9940-98e57cbb6aa4-kube-api-access-5pr8c\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:51.083385 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:51.083309    4645 patch_prober.go:29] interesting pod/e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 31 19:40:51.083551 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:51.083410    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg" podUID="9bacc59a-7abc-43b3-81dd-a8829710ad35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 31 19:40:51.147868 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:51.147839    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm"]
Oct 31 19:40:51.148572 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:51.148557    4645 cpu_manager.go:414] "containerMap: removing container" podUID="49ea0b50-2dfa-435c-9940-98e57cbb6aa4" containerName="payload-processing"
Oct 31 19:40:51.148623 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:51.148574    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49ea0b50-2dfa-435c-9940-98e57cbb6aa4" containerName="payload-processing"
Oct 31 19:40:51.148623 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:51.148601    4645 cpu_manager.go:414] "containerMap: removing container" podUID="ef5a2c9c-2e89-4945-a70d-8eea47d0b517" containerName="payload-pre-processing"
Oct 31 19:40:51.148623 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:51.148606    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef5a2c9c-2e89-4945-a70d-8eea47d0b517" containerName="payload-pre-processing"
Oct 31 19:40:51.148718 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:51.148626    4645 cpu_manager.go:414] "containerMap: removing container" podUID="b9153d96-5237-47a0-8bf4-1e77f69b071a" containerName="main"
Oct 31 19:40:51.148718 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:51.148631    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9153d96-5237-47a0-8bf4-1e77f69b071a" containerName="main"
Oct 31 19:40:51.148774 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:51.148729    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef5a2c9c-2e89-4945-a70d-8eea47d0b517" containerName="payload-pre-processing"
Oct 31 19:40:51.148774 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:51.148739    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9153d96-5237-47a0-8bf4-1e77f69b071a" containerName="main"
Oct 31 19:40:51.148774 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:51.148750    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="49ea0b50-2dfa-435c-9940-98e57cbb6aa4" containerName="payload-processing"
Oct 31 19:40:51.152592 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:51.152572    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm"
Oct 31 19:40:51.186356 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:51.184454    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-2d319a-openshift-default-dockercfg-47fkp\""
Oct 31 19:40:51.192740 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:40:51.192719    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod813ae008_e079_4406_83f7_6eae7acc3ce1.slice/crio-202d09c571de6ab9e4f9e33edbaddd0870663244742cc67b4c0d80cce9a934c5 WatchSource:0}: Error finding container 202d09c571de6ab9e4f9e33edbaddd0870663244742cc67b4c0d80cce9a934c5: Status 404 returned error can't find the container with id 202d09c571de6ab9e4f9e33edbaddd0870663244742cc67b4c0d80cce9a934c5
Oct 31 19:40:51.195778 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:51.195759    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm"]
Oct 31 19:40:51.197442 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:51.197400    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:40:51.197575 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:51.197478    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:40:51.197575 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:51.197523    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:40:51.202557 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:51.202523    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:51.206236 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:51.206212    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2"]
Oct 31 19:40:51.237203 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:51.237178    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/773450c2-8daa-4448-a78a-8ede38d60d69-workload-socket\") pod \"e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm\" (UID: \"773450c2-8daa-4448-a78a-8ede38d60d69\") " pod="openshift-ingress/e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm"
Oct 31 19:40:51.237364 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:51.237320    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/773450c2-8daa-4448-a78a-8ede38d60d69-istio-podinfo\") pod \"e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm\" (UID: \"773450c2-8daa-4448-a78a-8ede38d60d69\") " pod="openshift-ingress/e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm"
Oct 31 19:40:51.237438 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:51.237387    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/773450c2-8daa-4448-a78a-8ede38d60d69-workload-certs\") pod \"e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm\" (UID: \"773450c2-8daa-4448-a78a-8ede38d60d69\") " pod="openshift-ingress/e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm"
Oct 31 19:40:51.237438 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:51.237406    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/773450c2-8daa-4448-a78a-8ede38d60d69-istio-envoy\") pod \"e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm\" (UID: \"773450c2-8daa-4448-a78a-8ede38d60d69\") " pod="openshift-ingress/e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm"
Oct 31 19:40:51.237539 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:51.237464    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/773450c2-8daa-4448-a78a-8ede38d60d69-istiod-ca-cert\") pod \"e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm\" (UID: \"773450c2-8daa-4448-a78a-8ede38d60d69\") " pod="openshift-ingress/e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm"
Oct 31 19:40:51.237632 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:51.237612    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/773450c2-8daa-4448-a78a-8ede38d60d69-istio-token\") pod \"e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm\" (UID: \"773450c2-8daa-4448-a78a-8ede38d60d69\") " pod="openshift-ingress/e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm"
Oct 31 19:40:51.237790 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:51.237773    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/773450c2-8daa-4448-a78a-8ede38d60d69-istio-data\") pod \"e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm\" (UID: \"773450c2-8daa-4448-a78a-8ede38d60d69\") " pod="openshift-ingress/e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm"
Oct 31 19:40:51.237857 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:51.237840    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bgh8\" (UniqueName: \"kubernetes.io/projected/773450c2-8daa-4448-a78a-8ede38d60d69-kube-api-access-8bgh8\") pod \"e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm\" (UID: \"773450c2-8daa-4448-a78a-8ede38d60d69\") " pod="openshift-ingress/e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm"
Oct 31 19:40:51.237918 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:51.237897    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/773450c2-8daa-4448-a78a-8ede38d60d69-credential-socket\") pod \"e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm\" (UID: \"773450c2-8daa-4448-a78a-8ede38d60d69\") " pod="openshift-ingress/e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm"
Oct 31 19:40:51.335962 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:51.335934    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:51.339610 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:51.339585    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/773450c2-8daa-4448-a78a-8ede38d60d69-istio-podinfo\") pod \"e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm\" (UID: \"773450c2-8daa-4448-a78a-8ede38d60d69\") " pod="openshift-ingress/e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm"
Oct 31 19:40:51.339721 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:51.339632    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/773450c2-8daa-4448-a78a-8ede38d60d69-workload-certs\") pod \"e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm\" (UID: \"773450c2-8daa-4448-a78a-8ede38d60d69\") " pod="openshift-ingress/e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm"
Oct 31 19:40:51.339721 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:51.339655    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/773450c2-8daa-4448-a78a-8ede38d60d69-istio-envoy\") pod \"e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm\" (UID: \"773450c2-8daa-4448-a78a-8ede38d60d69\") " pod="openshift-ingress/e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm"
Oct 31 19:40:51.339807 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:51.339720    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/773450c2-8daa-4448-a78a-8ede38d60d69-istiod-ca-cert\") pod \"e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm\" (UID: \"773450c2-8daa-4448-a78a-8ede38d60d69\") " pod="openshift-ingress/e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm"
Oct 31 19:40:51.339807 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:51.339794    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/773450c2-8daa-4448-a78a-8ede38d60d69-istio-token\") pod \"e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm\" (UID: \"773450c2-8daa-4448-a78a-8ede38d60d69\") " pod="openshift-ingress/e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm"
Oct 31 19:40:51.340069 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:51.340049    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/773450c2-8daa-4448-a78a-8ede38d60d69-workload-certs\") pod \"e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm\" (UID: \"773450c2-8daa-4448-a78a-8ede38d60d69\") " pod="openshift-ingress/e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm"
Oct 31 19:40:51.341220 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:51.340247    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/773450c2-8daa-4448-a78a-8ede38d60d69-istio-data\") pod \"e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm\" (UID: \"773450c2-8daa-4448-a78a-8ede38d60d69\") " pod="openshift-ingress/e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm"
Oct 31 19:40:51.341220 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:51.340360    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bgh8\" (UniqueName: \"kubernetes.io/projected/773450c2-8daa-4448-a78a-8ede38d60d69-kube-api-access-8bgh8\") pod \"e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm\" (UID: \"773450c2-8daa-4448-a78a-8ede38d60d69\") " pod="openshift-ingress/e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm"
Oct 31 19:40:51.341220 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:51.340415    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/773450c2-8daa-4448-a78a-8ede38d60d69-credential-socket\") pod \"e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm\" (UID: \"773450c2-8daa-4448-a78a-8ede38d60d69\") " pod="openshift-ingress/e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm"
Oct 31 19:40:51.341220 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:51.340472    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/773450c2-8daa-4448-a78a-8ede38d60d69-workload-socket\") pod \"e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm\" (UID: \"773450c2-8daa-4448-a78a-8ede38d60d69\") " pod="openshift-ingress/e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm"
Oct 31 19:40:51.341220 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:51.340647    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/773450c2-8daa-4448-a78a-8ede38d60d69-istio-data\") pod \"e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm\" (UID: \"773450c2-8daa-4448-a78a-8ede38d60d69\") " pod="openshift-ingress/e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm"
Oct 31 19:40:51.341220 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:51.340860    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/773450c2-8daa-4448-a78a-8ede38d60d69-workload-socket\") pod \"e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm\" (UID: \"773450c2-8daa-4448-a78a-8ede38d60d69\") " pod="openshift-ingress/e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm"
Oct 31 19:40:51.341220 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:51.341111    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/773450c2-8daa-4448-a78a-8ede38d60d69-credential-socket\") pod \"e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm\" (UID: \"773450c2-8daa-4448-a78a-8ede38d60d69\") " pod="openshift-ingress/e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm"
Oct 31 19:40:51.342365 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:51.342345    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/773450c2-8daa-4448-a78a-8ede38d60d69-istiod-ca-cert\") pod \"e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm\" (UID: \"773450c2-8daa-4448-a78a-8ede38d60d69\") " pod="openshift-ingress/e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm"
Oct 31 19:40:51.342936 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:51.342913    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/773450c2-8daa-4448-a78a-8ede38d60d69-istio-envoy\") pod \"e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm\" (UID: \"773450c2-8daa-4448-a78a-8ede38d60d69\") " pod="openshift-ingress/e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm"
Oct 31 19:40:51.346021 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:51.345998    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/773450c2-8daa-4448-a78a-8ede38d60d69-istio-podinfo\") pod \"e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm\" (UID: \"773450c2-8daa-4448-a78a-8ede38d60d69\") " pod="openshift-ingress/e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm"
Oct 31 19:40:51.393345 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:51.393252    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/773450c2-8daa-4448-a78a-8ede38d60d69-istio-token\") pod \"e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm\" (UID: \"773450c2-8daa-4448-a78a-8ede38d60d69\") " pod="openshift-ingress/e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm"
Oct 31 19:40:51.408955 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:51.407424    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bgh8\" (UniqueName: \"kubernetes.io/projected/773450c2-8daa-4448-a78a-8ede38d60d69-kube-api-access-8bgh8\") pod \"e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm\" (UID: \"773450c2-8daa-4448-a78a-8ede38d60d69\") " pod="openshift-ingress/e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm"
Oct 31 19:40:51.506786 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:51.506743    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm"
Oct 31 19:40:51.654383 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:51.654312    4645 generic.go:356] "Generic (PLEG): container finished" podID="ef5a2c9c-2e89-4945-a70d-8eea47d0b517" containerID="90ec39e391aaa91fe1831b72bfa7f80b0edf99e521d255b8a67bd6c94f1d9be6" exitCode=0
Oct 31 19:40:51.655209 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:51.655173    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d4b21c-6bf64967d7-rj87s"
Oct 31 19:40:51.756401 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:51.755784    4645 generic.go:356] "Generic (PLEG): container finished" podID="49ea0b50-2dfa-435c-9940-98e57cbb6aa4" containerID="2b40fe03181750a4a1389e65a8e88682a3e083c5d4955a733b603f9e2d954778" exitCode=0
Oct 31 19:40:51.757812 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:51.757788    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d4b21c-764698f6f-dt2q6"
Oct 31 19:40:52.046432 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:52.046397    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9153d96-5237-47a0-8bf4-1e77f69b071a" path="/var/lib/kubelet/pods/b9153d96-5237-47a0-8bf4-1e77f69b071a/volumes"
Oct 31 19:40:52.053636 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:52.053414    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d4b21c-6bf64967d7-rj87s" event={"ID":"ef5a2c9c-2e89-4945-a70d-8eea47d0b517","Type":"ContainerDied","Data":"90ec39e391aaa91fe1831b72bfa7f80b0edf99e521d255b8a67bd6c94f1d9be6"}
Oct 31 19:40:52.053636 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:52.053517    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2"
Oct 31 19:40:52.053636 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:52.053583    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d4b21c-6bf64967d7-rj87s" event={"ID":"ef5a2c9c-2e89-4945-a70d-8eea47d0b517","Type":"ContainerDied","Data":"42265bdca8cce7c2b337598ae8ffb6a056d12b343a7bffc366ce1c2dcd02e528"}
Oct 31 19:40:52.053636 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:52.053602    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2" event={"ID":"813ae008-e079-4406-83f7-6eae7acc3ce1","Type":"ContainerStarted","Data":"3c0d8d30d39213108ab50f74714ca089e81860c63ad687013e351e6d86717a2c"}
Oct 31 19:40:52.053636 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:52.053628    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2" event={"ID":"813ae008-e079-4406-83f7-6eae7acc3ce1","Type":"ContainerStarted","Data":"202d09c571de6ab9e4f9e33edbaddd0870663244742cc67b4c0d80cce9a934c5"}
Oct 31 19:40:52.067495 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:52.053656    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d4b21c-764698f6f-dt2q6" event={"ID":"49ea0b50-2dfa-435c-9940-98e57cbb6aa4","Type":"ContainerDied","Data":"2b40fe03181750a4a1389e65a8e88682a3e083c5d4955a733b603f9e2d954778"}
Oct 31 19:40:52.067495 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:52.053687    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d4b21c-764698f6f-dt2q6" event={"ID":"49ea0b50-2dfa-435c-9940-98e57cbb6aa4","Type":"ContainerDied","Data":"3b487d82dfcdd62b92838b747716f5242c1ef1d98f4ef2275b3d672865079daa"}
Oct 31 19:40:52.067495 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:52.053735    4645 scope.go:122] "RemoveContainer" containerID="90ec39e391aaa91fe1831b72bfa7f80b0edf99e521d255b8a67bd6c94f1d9be6"
Oct 31 19:40:52.097458 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:52.094232    4645 patch_prober.go:29] interesting pod/e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 31 19:40:52.097458 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:52.094283    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg" podUID="9bacc59a-7abc-43b3-81dd-a8829710ad35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 31 19:40:52.099312 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:52.099265    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2" podStartSLOduration=3.099248769 podStartE2EDuration="3.099248769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:40:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:40:51.834291466 +0000 UTC m=+1282.407525594" watchObservedRunningTime="2026-10-31 19:40:52.099248769 +0000 UTC m=+1282.672482897"
Oct 31 19:40:52.108154 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:52.105762    4645 scope.go:122] "RemoveContainer" containerID="90ec39e391aaa91fe1831b72bfa7f80b0edf99e521d255b8a67bd6c94f1d9be6"
Oct 31 19:40:52.108154 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:52.105866    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d4b21c-6bf64967d7-rj87s"]
Oct 31 19:40:52.108154 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:52.107810    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d4b21c-6bf64967d7-rj87s"]
Oct 31 19:40:52.112887 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:40:52.112853    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90ec39e391aaa91fe1831b72bfa7f80b0edf99e521d255b8a67bd6c94f1d9be6\": container with ID starting with 90ec39e391aaa91fe1831b72bfa7f80b0edf99e521d255b8a67bd6c94f1d9be6 not found: ID does not exist" containerID="90ec39e391aaa91fe1831b72bfa7f80b0edf99e521d255b8a67bd6c94f1d9be6"
Oct 31 19:40:52.113002 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:52.112898    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90ec39e391aaa91fe1831b72bfa7f80b0edf99e521d255b8a67bd6c94f1d9be6"} err="failed to get container status \"90ec39e391aaa91fe1831b72bfa7f80b0edf99e521d255b8a67bd6c94f1d9be6\": rpc error: code = NotFound desc = could not find container \"90ec39e391aaa91fe1831b72bfa7f80b0edf99e521d255b8a67bd6c94f1d9be6\": container with ID starting with 90ec39e391aaa91fe1831b72bfa7f80b0edf99e521d255b8a67bd6c94f1d9be6 not found: ID does not exist"
Oct 31 19:40:52.113002 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:52.112922    4645 scope.go:122] "RemoveContainer" containerID="2b40fe03181750a4a1389e65a8e88682a3e083c5d4955a733b603f9e2d954778"
Oct 31 19:40:52.118780 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:52.118747    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d4b21c-764698f6f-dt2q6"]
Oct 31 19:40:52.123915 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:52.123863    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d4b21c-764698f6f-dt2q6"]
Oct 31 19:40:52.133730 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:52.133711    4645 scope.go:122] "RemoveContainer" containerID="2b40fe03181750a4a1389e65a8e88682a3e083c5d4955a733b603f9e2d954778"
Oct 31 19:40:52.134106 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:40:52.134077    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b40fe03181750a4a1389e65a8e88682a3e083c5d4955a733b603f9e2d954778\": container with ID starting with 2b40fe03181750a4a1389e65a8e88682a3e083c5d4955a733b603f9e2d954778 not found: ID does not exist" containerID="2b40fe03181750a4a1389e65a8e88682a3e083c5d4955a733b603f9e2d954778"
Oct 31 19:40:52.134157 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:52.134120    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b40fe03181750a4a1389e65a8e88682a3e083c5d4955a733b603f9e2d954778"} err="failed to get container status \"2b40fe03181750a4a1389e65a8e88682a3e083c5d4955a733b603f9e2d954778\": rpc error: code = NotFound desc = could not find container \"2b40fe03181750a4a1389e65a8e88682a3e083c5d4955a733b603f9e2d954778\": container with ID starting with 2b40fe03181750a4a1389e65a8e88682a3e083c5d4955a733b603f9e2d954778 not found: ID does not exist"
Oct 31 19:40:52.147679 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:40:52.147652    4645 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef5a2c9c_2e89_4945_a70d_8eea47d0b517.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49ea0b50_2dfa_435c_9940_98e57cbb6aa4.slice\": RecentStats: unable to find data in memory cache]"
Oct 31 19:40:52.290054 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:40:52.290011    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod773450c2_8daa_4448_a78a_8ede38d60d69.slice/crio-f8eb68858bd1fa87910f43ec36038e85e96ebbc1ab4f5d24ded9790d8356c4ef WatchSource:0}: Error finding container f8eb68858bd1fa87910f43ec36038e85e96ebbc1ab4f5d24ded9790d8356c4ef: Status 404 returned error can't find the container with id f8eb68858bd1fa87910f43ec36038e85e96ebbc1ab4f5d24ded9790d8356c4ef
Oct 31 19:40:52.292423 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:52.292158    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm"]
Oct 31 19:40:52.293309 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:52.293273    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:40:52.293421 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:52.293356    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:40:52.293421 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:52.293390    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:40:52.311865 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:52.311840    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:52.791463 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:52.791422    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm" event={"ID":"773450c2-8daa-4448-a78a-8ede38d60d69","Type":"ContainerStarted","Data":"523222635d65f76e0c47c008d6856668c03854cd36a652188d27282f7d6a968d"}
Oct 31 19:40:52.791905 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:52.791486    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm" event={"ID":"773450c2-8daa-4448-a78a-8ede38d60d69","Type":"ContainerStarted","Data":"f8eb68858bd1fa87910f43ec36038e85e96ebbc1ab4f5d24ded9790d8356c4ef"}
Oct 31 19:40:52.791961 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:52.791930    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2"
Oct 31 19:40:52.794021 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:52.794000    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2"
Oct 31 19:40:52.795006 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:52.794985    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2"
Oct 31 19:40:52.818527 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:52.818481    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm" podStartSLOduration=1.8184664609999999 podStartE2EDuration="1.818466461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:40:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:40:52.81283909 +0000 UTC m=+1283.386073219" watchObservedRunningTime="2026-10-31 19:40:52.818466461 +0000 UTC m=+1283.391700590"
Oct 31 19:40:53.091482 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:53.091367    4645 patch_prober.go:29] interesting pod/e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 31 19:40:53.091482 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:53.091422    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg" podUID="9bacc59a-7abc-43b3-81dd-a8829710ad35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 31 19:40:53.528350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:53.512499    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm"
Oct 31 19:40:53.558397 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:53.557746    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm"
Oct 31 19:40:53.619814 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:53.619780    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49ea0b50-2dfa-435c-9940-98e57cbb6aa4" path="/var/lib/kubelet/pods/49ea0b50-2dfa-435c-9940-98e57cbb6aa4/volumes"
Oct 31 19:40:53.620241 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:53.620225    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef5a2c9c-2e89-4945-a70d-8eea47d0b517" path="/var/lib/kubelet/pods/ef5a2c9c-2e89-4945-a70d-8eea47d0b517/volumes"
Oct 31 19:40:53.801879 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:53.801789    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm"
Oct 31 19:40:53.802857 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:53.802836    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm"
Oct 31 19:40:53.867067 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:53.866974    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:54.083638 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:54.083550    4645 patch_prober.go:29] interesting pod/e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 31 19:40:54.083905 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:54.083874    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg" podUID="9bacc59a-7abc-43b3-81dd-a8829710ad35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 31 19:40:54.192778 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:54.192738    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:54.306671 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:54.306642    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:54.703482 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:54.703458    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf"
Oct 31 19:40:54.801235 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:54.801199    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2a68f87-af40-4d21-9ef0-620f397e13be-istio-token\" (UniqueName: \"kubernetes.io/projected/b2a68f87-af40-4d21-9ef0-620f397e13be-istio-token\") pod \"b2a68f87-af40-4d21-9ef0-620f397e13be\" (UID: \"b2a68f87-af40-4d21-9ef0-620f397e13be\") "
Oct 31 19:40:54.801422 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:54.801389    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2a68f87-af40-4d21-9ef0-620f397e13be-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2a68f87-af40-4d21-9ef0-620f397e13be-workload-certs\") pod \"b2a68f87-af40-4d21-9ef0-620f397e13be\" (UID: \"b2a68f87-af40-4d21-9ef0-620f397e13be\") "
Oct 31 19:40:54.801422 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:54.801420    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b2a68f87-af40-4d21-9ef0-620f397e13be-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2a68f87-af40-4d21-9ef0-620f397e13be-istio-podinfo\") pod \"b2a68f87-af40-4d21-9ef0-620f397e13be\" (UID: \"b2a68f87-af40-4d21-9ef0-620f397e13be\") "
Oct 31 19:40:54.801545 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:54.801444    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2a68f87-af40-4d21-9ef0-620f397e13be-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2a68f87-af40-4d21-9ef0-620f397e13be-istiod-ca-cert\") pod \"b2a68f87-af40-4d21-9ef0-620f397e13be\" (UID: \"b2a68f87-af40-4d21-9ef0-620f397e13be\") "
Oct 31 19:40:54.801545 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:54.801459    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2a68f87-af40-4d21-9ef0-620f397e13be-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2a68f87-af40-4d21-9ef0-620f397e13be-workload-socket\") pod \"b2a68f87-af40-4d21-9ef0-620f397e13be\" (UID: \"b2a68f87-af40-4d21-9ef0-620f397e13be\") "
Oct 31 19:40:54.801545 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:54.801500    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2a68f87-af40-4d21-9ef0-620f397e13be-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2a68f87-af40-4d21-9ef0-620f397e13be-credential-socket\") pod \"b2a68f87-af40-4d21-9ef0-620f397e13be\" (UID: \"b2a68f87-af40-4d21-9ef0-620f397e13be\") "
Oct 31 19:40:54.801718 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:54.801581    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2a68f87-af40-4d21-9ef0-620f397e13be-kube-api-access-6m2zf\" (UniqueName: \"kubernetes.io/projected/b2a68f87-af40-4d21-9ef0-620f397e13be-kube-api-access-6m2zf\") pod \"b2a68f87-af40-4d21-9ef0-620f397e13be\" (UID: \"b2a68f87-af40-4d21-9ef0-620f397e13be\") "
Oct 31 19:40:54.801718 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:54.801686    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2a68f87-af40-4d21-9ef0-620f397e13be-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2a68f87-af40-4d21-9ef0-620f397e13be-istio-envoy\") pod \"b2a68f87-af40-4d21-9ef0-620f397e13be\" (UID: \"b2a68f87-af40-4d21-9ef0-620f397e13be\") "
Oct 31 19:40:54.801833 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:54.801810    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2a68f87-af40-4d21-9ef0-620f397e13be-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2a68f87-af40-4d21-9ef0-620f397e13be-istio-data\") pod \"b2a68f87-af40-4d21-9ef0-620f397e13be\" (UID: \"b2a68f87-af40-4d21-9ef0-620f397e13be\") "
Oct 31 19:40:54.801971 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:54.801742    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2a68f87-af40-4d21-9ef0-620f397e13be-workload-certs" pod "b2a68f87-af40-4d21-9ef0-620f397e13be" (UID: "b2a68f87-af40-4d21-9ef0-620f397e13be"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:54.802432 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:54.802092    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2a68f87-af40-4d21-9ef0-620f397e13be-credential-socket" pod "b2a68f87-af40-4d21-9ef0-620f397e13be" (UID: "b2a68f87-af40-4d21-9ef0-620f397e13be"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:54.802432 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:54.802273    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2a68f87-af40-4d21-9ef0-620f397e13be-istio-data" pod "b2a68f87-af40-4d21-9ef0-620f397e13be" (UID: "b2a68f87-af40-4d21-9ef0-620f397e13be"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:54.803310 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:54.803231    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2a68f87-af40-4d21-9ef0-620f397e13be-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:54.803310 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:54.803263    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2a68f87-af40-4d21-9ef0-620f397e13be-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:54.803310 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:54.803278    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2a68f87-af40-4d21-9ef0-620f397e13be-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:54.803896 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:54.803873    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2a68f87-af40-4d21-9ef0-620f397e13be-workload-socket" pod "b2a68f87-af40-4d21-9ef0-620f397e13be" (UID: "b2a68f87-af40-4d21-9ef0-620f397e13be"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:54.805031 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:54.805005    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2a68f87-af40-4d21-9ef0-620f397e13be-istiod-ca-cert" pod "b2a68f87-af40-4d21-9ef0-620f397e13be" (UID: "b2a68f87-af40-4d21-9ef0-620f397e13be"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:40:54.805198 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:54.805177    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2a68f87-af40-4d21-9ef0-620f397e13be-istio-token" pod "b2a68f87-af40-4d21-9ef0-620f397e13be" (UID: "b2a68f87-af40-4d21-9ef0-620f397e13be"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:40:54.805705 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:54.805562    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2a68f87-af40-4d21-9ef0-620f397e13be-istio-envoy" pod "b2a68f87-af40-4d21-9ef0-620f397e13be" (UID: "b2a68f87-af40-4d21-9ef0-620f397e13be"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:54.805705 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:54.805676    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2a68f87-af40-4d21-9ef0-620f397e13be-kube-api-access-6m2zf" pod "b2a68f87-af40-4d21-9ef0-620f397e13be" (UID: "b2a68f87-af40-4d21-9ef0-620f397e13be"). InnerVolumeSpecName "kube-api-access-6m2zf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:40:54.806141 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:54.806113    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b2a68f87-af40-4d21-9ef0-620f397e13be-istio-podinfo" pod "b2a68f87-af40-4d21-9ef0-620f397e13be" (UID: "b2a68f87-af40-4d21-9ef0-620f397e13be"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:40:54.811698 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:54.811670    4645 generic.go:356] "Generic (PLEG): container finished" podID="b2a68f87-af40-4d21-9ef0-620f397e13be" containerID="aaa54faf936afb8ee998be89fa312e75432531e53d0905dcca6e7f2fe700885c" exitCode=0
Oct 31 19:40:54.811795 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:54.811725    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf"
Oct 31 19:40:54.811795 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:54.811751    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf" event={"ID":"b2a68f87-af40-4d21-9ef0-620f397e13be","Type":"ContainerDied","Data":"aaa54faf936afb8ee998be89fa312e75432531e53d0905dcca6e7f2fe700885c"}
Oct 31 19:40:54.811889 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:54.811860    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf" event={"ID":"b2a68f87-af40-4d21-9ef0-620f397e13be","Type":"ContainerDied","Data":"257c7275cf3b4f16f7ece3ef65a7350b60731681b35ff06cb32a443a520a5b5c"}
Oct 31 19:40:54.811936 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:54.811890    4645 scope.go:122] "RemoveContainer" containerID="aaa54faf936afb8ee998be89fa312e75432531e53d0905dcca6e7f2fe700885c"
Oct 31 19:40:54.814182 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:54.814156    4645 generic.go:356] "Generic (PLEG): container finished" podID="619ab3b5-8258-4aea-90fe-a351097d91cd" containerID="91eaff5584a21d4367366fb3491e9729eeaad0a898f0406a181141ad0977f780" exitCode=0
Oct 31 19:40:54.814253 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:54.814191    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl" event={"ID":"619ab3b5-8258-4aea-90fe-a351097d91cd","Type":"ContainerDied","Data":"91eaff5584a21d4367366fb3491e9729eeaad0a898f0406a181141ad0977f780"}
Oct 31 19:40:54.816528 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:54.816494    4645 generic.go:356] "Generic (PLEG): container finished" podID="0f482c94-43ba-4570-9b2a-ef4886627127" containerID="185cdf71b9e21f8f49d2323c2f6de1ed0c6a9f1d6544be04ca5065dd4c992af1" exitCode=0
Oct 31 19:40:54.816680 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:54.816659    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm" event={"ID":"0f482c94-43ba-4570-9b2a-ef4886627127","Type":"ContainerDied","Data":"185cdf71b9e21f8f49d2323c2f6de1ed0c6a9f1d6544be04ca5065dd4c992af1"}
Oct 31 19:40:54.826105 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:54.826086    4645 scope.go:122] "RemoveContainer" containerID="aaa54faf936afb8ee998be89fa312e75432531e53d0905dcca6e7f2fe700885c"
Oct 31 19:40:54.826378 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:40:54.826323    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aaa54faf936afb8ee998be89fa312e75432531e53d0905dcca6e7f2fe700885c\": container with ID starting with aaa54faf936afb8ee998be89fa312e75432531e53d0905dcca6e7f2fe700885c not found: ID does not exist" containerID="aaa54faf936afb8ee998be89fa312e75432531e53d0905dcca6e7f2fe700885c"
Oct 31 19:40:54.826425 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:54.826386    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaa54faf936afb8ee998be89fa312e75432531e53d0905dcca6e7f2fe700885c"} err="failed to get container status \"aaa54faf936afb8ee998be89fa312e75432531e53d0905dcca6e7f2fe700885c\": rpc error: code = NotFound desc = could not find container \"aaa54faf936afb8ee998be89fa312e75432531e53d0905dcca6e7f2fe700885c\": container with ID starting with aaa54faf936afb8ee998be89fa312e75432531e53d0905dcca6e7f2fe700885c not found: ID does not exist"
Oct 31 19:40:54.838508 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:54.838484    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf"]
Oct 31 19:40:54.845904 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:54.845885    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-5n8bf"]
Oct 31 19:40:54.904570 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:54.904538    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2a68f87-af40-4d21-9ef0-620f397e13be-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:54.904570 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:54.904566    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2a68f87-af40-4d21-9ef0-620f397e13be-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:54.904716 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:54.904578    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2a68f87-af40-4d21-9ef0-620f397e13be-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:54.904716 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:54.904587    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6m2zf\" (UniqueName: \"kubernetes.io/projected/b2a68f87-af40-4d21-9ef0-620f397e13be-kube-api-access-6m2zf\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:54.904716 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:54.904597    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2a68f87-af40-4d21-9ef0-620f397e13be-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:54.904716 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:54.904606    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2a68f87-af40-4d21-9ef0-620f397e13be-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:55.088368 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.088310    4645 patch_prober.go:29] interesting pod/e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body=
Oct 31 19:40:55.088524 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.088388    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg" podUID="9bacc59a-7abc-43b3-81dd-a8829710ad35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused"
Oct 31 19:40:55.117187 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.116881    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm"
Oct 31 19:40:55.143612 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.143589    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl"
Oct 31 19:40:55.210609 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.210579    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f482c94-43ba-4570-9b2a-ef4886627127-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f482c94-43ba-4570-9b2a-ef4886627127-istio-envoy\") pod \"0f482c94-43ba-4570-9b2a-ef4886627127\" (UID: \"0f482c94-43ba-4570-9b2a-ef4886627127\") "
Oct 31 19:40:55.210781 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.210617    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/619ab3b5-8258-4aea-90fe-a351097d91cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/619ab3b5-8258-4aea-90fe-a351097d91cd-workload-socket\") pod \"619ab3b5-8258-4aea-90fe-a351097d91cd\" (UID: \"619ab3b5-8258-4aea-90fe-a351097d91cd\") "
Oct 31 19:40:55.210781 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.210652    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/619ab3b5-8258-4aea-90fe-a351097d91cd-kube-api-access-4x7tg\" (UniqueName: \"kubernetes.io/projected/619ab3b5-8258-4aea-90fe-a351097d91cd-kube-api-access-4x7tg\") pod \"619ab3b5-8258-4aea-90fe-a351097d91cd\" (UID: \"619ab3b5-8258-4aea-90fe-a351097d91cd\") "
Oct 31 19:40:55.210781 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.210682    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0f482c94-43ba-4570-9b2a-ef4886627127-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f482c94-43ba-4570-9b2a-ef4886627127-istio-podinfo\") pod \"0f482c94-43ba-4570-9b2a-ef4886627127\" (UID: \"0f482c94-43ba-4570-9b2a-ef4886627127\") "
Oct 31 19:40:55.210781 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.210731    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/619ab3b5-8258-4aea-90fe-a351097d91cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/619ab3b5-8258-4aea-90fe-a351097d91cd-istio-envoy\") pod \"619ab3b5-8258-4aea-90fe-a351097d91cd\" (UID: \"619ab3b5-8258-4aea-90fe-a351097d91cd\") "
Oct 31 19:40:55.210999 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.210862    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f482c94-43ba-4570-9b2a-ef4886627127-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f482c94-43ba-4570-9b2a-ef4886627127-workload-certs\") pod \"0f482c94-43ba-4570-9b2a-ef4886627127\" (UID: \"0f482c94-43ba-4570-9b2a-ef4886627127\") "
Oct 31 19:40:55.210999 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.210906    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/619ab3b5-8258-4aea-90fe-a351097d91cd-workload-socket" pod "619ab3b5-8258-4aea-90fe-a351097d91cd" (UID: "619ab3b5-8258-4aea-90fe-a351097d91cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:55.210999 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.210935    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/619ab3b5-8258-4aea-90fe-a351097d91cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/619ab3b5-8258-4aea-90fe-a351097d91cd-workload-certs\") pod \"619ab3b5-8258-4aea-90fe-a351097d91cd\" (UID: \"619ab3b5-8258-4aea-90fe-a351097d91cd\") "
Oct 31 19:40:55.211143 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.211049    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/619ab3b5-8258-4aea-90fe-a351097d91cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/619ab3b5-8258-4aea-90fe-a351097d91cd-istio-podinfo\") pod \"619ab3b5-8258-4aea-90fe-a351097d91cd\" (UID: \"619ab3b5-8258-4aea-90fe-a351097d91cd\") "
Oct 31 19:40:55.211143 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.211090    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f482c94-43ba-4570-9b2a-ef4886627127-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f482c94-43ba-4570-9b2a-ef4886627127-credential-socket\") pod \"0f482c94-43ba-4570-9b2a-ef4886627127\" (UID: \"0f482c94-43ba-4570-9b2a-ef4886627127\") "
Oct 31 19:40:55.211143 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.211121    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f482c94-43ba-4570-9b2a-ef4886627127-workload-certs" pod "0f482c94-43ba-4570-9b2a-ef4886627127" (UID: "0f482c94-43ba-4570-9b2a-ef4886627127"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:55.212158 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.211321    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/619ab3b5-8258-4aea-90fe-a351097d91cd-workload-certs" pod "619ab3b5-8258-4aea-90fe-a351097d91cd" (UID: "619ab3b5-8258-4aea-90fe-a351097d91cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:55.212158 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.211343    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f482c94-43ba-4570-9b2a-ef4886627127-credential-socket" pod "0f482c94-43ba-4570-9b2a-ef4886627127" (UID: "0f482c94-43ba-4570-9b2a-ef4886627127"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:55.212158 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.211371    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/619ab3b5-8258-4aea-90fe-a351097d91cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/619ab3b5-8258-4aea-90fe-a351097d91cd-istiod-ca-cert\") pod \"619ab3b5-8258-4aea-90fe-a351097d91cd\" (UID: \"619ab3b5-8258-4aea-90fe-a351097d91cd\") "
Oct 31 19:40:55.212158 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.211452    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/619ab3b5-8258-4aea-90fe-a351097d91cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/619ab3b5-8258-4aea-90fe-a351097d91cd-istio-data\") pod \"619ab3b5-8258-4aea-90fe-a351097d91cd\" (UID: \"619ab3b5-8258-4aea-90fe-a351097d91cd\") "
Oct 31 19:40:55.212158 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.211516    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f482c94-43ba-4570-9b2a-ef4886627127-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f482c94-43ba-4570-9b2a-ef4886627127-istio-data\") pod \"0f482c94-43ba-4570-9b2a-ef4886627127\" (UID: \"0f482c94-43ba-4570-9b2a-ef4886627127\") "
Oct 31 19:40:55.212158 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.211547    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/619ab3b5-8258-4aea-90fe-a351097d91cd-istio-token\" (UniqueName: \"kubernetes.io/projected/619ab3b5-8258-4aea-90fe-a351097d91cd-istio-token\") pod \"619ab3b5-8258-4aea-90fe-a351097d91cd\" (UID: \"619ab3b5-8258-4aea-90fe-a351097d91cd\") "
Oct 31 19:40:55.212158 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.211610    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f482c94-43ba-4570-9b2a-ef4886627127-kube-api-access-62tnz\" (UniqueName: \"kubernetes.io/projected/0f482c94-43ba-4570-9b2a-ef4886627127-kube-api-access-62tnz\") pod \"0f482c94-43ba-4570-9b2a-ef4886627127\" (UID: \"0f482c94-43ba-4570-9b2a-ef4886627127\") "
Oct 31 19:40:55.212158 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.211664    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f482c94-43ba-4570-9b2a-ef4886627127-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f482c94-43ba-4570-9b2a-ef4886627127-istiod-ca-cert\") pod \"0f482c94-43ba-4570-9b2a-ef4886627127\" (UID: \"0f482c94-43ba-4570-9b2a-ef4886627127\") "
Oct 31 19:40:55.212158 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.211699    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/619ab3b5-8258-4aea-90fe-a351097d91cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/619ab3b5-8258-4aea-90fe-a351097d91cd-credential-socket\") pod \"619ab3b5-8258-4aea-90fe-a351097d91cd\" (UID: \"619ab3b5-8258-4aea-90fe-a351097d91cd\") "
Oct 31 19:40:55.212158 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.211721    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f482c94-43ba-4570-9b2a-ef4886627127-istio-token\" (UniqueName: \"kubernetes.io/projected/0f482c94-43ba-4570-9b2a-ef4886627127-istio-token\") pod \"0f482c94-43ba-4570-9b2a-ef4886627127\" (UID: \"0f482c94-43ba-4570-9b2a-ef4886627127\") "
Oct 31 19:40:55.212158 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.211750    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f482c94-43ba-4570-9b2a-ef4886627127-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f482c94-43ba-4570-9b2a-ef4886627127-workload-socket\") pod \"0f482c94-43ba-4570-9b2a-ef4886627127\" (UID: \"0f482c94-43ba-4570-9b2a-ef4886627127\") "
Oct 31 19:40:55.212158 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.211783    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/619ab3b5-8258-4aea-90fe-a351097d91cd-istiod-ca-cert" pod "619ab3b5-8258-4aea-90fe-a351097d91cd" (UID: "619ab3b5-8258-4aea-90fe-a351097d91cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:40:55.212158 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.211795    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/619ab3b5-8258-4aea-90fe-a351097d91cd-istio-data" pod "619ab3b5-8258-4aea-90fe-a351097d91cd" (UID: "619ab3b5-8258-4aea-90fe-a351097d91cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:55.212158 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.212008    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/619ab3b5-8258-4aea-90fe-a351097d91cd-credential-socket" pod "619ab3b5-8258-4aea-90fe-a351097d91cd" (UID: "619ab3b5-8258-4aea-90fe-a351097d91cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:55.212823 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.212419    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f482c94-43ba-4570-9b2a-ef4886627127-workload-socket" pod "0f482c94-43ba-4570-9b2a-ef4886627127" (UID: "0f482c94-43ba-4570-9b2a-ef4886627127"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:55.212823 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.212740    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f482c94-43ba-4570-9b2a-ef4886627127-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:55.212823 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.212763    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/619ab3b5-8258-4aea-90fe-a351097d91cd-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:55.212823 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.212779    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/619ab3b5-8258-4aea-90fe-a351097d91cd-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:55.212823 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.212794    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/619ab3b5-8258-4aea-90fe-a351097d91cd-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:55.212823 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.212809    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f482c94-43ba-4570-9b2a-ef4886627127-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:55.212823 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.212823    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/619ab3b5-8258-4aea-90fe-a351097d91cd-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:55.213080 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.212836    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f482c94-43ba-4570-9b2a-ef4886627127-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:55.213080 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.212847    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/619ab3b5-8258-4aea-90fe-a351097d91cd-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:55.213209 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.213186    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f482c94-43ba-4570-9b2a-ef4886627127-istiod-ca-cert" pod "0f482c94-43ba-4570-9b2a-ef4886627127" (UID: "0f482c94-43ba-4570-9b2a-ef4886627127"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:40:55.213529 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.213505    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f482c94-43ba-4570-9b2a-ef4886627127-istio-data" pod "0f482c94-43ba-4570-9b2a-ef4886627127" (UID: "0f482c94-43ba-4570-9b2a-ef4886627127"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:55.219528 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.219495    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/619ab3b5-8258-4aea-90fe-a351097d91cd-istio-podinfo" pod "619ab3b5-8258-4aea-90fe-a351097d91cd" (UID: "619ab3b5-8258-4aea-90fe-a351097d91cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:40:55.219528 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.219504    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f482c94-43ba-4570-9b2a-ef4886627127-kube-api-access-62tnz" pod "0f482c94-43ba-4570-9b2a-ef4886627127" (UID: "0f482c94-43ba-4570-9b2a-ef4886627127"). InnerVolumeSpecName "kube-api-access-62tnz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:40:55.219528 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.219520    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/619ab3b5-8258-4aea-90fe-a351097d91cd-istio-token" pod "619ab3b5-8258-4aea-90fe-a351097d91cd" (UID: "619ab3b5-8258-4aea-90fe-a351097d91cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:40:55.219716 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.219567    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/619ab3b5-8258-4aea-90fe-a351097d91cd-kube-api-access-4x7tg" pod "619ab3b5-8258-4aea-90fe-a351097d91cd" (UID: "619ab3b5-8258-4aea-90fe-a351097d91cd"). InnerVolumeSpecName "kube-api-access-4x7tg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:40:55.219716 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.219617    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0f482c94-43ba-4570-9b2a-ef4886627127-istio-podinfo" pod "0f482c94-43ba-4570-9b2a-ef4886627127" (UID: "0f482c94-43ba-4570-9b2a-ef4886627127"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:40:55.219783 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.219716    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f482c94-43ba-4570-9b2a-ef4886627127-istio-token" pod "0f482c94-43ba-4570-9b2a-ef4886627127" (UID: "0f482c94-43ba-4570-9b2a-ef4886627127"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:40:55.219928 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.219897    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f482c94-43ba-4570-9b2a-ef4886627127-istio-envoy" pod "0f482c94-43ba-4570-9b2a-ef4886627127" (UID: "0f482c94-43ba-4570-9b2a-ef4886627127"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:55.220006 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.219978    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/619ab3b5-8258-4aea-90fe-a351097d91cd-istio-envoy" pod "619ab3b5-8258-4aea-90fe-a351097d91cd" (UID: "619ab3b5-8258-4aea-90fe-a351097d91cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:40:55.220611 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.220586    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:55.325800 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.322938    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/619ab3b5-8258-4aea-90fe-a351097d91cd-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:55.325800 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.322975    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62tnz\" (UniqueName: \"kubernetes.io/projected/0f482c94-43ba-4570-9b2a-ef4886627127-kube-api-access-62tnz\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:55.325800 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.322991    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f482c94-43ba-4570-9b2a-ef4886627127-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:55.325800 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.323004    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f482c94-43ba-4570-9b2a-ef4886627127-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:55.325800 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.323019    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f482c94-43ba-4570-9b2a-ef4886627127-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:55.325800 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.323032    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4x7tg\" (UniqueName: \"kubernetes.io/projected/619ab3b5-8258-4aea-90fe-a351097d91cd-kube-api-access-4x7tg\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:55.325800 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.323045    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f482c94-43ba-4570-9b2a-ef4886627127-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:55.325800 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.323058    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/619ab3b5-8258-4aea-90fe-a351097d91cd-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:55.325800 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.323073    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/619ab3b5-8258-4aea-90fe-a351097d91cd-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:55.325800 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.323087    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f482c94-43ba-4570-9b2a-ef4886627127-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:40:55.622794 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.622693    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2a68f87-af40-4d21-9ef0-620f397e13be" path="/var/lib/kubelet/pods/b2a68f87-af40-4d21-9ef0-620f397e13be/volumes"
Oct 31 19:40:55.676305 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.676276    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:55.813073 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.813037    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:55.826772 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.826734    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm"
Oct 31 19:40:55.826772 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.826758    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm" event={"ID":"0f482c94-43ba-4570-9b2a-ef4886627127","Type":"ContainerDied","Data":"f49f4fa577c3cdf3a2cc3020d2cdf095c730b59e842de80990c07268ea35067d"}
Oct 31 19:40:55.827012 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.826816    4645 scope.go:122] "RemoveContainer" containerID="185cdf71b9e21f8f49d2323c2f6de1ed0c6a9f1d6544be04ca5065dd4c992af1"
Oct 31 19:40:55.829949 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.829932    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl"
Oct 31 19:40:55.830068 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.829969    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl" event={"ID":"619ab3b5-8258-4aea-90fe-a351097d91cd","Type":"ContainerDied","Data":"7e9d64b66e013c70b54ff818553434604cb00c55cd25fe70542be78cec55cd92"}
Oct 31 19:40:55.839023 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.839000    4645 scope.go:122] "RemoveContainer" containerID="91eaff5584a21d4367366fb3491e9729eeaad0a898f0406a181141ad0977f780"
Oct 31 19:40:55.839785 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.839760    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f5c1ab-68b5bdf4d7-jwcll"]
Oct 31 19:40:55.840589 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.840573    4645 cpu_manager.go:414] "containerMap: removing container" podUID="0f482c94-43ba-4570-9b2a-ef4886627127" containerName="istio-proxy"
Oct 31 19:40:55.840642 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.840591    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f482c94-43ba-4570-9b2a-ef4886627127" containerName="istio-proxy"
Oct 31 19:40:55.840642 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.840607    4645 cpu_manager.go:414] "containerMap: removing container" podUID="619ab3b5-8258-4aea-90fe-a351097d91cd" containerName="istio-proxy"
Oct 31 19:40:55.840642 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.840612    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="619ab3b5-8258-4aea-90fe-a351097d91cd" containerName="istio-proxy"
Oct 31 19:40:55.840642 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.840641    4645 cpu_manager.go:414] "containerMap: removing container" podUID="b2a68f87-af40-4d21-9ef0-620f397e13be" containerName="istio-proxy"
Oct 31 19:40:55.840768 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.840646    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2a68f87-af40-4d21-9ef0-620f397e13be" containerName="istio-proxy"
Oct 31 19:40:55.840768 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.840748    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f482c94-43ba-4570-9b2a-ef4886627127" containerName="istio-proxy"
Oct 31 19:40:55.840768 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.840756    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="619ab3b5-8258-4aea-90fe-a351097d91cd" containerName="istio-proxy"
Oct 31 19:40:55.840768 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.840767    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2a68f87-af40-4d21-9ef0-620f397e13be" containerName="istio-proxy"
Oct 31 19:40:55.845776 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.845760    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f5c1ab-68b5bdf4d7-jwcll"
Oct 31 19:40:55.857531 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.857507    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-f5c1ab-dockercfg-fqmjq\""
Oct 31 19:40:55.865819 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.865798    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm"]
Oct 31 19:40:55.866910 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.866891    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-f5c1ab\""
Oct 31 19:40:55.885041 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.884951    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f5c1ab-68b5bdf4d7-jwcll"]
Oct 31 19:40:55.885297 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.885276    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-wj6fm"]
Oct 31 19:40:55.913736 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.913575    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl"]
Oct 31 19:40:56.001350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.976058    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-k9knl"]
Oct 31 19:40:56.001350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.999698    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d05d0a8-95d5-4f44-b7d9-bae7800b7176-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-f5c1ab-68b5bdf4d7-jwcll\" (UID: \"4d05d0a8-95d5-4f44-b7d9-bae7800b7176\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f5c1ab-68b5bdf4d7-jwcll"
Oct 31 19:40:56.001350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:55.999809    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zml6b\" (UniqueName: \"kubernetes.io/projected/4d05d0a8-95d5-4f44-b7d9-bae7800b7176-kube-api-access-zml6b\") pod \"payload-pre-processing-e2e-shared-b-w0-f5c1ab-68b5bdf4d7-jwcll\" (UID: \"4d05d0a8-95d5-4f44-b7d9-bae7800b7176\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f5c1ab-68b5bdf4d7-jwcll"
Oct 31 19:40:56.232083 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:56.185380    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-f5c1ab-54c4d89f99-b7fr8"]
Oct 31 19:40:56.249360 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:56.235130    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f5c1ab-54c4d89f99-b7fr8"
Oct 31 19:40:56.281842 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:56.281811    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:56.290604 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:56.285232    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d05d0a8-95d5-4f44-b7d9-bae7800b7176-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-f5c1ab-68b5bdf4d7-jwcll\" (UID: \"4d05d0a8-95d5-4f44-b7d9-bae7800b7176\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f5c1ab-68b5bdf4d7-jwcll"
Oct 31 19:40:56.292024 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:56.291293    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zml6b\" (UniqueName: \"kubernetes.io/projected/4d05d0a8-95d5-4f44-b7d9-bae7800b7176-kube-api-access-zml6b\") pod \"payload-pre-processing-e2e-shared-b-w0-f5c1ab-68b5bdf4d7-jwcll\" (UID: \"4d05d0a8-95d5-4f44-b7d9-bae7800b7176\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f5c1ab-68b5bdf4d7-jwcll"
Oct 31 19:40:56.292024 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:56.291000    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-f5c1ab-54c4d89f99-b7fr8"]
Oct 31 19:40:56.292024 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:56.286357    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d05d0a8-95d5-4f44-b7d9-bae7800b7176-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-f5c1ab-68b5bdf4d7-jwcll\" (UID: \"4d05d0a8-95d5-4f44-b7d9-bae7800b7176\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f5c1ab-68b5bdf4d7-jwcll"
Oct 31 19:40:56.330047 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:56.329923    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zml6b\" (UniqueName: \"kubernetes.io/projected/4d05d0a8-95d5-4f44-b7d9-bae7800b7176-kube-api-access-zml6b\") pod \"payload-pre-processing-e2e-shared-b-w0-f5c1ab-68b5bdf4d7-jwcll\" (UID: \"4d05d0a8-95d5-4f44-b7d9-bae7800b7176\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f5c1ab-68b5bdf4d7-jwcll"
Oct 31 19:40:56.392841 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:56.392812    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5005342-3674-4428-ac17-a42f0dc6d8b2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-f5c1ab-54c4d89f99-b7fr8\" (UID: \"a5005342-3674-4428-ac17-a42f0dc6d8b2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f5c1ab-54c4d89f99-b7fr8"
Oct 31 19:40:56.392989 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:56.392976    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rptqq\" (UniqueName: \"kubernetes.io/projected/a5005342-3674-4428-ac17-a42f0dc6d8b2-kube-api-access-rptqq\") pod \"payload-processing-e2e-shared-b-w0-f5c1ab-54c4d89f99-b7fr8\" (UID: \"a5005342-3674-4428-ac17-a42f0dc6d8b2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f5c1ab-54c4d89f99-b7fr8"
Oct 31 19:40:56.426856 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:56.426822    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:56.464539 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:56.464509    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f5c1ab-68b5bdf4d7-jwcll"
Oct 31 19:40:56.471268 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:56.471239    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1c0c1d-65fc757fd7-c79ts"
Oct 31 19:40:56.471961 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:56.471940    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg"
Oct 31 19:40:56.472294 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:56.472271    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg"
Oct 31 19:40:56.473392 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:56.473370    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg"
Oct 31 19:40:56.494343 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:56.494253    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5005342-3674-4428-ac17-a42f0dc6d8b2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-f5c1ab-54c4d89f99-b7fr8\" (UID: \"a5005342-3674-4428-ac17-a42f0dc6d8b2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f5c1ab-54c4d89f99-b7fr8"
Oct 31 19:40:56.494461 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:56.494419    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rptqq\" (UniqueName: \"kubernetes.io/projected/a5005342-3674-4428-ac17-a42f0dc6d8b2-kube-api-access-rptqq\") pod \"payload-processing-e2e-shared-b-w0-f5c1ab-54c4d89f99-b7fr8\" (UID: \"a5005342-3674-4428-ac17-a42f0dc6d8b2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f5c1ab-54c4d89f99-b7fr8"
Oct 31 19:40:56.496219 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:56.495924    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5005342-3674-4428-ac17-a42f0dc6d8b2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-f5c1ab-54c4d89f99-b7fr8\" (UID: \"a5005342-3674-4428-ac17-a42f0dc6d8b2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f5c1ab-54c4d89f99-b7fr8"
Oct 31 19:40:56.509027 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:56.509005    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rptqq\" (UniqueName: \"kubernetes.io/projected/a5005342-3674-4428-ac17-a42f0dc6d8b2-kube-api-access-rptqq\") pod \"payload-processing-e2e-shared-b-w0-f5c1ab-54c4d89f99-b7fr8\" (UID: \"a5005342-3674-4428-ac17-a42f0dc6d8b2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f5c1ab-54c4d89f99-b7fr8"
Oct 31 19:40:56.593462 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:56.593432    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f5c1ab-54c4d89f99-b7fr8"
Oct 31 19:40:56.765760 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:40:56.765733    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d05d0a8_95d5_4f44_b7d9_bae7800b7176.slice/crio-11750d36a0ef565a9334ffc033303060193297e43684ab3e61237879f50a8cb5 WatchSource:0}: Error finding container 11750d36a0ef565a9334ffc033303060193297e43684ab3e61237879f50a8cb5: Status 404 returned error can't find the container with id 11750d36a0ef565a9334ffc033303060193297e43684ab3e61237879f50a8cb5
Oct 31 19:40:56.766086 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:56.766067    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f5c1ab-68b5bdf4d7-jwcll"]
Oct 31 19:40:56.839685 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:56.839654    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f5c1ab-68b5bdf4d7-jwcll" event={"ID":"4d05d0a8-95d5-4f44-b7d9-bae7800b7176","Type":"ContainerStarted","Data":"11750d36a0ef565a9334ffc033303060193297e43684ab3e61237879f50a8cb5"}
Oct 31 19:40:56.907472 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:56.907448    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-f5c1ab-54c4d89f99-b7fr8"]
Oct 31 19:40:56.908802 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:40:56.908775    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5005342_3674_4428_ac17_a42f0dc6d8b2.slice/crio-78dadf43a1eecc67c317f0d4b2e6f7ed9aa13b3150cdedae0fb0e7fd2ec6e4f5 WatchSource:0}: Error finding container 78dadf43a1eecc67c317f0d4b2e6f7ed9aa13b3150cdedae0fb0e7fd2ec6e4f5: Status 404 returned error can't find the container with id 78dadf43a1eecc67c317f0d4b2e6f7ed9aa13b3150cdedae0fb0e7fd2ec6e4f5
Oct 31 19:40:56.964504 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:56.964471    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:57.012681 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:57.012619    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2d319a-d95b54579-hzwkl"]
Oct 31 19:40:57.017884 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:57.017864    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2d319a-d95b54579-hzwkl"]
Oct 31 19:40:57.017987 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:57.017963    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2d319a-d95b54579-hzwkl"
Oct 31 19:40:57.025548 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:57.025504    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-2d319a\""
Oct 31 19:40:57.026620 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:57.026594    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2d319a-7dcc698947-479ft"]
Oct 31 19:40:57.032200 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:57.032170    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2d319a-7dcc698947-479ft"
Oct 31 19:40:57.042844 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:57.042822    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2d319a-7dcc698947-479ft"]
Oct 31 19:40:57.101799 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:57.101766    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:57.108813 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:57.108790    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlc8d\" (UniqueName: \"kubernetes.io/projected/c3957000-2a12-4ca9-9cd5-3464f5b6146e-kube-api-access-mlc8d\") pod \"payload-pre-processing-e2e-api-a-2d319a-d95b54579-hzwkl\" (UID: \"c3957000-2a12-4ca9-9cd5-3464f5b6146e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2d319a-d95b54579-hzwkl"
Oct 31 19:40:57.108919 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:57.108891    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3957000-2a12-4ca9-9cd5-3464f5b6146e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-2d319a-d95b54579-hzwkl\" (UID: \"c3957000-2a12-4ca9-9cd5-3464f5b6146e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2d319a-d95b54579-hzwkl"
Oct 31 19:40:57.109025 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:57.109008    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6043e494-bfb6-4b33-82cc-f93a504e36e6-plugins-config-volume\") pod \"payload-processing-e2e-api-a-2d319a-7dcc698947-479ft\" (UID: \"6043e494-bfb6-4b33-82cc-f93a504e36e6\") " pod="openshift-ingress/payload-processing-e2e-api-a-2d319a-7dcc698947-479ft"
Oct 31 19:40:57.109166 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:57.109152    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9768f\" (UniqueName: \"kubernetes.io/projected/6043e494-bfb6-4b33-82cc-f93a504e36e6-kube-api-access-9768f\") pod \"payload-processing-e2e-api-a-2d319a-7dcc698947-479ft\" (UID: \"6043e494-bfb6-4b33-82cc-f93a504e36e6\") " pod="openshift-ingress/payload-processing-e2e-api-a-2d319a-7dcc698947-479ft"
Oct 31 19:40:57.119256 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:57.119234    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-2d319a-dockercfg-xgp5z\""
Oct 31 19:40:57.224349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:57.218756    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6043e494-bfb6-4b33-82cc-f93a504e36e6-plugins-config-volume\") pod \"payload-processing-e2e-api-a-2d319a-7dcc698947-479ft\" (UID: \"6043e494-bfb6-4b33-82cc-f93a504e36e6\") " pod="openshift-ingress/payload-processing-e2e-api-a-2d319a-7dcc698947-479ft"
Oct 31 19:40:57.224349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:57.218995    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9768f\" (UniqueName: \"kubernetes.io/projected/6043e494-bfb6-4b33-82cc-f93a504e36e6-kube-api-access-9768f\") pod \"payload-processing-e2e-api-a-2d319a-7dcc698947-479ft\" (UID: \"6043e494-bfb6-4b33-82cc-f93a504e36e6\") " pod="openshift-ingress/payload-processing-e2e-api-a-2d319a-7dcc698947-479ft"
Oct 31 19:40:57.224349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:57.219141    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlc8d\" (UniqueName: \"kubernetes.io/projected/c3957000-2a12-4ca9-9cd5-3464f5b6146e-kube-api-access-mlc8d\") pod \"payload-pre-processing-e2e-api-a-2d319a-d95b54579-hzwkl\" (UID: \"c3957000-2a12-4ca9-9cd5-3464f5b6146e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2d319a-d95b54579-hzwkl"
Oct 31 19:40:57.224349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:57.219187    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3957000-2a12-4ca9-9cd5-3464f5b6146e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-2d319a-d95b54579-hzwkl\" (UID: \"c3957000-2a12-4ca9-9cd5-3464f5b6146e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2d319a-d95b54579-hzwkl"
Oct 31 19:40:57.224349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:57.220245    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3957000-2a12-4ca9-9cd5-3464f5b6146e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-2d319a-d95b54579-hzwkl\" (UID: \"c3957000-2a12-4ca9-9cd5-3464f5b6146e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2d319a-d95b54579-hzwkl"
Oct 31 19:40:57.271403 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:57.258447    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6043e494-bfb6-4b33-82cc-f93a504e36e6-plugins-config-volume\") pod \"payload-processing-e2e-api-a-2d319a-7dcc698947-479ft\" (UID: \"6043e494-bfb6-4b33-82cc-f93a504e36e6\") " pod="openshift-ingress/payload-processing-e2e-api-a-2d319a-7dcc698947-479ft"
Oct 31 19:40:57.290353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:57.289042    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlc8d\" (UniqueName: \"kubernetes.io/projected/c3957000-2a12-4ca9-9cd5-3464f5b6146e-kube-api-access-mlc8d\") pod \"payload-pre-processing-e2e-api-a-2d319a-d95b54579-hzwkl\" (UID: \"c3957000-2a12-4ca9-9cd5-3464f5b6146e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2d319a-d95b54579-hzwkl"
Oct 31 19:40:57.331353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:57.325987    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9768f\" (UniqueName: \"kubernetes.io/projected/6043e494-bfb6-4b33-82cc-f93a504e36e6-kube-api-access-9768f\") pod \"payload-processing-e2e-api-a-2d319a-7dcc698947-479ft\" (UID: \"6043e494-bfb6-4b33-82cc-f93a504e36e6\") " pod="openshift-ingress/payload-processing-e2e-api-a-2d319a-7dcc698947-479ft"
Oct 31 19:40:57.346863 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:57.345660    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2d319a-d95b54579-hzwkl"
Oct 31 19:40:57.368016 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:57.366127    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2d319a-7dcc698947-479ft"
Oct 31 19:40:57.495366 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:57.415741    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:57.632493 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:57.632407    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f482c94-43ba-4570-9b2a-ef4886627127" path="/var/lib/kubelet/pods/0f482c94-43ba-4570-9b2a-ef4886627127/volumes"
Oct 31 19:40:57.634228 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:57.633036    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="619ab3b5-8258-4aea-90fe-a351097d91cd" path="/var/lib/kubelet/pods/619ab3b5-8258-4aea-90fe-a351097d91cd/volumes"
Oct 31 19:40:57.812218 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:57.812140    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:57.853200 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:57.853168    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f5c1ab-54c4d89f99-b7fr8" event={"ID":"a5005342-3674-4428-ac17-a42f0dc6d8b2","Type":"ContainerStarted","Data":"abd5bf92064b245b03b7801a3dd5a63c6a7f2b1eb9014bbe05b10b68f0f5ad60"}
Oct 31 19:40:57.853610 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:57.853230    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f5c1ab-54c4d89f99-b7fr8"
Oct 31 19:40:57.853610 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:57.853258    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f5c1ab-54c4d89f99-b7fr8" event={"ID":"a5005342-3674-4428-ac17-a42f0dc6d8b2","Type":"ContainerStarted","Data":"78dadf43a1eecc67c317f0d4b2e6f7ed9aa13b3150cdedae0fb0e7fd2ec6e4f5"}
Oct 31 19:40:57.854860 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:57.854835    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f5c1ab-68b5bdf4d7-jwcll" event={"ID":"4d05d0a8-95d5-4f44-b7d9-bae7800b7176","Type":"ContainerStarted","Data":"1edb64f1761ac82b92da42c3df5f7d5674bbac3840d46c187920d9ab1a8d6648"}
Oct 31 19:40:57.854965 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:57.854916    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f5c1ab-68b5bdf4d7-jwcll"
Oct 31 19:40:57.862926 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:57.862886    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2d319a-7dcc698947-479ft"]
Oct 31 19:40:57.890821 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:57.890786    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2d319a-d95b54579-hzwkl"]
Oct 31 19:40:57.900165 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:57.900111    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f5c1ab-54c4d89f99-b7fr8" podStartSLOduration=2.900092778 podStartE2EDuration="2.900092778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:40:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:40:57.896008771 +0000 UTC m=+1288.469242901" watchObservedRunningTime="2026-10-31 19:40:57.900092778 +0000 UTC m=+1288.473326908"
Oct 31 19:40:57.903792 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:40:57.903768    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3957000_2a12_4ca9_9cd5_3464f5b6146e.slice/crio-7f01ac7fec817dd16a02bf4f65a15ab2483688d0175611c80ac6abe7940a17c7 WatchSource:0}: Error finding container 7f01ac7fec817dd16a02bf4f65a15ab2483688d0175611c80ac6abe7940a17c7: Status 404 returned error can't find the container with id 7f01ac7fec817dd16a02bf4f65a15ab2483688d0175611c80ac6abe7940a17c7
Oct 31 19:40:57.922247 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:57.922206    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f5c1ab-68b5bdf4d7-jwcll" podStartSLOduration=2.922196481 podStartE2EDuration="2.922196481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:40:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:40:57.920289736 +0000 UTC m=+1288.493523866" watchObservedRunningTime="2026-10-31 19:40:57.922196481 +0000 UTC m=+1288.495430610"
Oct 31 19:40:58.388653 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:58.388620    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:58.518608 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:58.518576    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:58.630966 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:58.630931    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:40:58.867257 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:58.867220    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2d319a-7dcc698947-479ft" event={"ID":"6043e494-bfb6-4b33-82cc-f93a504e36e6","Type":"ContainerStarted","Data":"a753eeba7ef2b69bf03b42b934bddf440f844bb80fa3d42ed7833156460270bf"}
Oct 31 19:40:58.867844 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:58.867814    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2d319a-7dcc698947-479ft" event={"ID":"6043e494-bfb6-4b33-82cc-f93a504e36e6","Type":"ContainerStarted","Data":"7317aeb309c1fb0d90eeb3ae53acfba55dc8617df2320a60577bcd9fb6dad159"}
Oct 31 19:40:58.867976 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:58.867857    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-2d319a-7dcc698947-479ft"
Oct 31 19:40:58.869314 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:58.869291    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2d319a-d95b54579-hzwkl" event={"ID":"c3957000-2a12-4ca9-9cd5-3464f5b6146e","Type":"ContainerStarted","Data":"209a6a7af0cbae1060246203a66b2194f53c2e9c722b68b440a6593b1e54652c"}
Oct 31 19:40:58.869668 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:58.869645    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2d319a-d95b54579-hzwkl"
Oct 31 19:40:58.869769 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:58.869694    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2d319a-d95b54579-hzwkl" event={"ID":"c3957000-2a12-4ca9-9cd5-3464f5b6146e","Type":"ContainerStarted","Data":"7f01ac7fec817dd16a02bf4f65a15ab2483688d0175611c80ac6abe7940a17c7"}
Oct 31 19:40:58.885941 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:58.885880    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-2d319a-7dcc698947-479ft" podStartSLOduration=2.8858622350000003 podStartE2EDuration="2.885862235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:40:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:40:58.885386839 +0000 UTC m=+1289.458620970" watchObservedRunningTime="2026-10-31 19:40:58.885862235 +0000 UTC m=+1289.459096366"
Oct 31 19:40:58.909409 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:58.909347    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2d319a-d95b54579-hzwkl" podStartSLOduration=2.909312908 podStartE2EDuration="2.909312908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:40:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:40:58.903978169 +0000 UTC m=+1289.477212298" watchObservedRunningTime="2026-10-31 19:40:58.909312908 +0000 UTC m=+1289.482547038"
Oct 31 19:40:59.754894 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:40:59.754866    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:00.632463 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:00.632426    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:01.589771 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:01.589737    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:01.999924 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:01.999890    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:02.114247 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:02.114223    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:02.593293 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:02.591248    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1c0c1d-fcd798f5d-x5rgq"]
Oct 31 19:41:02.597262 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:02.597235    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c0c1d-fcd798f5d-x5rgq"
Oct 31 19:41:02.605203 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:02.604903    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-1c0c1d-serving-cert\""
Oct 31 19:41:02.615744 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:02.615719    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1c0c1d-fcd798f5d-x5rgq"]
Oct 31 19:41:02.662982 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:02.662922    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:02.749253 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:02.743465    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/accc06a5-6668-4b44-9578-d57b054280e0-metrics-certs\") pod \"maas-api-e2e-api-b-1c0c1d-fcd798f5d-x5rgq\" (UID: \"accc06a5-6668-4b44-9578-d57b054280e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c0c1d-fcd798f5d-x5rgq"
Oct 31 19:41:02.749253 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:02.743588    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/accc06a5-6668-4b44-9578-d57b054280e0-openshift-service-ca\") pod \"maas-api-e2e-api-b-1c0c1d-fcd798f5d-x5rgq\" (UID: \"accc06a5-6668-4b44-9578-d57b054280e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c0c1d-fcd798f5d-x5rgq"
Oct 31 19:41:02.749253 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:02.743710    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/accc06a5-6668-4b44-9578-d57b054280e0-maas-api-tls\") pod \"maas-api-e2e-api-b-1c0c1d-fcd798f5d-x5rgq\" (UID: \"accc06a5-6668-4b44-9578-d57b054280e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c0c1d-fcd798f5d-x5rgq"
Oct 31 19:41:02.749253 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:02.743739    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-297bm\" (UniqueName: \"kubernetes.io/projected/accc06a5-6668-4b44-9578-d57b054280e0-kube-api-access-297bm\") pod \"maas-api-e2e-api-b-1c0c1d-fcd798f5d-x5rgq\" (UID: \"accc06a5-6668-4b44-9578-d57b054280e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c0c1d-fcd798f5d-x5rgq"
Oct 31 19:41:02.877112 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:02.877016    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/accc06a5-6668-4b44-9578-d57b054280e0-metrics-certs\") pod \"maas-api-e2e-api-b-1c0c1d-fcd798f5d-x5rgq\" (UID: \"accc06a5-6668-4b44-9578-d57b054280e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c0c1d-fcd798f5d-x5rgq"
Oct 31 19:41:02.877280 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:02.877116    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/accc06a5-6668-4b44-9578-d57b054280e0-openshift-service-ca\") pod \"maas-api-e2e-api-b-1c0c1d-fcd798f5d-x5rgq\" (UID: \"accc06a5-6668-4b44-9578-d57b054280e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c0c1d-fcd798f5d-x5rgq"
Oct 31 19:41:02.877280 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:02.877229    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/accc06a5-6668-4b44-9578-d57b054280e0-maas-api-tls\") pod \"maas-api-e2e-api-b-1c0c1d-fcd798f5d-x5rgq\" (UID: \"accc06a5-6668-4b44-9578-d57b054280e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c0c1d-fcd798f5d-x5rgq"
Oct 31 19:41:02.877280 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:02.877260    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-297bm\" (UniqueName: \"kubernetes.io/projected/accc06a5-6668-4b44-9578-d57b054280e0-kube-api-access-297bm\") pod \"maas-api-e2e-api-b-1c0c1d-fcd798f5d-x5rgq\" (UID: \"accc06a5-6668-4b44-9578-d57b054280e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c0c1d-fcd798f5d-x5rgq"
Oct 31 19:41:02.877966 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:41:02.877941    4645 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-1c0c1d-serving-cert: secret "maas-api-e2e-api-b-1c0c1d-serving-cert" not found
Oct 31 19:41:02.877966 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:02.877958    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/accc06a5-6668-4b44-9578-d57b054280e0-openshift-service-ca\") pod \"maas-api-e2e-api-b-1c0c1d-fcd798f5d-x5rgq\" (UID: \"accc06a5-6668-4b44-9578-d57b054280e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c0c1d-fcd798f5d-x5rgq"
Oct 31 19:41:02.878103 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:41:02.878030    4645 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/accc06a5-6668-4b44-9578-d57b054280e0-maas-api-tls podName:accc06a5-6668-4b44-9578-d57b054280e0 nodeName:}" failed. No retries permitted until 2026-10-31 19:41:03.378002451 +0000 UTC m=+1293.951236558 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/accc06a5-6668-4b44-9578-d57b054280e0-maas-api-tls") pod "maas-api-e2e-api-b-1c0c1d-fcd798f5d-x5rgq" (UID: "accc06a5-6668-4b44-9578-d57b054280e0") : secret "maas-api-e2e-api-b-1c0c1d-serving-cert" not found
Oct 31 19:41:02.879003 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:02.878976    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:02.881838 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:02.881813    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/accc06a5-6668-4b44-9578-d57b054280e0-metrics-certs\") pod \"maas-api-e2e-api-b-1c0c1d-fcd798f5d-x5rgq\" (UID: \"accc06a5-6668-4b44-9578-d57b054280e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c0c1d-fcd798f5d-x5rgq"
Oct 31 19:41:02.891342 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:02.891306    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-297bm\" (UniqueName: \"kubernetes.io/projected/accc06a5-6668-4b44-9578-d57b054280e0-kube-api-access-297bm\") pod \"maas-api-e2e-api-b-1c0c1d-fcd798f5d-x5rgq\" (UID: \"accc06a5-6668-4b44-9578-d57b054280e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c0c1d-fcd798f5d-x5rgq"
Oct 31 19:41:03.073261 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:03.073228    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:03.283432 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:03.282876    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:03.374235 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:03.374198    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-88eaf9-697ffc695-b9ds8"]
Oct 31 19:41:03.379099 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:03.379080    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-88eaf9-697ffc695-b9ds8"
Oct 31 19:41:03.386942 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:03.386922    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-88eaf9\""
Oct 31 19:41:03.393774 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:03.393752    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-88eaf9-697ffc695-b9ds8"]
Oct 31 19:41:03.394371 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:03.394352    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-88eaf9-dockercfg-6zm6t\""
Oct 31 19:41:03.412757 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:03.412536    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-88eaf9-56b45b6577-m674p"]
Oct 31 19:41:03.416796 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:03.414850    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/accc06a5-6668-4b44-9578-d57b054280e0-maas-api-tls\") pod \"maas-api-e2e-api-b-1c0c1d-fcd798f5d-x5rgq\" (UID: \"accc06a5-6668-4b44-9578-d57b054280e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c0c1d-fcd798f5d-x5rgq"
Oct 31 19:41:03.416796 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:03.415944    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2c7c458-2fc5-4674-b080-7f0c9f29a3b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-88eaf9-697ffc695-b9ds8\" (UID: \"f2c7c458-2fc5-4674-b080-7f0c9f29a3b9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-88eaf9-697ffc695-b9ds8"
Oct 31 19:41:03.416796 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:03.415979    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bb4mv\" (UniqueName: \"kubernetes.io/projected/f2c7c458-2fc5-4674-b080-7f0c9f29a3b9-kube-api-access-bb4mv\") pod \"payload-pre-processing-e2e-inf-b-88eaf9-697ffc695-b9ds8\" (UID: \"f2c7c458-2fc5-4674-b080-7f0c9f29a3b9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-88eaf9-697ffc695-b9ds8"
Oct 31 19:41:03.417913 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:03.417865    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:03.419476 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:03.419455    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/accc06a5-6668-4b44-9578-d57b054280e0-maas-api-tls\") pod \"maas-api-e2e-api-b-1c0c1d-fcd798f5d-x5rgq\" (UID: \"accc06a5-6668-4b44-9578-d57b054280e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c0c1d-fcd798f5d-x5rgq"
Oct 31 19:41:03.431415 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:03.431395    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-88eaf9-56b45b6577-m674p"]
Oct 31 19:41:03.431575 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:03.431502    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-88eaf9-56b45b6577-m674p"
Oct 31 19:41:03.536399 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:03.535829    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c0c1d-fcd798f5d-x5rgq"
Oct 31 19:41:03.561353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:03.547749    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2c7c458-2fc5-4674-b080-7f0c9f29a3b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-88eaf9-697ffc695-b9ds8\" (UID: \"f2c7c458-2fc5-4674-b080-7f0c9f29a3b9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-88eaf9-697ffc695-b9ds8"
Oct 31 19:41:03.561353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:03.547802    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb4mv\" (UniqueName: \"kubernetes.io/projected/f2c7c458-2fc5-4674-b080-7f0c9f29a3b9-kube-api-access-bb4mv\") pod \"payload-pre-processing-e2e-inf-b-88eaf9-697ffc695-b9ds8\" (UID: \"f2c7c458-2fc5-4674-b080-7f0c9f29a3b9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-88eaf9-697ffc695-b9ds8"
Oct 31 19:41:03.705031 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:03.667616    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ae56bf5-1961-48fc-9c77-bb64e0d791cf-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-88eaf9-56b45b6577-m674p\" (UID: \"7ae56bf5-1961-48fc-9c77-bb64e0d791cf\") " pod="openshift-ingress/payload-processing-e2e-inf-b-88eaf9-56b45b6577-m674p"
Oct 31 19:41:03.705031 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:03.667686    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc5dm\" (UniqueName: \"kubernetes.io/projected/7ae56bf5-1961-48fc-9c77-bb64e0d791cf-kube-api-access-jc5dm\") pod \"payload-processing-e2e-inf-b-88eaf9-56b45b6577-m674p\" (UID: \"7ae56bf5-1961-48fc-9c77-bb64e0d791cf\") " pod="openshift-ingress/payload-processing-e2e-inf-b-88eaf9-56b45b6577-m674p"
Oct 31 19:41:03.705031 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:03.671367    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2c7c458-2fc5-4674-b080-7f0c9f29a3b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-88eaf9-697ffc695-b9ds8\" (UID: \"f2c7c458-2fc5-4674-b080-7f0c9f29a3b9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-88eaf9-697ffc695-b9ds8"
Oct 31 19:41:03.705031 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:03.700586    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb4mv\" (UniqueName: \"kubernetes.io/projected/f2c7c458-2fc5-4674-b080-7f0c9f29a3b9-kube-api-access-bb4mv\") pod \"payload-pre-processing-e2e-inf-b-88eaf9-697ffc695-b9ds8\" (UID: \"f2c7c458-2fc5-4674-b080-7f0c9f29a3b9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-88eaf9-697ffc695-b9ds8"
Oct 31 19:41:03.770126 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:03.770092    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ae56bf5-1961-48fc-9c77-bb64e0d791cf-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-88eaf9-56b45b6577-m674p\" (UID: \"7ae56bf5-1961-48fc-9c77-bb64e0d791cf\") " pod="openshift-ingress/payload-processing-e2e-inf-b-88eaf9-56b45b6577-m674p"
Oct 31 19:41:03.770455 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:03.770428    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc5dm\" (UniqueName: \"kubernetes.io/projected/7ae56bf5-1961-48fc-9c77-bb64e0d791cf-kube-api-access-jc5dm\") pod \"payload-processing-e2e-inf-b-88eaf9-56b45b6577-m674p\" (UID: \"7ae56bf5-1961-48fc-9c77-bb64e0d791cf\") " pod="openshift-ingress/payload-processing-e2e-inf-b-88eaf9-56b45b6577-m674p"
Oct 31 19:41:03.771153 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:03.771090    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ae56bf5-1961-48fc-9c77-bb64e0d791cf-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-88eaf9-56b45b6577-m674p\" (UID: \"7ae56bf5-1961-48fc-9c77-bb64e0d791cf\") " pod="openshift-ingress/payload-processing-e2e-inf-b-88eaf9-56b45b6577-m674p"
Oct 31 19:41:03.786560 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:03.786486    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc5dm\" (UniqueName: \"kubernetes.io/projected/7ae56bf5-1961-48fc-9c77-bb64e0d791cf-kube-api-access-jc5dm\") pod \"payload-processing-e2e-inf-b-88eaf9-56b45b6577-m674p\" (UID: \"7ae56bf5-1961-48fc-9c77-bb64e0d791cf\") " pod="openshift-ingress/payload-processing-e2e-inf-b-88eaf9-56b45b6577-m674p"
Oct 31 19:41:03.998362 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:03.998090    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-88eaf9-697ffc695-b9ds8"
Oct 31 19:41:04.027106 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:04.027072    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1c0c1d-fcd798f5d-x5rgq"]
Oct 31 19:41:04.055485 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:04.055417    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-88eaf9-56b45b6577-m674p"
Oct 31 19:41:04.267174 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:04.267086    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:04.439705 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:04.439671    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-88eaf9-697ffc695-b9ds8"]
Oct 31 19:41:04.516078 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:04.515549    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-88eaf9-56b45b6577-m674p"]
Oct 31 19:41:04.945899 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:04.945863    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-88eaf9-56b45b6577-m674p" event={"ID":"7ae56bf5-1961-48fc-9c77-bb64e0d791cf","Type":"ContainerStarted","Data":"622bd5e7b4d3fcab0d71e79b1c5c49e26fbf7866c1213da8d8181067d101fada"}
Oct 31 19:41:04.946197 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:04.945928    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-88eaf9-56b45b6577-m674p"
Oct 31 19:41:04.946197 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:04.945956    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-88eaf9-56b45b6577-m674p" event={"ID":"7ae56bf5-1961-48fc-9c77-bb64e0d791cf","Type":"ContainerStarted","Data":"712e981cf64566e226e2470cb5a11f000409187b7a57ab42ef6de8227da08895"}
Oct 31 19:41:04.947728 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:04.947698    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c0c1d-fcd798f5d-x5rgq" event={"ID":"accc06a5-6668-4b44-9578-d57b054280e0","Type":"ContainerStarted","Data":"c6ba231d8e233ddd10c699593b0fd5ca8c23ec182d89e0b064f7402066de39fc"}
Oct 31 19:41:04.947888 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:04.947751    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c0c1d-fcd798f5d-x5rgq" event={"ID":"accc06a5-6668-4b44-9578-d57b054280e0","Type":"ContainerStarted","Data":"6684d50fa46481d7d2bb39be7afa89a43cf53a7ffbce88f5e28b9980a110d519"}
Oct 31 19:41:04.947888 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:04.947781    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c0c1d-fcd798f5d-x5rgq"
Oct 31 19:41:04.949539 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:04.949514    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-88eaf9-697ffc695-b9ds8" event={"ID":"f2c7c458-2fc5-4674-b080-7f0c9f29a3b9","Type":"ContainerStarted","Data":"210f25b91f3fcff0b9fda607493ebc2484d5d6ec1dd03de55ce184c9b5dbd59b"}
Oct 31 19:41:04.949651 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:04.949551    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-88eaf9-697ffc695-b9ds8" event={"ID":"f2c7c458-2fc5-4674-b080-7f0c9f29a3b9","Type":"ContainerStarted","Data":"b1d00891dd444836d1f16feb5196eae9802128075eae313b1677b8e30e2e93f0"}
Oct 31 19:41:04.949693 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:04.949644    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-88eaf9-697ffc695-b9ds8"
Oct 31 19:41:04.967319 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:04.967261    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-88eaf9-56b45b6577-m674p" podStartSLOduration=1.9672451 podStartE2EDuration="1.9672451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:41:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:41:04.966997581 +0000 UTC m=+1295.540231710" watchObservedRunningTime="2026-10-31 19:41:04.9672451 +0000 UTC m=+1295.540479230"
Oct 31 19:41:04.987852 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:04.987800    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-88eaf9-697ffc695-b9ds8" podStartSLOduration=1.987783808 podStartE2EDuration="1.987783808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:41:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:41:04.984819261 +0000 UTC m=+1295.558053390" watchObservedRunningTime="2026-10-31 19:41:04.987783808 +0000 UTC m=+1295.561017939"
Oct 31 19:41:05.012741 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:05.012696    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c0c1d-fcd798f5d-x5rgq" podStartSLOduration=2.34877411 podStartE2EDuration="3.012683011s" totalImagesPullingTime="663.908901ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:41:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:41:05.011011768 +0000 UTC m=+1295.584245894" watchObservedRunningTime="2026-10-31 19:41:05.012683011 +0000 UTC m=+1295.585917137"
Oct 31 19:41:05.271498 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:05.271415    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:06.248012 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:41:06.247974    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 19:41:06.490038 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:06.490009    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:07.449622 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:07.449588    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:08.249695 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:08.249654    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:08.884792 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:08.884766    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f5c1ab-54c4d89f99-b7fr8"
Oct 31 19:41:08.884963 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:08.884829    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f5c1ab-68b5bdf4d7-jwcll"
Oct 31 19:41:09.181714 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:09.181681    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:09.887724 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:09.887697    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-2d319a-7dcc698947-479ft"
Oct 31 19:41:09.888181 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:09.887773    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2d319a-d95b54579-hzwkl"
Oct 31 19:41:10.181820 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:10.181769    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:10.781934 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:10.781901    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:11.053408 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:11.052682    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c0c1d-fcd798f5d-x5rgq"
Oct 31 19:41:11.180350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:11.147784    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:12.195409 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:12.193455    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:13.157660 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:13.157627    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:13.409925 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:13.409896    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:13.939773 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:13.939737    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:14.046219 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:14.046184    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:14.159007 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:14.158808    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:14.753668 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:14.753634    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm"]
Oct 31 19:41:14.793302 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:14.793268    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm"]
Oct 31 19:41:14.793488 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:14.793412    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm"
Oct 31 19:41:14.875813 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:14.875781    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/940375be-4b29-43ca-b0de-1643eaa82341-credential-socket\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm\" (UID: \"940375be-4b29-43ca-b0de-1643eaa82341\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm"
Oct 31 19:41:14.875985 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:14.875824    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/940375be-4b29-43ca-b0de-1643eaa82341-istio-token\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm\" (UID: \"940375be-4b29-43ca-b0de-1643eaa82341\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm"
Oct 31 19:41:14.875985 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:14.875931    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/940375be-4b29-43ca-b0de-1643eaa82341-istiod-ca-cert\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm\" (UID: \"940375be-4b29-43ca-b0de-1643eaa82341\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm"
Oct 31 19:41:14.875985 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:14.875961    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/940375be-4b29-43ca-b0de-1643eaa82341-workload-certs\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm\" (UID: \"940375be-4b29-43ca-b0de-1643eaa82341\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm"
Oct 31 19:41:14.876111 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:14.875991    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/940375be-4b29-43ca-b0de-1643eaa82341-istio-envoy\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm\" (UID: \"940375be-4b29-43ca-b0de-1643eaa82341\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm"
Oct 31 19:41:14.876146 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:14.876099    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/940375be-4b29-43ca-b0de-1643eaa82341-istio-data\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm\" (UID: \"940375be-4b29-43ca-b0de-1643eaa82341\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm"
Oct 31 19:41:14.876269 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:14.876247    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhwjb\" (UniqueName: \"kubernetes.io/projected/940375be-4b29-43ca-b0de-1643eaa82341-kube-api-access-qhwjb\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm\" (UID: \"940375be-4b29-43ca-b0de-1643eaa82341\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm"
Oct 31 19:41:14.876324 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:14.876306    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/940375be-4b29-43ca-b0de-1643eaa82341-workload-socket\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm\" (UID: \"940375be-4b29-43ca-b0de-1643eaa82341\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm"
Oct 31 19:41:14.876435 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:14.876420    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/940375be-4b29-43ca-b0de-1643eaa82341-istio-podinfo\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm\" (UID: \"940375be-4b29-43ca-b0de-1643eaa82341\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm"
Oct 31 19:41:14.978208 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:14.978181    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhwjb\" (UniqueName: \"kubernetes.io/projected/940375be-4b29-43ca-b0de-1643eaa82341-kube-api-access-qhwjb\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm\" (UID: \"940375be-4b29-43ca-b0de-1643eaa82341\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm"
Oct 31 19:41:14.978371 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:14.978232    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/940375be-4b29-43ca-b0de-1643eaa82341-workload-socket\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm\" (UID: \"940375be-4b29-43ca-b0de-1643eaa82341\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm"
Oct 31 19:41:14.978448 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:14.978373    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/940375be-4b29-43ca-b0de-1643eaa82341-istio-podinfo\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm\" (UID: \"940375be-4b29-43ca-b0de-1643eaa82341\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm"
Oct 31 19:41:14.978551 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:14.978531    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/940375be-4b29-43ca-b0de-1643eaa82341-credential-socket\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm\" (UID: \"940375be-4b29-43ca-b0de-1643eaa82341\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm"
Oct 31 19:41:14.978680 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:14.978552    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/940375be-4b29-43ca-b0de-1643eaa82341-workload-socket\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm\" (UID: \"940375be-4b29-43ca-b0de-1643eaa82341\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm"
Oct 31 19:41:14.978680 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:14.978576    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/940375be-4b29-43ca-b0de-1643eaa82341-istio-token\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm\" (UID: \"940375be-4b29-43ca-b0de-1643eaa82341\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm"
Oct 31 19:41:14.978680 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:14.978623    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/940375be-4b29-43ca-b0de-1643eaa82341-istiod-ca-cert\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm\" (UID: \"940375be-4b29-43ca-b0de-1643eaa82341\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm"
Oct 31 19:41:14.978680 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:14.978643    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/940375be-4b29-43ca-b0de-1643eaa82341-workload-certs\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm\" (UID: \"940375be-4b29-43ca-b0de-1643eaa82341\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm"
Oct 31 19:41:14.978680 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:14.978670    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/940375be-4b29-43ca-b0de-1643eaa82341-istio-envoy\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm\" (UID: \"940375be-4b29-43ca-b0de-1643eaa82341\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm"
Oct 31 19:41:14.978971 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:14.978741    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/940375be-4b29-43ca-b0de-1643eaa82341-istio-data\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm\" (UID: \"940375be-4b29-43ca-b0de-1643eaa82341\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm"
Oct 31 19:41:14.978971 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:14.978795    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/940375be-4b29-43ca-b0de-1643eaa82341-credential-socket\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm\" (UID: \"940375be-4b29-43ca-b0de-1643eaa82341\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm"
Oct 31 19:41:14.979059 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:14.979022    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/940375be-4b29-43ca-b0de-1643eaa82341-workload-certs\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm\" (UID: \"940375be-4b29-43ca-b0de-1643eaa82341\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm"
Oct 31 19:41:14.979189 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:14.979166    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/940375be-4b29-43ca-b0de-1643eaa82341-istio-data\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm\" (UID: \"940375be-4b29-43ca-b0de-1643eaa82341\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm"
Oct 31 19:41:14.979482 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:14.979463    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/940375be-4b29-43ca-b0de-1643eaa82341-istiod-ca-cert\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm\" (UID: \"940375be-4b29-43ca-b0de-1643eaa82341\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm"
Oct 31 19:41:14.980943 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:14.980922    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/940375be-4b29-43ca-b0de-1643eaa82341-istio-envoy\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm\" (UID: \"940375be-4b29-43ca-b0de-1643eaa82341\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm"
Oct 31 19:41:14.980943 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:14.980938    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/940375be-4b29-43ca-b0de-1643eaa82341-istio-podinfo\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm\" (UID: \"940375be-4b29-43ca-b0de-1643eaa82341\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm"
Oct 31 19:41:14.994398 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:14.994374    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhwjb\" (UniqueName: \"kubernetes.io/projected/940375be-4b29-43ca-b0de-1643eaa82341-kube-api-access-qhwjb\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm\" (UID: \"940375be-4b29-43ca-b0de-1643eaa82341\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm"
Oct 31 19:41:15.001424 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:15.001395    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/940375be-4b29-43ca-b0de-1643eaa82341-istio-token\") pod \"e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm\" (UID: \"940375be-4b29-43ca-b0de-1643eaa82341\") " pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm"
Oct 31 19:41:15.124034 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:15.123953    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm"
Oct 31 19:41:15.218223 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:15.217627    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz"]
Oct 31 19:41:15.227386 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:15.224568    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:15.267275 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:15.267243    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz"]
Oct 31 19:41:15.267510 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:15.267368    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz"
Oct 31 19:41:15.286868 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:15.286835    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nprt\" (UniqueName: \"kubernetes.io/projected/79f81748-b7d5-4ed1-8f28-4386bba24a79-kube-api-access-5nprt\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz\" (UID: \"79f81748-b7d5-4ed1-8f28-4386bba24a79\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz"
Oct 31 19:41:15.286999 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:15.286884    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79f81748-b7d5-4ed1-8f28-4386bba24a79-istiod-ca-cert\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz\" (UID: \"79f81748-b7d5-4ed1-8f28-4386bba24a79\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz"
Oct 31 19:41:15.286999 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:15.286924    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79f81748-b7d5-4ed1-8f28-4386bba24a79-istio-envoy\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz\" (UID: \"79f81748-b7d5-4ed1-8f28-4386bba24a79\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz"
Oct 31 19:41:15.287108 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:15.287084    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79f81748-b7d5-4ed1-8f28-4386bba24a79-workload-socket\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz\" (UID: \"79f81748-b7d5-4ed1-8f28-4386bba24a79\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz"
Oct 31 19:41:15.287187 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:15.287171    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79f81748-b7d5-4ed1-8f28-4386bba24a79-istio-token\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz\" (UID: \"79f81748-b7d5-4ed1-8f28-4386bba24a79\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz"
Oct 31 19:41:15.287257 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:15.287201    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79f81748-b7d5-4ed1-8f28-4386bba24a79-istio-podinfo\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz\" (UID: \"79f81748-b7d5-4ed1-8f28-4386bba24a79\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz"
Oct 31 19:41:15.287257 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:15.287238    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79f81748-b7d5-4ed1-8f28-4386bba24a79-credential-socket\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz\" (UID: \"79f81748-b7d5-4ed1-8f28-4386bba24a79\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz"
Oct 31 19:41:15.287415 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:15.287297    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79f81748-b7d5-4ed1-8f28-4386bba24a79-istio-data\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz\" (UID: \"79f81748-b7d5-4ed1-8f28-4386bba24a79\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz"
Oct 31 19:41:15.287415 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:15.287383    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79f81748-b7d5-4ed1-8f28-4386bba24a79-workload-certs\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz\" (UID: \"79f81748-b7d5-4ed1-8f28-4386bba24a79\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz"
Oct 31 19:41:15.389616 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:15.389531    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79f81748-b7d5-4ed1-8f28-4386bba24a79-workload-certs\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz\" (UID: \"79f81748-b7d5-4ed1-8f28-4386bba24a79\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz"
Oct 31 19:41:15.389809 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:15.389748    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nprt\" (UniqueName: \"kubernetes.io/projected/79f81748-b7d5-4ed1-8f28-4386bba24a79-kube-api-access-5nprt\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz\" (UID: \"79f81748-b7d5-4ed1-8f28-4386bba24a79\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz"
Oct 31 19:41:15.389809 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:15.389784    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79f81748-b7d5-4ed1-8f28-4386bba24a79-istiod-ca-cert\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz\" (UID: \"79f81748-b7d5-4ed1-8f28-4386bba24a79\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz"
Oct 31 19:41:15.389928 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:15.389816    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79f81748-b7d5-4ed1-8f28-4386bba24a79-istio-envoy\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz\" (UID: \"79f81748-b7d5-4ed1-8f28-4386bba24a79\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz"
Oct 31 19:41:15.389983 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:15.389960    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79f81748-b7d5-4ed1-8f28-4386bba24a79-workload-socket\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz\" (UID: \"79f81748-b7d5-4ed1-8f28-4386bba24a79\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz"
Oct 31 19:41:15.389983 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:15.389964    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79f81748-b7d5-4ed1-8f28-4386bba24a79-workload-certs\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz\" (UID: \"79f81748-b7d5-4ed1-8f28-4386bba24a79\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz"
Oct 31 19:41:15.390110 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:15.390085    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79f81748-b7d5-4ed1-8f28-4386bba24a79-istio-token\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz\" (UID: \"79f81748-b7d5-4ed1-8f28-4386bba24a79\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz"
Oct 31 19:41:15.390177 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:15.390130    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79f81748-b7d5-4ed1-8f28-4386bba24a79-istio-podinfo\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz\" (UID: \"79f81748-b7d5-4ed1-8f28-4386bba24a79\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz"
Oct 31 19:41:15.390177 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:15.390160    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79f81748-b7d5-4ed1-8f28-4386bba24a79-credential-socket\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz\" (UID: \"79f81748-b7d5-4ed1-8f28-4386bba24a79\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz"
Oct 31 19:41:15.390308 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:15.390209    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79f81748-b7d5-4ed1-8f28-4386bba24a79-istio-data\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz\" (UID: \"79f81748-b7d5-4ed1-8f28-4386bba24a79\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz"
Oct 31 19:41:15.390308 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:15.390279    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79f81748-b7d5-4ed1-8f28-4386bba24a79-workload-socket\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz\" (UID: \"79f81748-b7d5-4ed1-8f28-4386bba24a79\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz"
Oct 31 19:41:15.390664 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:15.390641    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79f81748-b7d5-4ed1-8f28-4386bba24a79-istio-data\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz\" (UID: \"79f81748-b7d5-4ed1-8f28-4386bba24a79\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz"
Oct 31 19:41:15.390751 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:15.390718    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79f81748-b7d5-4ed1-8f28-4386bba24a79-istiod-ca-cert\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz\" (UID: \"79f81748-b7d5-4ed1-8f28-4386bba24a79\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz"
Oct 31 19:41:15.391071 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:15.391039    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79f81748-b7d5-4ed1-8f28-4386bba24a79-credential-socket\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz\" (UID: \"79f81748-b7d5-4ed1-8f28-4386bba24a79\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz"
Oct 31 19:41:15.392982 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:15.392959    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79f81748-b7d5-4ed1-8f28-4386bba24a79-istio-podinfo\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz\" (UID: \"79f81748-b7d5-4ed1-8f28-4386bba24a79\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz"
Oct 31 19:41:15.393286 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:15.393265    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79f81748-b7d5-4ed1-8f28-4386bba24a79-istio-envoy\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz\" (UID: \"79f81748-b7d5-4ed1-8f28-4386bba24a79\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz"
Oct 31 19:41:15.401876 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:15.401854    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nprt\" (UniqueName: \"kubernetes.io/projected/79f81748-b7d5-4ed1-8f28-4386bba24a79-kube-api-access-5nprt\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz\" (UID: \"79f81748-b7d5-4ed1-8f28-4386bba24a79\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz"
Oct 31 19:41:15.402064 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:15.402043    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79f81748-b7d5-4ed1-8f28-4386bba24a79-istio-token\") pod \"e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz\" (UID: \"79f81748-b7d5-4ed1-8f28-4386bba24a79\") " pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz"
Oct 31 19:41:15.521359 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:15.521315    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm"]
Oct 31 19:41:15.521885 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:41:15.521857    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940375be_4b29_43ca_b0de_1643eaa82341.slice/crio-8df25cfd4d3a9c7af50b381668759ec5847688a778ef448463883a3654c97c38 WatchSource:0}: Error finding container 8df25cfd4d3a9c7af50b381668759ec5847688a778ef448463883a3654c97c38: Status 404 returned error can't find the container with id 8df25cfd4d3a9c7af50b381668759ec5847688a778ef448463883a3654c97c38
Oct 31 19:41:15.524766 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:15.524731    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:41:15.524867 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:15.524808    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:41:15.524867 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:15.524856    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:41:15.634682 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:15.634656    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz"
Oct 31 19:41:15.960569 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:15.960545    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-88eaf9-56b45b6577-m674p"
Oct 31 19:41:15.966723 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:15.966702    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-88eaf9-697ffc695-b9ds8"
Oct 31 19:41:15.981112 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:41:15.981083    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79f81748_b7d5_4ed1_8f28_4386bba24a79.slice/crio-02fd2bcdada5d507afff7fc4e35fd88cb7c368806f7a4350fc2d48213abc11de WatchSource:0}: Error finding container 02fd2bcdada5d507afff7fc4e35fd88cb7c368806f7a4350fc2d48213abc11de: Status 404 returned error can't find the container with id 02fd2bcdada5d507afff7fc4e35fd88cb7c368806f7a4350fc2d48213abc11de
Oct 31 19:41:15.984543 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:15.984519    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz"]
Oct 31 19:41:15.991211 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:15.988232    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:41:15.991211 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:15.988379    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:41:15.991211 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:15.988473    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:41:16.200216 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:16.200168    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm" event={"ID":"940375be-4b29-43ca-b0de-1643eaa82341","Type":"ContainerStarted","Data":"33261bccefb45b8390c33d027c8835feecd482139c806933f79afd02a02e4d37"}
Oct 31 19:41:16.200493 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:16.200253    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm" event={"ID":"940375be-4b29-43ca-b0de-1643eaa82341","Type":"ContainerStarted","Data":"8df25cfd4d3a9c7af50b381668759ec5847688a778ef448463883a3654c97c38"}
Oct 31 19:41:16.202766 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:16.202622    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz" event={"ID":"79f81748-b7d5-4ed1-8f28-4386bba24a79","Type":"ContainerStarted","Data":"eb4ad62af9dbd4c98b726505c351c8a7c15c0a288e34167527327263927b3e8b"}
Oct 31 19:41:16.202766 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:16.202675    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz" event={"ID":"79f81748-b7d5-4ed1-8f28-4386bba24a79","Type":"ContainerStarted","Data":"02fd2bcdada5d507afff7fc4e35fd88cb7c368806f7a4350fc2d48213abc11de"}
Oct 31 19:41:16.230519 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:16.230433    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm" podStartSLOduration=2.230416461 podStartE2EDuration="2.230416461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:41:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:41:16.222080145 +0000 UTC m=+1306.795314274" watchObservedRunningTime="2026-10-31 19:41:16.230416461 +0000 UTC m=+1306.803650592"
Oct 31 19:41:16.252938 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:16.252884    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz" podStartSLOduration=1.252867092 podStartE2EDuration="1.252867092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:41:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:41:16.24588603 +0000 UTC m=+1306.819120172" watchObservedRunningTime="2026-10-31 19:41:16.252867092 +0000 UTC m=+1306.826101223"
Oct 31 19:41:16.263058 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:16.263029    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:16.318055 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:16.317995    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2d319a-bdff7977-7zzh4"]
Oct 31 19:41:16.349348 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:16.343967    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2d319a-bdff7977-7zzh4"
Oct 31 19:41:16.349348 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:16.347531    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2d319a-bdff7977-7zzh4"]
Oct 31 19:41:16.355787 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:16.355744    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-2d319a-serving-cert\""
Oct 31 19:41:16.383084 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:16.383007    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:16.410686 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:16.410658    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92811d3a-a4be-49a8-a815-d987ff5ccc1a-metrics-certs\") pod \"maas-api-e2e-api-a-2d319a-bdff7977-7zzh4\" (UID: \"92811d3a-a4be-49a8-a815-d987ff5ccc1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2d319a-bdff7977-7zzh4"
Oct 31 19:41:16.411116 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:16.411091    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92811d3a-a4be-49a8-a815-d987ff5ccc1a-openshift-service-ca\") pod \"maas-api-e2e-api-a-2d319a-bdff7977-7zzh4\" (UID: \"92811d3a-a4be-49a8-a815-d987ff5ccc1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2d319a-bdff7977-7zzh4"
Oct 31 19:41:16.411242 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:16.411155    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92811d3a-a4be-49a8-a815-d987ff5ccc1a-maas-api-tls\") pod \"maas-api-e2e-api-a-2d319a-bdff7977-7zzh4\" (UID: \"92811d3a-a4be-49a8-a815-d987ff5ccc1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2d319a-bdff7977-7zzh4"
Oct 31 19:41:16.411242 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:16.411196    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r4dr\" (UniqueName: \"kubernetes.io/projected/92811d3a-a4be-49a8-a815-d987ff5ccc1a-kube-api-access-6r4dr\") pod \"maas-api-e2e-api-a-2d319a-bdff7977-7zzh4\" (UID: \"92811d3a-a4be-49a8-a815-d987ff5ccc1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2d319a-bdff7977-7zzh4"
Oct 31 19:41:16.499123 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:16.499029    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:16.530048 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:16.522946    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92811d3a-a4be-49a8-a815-d987ff5ccc1a-openshift-service-ca\") pod \"maas-api-e2e-api-a-2d319a-bdff7977-7zzh4\" (UID: \"92811d3a-a4be-49a8-a815-d987ff5ccc1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2d319a-bdff7977-7zzh4"
Oct 31 19:41:16.530524 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:16.530480    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92811d3a-a4be-49a8-a815-d987ff5ccc1a-maas-api-tls\") pod \"maas-api-e2e-api-a-2d319a-bdff7977-7zzh4\" (UID: \"92811d3a-a4be-49a8-a815-d987ff5ccc1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2d319a-bdff7977-7zzh4"
Oct 31 19:41:16.538809 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:16.538749    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92811d3a-a4be-49a8-a815-d987ff5ccc1a-openshift-service-ca\") pod \"maas-api-e2e-api-a-2d319a-bdff7977-7zzh4\" (UID: \"92811d3a-a4be-49a8-a815-d987ff5ccc1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2d319a-bdff7977-7zzh4"
Oct 31 19:41:16.548468 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:16.548433    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r4dr\" (UniqueName: \"kubernetes.io/projected/92811d3a-a4be-49a8-a815-d987ff5ccc1a-kube-api-access-6r4dr\") pod \"maas-api-e2e-api-a-2d319a-bdff7977-7zzh4\" (UID: \"92811d3a-a4be-49a8-a815-d987ff5ccc1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2d319a-bdff7977-7zzh4"
Oct 31 19:41:16.551172 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:16.550480    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92811d3a-a4be-49a8-a815-d987ff5ccc1a-metrics-certs\") pod \"maas-api-e2e-api-a-2d319a-bdff7977-7zzh4\" (UID: \"92811d3a-a4be-49a8-a815-d987ff5ccc1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2d319a-bdff7977-7zzh4"
Oct 31 19:41:16.563959 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:16.562996    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92811d3a-a4be-49a8-a815-d987ff5ccc1a-metrics-certs\") pod \"maas-api-e2e-api-a-2d319a-bdff7977-7zzh4\" (UID: \"92811d3a-a4be-49a8-a815-d987ff5ccc1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2d319a-bdff7977-7zzh4"
Oct 31 19:41:16.573484 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:16.573407    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r4dr\" (UniqueName: \"kubernetes.io/projected/92811d3a-a4be-49a8-a815-d987ff5ccc1a-kube-api-access-6r4dr\") pod \"maas-api-e2e-api-a-2d319a-bdff7977-7zzh4\" (UID: \"92811d3a-a4be-49a8-a815-d987ff5ccc1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2d319a-bdff7977-7zzh4"
Oct 31 19:41:16.576860 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:16.576747    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92811d3a-a4be-49a8-a815-d987ff5ccc1a-maas-api-tls\") pod \"maas-api-e2e-api-a-2d319a-bdff7977-7zzh4\" (UID: \"92811d3a-a4be-49a8-a815-d987ff5ccc1a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2d319a-bdff7977-7zzh4"
Oct 31 19:41:16.638039 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:16.637484    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz"
Oct 31 19:41:16.712364 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:16.712231    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2d319a-bdff7977-7zzh4"
Oct 31 19:41:17.160962 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:17.156562    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm"
Oct 31 19:41:17.160962 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:17.157168    4645 patch_prober.go:29] interesting pod/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Oct 31 19:41:17.160962 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:17.157227    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm" podUID="940375be-4b29-43ca-b0de-1643eaa82341" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Oct 31 19:41:17.436596 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:17.436545    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:17.512776 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:17.512743    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2d319a-bdff7977-7zzh4"]
Oct 31 19:41:17.513921 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:41:17.513886    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92811d3a_a4be_49a8_a815_d987ff5ccc1a.slice/crio-ad85df88458b605a304d06e26eebf070f26f07ca9d58382d92252301d2c8487d WatchSource:0}: Error finding container ad85df88458b605a304d06e26eebf070f26f07ca9d58382d92252301d2c8487d: Status 404 returned error can't find the container with id ad85df88458b605a304d06e26eebf070f26f07ca9d58382d92252301d2c8487d
Oct 31 19:41:17.635955 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:17.635931    4645 patch_prober.go:29] interesting pod/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": read tcp 10.133.0.2:56576->10.133.0.105:15021: read: connection reset by peer" start-of-body=
Oct 31 19:41:17.636218 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:17.636190    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c" podUID="29a9754a-f686-404c-b428-a3b20ba0aa75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": read tcp 10.133.0.2:56576->10.133.0.105:15021: read: connection reset by peer"
Oct 31 19:41:17.637451 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:17.637289    4645 patch_prober.go:29] interesting pod/e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Oct 31 19:41:17.637451 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:17.637400    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz" podUID="79f81748-b7d5-4ed1-8f28-4386bba24a79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Oct 31 19:41:17.749401 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:17.742446    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:18.003401 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:17.980137    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:18.125359 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:18.125309    4645 patch_prober.go:29] interesting pod/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Oct 31 19:41:18.125535 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:18.125487    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm" podUID="940375be-4b29-43ca-b0de-1643eaa82341" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Oct 31 19:41:18.436090 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:18.436057    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2d319a-bdff7977-7zzh4" event={"ID":"92811d3a-a4be-49a8-a815-d987ff5ccc1a","Type":"ContainerStarted","Data":"41cd9734f12733f63edc7d14cd795477eddfbed3f86a8989f786f0fc29e7dbc7"}
Oct 31 19:41:18.436542 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:18.436110    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2d319a-bdff7977-7zzh4" event={"ID":"92811d3a-a4be-49a8-a815-d987ff5ccc1a","Type":"ContainerStarted","Data":"ad85df88458b605a304d06e26eebf070f26f07ca9d58382d92252301d2c8487d"}
Oct 31 19:41:18.436542 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:18.436439    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2d319a-bdff7977-7zzh4"
Oct 31 19:41:18.437994 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:18.437974    4645 generic.go:356] "Generic (PLEG): container finished" podID="29a9754a-f686-404c-b428-a3b20ba0aa75" containerID="f64fb7efbcea841caf806db1831039c30ba96412296c4726fdb2790508abea98" exitCode=0
Oct 31 19:41:18.438258 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:18.438238    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c" event={"ID":"29a9754a-f686-404c-b428-a3b20ba0aa75","Type":"ContainerDied","Data":"f64fb7efbcea841caf806db1831039c30ba96412296c4726fdb2790508abea98"}
Oct 31 19:41:18.438607 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:18.438589    4645 scope.go:122] "RemoveContainer" containerID="f64fb7efbcea841caf806db1831039c30ba96412296c4726fdb2790508abea98"
Oct 31 19:41:18.440180 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:18.440138    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:41:18.440268 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:18.440216    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:41:18.440268 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:18.440250    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:41:18.504965 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:18.504871    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2d319a-bdff7977-7zzh4" podStartSLOduration=2.014290997 podStartE2EDuration="2.504852568s" totalImagesPullingTime="490.561571ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:41:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:41:18.503711666 +0000 UTC m=+1309.076945796" watchObservedRunningTime="2026-10-31 19:41:18.504852568 +0000 UTC m=+1309.078086697"
Oct 31 19:41:18.635951 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:18.635847    4645 patch_prober.go:29] interesting pod/e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Oct 31 19:41:18.635951 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:18.635901    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz" podUID="79f81748-b7d5-4ed1-8f28-4386bba24a79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Oct 31 19:41:18.716408 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:18.712099    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:19.127387 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:19.127275    4645 patch_prober.go:29] interesting pod/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Oct 31 19:41:19.127387 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:19.127353    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm" podUID="940375be-4b29-43ca-b0de-1643eaa82341" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Oct 31 19:41:19.449030 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:19.448997    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c" event={"ID":"29a9754a-f686-404c-b428-a3b20ba0aa75","Type":"ContainerStarted","Data":"c9eb86dec8f35ac16479d9aa626e092211036e216812eb26aea6b69a9231c9d4"}
Oct 31 19:41:19.622573 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:19.622541    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422"]
Oct 31 19:41:19.622757 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:19.622717    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422" podUID="2bc08948-e33b-4897-b931-021255991026" containerName="istio-proxy" containerID="cri-o://62b3fbf61bfd0d8884480ba17cffc179991acb864b1dd1e4d6ab09ff82b2ac72" gracePeriod=30
Oct 31 19:41:19.635871 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:19.635596    4645 patch_prober.go:29] interesting pod/e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Oct 31 19:41:19.635871 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:19.635644    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz" podUID="79f81748-b7d5-4ed1-8f28-4386bba24a79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Oct 31 19:41:19.900528 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:19.900432    4645 patch_prober.go:29] interesting pod/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 31 19:41:19.900906 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:19.900876    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c" podUID="29a9754a-f686-404c-b428-a3b20ba0aa75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 31 19:41:19.922832 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:19.900775    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c"
Oct 31 19:41:19.950440 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:19.947478    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:20.127353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:20.126745    4645 patch_prober.go:29] interesting pod/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Oct 31 19:41:20.127353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:20.126812    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm" podUID="940375be-4b29-43ca-b0de-1643eaa82341" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Oct 31 19:41:20.635834 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:20.635799    4645 patch_prober.go:29] interesting pod/e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Oct 31 19:41:20.636605 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:20.636437    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz" podUID="79f81748-b7d5-4ed1-8f28-4386bba24a79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Oct 31 19:41:20.847011 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:20.846968    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:20.924635 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:20.924598    4645 patch_prober.go:29] interesting pod/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 31 19:41:20.936915 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:20.936878    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c" podUID="29a9754a-f686-404c-b428-a3b20ba0aa75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 31 19:41:21.071355 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:21.066830    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:21.136292 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:21.136243    4645 patch_prober.go:29] interesting pod/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Oct 31 19:41:21.150516 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:21.150461    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm" podUID="940375be-4b29-43ca-b0de-1643eaa82341" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Oct 31 19:41:21.592373 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:21.585838    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:21.664389 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:21.641283    4645 patch_prober.go:29] interesting pod/e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Oct 31 19:41:21.664389 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:21.641358    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz" podUID="79f81748-b7d5-4ed1-8f28-4386bba24a79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Oct 31 19:41:21.948386 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:21.929809    4645 patch_prober.go:29] interesting pod/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 31 19:41:21.948386 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:21.929867    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c" podUID="29a9754a-f686-404c-b428-a3b20ba0aa75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 31 19:41:22.125590 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:22.125546    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:22.129361 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:22.126658    4645 patch_prober.go:29] interesting pod/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Oct 31 19:41:22.129361 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:22.126711    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm" podUID="940375be-4b29-43ca-b0de-1643eaa82341" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Oct 31 19:41:22.267405 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:22.262936    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:22.535413 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:22.523820    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:22.699353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:22.674159    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:23.037608 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:22.752228    4645 patch_prober.go:29] interesting pod/e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Oct 31 19:41:23.037890 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:23.037849    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz" podUID="79f81748-b7d5-4ed1-8f28-4386bba24a79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Oct 31 19:41:23.038240 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:23.029288    4645 patch_prober.go:29] interesting pod/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 31 19:41:23.038484 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:23.038274    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c" podUID="29a9754a-f686-404c-b428-a3b20ba0aa75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 31 19:41:23.040699 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:23.039989    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg"]
Oct 31 19:41:23.040699 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:23.040288    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg" podUID="9bacc59a-7abc-43b3-81dd-a8829710ad35" containerName="istio-proxy" containerID="cri-o://b8dce505a1fc1201cc5c6775c022b0b334366af36acbbff66fd883f23b33be5f" gracePeriod=30
Oct 31 19:41:23.054047 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:23.054021    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7"]
Oct 31 19:41:23.054505 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:23.054459    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7" podUID="fd915c45-10cb-4d0f-a519-44bbe338e43d" containerName="istio-proxy" containerID="cri-o://738c6800661a462c22ac22e82d1ec79ffa27edcd8faf80becbf78ae8e49c0200" gracePeriod=30
Oct 31 19:41:23.152399 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:23.143499    4645 patch_prober.go:29] interesting pod/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Oct 31 19:41:23.190350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:23.168581    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm" podUID="940375be-4b29-43ca-b0de-1643eaa82341" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Oct 31 19:41:23.190350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:23.172872    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt"]
Oct 31 19:41:23.190350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:23.173107    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt" podUID="4abbcb7a-0390-4ed4-8d3b-468921489dc9" containerName="istio-proxy" containerID="cri-o://d62594b322266a964f7e928ec70564d883404edd2ae582b721e271ffbc01f14f" gracePeriod=30
Oct 31 19:41:23.460541 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:23.460319    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:23.659353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:23.638423    4645 patch_prober.go:29] interesting pod/e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body=
Oct 31 19:41:23.659353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:23.638487    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz" podUID="79f81748-b7d5-4ed1-8f28-4386bba24a79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused"
Oct 31 19:41:23.898910 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:23.898820    4645 patch_prober.go:29] interesting pod/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 31 19:41:23.898910 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:23.898889    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c" podUID="29a9754a-f686-404c-b428-a3b20ba0aa75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 31 19:41:24.145351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:24.145082    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm"
Oct 31 19:41:24.146197 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:24.146150    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm"
Oct 31 19:41:24.147276 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:24.147254    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm"
Oct 31 19:41:24.454561 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:24.454531    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2d319a-bdff7977-7zzh4"
Oct 31 19:41:24.640427 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:24.640400    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz"
Oct 31 19:41:24.640866 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:24.640841    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz"
Oct 31 19:41:24.642130 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:24.642107    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz"
Oct 31 19:41:24.897792 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:24.896782    4645 patch_prober.go:29] interesting pod/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 31 19:41:24.897792 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:24.896840    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c" podUID="29a9754a-f686-404c-b428-a3b20ba0aa75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 31 19:41:25.017971 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:25.017944    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422"
Oct 31 19:41:25.111352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:25.110975    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:25.133841 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:25.133522    4645 generic.go:356] "Generic (PLEG): container finished" podID="2bc08948-e33b-4897-b931-021255991026" containerID="62b3fbf61bfd0d8884480ba17cffc179991acb864b1dd1e4d6ab09ff82b2ac72" exitCode=0
Oct 31 19:41:25.135710 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:25.134493    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422"
Oct 31 19:41:25.135710 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:25.135408    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422" event={"ID":"2bc08948-e33b-4897-b931-021255991026","Type":"ContainerDied","Data":"62b3fbf61bfd0d8884480ba17cffc179991acb864b1dd1e4d6ab09ff82b2ac72"}
Oct 31 19:41:25.135710 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:25.135516    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422" event={"ID":"2bc08948-e33b-4897-b931-021255991026","Type":"ContainerDied","Data":"543245d4d93f5662f8b4b70791a55b493a8893a7734997b0ad3cb21fd821d5f0"}
Oct 31 19:41:25.135710 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:25.135550    4645 scope.go:122] "RemoveContainer" containerID="62b3fbf61bfd0d8884480ba17cffc179991acb864b1dd1e4d6ab09ff82b2ac72"
Oct 31 19:41:25.143373 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:25.143349    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2bc08948-e33b-4897-b931-021255991026-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bc08948-e33b-4897-b931-021255991026-istio-podinfo\") pod \"2bc08948-e33b-4897-b931-021255991026\" (UID: \"2bc08948-e33b-4897-b931-021255991026\") "
Oct 31 19:41:25.143512 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:25.143383    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bc08948-e33b-4897-b931-021255991026-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bc08948-e33b-4897-b931-021255991026-istiod-ca-cert\") pod \"2bc08948-e33b-4897-b931-021255991026\" (UID: \"2bc08948-e33b-4897-b931-021255991026\") "
Oct 31 19:41:25.143512 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:25.143451    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bc08948-e33b-4897-b931-021255991026-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bc08948-e33b-4897-b931-021255991026-workload-socket\") pod \"2bc08948-e33b-4897-b931-021255991026\" (UID: \"2bc08948-e33b-4897-b931-021255991026\") "
Oct 31 19:41:25.143512 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:25.143502    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bc08948-e33b-4897-b931-021255991026-kube-api-access-2xfwd\" (UniqueName: \"kubernetes.io/projected/2bc08948-e33b-4897-b931-021255991026-kube-api-access-2xfwd\") pod \"2bc08948-e33b-4897-b931-021255991026\" (UID: \"2bc08948-e33b-4897-b931-021255991026\") "
Oct 31 19:41:25.143720 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:25.143520    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bc08948-e33b-4897-b931-021255991026-istio-token\" (UniqueName: \"kubernetes.io/projected/2bc08948-e33b-4897-b931-021255991026-istio-token\") pod \"2bc08948-e33b-4897-b931-021255991026\" (UID: \"2bc08948-e33b-4897-b931-021255991026\") "
Oct 31 19:41:25.143720 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:25.143618    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bc08948-e33b-4897-b931-021255991026-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bc08948-e33b-4897-b931-021255991026-credential-socket\") pod \"2bc08948-e33b-4897-b931-021255991026\" (UID: \"2bc08948-e33b-4897-b931-021255991026\") "
Oct 31 19:41:25.143720 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:25.143714    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bc08948-e33b-4897-b931-021255991026-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bc08948-e33b-4897-b931-021255991026-istio-envoy\") pod \"2bc08948-e33b-4897-b931-021255991026\" (UID: \"2bc08948-e33b-4897-b931-021255991026\") "
Oct 31 19:41:25.143904 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:25.143791    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bc08948-e33b-4897-b931-021255991026-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bc08948-e33b-4897-b931-021255991026-istio-data\") pod \"2bc08948-e33b-4897-b931-021255991026\" (UID: \"2bc08948-e33b-4897-b931-021255991026\") "
Oct 31 19:41:25.143904 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:25.143844    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bc08948-e33b-4897-b931-021255991026-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bc08948-e33b-4897-b931-021255991026-workload-certs\") pod \"2bc08948-e33b-4897-b931-021255991026\" (UID: \"2bc08948-e33b-4897-b931-021255991026\") "
Oct 31 19:41:25.144019 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:25.143925    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bc08948-e33b-4897-b931-021255991026-workload-socket" pod "2bc08948-e33b-4897-b931-021255991026" (UID: "2bc08948-e33b-4897-b931-021255991026"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:25.144273 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:25.144246    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bc08948-e33b-4897-b931-021255991026-istiod-ca-cert" pod "2bc08948-e33b-4897-b931-021255991026" (UID: "2bc08948-e33b-4897-b931-021255991026"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:41:25.144577 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:25.144552    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bc08948-e33b-4897-b931-021255991026-workload-certs" pod "2bc08948-e33b-4897-b931-021255991026" (UID: "2bc08948-e33b-4897-b931-021255991026"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:25.145063 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:25.144778    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bc08948-e33b-4897-b931-021255991026-credential-socket" pod "2bc08948-e33b-4897-b931-021255991026" (UID: "2bc08948-e33b-4897-b931-021255991026"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:25.145063 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:25.144839    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bc08948-e33b-4897-b931-021255991026-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:25.145063 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:25.144859    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bc08948-e33b-4897-b931-021255991026-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:25.145063 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:25.144873    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bc08948-e33b-4897-b931-021255991026-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:25.146662 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:25.145462    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bc08948-e33b-4897-b931-021255991026-istio-data" pod "2bc08948-e33b-4897-b931-021255991026" (UID: "2bc08948-e33b-4897-b931-021255991026"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:25.149896 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:25.149877    4645 scope.go:122] "RemoveContainer" containerID="62b3fbf61bfd0d8884480ba17cffc179991acb864b1dd1e4d6ab09ff82b2ac72"
Oct 31 19:41:25.150139 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:25.150119    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bc08948-e33b-4897-b931-021255991026-kube-api-access-2xfwd" pod "2bc08948-e33b-4897-b931-021255991026" (UID: "2bc08948-e33b-4897-b931-021255991026"). InnerVolumeSpecName "kube-api-access-2xfwd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:41:25.150322 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:41:25.150293    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62b3fbf61bfd0d8884480ba17cffc179991acb864b1dd1e4d6ab09ff82b2ac72\": container with ID starting with 62b3fbf61bfd0d8884480ba17cffc179991acb864b1dd1e4d6ab09ff82b2ac72 not found: ID does not exist" containerID="62b3fbf61bfd0d8884480ba17cffc179991acb864b1dd1e4d6ab09ff82b2ac72"
Oct 31 19:41:25.150449 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:25.150367    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62b3fbf61bfd0d8884480ba17cffc179991acb864b1dd1e4d6ab09ff82b2ac72"} err="failed to get container status \"62b3fbf61bfd0d8884480ba17cffc179991acb864b1dd1e4d6ab09ff82b2ac72\": rpc error: code = NotFound desc = could not find container \"62b3fbf61bfd0d8884480ba17cffc179991acb864b1dd1e4d6ab09ff82b2ac72\": container with ID starting with 62b3fbf61bfd0d8884480ba17cffc179991acb864b1dd1e4d6ab09ff82b2ac72 not found: ID does not exist"
Oct 31 19:41:25.150968 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:25.150940    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bc08948-e33b-4897-b931-021255991026-istio-envoy" pod "2bc08948-e33b-4897-b931-021255991026" (UID: "2bc08948-e33b-4897-b931-021255991026"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:25.152116 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:25.152084    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2bc08948-e33b-4897-b931-021255991026-istio-podinfo" pod "2bc08948-e33b-4897-b931-021255991026" (UID: "2bc08948-e33b-4897-b931-021255991026"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:41:25.153629 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:25.153601    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bc08948-e33b-4897-b931-021255991026-istio-token" pod "2bc08948-e33b-4897-b931-021255991026" (UID: "2bc08948-e33b-4897-b931-021255991026"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:41:25.247792 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:25.247711    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xfwd\" (UniqueName: \"kubernetes.io/projected/2bc08948-e33b-4897-b931-021255991026-kube-api-access-2xfwd\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:25.247792 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:25.247737    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bc08948-e33b-4897-b931-021255991026-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:25.247792 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:25.247751    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bc08948-e33b-4897-b931-021255991026-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:25.247792 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:25.247763    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bc08948-e33b-4897-b931-021255991026-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:25.247792 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:25.247774    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bc08948-e33b-4897-b931-021255991026-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:25.247792 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:25.247789    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bc08948-e33b-4897-b931-021255991026-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:25.488021 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:25.487986    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422"]
Oct 31 19:41:25.492130 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:25.492107    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-qv422"]
Oct 31 19:41:25.622680 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:25.622601    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bc08948-e33b-4897-b931-021255991026" path="/var/lib/kubelet/pods/2bc08948-e33b-4897-b931-021255991026/volumes"
Oct 31 19:41:25.846962 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:25.846924    4645 patch_prober.go:29] interesting pod/e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 19:41:25.847168 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:25.846986    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt" podUID="4abbcb7a-0390-4ed4-8d3b-468921489dc9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 19:41:25.896942 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:25.896866    4645 patch_prober.go:29] interesting pod/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 31 19:41:25.896942 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:25.896916    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c" podUID="29a9754a-f686-404c-b428-a3b20ba0aa75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 31 19:41:26.500353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:26.491573    4645 patch_prober.go:29] interesting pod/e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 19:41:26.500353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:26.491637    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg" podUID="9bacc59a-7abc-43b3-81dd-a8829710ad35" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 19:41:26.901501 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:26.901418    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c"
Oct 31 19:41:26.901802 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:26.901780    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c"
Oct 31 19:41:26.902515 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:26.902492    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c"
Oct 31 19:41:26.948337 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:26.948306    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-88eaf9-9988c7848-fqszv"]
Oct 31 19:41:26.949218 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:26.949197    4645 cpu_manager.go:414] "containerMap: removing container" podUID="2bc08948-e33b-4897-b931-021255991026" containerName="istio-proxy"
Oct 31 19:41:26.949218 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:26.949217    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bc08948-e33b-4897-b931-021255991026" containerName="istio-proxy"
Oct 31 19:41:26.949401 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:26.949384    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bc08948-e33b-4897-b931-021255991026" containerName="istio-proxy"
Oct 31 19:41:26.954601 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:26.954529    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-88eaf9-9988c7848-fqszv"
Oct 31 19:41:26.962641 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:26.962619    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-88eaf9-serving-cert\""
Oct 31 19:41:26.995284 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:26.995252    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-88eaf9-9988c7848-fqszv"]
Oct 31 19:41:27.034375 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:27.034346    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:27.048590 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:27.048559    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4abee805-15d3-4b54-bd38-889f562f0c57-maas-api-tls\") pod \"maas-api-e2e-inf-b-88eaf9-9988c7848-fqszv\" (UID: \"4abee805-15d3-4b54-bd38-889f562f0c57\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-88eaf9-9988c7848-fqszv"
Oct 31 19:41:27.048743 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:27.048725    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4abee805-15d3-4b54-bd38-889f562f0c57-metrics-certs\") pod \"maas-api-e2e-inf-b-88eaf9-9988c7848-fqszv\" (UID: \"4abee805-15d3-4b54-bd38-889f562f0c57\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-88eaf9-9988c7848-fqszv"
Oct 31 19:41:27.048813 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:27.048746    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4abee805-15d3-4b54-bd38-889f562f0c57-openshift-service-ca\") pod \"maas-api-e2e-inf-b-88eaf9-9988c7848-fqszv\" (UID: \"4abee805-15d3-4b54-bd38-889f562f0c57\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-88eaf9-9988c7848-fqszv"
Oct 31 19:41:27.049061 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:27.048847    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nknl4\" (UniqueName: \"kubernetes.io/projected/4abee805-15d3-4b54-bd38-889f562f0c57-kube-api-access-nknl4\") pod \"maas-api-e2e-inf-b-88eaf9-9988c7848-fqszv\" (UID: \"4abee805-15d3-4b54-bd38-889f562f0c57\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-88eaf9-9988c7848-fqszv"
Oct 31 19:41:27.150439 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:27.150402    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4abee805-15d3-4b54-bd38-889f562f0c57-metrics-certs\") pod \"maas-api-e2e-inf-b-88eaf9-9988c7848-fqszv\" (UID: \"4abee805-15d3-4b54-bd38-889f562f0c57\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-88eaf9-9988c7848-fqszv"
Oct 31 19:41:27.150609 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:27.150454    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4abee805-15d3-4b54-bd38-889f562f0c57-openshift-service-ca\") pod \"maas-api-e2e-inf-b-88eaf9-9988c7848-fqszv\" (UID: \"4abee805-15d3-4b54-bd38-889f562f0c57\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-88eaf9-9988c7848-fqszv"
Oct 31 19:41:27.150609 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:27.150591    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nknl4\" (UniqueName: \"kubernetes.io/projected/4abee805-15d3-4b54-bd38-889f562f0c57-kube-api-access-nknl4\") pod \"maas-api-e2e-inf-b-88eaf9-9988c7848-fqszv\" (UID: \"4abee805-15d3-4b54-bd38-889f562f0c57\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-88eaf9-9988c7848-fqszv"
Oct 31 19:41:27.150796 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:27.150776    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4abee805-15d3-4b54-bd38-889f562f0c57-maas-api-tls\") pod \"maas-api-e2e-inf-b-88eaf9-9988c7848-fqszv\" (UID: \"4abee805-15d3-4b54-bd38-889f562f0c57\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-88eaf9-9988c7848-fqszv"
Oct 31 19:41:27.151238 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:27.151205    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4abee805-15d3-4b54-bd38-889f562f0c57-openshift-service-ca\") pod \"maas-api-e2e-inf-b-88eaf9-9988c7848-fqszv\" (UID: \"4abee805-15d3-4b54-bd38-889f562f0c57\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-88eaf9-9988c7848-fqszv"
Oct 31 19:41:27.153362 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:27.153291    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4abee805-15d3-4b54-bd38-889f562f0c57-maas-api-tls\") pod \"maas-api-e2e-inf-b-88eaf9-9988c7848-fqszv\" (UID: \"4abee805-15d3-4b54-bd38-889f562f0c57\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-88eaf9-9988c7848-fqszv"
Oct 31 19:41:27.153500 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:27.153480    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4abee805-15d3-4b54-bd38-889f562f0c57-metrics-certs\") pod \"maas-api-e2e-inf-b-88eaf9-9988c7848-fqszv\" (UID: \"4abee805-15d3-4b54-bd38-889f562f0c57\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-88eaf9-9988c7848-fqszv"
Oct 31 19:41:27.169858 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:27.169833    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nknl4\" (UniqueName: \"kubernetes.io/projected/4abee805-15d3-4b54-bd38-889f562f0c57-kube-api-access-nknl4\") pod \"maas-api-e2e-inf-b-88eaf9-9988c7848-fqszv\" (UID: \"4abee805-15d3-4b54-bd38-889f562f0c57\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-88eaf9-9988c7848-fqszv"
Oct 31 19:41:27.231258 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:27.231213    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:27.297478 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:27.284231    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-88eaf9-9988c7848-fqszv"
Oct 31 19:41:27.353975 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:27.353935    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:27.678673 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:27.668094    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:27.717793 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:27.717761    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-88eaf9-9988c7848-fqszv"]
Oct 31 19:41:28.080644 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.067203    4645 patch_prober.go:29] interesting pod/e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 19:41:28.080644 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.067267    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7" podUID="fd915c45-10cb-4d0f-a519-44bbe338e43d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 19:41:28.318207 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.317708    4645 generic.go:356] "Generic (PLEG): container finished" podID="fd915c45-10cb-4d0f-a519-44bbe338e43d" containerID="738c6800661a462c22ac22e82d1ec79ffa27edcd8faf80becbf78ae8e49c0200" exitCode=0
Oct 31 19:41:28.318207 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.317778    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7" event={"ID":"fd915c45-10cb-4d0f-a519-44bbe338e43d","Type":"ContainerDied","Data":"738c6800661a462c22ac22e82d1ec79ffa27edcd8faf80becbf78ae8e49c0200"}
Oct 31 19:41:28.322822 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.319601    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-88eaf9-9988c7848-fqszv" event={"ID":"4abee805-15d3-4b54-bd38-889f562f0c57","Type":"ContainerStarted","Data":"7a430cffc4e88ccce6a67d0dbaf4dfdaa1de7f714c09b19a4daa39048077e0a6"}
Oct 31 19:41:28.322822 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.321240    4645 generic.go:356] "Generic (PLEG): container finished" podID="9bacc59a-7abc-43b3-81dd-a8829710ad35" containerID="b8dce505a1fc1201cc5c6775c022b0b334366af36acbbff66fd883f23b33be5f" exitCode=0
Oct 31 19:41:28.322822 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.321402    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg" event={"ID":"9bacc59a-7abc-43b3-81dd-a8829710ad35","Type":"ContainerDied","Data":"b8dce505a1fc1201cc5c6775c022b0b334366af36acbbff66fd883f23b33be5f"}
Oct 31 19:41:28.498443 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.498417    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg"
Oct 31 19:41:28.518709 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.518676    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bacc59a-7abc-43b3-81dd-a8829710ad35-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bacc59a-7abc-43b3-81dd-a8829710ad35-credential-socket\") pod \"9bacc59a-7abc-43b3-81dd-a8829710ad35\" (UID: \"9bacc59a-7abc-43b3-81dd-a8829710ad35\") "
Oct 31 19:41:28.518880 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.518742    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bacc59a-7abc-43b3-81dd-a8829710ad35-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bacc59a-7abc-43b3-81dd-a8829710ad35-istio-envoy\") pod \"9bacc59a-7abc-43b3-81dd-a8829710ad35\" (UID: \"9bacc59a-7abc-43b3-81dd-a8829710ad35\") "
Oct 31 19:41:28.518880 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.518786    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bacc59a-7abc-43b3-81dd-a8829710ad35-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bacc59a-7abc-43b3-81dd-a8829710ad35-istio-data\") pod \"9bacc59a-7abc-43b3-81dd-a8829710ad35\" (UID: \"9bacc59a-7abc-43b3-81dd-a8829710ad35\") "
Oct 31 19:41:28.519004 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.518895    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bacc59a-7abc-43b3-81dd-a8829710ad35-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bacc59a-7abc-43b3-81dd-a8829710ad35-workload-certs\") pod \"9bacc59a-7abc-43b3-81dd-a8829710ad35\" (UID: \"9bacc59a-7abc-43b3-81dd-a8829710ad35\") "
Oct 31 19:41:28.519004 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.518966    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bacc59a-7abc-43b3-81dd-a8829710ad35-credential-socket" pod "9bacc59a-7abc-43b3-81dd-a8829710ad35" (UID: "9bacc59a-7abc-43b3-81dd-a8829710ad35"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:28.519119 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.519004    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9bacc59a-7abc-43b3-81dd-a8829710ad35-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bacc59a-7abc-43b3-81dd-a8829710ad35-istio-podinfo\") pod \"9bacc59a-7abc-43b3-81dd-a8829710ad35\" (UID: \"9bacc59a-7abc-43b3-81dd-a8829710ad35\") "
Oct 31 19:41:28.519177 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.519119    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bacc59a-7abc-43b3-81dd-a8829710ad35-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bacc59a-7abc-43b3-81dd-a8829710ad35-istiod-ca-cert\") pod \"9bacc59a-7abc-43b3-81dd-a8829710ad35\" (UID: \"9bacc59a-7abc-43b3-81dd-a8829710ad35\") "
Oct 31 19:41:28.519236 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.519200    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bacc59a-7abc-43b3-81dd-a8829710ad35-kube-api-access-wt4fl\" (UniqueName: \"kubernetes.io/projected/9bacc59a-7abc-43b3-81dd-a8829710ad35-kube-api-access-wt4fl\") pod \"9bacc59a-7abc-43b3-81dd-a8829710ad35\" (UID: \"9bacc59a-7abc-43b3-81dd-a8829710ad35\") "
Oct 31 19:41:28.519291 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.519246    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bacc59a-7abc-43b3-81dd-a8829710ad35-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bacc59a-7abc-43b3-81dd-a8829710ad35-workload-socket\") pod \"9bacc59a-7abc-43b3-81dd-a8829710ad35\" (UID: \"9bacc59a-7abc-43b3-81dd-a8829710ad35\") "
Oct 31 19:41:28.519422 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.519385    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bacc59a-7abc-43b3-81dd-a8829710ad35-istio-token\" (UniqueName: \"kubernetes.io/projected/9bacc59a-7abc-43b3-81dd-a8829710ad35-istio-token\") pod \"9bacc59a-7abc-43b3-81dd-a8829710ad35\" (UID: \"9bacc59a-7abc-43b3-81dd-a8829710ad35\") "
Oct 31 19:41:28.519525 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.519456    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bacc59a-7abc-43b3-81dd-a8829710ad35-workload-certs" pod "9bacc59a-7abc-43b3-81dd-a8829710ad35" (UID: "9bacc59a-7abc-43b3-81dd-a8829710ad35"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:28.519661 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.519642    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bacc59a-7abc-43b3-81dd-a8829710ad35-istio-data" pod "9bacc59a-7abc-43b3-81dd-a8829710ad35" (UID: "9bacc59a-7abc-43b3-81dd-a8829710ad35"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:28.520277 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.520249    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bacc59a-7abc-43b3-81dd-a8829710ad35-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:28.520277 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.520279    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bacc59a-7abc-43b3-81dd-a8829710ad35-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:28.520277 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.520294    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bacc59a-7abc-43b3-81dd-a8829710ad35-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:28.520792 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.520557    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bacc59a-7abc-43b3-81dd-a8829710ad35-workload-socket" pod "9bacc59a-7abc-43b3-81dd-a8829710ad35" (UID: "9bacc59a-7abc-43b3-81dd-a8829710ad35"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:28.522480 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.522433    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bacc59a-7abc-43b3-81dd-a8829710ad35-istiod-ca-cert" pod "9bacc59a-7abc-43b3-81dd-a8829710ad35" (UID: "9bacc59a-7abc-43b3-81dd-a8829710ad35"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:41:28.525613 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.525571    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bacc59a-7abc-43b3-81dd-a8829710ad35-istio-token" pod "9bacc59a-7abc-43b3-81dd-a8829710ad35" (UID: "9bacc59a-7abc-43b3-81dd-a8829710ad35"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:41:28.532027 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.531988    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bacc59a-7abc-43b3-81dd-a8829710ad35-istio-envoy" pod "9bacc59a-7abc-43b3-81dd-a8829710ad35" (UID: "9bacc59a-7abc-43b3-81dd-a8829710ad35"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:28.540426 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.540389    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9bacc59a-7abc-43b3-81dd-a8829710ad35-istio-podinfo" pod "9bacc59a-7abc-43b3-81dd-a8829710ad35" (UID: "9bacc59a-7abc-43b3-81dd-a8829710ad35"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:41:28.554447 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.554392    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bacc59a-7abc-43b3-81dd-a8829710ad35-kube-api-access-wt4fl" pod "9bacc59a-7abc-43b3-81dd-a8829710ad35" (UID: "9bacc59a-7abc-43b3-81dd-a8829710ad35"). InnerVolumeSpecName "kube-api-access-wt4fl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:41:28.622087 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.622009    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bacc59a-7abc-43b3-81dd-a8829710ad35-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:28.622087 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.622038    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bacc59a-7abc-43b3-81dd-a8829710ad35-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:28.622087 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.622048    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wt4fl\" (UniqueName: \"kubernetes.io/projected/9bacc59a-7abc-43b3-81dd-a8829710ad35-kube-api-access-wt4fl\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:28.622087 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.622057    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bacc59a-7abc-43b3-81dd-a8829710ad35-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:28.622087 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.622068    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bacc59a-7abc-43b3-81dd-a8829710ad35-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:28.622087 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.622077    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bacc59a-7abc-43b3-81dd-a8829710ad35-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:28.641616 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.641595    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7"
Oct 31 19:41:28.723238 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.723202    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd915c45-10cb-4d0f-a519-44bbe338e43d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd915c45-10cb-4d0f-a519-44bbe338e43d-workload-certs\") pod \"fd915c45-10cb-4d0f-a519-44bbe338e43d\" (UID: \"fd915c45-10cb-4d0f-a519-44bbe338e43d\") "
Oct 31 19:41:28.723676 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.723283    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd915c45-10cb-4d0f-a519-44bbe338e43d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd915c45-10cb-4d0f-a519-44bbe338e43d-credential-socket\") pod \"fd915c45-10cb-4d0f-a519-44bbe338e43d\" (UID: \"fd915c45-10cb-4d0f-a519-44bbe338e43d\") "
Oct 31 19:41:28.723676 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.723356    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd915c45-10cb-4d0f-a519-44bbe338e43d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd915c45-10cb-4d0f-a519-44bbe338e43d-istiod-ca-cert\") pod \"fd915c45-10cb-4d0f-a519-44bbe338e43d\" (UID: \"fd915c45-10cb-4d0f-a519-44bbe338e43d\") "
Oct 31 19:41:28.723676 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.723459    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd915c45-10cb-4d0f-a519-44bbe338e43d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd915c45-10cb-4d0f-a519-44bbe338e43d-istio-data\") pod \"fd915c45-10cb-4d0f-a519-44bbe338e43d\" (UID: \"fd915c45-10cb-4d0f-a519-44bbe338e43d\") "
Oct 31 19:41:28.723676 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.723487    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd915c45-10cb-4d0f-a519-44bbe338e43d-workload-certs" pod "fd915c45-10cb-4d0f-a519-44bbe338e43d" (UID: "fd915c45-10cb-4d0f-a519-44bbe338e43d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:28.723676 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.723495    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd915c45-10cb-4d0f-a519-44bbe338e43d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd915c45-10cb-4d0f-a519-44bbe338e43d-istio-envoy\") pod \"fd915c45-10cb-4d0f-a519-44bbe338e43d\" (UID: \"fd915c45-10cb-4d0f-a519-44bbe338e43d\") "
Oct 31 19:41:28.723676 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.723621    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd915c45-10cb-4d0f-a519-44bbe338e43d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd915c45-10cb-4d0f-a519-44bbe338e43d-workload-socket\") pod \"fd915c45-10cb-4d0f-a519-44bbe338e43d\" (UID: \"fd915c45-10cb-4d0f-a519-44bbe338e43d\") "
Oct 31 19:41:28.723975 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.723792    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd915c45-10cb-4d0f-a519-44bbe338e43d-istio-token\" (UniqueName: \"kubernetes.io/projected/fd915c45-10cb-4d0f-a519-44bbe338e43d-istio-token\") pod \"fd915c45-10cb-4d0f-a519-44bbe338e43d\" (UID: \"fd915c45-10cb-4d0f-a519-44bbe338e43d\") "
Oct 31 19:41:28.723975 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.723888    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fd915c45-10cb-4d0f-a519-44bbe338e43d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd915c45-10cb-4d0f-a519-44bbe338e43d-istio-podinfo\") pod \"fd915c45-10cb-4d0f-a519-44bbe338e43d\" (UID: \"fd915c45-10cb-4d0f-a519-44bbe338e43d\") "
Oct 31 19:41:28.724085 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.724017    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd915c45-10cb-4d0f-a519-44bbe338e43d-kube-api-access-zctzl\" (UniqueName: \"kubernetes.io/projected/fd915c45-10cb-4d0f-a519-44bbe338e43d-kube-api-access-zctzl\") pod \"fd915c45-10cb-4d0f-a519-44bbe338e43d\" (UID: \"fd915c45-10cb-4d0f-a519-44bbe338e43d\") "
Oct 31 19:41:28.724424 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.724138    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd915c45-10cb-4d0f-a519-44bbe338e43d-istiod-ca-cert" pod "fd915c45-10cb-4d0f-a519-44bbe338e43d" (UID: "fd915c45-10cb-4d0f-a519-44bbe338e43d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:41:28.724940 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.724609    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd915c45-10cb-4d0f-a519-44bbe338e43d-credential-socket" pod "fd915c45-10cb-4d0f-a519-44bbe338e43d" (UID: "fd915c45-10cb-4d0f-a519-44bbe338e43d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:28.725394 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.725145    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd915c45-10cb-4d0f-a519-44bbe338e43d-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:28.725394 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.725172    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd915c45-10cb-4d0f-a519-44bbe338e43d-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:28.725394 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.725187    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd915c45-10cb-4d0f-a519-44bbe338e43d-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:28.725621 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.725465    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd915c45-10cb-4d0f-a519-44bbe338e43d-workload-socket" pod "fd915c45-10cb-4d0f-a519-44bbe338e43d" (UID: "fd915c45-10cb-4d0f-a519-44bbe338e43d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:28.725789 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.725745    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd915c45-10cb-4d0f-a519-44bbe338e43d-istio-data" pod "fd915c45-10cb-4d0f-a519-44bbe338e43d" (UID: "fd915c45-10cb-4d0f-a519-44bbe338e43d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:28.726732 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.726705    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd915c45-10cb-4d0f-a519-44bbe338e43d-istio-envoy" pod "fd915c45-10cb-4d0f-a519-44bbe338e43d" (UID: "fd915c45-10cb-4d0f-a519-44bbe338e43d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:28.727785 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.727759    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd915c45-10cb-4d0f-a519-44bbe338e43d-kube-api-access-zctzl" pod "fd915c45-10cb-4d0f-a519-44bbe338e43d" (UID: "fd915c45-10cb-4d0f-a519-44bbe338e43d"). InnerVolumeSpecName "kube-api-access-zctzl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:41:28.728527 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.728501    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd915c45-10cb-4d0f-a519-44bbe338e43d-istio-token" pod "fd915c45-10cb-4d0f-a519-44bbe338e43d" (UID: "fd915c45-10cb-4d0f-a519-44bbe338e43d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:41:28.728696 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.728674    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd915c45-10cb-4d0f-a519-44bbe338e43d-istio-podinfo" pod "fd915c45-10cb-4d0f-a519-44bbe338e43d" (UID: "fd915c45-10cb-4d0f-a519-44bbe338e43d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:41:28.813526 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.813495    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:28.816653 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.816624    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt"
Oct 31 19:41:28.826338 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.826287    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4abbcb7a-0390-4ed4-8d3b-468921489dc9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4abbcb7a-0390-4ed4-8d3b-468921489dc9-istio-podinfo\") pod \"4abbcb7a-0390-4ed4-8d3b-468921489dc9\" (UID: \"4abbcb7a-0390-4ed4-8d3b-468921489dc9\") "
Oct 31 19:41:28.826465 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.826409    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4abbcb7a-0390-4ed4-8d3b-468921489dc9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4abbcb7a-0390-4ed4-8d3b-468921489dc9-istiod-ca-cert\") pod \"4abbcb7a-0390-4ed4-8d3b-468921489dc9\" (UID: \"4abbcb7a-0390-4ed4-8d3b-468921489dc9\") "
Oct 31 19:41:28.826465 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.826452    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4abbcb7a-0390-4ed4-8d3b-468921489dc9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4abbcb7a-0390-4ed4-8d3b-468921489dc9-istio-data\") pod \"4abbcb7a-0390-4ed4-8d3b-468921489dc9\" (UID: \"4abbcb7a-0390-4ed4-8d3b-468921489dc9\") "
Oct 31 19:41:28.826569 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.826507    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4abbcb7a-0390-4ed4-8d3b-468921489dc9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4abbcb7a-0390-4ed4-8d3b-468921489dc9-istio-envoy\") pod \"4abbcb7a-0390-4ed4-8d3b-468921489dc9\" (UID: \"4abbcb7a-0390-4ed4-8d3b-468921489dc9\") "
Oct 31 19:41:28.826623 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.826574    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4abbcb7a-0390-4ed4-8d3b-468921489dc9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4abbcb7a-0390-4ed4-8d3b-468921489dc9-workload-socket\") pod \"4abbcb7a-0390-4ed4-8d3b-468921489dc9\" (UID: \"4abbcb7a-0390-4ed4-8d3b-468921489dc9\") "
Oct 31 19:41:28.826672 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.826633    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4abbcb7a-0390-4ed4-8d3b-468921489dc9-istio-token\" (UniqueName: \"kubernetes.io/projected/4abbcb7a-0390-4ed4-8d3b-468921489dc9-istio-token\") pod \"4abbcb7a-0390-4ed4-8d3b-468921489dc9\" (UID: \"4abbcb7a-0390-4ed4-8d3b-468921489dc9\") "
Oct 31 19:41:28.828208 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.826718    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4abbcb7a-0390-4ed4-8d3b-468921489dc9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4abbcb7a-0390-4ed4-8d3b-468921489dc9-credential-socket\") pod \"4abbcb7a-0390-4ed4-8d3b-468921489dc9\" (UID: \"4abbcb7a-0390-4ed4-8d3b-468921489dc9\") "
Oct 31 19:41:28.828208 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.826766    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4abbcb7a-0390-4ed4-8d3b-468921489dc9-kube-api-access-n667j\" (UniqueName: \"kubernetes.io/projected/4abbcb7a-0390-4ed4-8d3b-468921489dc9-kube-api-access-n667j\") pod \"4abbcb7a-0390-4ed4-8d3b-468921489dc9\" (UID: \"4abbcb7a-0390-4ed4-8d3b-468921489dc9\") "
Oct 31 19:41:28.828208 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.826849    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4abbcb7a-0390-4ed4-8d3b-468921489dc9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4abbcb7a-0390-4ed4-8d3b-468921489dc9-workload-certs\") pod \"4abbcb7a-0390-4ed4-8d3b-468921489dc9\" (UID: \"4abbcb7a-0390-4ed4-8d3b-468921489dc9\") "
Oct 31 19:41:28.828208 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.827910    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd915c45-10cb-4d0f-a519-44bbe338e43d-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:28.828208 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.827940    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd915c45-10cb-4d0f-a519-44bbe338e43d-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:28.828208 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.827955    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zctzl\" (UniqueName: \"kubernetes.io/projected/fd915c45-10cb-4d0f-a519-44bbe338e43d-kube-api-access-zctzl\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:28.828208 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.827969    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd915c45-10cb-4d0f-a519-44bbe338e43d-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:28.828208 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.827983    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd915c45-10cb-4d0f-a519-44bbe338e43d-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:28.828208 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.828000    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd915c45-10cb-4d0f-a519-44bbe338e43d-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:28.829005 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.828805    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4abbcb7a-0390-4ed4-8d3b-468921489dc9-workload-certs" pod "4abbcb7a-0390-4ed4-8d3b-468921489dc9" (UID: "4abbcb7a-0390-4ed4-8d3b-468921489dc9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:28.829230 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.829130    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4abbcb7a-0390-4ed4-8d3b-468921489dc9-workload-socket" pod "4abbcb7a-0390-4ed4-8d3b-468921489dc9" (UID: "4abbcb7a-0390-4ed4-8d3b-468921489dc9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:28.829645 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.829440    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4abbcb7a-0390-4ed4-8d3b-468921489dc9-istio-data" pod "4abbcb7a-0390-4ed4-8d3b-468921489dc9" (UID: "4abbcb7a-0390-4ed4-8d3b-468921489dc9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:28.829645 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.829385    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4abbcb7a-0390-4ed4-8d3b-468921489dc9-istiod-ca-cert" pod "4abbcb7a-0390-4ed4-8d3b-468921489dc9" (UID: "4abbcb7a-0390-4ed4-8d3b-468921489dc9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:41:28.829786 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.829664    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4abbcb7a-0390-4ed4-8d3b-468921489dc9-credential-socket" pod "4abbcb7a-0390-4ed4-8d3b-468921489dc9" (UID: "4abbcb7a-0390-4ed4-8d3b-468921489dc9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:28.832340 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.832292    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4abbcb7a-0390-4ed4-8d3b-468921489dc9-istio-podinfo" pod "4abbcb7a-0390-4ed4-8d3b-468921489dc9" (UID: "4abbcb7a-0390-4ed4-8d3b-468921489dc9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:41:28.842178 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.841443    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4abbcb7a-0390-4ed4-8d3b-468921489dc9-kube-api-access-n667j" pod "4abbcb7a-0390-4ed4-8d3b-468921489dc9" (UID: "4abbcb7a-0390-4ed4-8d3b-468921489dc9"). InnerVolumeSpecName "kube-api-access-n667j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:41:28.842497 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.842461    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4abbcb7a-0390-4ed4-8d3b-468921489dc9-istio-token" pod "4abbcb7a-0390-4ed4-8d3b-468921489dc9" (UID: "4abbcb7a-0390-4ed4-8d3b-468921489dc9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:41:28.842883 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.842859    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4abbcb7a-0390-4ed4-8d3b-468921489dc9-istio-envoy" pod "4abbcb7a-0390-4ed4-8d3b-468921489dc9" (UID: "4abbcb7a-0390-4ed4-8d3b-468921489dc9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:28.929811 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.929779    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4abbcb7a-0390-4ed4-8d3b-468921489dc9-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:28.929811 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.929806    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4abbcb7a-0390-4ed4-8d3b-468921489dc9-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:28.929811 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.929815    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4abbcb7a-0390-4ed4-8d3b-468921489dc9-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:28.930045 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.929825    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4abbcb7a-0390-4ed4-8d3b-468921489dc9-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:28.930045 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.929834    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4abbcb7a-0390-4ed4-8d3b-468921489dc9-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:28.930045 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.929843    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4abbcb7a-0390-4ed4-8d3b-468921489dc9-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:28.930045 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.929852    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n667j\" (UniqueName: \"kubernetes.io/projected/4abbcb7a-0390-4ed4-8d3b-468921489dc9-kube-api-access-n667j\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:28.930045 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.929862    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4abbcb7a-0390-4ed4-8d3b-468921489dc9-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:28.930045 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:28.929872    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4abbcb7a-0390-4ed4-8d3b-468921489dc9-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:29.330646 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:29.330616    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg"
Oct 31 19:41:29.330822 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:29.330603    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg" event={"ID":"9bacc59a-7abc-43b3-81dd-a8829710ad35","Type":"ContainerDied","Data":"5f3ddaf49dfbe7170868440577e352df5a231fcbed10844c66e99954fbca55c8"}
Oct 31 19:41:29.330911 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:29.330850    4645 scope.go:122] "RemoveContainer" containerID="b8dce505a1fc1201cc5c6775c022b0b334366af36acbbff66fd883f23b33be5f"
Oct 31 19:41:29.332704 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:29.332678    4645 generic.go:356] "Generic (PLEG): container finished" podID="4abbcb7a-0390-4ed4-8d3b-468921489dc9" containerID="d62594b322266a964f7e928ec70564d883404edd2ae582b721e271ffbc01f14f" exitCode=0
Oct 31 19:41:29.332814 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:29.332752    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt"
Oct 31 19:41:29.332869 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:29.332827    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt" event={"ID":"4abbcb7a-0390-4ed4-8d3b-468921489dc9","Type":"ContainerDied","Data":"d62594b322266a964f7e928ec70564d883404edd2ae582b721e271ffbc01f14f"}
Oct 31 19:41:29.332869 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:29.332864    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt" event={"ID":"4abbcb7a-0390-4ed4-8d3b-468921489dc9","Type":"ContainerDied","Data":"89d9d84e923b555cff4c606b0390e61062463eaa04c31fe62bb720eb57ca1776"}
Oct 31 19:41:29.334843 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:29.334819    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7" event={"ID":"fd915c45-10cb-4d0f-a519-44bbe338e43d","Type":"ContainerDied","Data":"9785b05bcc6df7ce3915597ea14ce88f5d04b6bdeb3254207677c09f9a9176c3"}
Oct 31 19:41:29.334977 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:29.334831    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7"
Oct 31 19:41:29.337866 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:29.337838    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-88eaf9-9988c7848-fqszv" event={"ID":"4abee805-15d3-4b54-bd38-889f562f0c57","Type":"ContainerStarted","Data":"dc4c9f61fb81ea995f5ec6c2308a699a654fca7214253a53df32f301272b4449"}
Oct 31 19:41:29.338670 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:29.338646    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-88eaf9-9988c7848-fqszv"
Oct 31 19:41:29.346590 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:29.346569    4645 scope.go:122] "RemoveContainer" containerID="d62594b322266a964f7e928ec70564d883404edd2ae582b721e271ffbc01f14f"
Oct 31 19:41:29.357279 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:29.357264    4645 scope.go:122] "RemoveContainer" containerID="d62594b322266a964f7e928ec70564d883404edd2ae582b721e271ffbc01f14f"
Oct 31 19:41:29.357546 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:41:29.357531    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d62594b322266a964f7e928ec70564d883404edd2ae582b721e271ffbc01f14f\": container with ID starting with d62594b322266a964f7e928ec70564d883404edd2ae582b721e271ffbc01f14f not found: ID does not exist" containerID="d62594b322266a964f7e928ec70564d883404edd2ae582b721e271ffbc01f14f"
Oct 31 19:41:29.357593 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:29.357551    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d62594b322266a964f7e928ec70564d883404edd2ae582b721e271ffbc01f14f"} err="failed to get container status \"d62594b322266a964f7e928ec70564d883404edd2ae582b721e271ffbc01f14f\": rpc error: code = NotFound desc = could not find container \"d62594b322266a964f7e928ec70564d883404edd2ae582b721e271ffbc01f14f\": container with ID starting with d62594b322266a964f7e928ec70564d883404edd2ae582b721e271ffbc01f14f not found: ID does not exist"
Oct 31 19:41:29.357593 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:29.357566    4645 scope.go:122] "RemoveContainer" containerID="738c6800661a462c22ac22e82d1ec79ffa27edcd8faf80becbf78ae8e49c0200"
Oct 31 19:41:29.362788 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:29.362745    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-88eaf9-9988c7848-fqszv" podStartSLOduration=2.817099307 podStartE2EDuration="3.36273433s" totalImagesPullingTime="545.635023ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:41:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:41:29.360537136 +0000 UTC m=+1319.933771265" watchObservedRunningTime="2026-10-31 19:41:29.36273433 +0000 UTC m=+1319.935968459"
Oct 31 19:41:29.386642 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:29.386616    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt"]
Oct 31 19:41:29.393980 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:29.393961    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-vx2qt"]
Oct 31 19:41:29.409255 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:29.409232    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg"]
Oct 31 19:41:29.414550 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:29.414531    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-49sbg"]
Oct 31 19:41:29.435310 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:29.435287    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7"]
Oct 31 19:41:29.440259 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:29.440211    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-67adab47-openshift-default-84fb98c68f-5t4q7"]
Oct 31 19:41:29.620414 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:29.620384    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4abbcb7a-0390-4ed4-8d3b-468921489dc9" path="/var/lib/kubelet/pods/4abbcb7a-0390-4ed4-8d3b-468921489dc9/volumes"
Oct 31 19:41:29.620943 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:29.620922    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bacc59a-7abc-43b3-81dd-a8829710ad35" path="/var/lib/kubelet/pods/9bacc59a-7abc-43b3-81dd-a8829710ad35/volumes"
Oct 31 19:41:29.621441 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:29.621422    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd915c45-10cb-4d0f-a519-44bbe338e43d" path="/var/lib/kubelet/pods/fd915c45-10cb-4d0f-a519-44bbe338e43d/volumes"
Oct 31 19:41:30.121907 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:30.121872    4645 scope.go:122] "RemoveContainer" containerID="e52c2c269e5f6b3e4adfaab3907d07922c229c377ea17a1460ffbe26970a1b21"
Oct 31 19:41:30.323400 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:30.323180    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:31.939159 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:31.939067    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:32.232783 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:32.232709    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:32.344228 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:32.344192    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:32.573032 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:32.572948    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc"]
Oct 31 19:41:32.573305 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:32.573234    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc" podUID="8844ebc7-a83d-4b83-859e-2eb4d1e816c2" containerName="istio-proxy" containerID="cri-o://74805777beab59124da8de48f2182a3636ead44b1a52773b0670ac65012c59d1" gracePeriod=30
Oct 31 19:41:32.602967 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:32.602935    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw"]
Oct 31 19:41:32.603194 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:32.603165    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw" podUID="585024d3-5768-463e-acdc-81151a7c0461" containerName="istio-proxy" containerID="cri-o://69ec18e93e369eb6f8cac06277a33448631ff62e4d2f446fecdc5e4899fdd649" gracePeriod=30
Oct 31 19:41:32.603973 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:32.603683    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:32.607401 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:32.607361    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz"]
Oct 31 19:41:32.607570 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:32.607551    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz" podUID="79f81748-b7d5-4ed1-8f28-4386bba24a79" containerName="istio-proxy" containerID="cri-o://eb4ad62af9dbd4c98b726505c351c8a7c15c0a288e34167527327263927b3e8b" gracePeriod=30
Oct 31 19:41:33.252948 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:33.252892    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:33.412093 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:33.412055    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:33.996488 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:33.996453    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:34.148354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:34.139320    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:34.260605 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:34.259882    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:34.795867 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:34.795833    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:35.834608 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:35.834573    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:36.246964 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:41:36.246931    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 19:41:36.411544 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:36.411518    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-88eaf9-9988c7848-fqszv"
Oct 31 19:41:36.643512 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:36.643428    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:37.653926 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:37.653879    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:37.901323 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:37.901294    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw"
Oct 31 19:41:37.933119 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:37.933086    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/585024d3-5768-463e-acdc-81151a7c0461-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/585024d3-5768-463e-acdc-81151a7c0461-credential-socket\") pod \"585024d3-5768-463e-acdc-81151a7c0461\" (UID: \"585024d3-5768-463e-acdc-81151a7c0461\") "
Oct 31 19:41:37.933268 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:37.933180    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/585024d3-5768-463e-acdc-81151a7c0461-istio-token\" (UniqueName: \"kubernetes.io/projected/585024d3-5768-463e-acdc-81151a7c0461-istio-token\") pod \"585024d3-5768-463e-acdc-81151a7c0461\" (UID: \"585024d3-5768-463e-acdc-81151a7c0461\") "
Oct 31 19:41:37.933404 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:37.933293    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/585024d3-5768-463e-acdc-81151a7c0461-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/585024d3-5768-463e-acdc-81151a7c0461-workload-socket\") pod \"585024d3-5768-463e-acdc-81151a7c0461\" (UID: \"585024d3-5768-463e-acdc-81151a7c0461\") "
Oct 31 19:41:37.933484 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:37.933399    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/585024d3-5768-463e-acdc-81151a7c0461-credential-socket" pod "585024d3-5768-463e-acdc-81151a7c0461" (UID: "585024d3-5768-463e-acdc-81151a7c0461"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:37.933484 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:37.933420    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/585024d3-5768-463e-acdc-81151a7c0461-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/585024d3-5768-463e-acdc-81151a7c0461-istio-podinfo\") pod \"585024d3-5768-463e-acdc-81151a7c0461\" (UID: \"585024d3-5768-463e-acdc-81151a7c0461\") "
Oct 31 19:41:37.933875 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:37.933851    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/585024d3-5768-463e-acdc-81151a7c0461-workload-socket" pod "585024d3-5768-463e-acdc-81151a7c0461" (UID: "585024d3-5768-463e-acdc-81151a7c0461"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:37.933979 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:37.933967    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/585024d3-5768-463e-acdc-81151a7c0461-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/585024d3-5768-463e-acdc-81151a7c0461-istio-data\") pod \"585024d3-5768-463e-acdc-81151a7c0461\" (UID: \"585024d3-5768-463e-acdc-81151a7c0461\") "
Oct 31 19:41:37.934026 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:37.934018    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/585024d3-5768-463e-acdc-81151a7c0461-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/585024d3-5768-463e-acdc-81151a7c0461-istiod-ca-cert\") pod \"585024d3-5768-463e-acdc-81151a7c0461\" (UID: \"585024d3-5768-463e-acdc-81151a7c0461\") "
Oct 31 19:41:37.934088 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:37.934075    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/585024d3-5768-463e-acdc-81151a7c0461-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/585024d3-5768-463e-acdc-81151a7c0461-istio-envoy\") pod \"585024d3-5768-463e-acdc-81151a7c0461\" (UID: \"585024d3-5768-463e-acdc-81151a7c0461\") "
Oct 31 19:41:37.934138 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:37.934125    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/585024d3-5768-463e-acdc-81151a7c0461-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/585024d3-5768-463e-acdc-81151a7c0461-workload-certs\") pod \"585024d3-5768-463e-acdc-81151a7c0461\" (UID: \"585024d3-5768-463e-acdc-81151a7c0461\") "
Oct 31 19:41:37.934207 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:37.934193    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/585024d3-5768-463e-acdc-81151a7c0461-kube-api-access-6qjv2\" (UniqueName: \"kubernetes.io/projected/585024d3-5768-463e-acdc-81151a7c0461-kube-api-access-6qjv2\") pod \"585024d3-5768-463e-acdc-81151a7c0461\" (UID: \"585024d3-5768-463e-acdc-81151a7c0461\") "
Oct 31 19:41:37.935244 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:37.934990    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/585024d3-5768-463e-acdc-81151a7c0461-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:37.935244 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:37.935013    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/585024d3-5768-463e-acdc-81151a7c0461-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:37.935615 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:37.935595    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/585024d3-5768-463e-acdc-81151a7c0461-istiod-ca-cert" pod "585024d3-5768-463e-acdc-81151a7c0461" (UID: "585024d3-5768-463e-acdc-81151a7c0461"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:41:37.935752 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:37.935729    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/585024d3-5768-463e-acdc-81151a7c0461-istio-data" pod "585024d3-5768-463e-acdc-81151a7c0461" (UID: "585024d3-5768-463e-acdc-81151a7c0461"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:37.935967 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:37.935943    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/585024d3-5768-463e-acdc-81151a7c0461-workload-certs" pod "585024d3-5768-463e-acdc-81151a7c0461" (UID: "585024d3-5768-463e-acdc-81151a7c0461"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:37.939974 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:37.939949    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/585024d3-5768-463e-acdc-81151a7c0461-istio-podinfo" pod "585024d3-5768-463e-acdc-81151a7c0461" (UID: "585024d3-5768-463e-acdc-81151a7c0461"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:41:37.941236 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:37.941194    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/585024d3-5768-463e-acdc-81151a7c0461-istio-token" pod "585024d3-5768-463e-acdc-81151a7c0461" (UID: "585024d3-5768-463e-acdc-81151a7c0461"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:41:37.941236 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:37.941228    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/585024d3-5768-463e-acdc-81151a7c0461-kube-api-access-6qjv2" pod "585024d3-5768-463e-acdc-81151a7c0461" (UID: "585024d3-5768-463e-acdc-81151a7c0461"). InnerVolumeSpecName "kube-api-access-6qjv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:41:37.952960 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:37.952781    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/585024d3-5768-463e-acdc-81151a7c0461-istio-envoy" pod "585024d3-5768-463e-acdc-81151a7c0461" (UID: "585024d3-5768-463e-acdc-81151a7c0461"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:37.983910 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:37.983885    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc"
Oct 31 19:41:37.991674 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:37.991654    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz"
Oct 31 19:41:38.035720 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.035693    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8844ebc7-a83d-4b83-859e-2eb4d1e816c2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8844ebc7-a83d-4b83-859e-2eb4d1e816c2-credential-socket\") pod \"8844ebc7-a83d-4b83-859e-2eb4d1e816c2\" (UID: \"8844ebc7-a83d-4b83-859e-2eb4d1e816c2\") "
Oct 31 19:41:38.035861 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.035729    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79f81748-b7d5-4ed1-8f28-4386bba24a79-istio-token\" (UniqueName: \"kubernetes.io/projected/79f81748-b7d5-4ed1-8f28-4386bba24a79-istio-token\") pod \"79f81748-b7d5-4ed1-8f28-4386bba24a79\" (UID: \"79f81748-b7d5-4ed1-8f28-4386bba24a79\") "
Oct 31 19:41:38.035861 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.035793    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79f81748-b7d5-4ed1-8f28-4386bba24a79-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79f81748-b7d5-4ed1-8f28-4386bba24a79-workload-socket\") pod \"79f81748-b7d5-4ed1-8f28-4386bba24a79\" (UID: \"79f81748-b7d5-4ed1-8f28-4386bba24a79\") "
Oct 31 19:41:38.035861 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.035847    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8844ebc7-a83d-4b83-859e-2eb4d1e816c2-istio-token\" (UniqueName: \"kubernetes.io/projected/8844ebc7-a83d-4b83-859e-2eb4d1e816c2-istio-token\") pod \"8844ebc7-a83d-4b83-859e-2eb4d1e816c2\" (UID: \"8844ebc7-a83d-4b83-859e-2eb4d1e816c2\") "
Oct 31 19:41:38.036037 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.035877    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8844ebc7-a83d-4b83-859e-2eb4d1e816c2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8844ebc7-a83d-4b83-859e-2eb4d1e816c2-istio-data\") pod \"8844ebc7-a83d-4b83-859e-2eb4d1e816c2\" (UID: \"8844ebc7-a83d-4b83-859e-2eb4d1e816c2\") "
Oct 31 19:41:38.036037 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.035925    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8844ebc7-a83d-4b83-859e-2eb4d1e816c2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8844ebc7-a83d-4b83-859e-2eb4d1e816c2-istio-envoy\") pod \"8844ebc7-a83d-4b83-859e-2eb4d1e816c2\" (UID: \"8844ebc7-a83d-4b83-859e-2eb4d1e816c2\") "
Oct 31 19:41:38.036037 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.035993    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8844ebc7-a83d-4b83-859e-2eb4d1e816c2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8844ebc7-a83d-4b83-859e-2eb4d1e816c2-istio-podinfo\") pod \"8844ebc7-a83d-4b83-859e-2eb4d1e816c2\" (UID: \"8844ebc7-a83d-4b83-859e-2eb4d1e816c2\") "
Oct 31 19:41:38.036201 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.036038    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8844ebc7-a83d-4b83-859e-2eb4d1e816c2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8844ebc7-a83d-4b83-859e-2eb4d1e816c2-istiod-ca-cert\") pod \"8844ebc7-a83d-4b83-859e-2eb4d1e816c2\" (UID: \"8844ebc7-a83d-4b83-859e-2eb4d1e816c2\") "
Oct 31 19:41:38.036201 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.036093    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8844ebc7-a83d-4b83-859e-2eb4d1e816c2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8844ebc7-a83d-4b83-859e-2eb4d1e816c2-workload-certs\") pod \"8844ebc7-a83d-4b83-859e-2eb4d1e816c2\" (UID: \"8844ebc7-a83d-4b83-859e-2eb4d1e816c2\") "
Oct 31 19:41:38.036374 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.036284    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8844ebc7-a83d-4b83-859e-2eb4d1e816c2-kube-api-access-9xk4v\" (UniqueName: \"kubernetes.io/projected/8844ebc7-a83d-4b83-859e-2eb4d1e816c2-kube-api-access-9xk4v\") pod \"8844ebc7-a83d-4b83-859e-2eb4d1e816c2\" (UID: \"8844ebc7-a83d-4b83-859e-2eb4d1e816c2\") "
Oct 31 19:41:38.036374 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.036319    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8844ebc7-a83d-4b83-859e-2eb4d1e816c2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8844ebc7-a83d-4b83-859e-2eb4d1e816c2-workload-socket\") pod \"8844ebc7-a83d-4b83-859e-2eb4d1e816c2\" (UID: \"8844ebc7-a83d-4b83-859e-2eb4d1e816c2\") "
Oct 31 19:41:38.036492 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.036376    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79f81748-b7d5-4ed1-8f28-4386bba24a79-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79f81748-b7d5-4ed1-8f28-4386bba24a79-credential-socket\") pod \"79f81748-b7d5-4ed1-8f28-4386bba24a79\" (UID: \"79f81748-b7d5-4ed1-8f28-4386bba24a79\") "
Oct 31 19:41:38.036492 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.036419    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79f81748-b7d5-4ed1-8f28-4386bba24a79-kube-api-access-5nprt\" (UniqueName: \"kubernetes.io/projected/79f81748-b7d5-4ed1-8f28-4386bba24a79-kube-api-access-5nprt\") pod \"79f81748-b7d5-4ed1-8f28-4386bba24a79\" (UID: \"79f81748-b7d5-4ed1-8f28-4386bba24a79\") "
Oct 31 19:41:38.036492 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.036444    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/79f81748-b7d5-4ed1-8f28-4386bba24a79-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79f81748-b7d5-4ed1-8f28-4386bba24a79-istio-podinfo\") pod \"79f81748-b7d5-4ed1-8f28-4386bba24a79\" (UID: \"79f81748-b7d5-4ed1-8f28-4386bba24a79\") "
Oct 31 19:41:38.036648 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.036538    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79f81748-b7d5-4ed1-8f28-4386bba24a79-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79f81748-b7d5-4ed1-8f28-4386bba24a79-workload-certs\") pod \"79f81748-b7d5-4ed1-8f28-4386bba24a79\" (UID: \"79f81748-b7d5-4ed1-8f28-4386bba24a79\") "
Oct 31 19:41:38.036648 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.036577    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79f81748-b7d5-4ed1-8f28-4386bba24a79-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79f81748-b7d5-4ed1-8f28-4386bba24a79-istio-envoy\") pod \"79f81748-b7d5-4ed1-8f28-4386bba24a79\" (UID: \"79f81748-b7d5-4ed1-8f28-4386bba24a79\") "
Oct 31 19:41:38.036648 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.036618    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79f81748-b7d5-4ed1-8f28-4386bba24a79-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79f81748-b7d5-4ed1-8f28-4386bba24a79-istiod-ca-cert\") pod \"79f81748-b7d5-4ed1-8f28-4386bba24a79\" (UID: \"79f81748-b7d5-4ed1-8f28-4386bba24a79\") "
Oct 31 19:41:38.036797 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.036663    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79f81748-b7d5-4ed1-8f28-4386bba24a79-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79f81748-b7d5-4ed1-8f28-4386bba24a79-istio-data\") pod \"79f81748-b7d5-4ed1-8f28-4386bba24a79\" (UID: \"79f81748-b7d5-4ed1-8f28-4386bba24a79\") "
Oct 31 19:41:38.037199 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.037168    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8844ebc7-a83d-4b83-859e-2eb4d1e816c2-workload-socket" pod "8844ebc7-a83d-4b83-859e-2eb4d1e816c2" (UID: "8844ebc7-a83d-4b83-859e-2eb4d1e816c2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:38.037464 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.037440    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79f81748-b7d5-4ed1-8f28-4386bba24a79-credential-socket" pod "79f81748-b7d5-4ed1-8f28-4386bba24a79" (UID: "79f81748-b7d5-4ed1-8f28-4386bba24a79"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:38.037544 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.037483    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8844ebc7-a83d-4b83-859e-2eb4d1e816c2-istiod-ca-cert" pod "8844ebc7-a83d-4b83-859e-2eb4d1e816c2" (UID: "8844ebc7-a83d-4b83-859e-2eb4d1e816c2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:41:38.037794 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.037769    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79f81748-b7d5-4ed1-8f28-4386bba24a79-workload-socket" pod "79f81748-b7d5-4ed1-8f28-4386bba24a79" (UID: "79f81748-b7d5-4ed1-8f28-4386bba24a79"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:38.038082 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.038057    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8844ebc7-a83d-4b83-859e-2eb4d1e816c2-workload-certs" pod "8844ebc7-a83d-4b83-859e-2eb4d1e816c2" (UID: "8844ebc7-a83d-4b83-859e-2eb4d1e816c2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:38.038375 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.038351    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8844ebc7-a83d-4b83-859e-2eb4d1e816c2-istio-data" pod "8844ebc7-a83d-4b83-859e-2eb4d1e816c2" (UID: "8844ebc7-a83d-4b83-859e-2eb4d1e816c2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:38.038930 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.038903    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8844ebc7-a83d-4b83-859e-2eb4d1e816c2-credential-socket" pod "8844ebc7-a83d-4b83-859e-2eb4d1e816c2" (UID: "8844ebc7-a83d-4b83-859e-2eb4d1e816c2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:38.039217 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.039086    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79f81748-b7d5-4ed1-8f28-4386bba24a79-istio-data" pod "79f81748-b7d5-4ed1-8f28-4386bba24a79" (UID: "79f81748-b7d5-4ed1-8f28-4386bba24a79"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:38.039385 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.039363    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79f81748-b7d5-4ed1-8f28-4386bba24a79-istiod-ca-cert" pod "79f81748-b7d5-4ed1-8f28-4386bba24a79" (UID: "79f81748-b7d5-4ed1-8f28-4386bba24a79"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:41:38.040566 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.040538    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8844ebc7-a83d-4b83-859e-2eb4d1e816c2-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:38.040690 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.040569    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/585024d3-5768-463e-acdc-81151a7c0461-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:38.040690 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.040584    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/585024d3-5768-463e-acdc-81151a7c0461-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:38.040690 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.040599    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79f81748-b7d5-4ed1-8f28-4386bba24a79-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:38.040690 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.040613    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/585024d3-5768-463e-acdc-81151a7c0461-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:38.040690 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.040628    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/585024d3-5768-463e-acdc-81151a7c0461-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:38.040690 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.040644    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/585024d3-5768-463e-acdc-81151a7c0461-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:38.040690 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.040658    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79f81748-b7d5-4ed1-8f28-4386bba24a79-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:38.040690 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.040671    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79f81748-b7d5-4ed1-8f28-4386bba24a79-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:38.040690 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.040685    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qjv2\" (UniqueName: \"kubernetes.io/projected/585024d3-5768-463e-acdc-81151a7c0461-kube-api-access-6qjv2\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:38.040690 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.040686    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79f81748-b7d5-4ed1-8f28-4386bba24a79-workload-certs" pod "79f81748-b7d5-4ed1-8f28-4386bba24a79" (UID: "79f81748-b7d5-4ed1-8f28-4386bba24a79"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:38.041007 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.040699    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8844ebc7-a83d-4b83-859e-2eb4d1e816c2-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:38.041007 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.040712    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79f81748-b7d5-4ed1-8f28-4386bba24a79-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:38.041007 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.040726    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8844ebc7-a83d-4b83-859e-2eb4d1e816c2-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:38.041007 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.040738    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/585024d3-5768-463e-acdc-81151a7c0461-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:38.041007 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.040752    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8844ebc7-a83d-4b83-859e-2eb4d1e816c2-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:38.041007 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.040765    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8844ebc7-a83d-4b83-859e-2eb4d1e816c2-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:38.042709 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.042686    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8844ebc7-a83d-4b83-859e-2eb4d1e816c2-istio-podinfo" pod "8844ebc7-a83d-4b83-859e-2eb4d1e816c2" (UID: "8844ebc7-a83d-4b83-859e-2eb4d1e816c2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:41:38.042709 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.042699    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8844ebc7-a83d-4b83-859e-2eb4d1e816c2-istio-envoy" pod "8844ebc7-a83d-4b83-859e-2eb4d1e816c2" (UID: "8844ebc7-a83d-4b83-859e-2eb4d1e816c2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:38.042857 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.042834    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79f81748-b7d5-4ed1-8f28-4386bba24a79-kube-api-access-5nprt" pod "79f81748-b7d5-4ed1-8f28-4386bba24a79" (UID: "79f81748-b7d5-4ed1-8f28-4386bba24a79"). InnerVolumeSpecName "kube-api-access-5nprt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:41:38.042990 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.042971    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8844ebc7-a83d-4b83-859e-2eb4d1e816c2-istio-token" pod "8844ebc7-a83d-4b83-859e-2eb4d1e816c2" (UID: "8844ebc7-a83d-4b83-859e-2eb4d1e816c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:41:38.043044 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.042998    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79f81748-b7d5-4ed1-8f28-4386bba24a79-istio-token" pod "79f81748-b7d5-4ed1-8f28-4386bba24a79" (UID: "79f81748-b7d5-4ed1-8f28-4386bba24a79"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:41:38.043314 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.043296    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79f81748-b7d5-4ed1-8f28-4386bba24a79-istio-podinfo" pod "79f81748-b7d5-4ed1-8f28-4386bba24a79" (UID: "79f81748-b7d5-4ed1-8f28-4386bba24a79"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:41:38.044537 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.044509    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8844ebc7-a83d-4b83-859e-2eb4d1e816c2-kube-api-access-9xk4v" pod "8844ebc7-a83d-4b83-859e-2eb4d1e816c2" (UID: "8844ebc7-a83d-4b83-859e-2eb4d1e816c2"). InnerVolumeSpecName "kube-api-access-9xk4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:41:38.044618 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.044589    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79f81748-b7d5-4ed1-8f28-4386bba24a79-istio-envoy" pod "79f81748-b7d5-4ed1-8f28-4386bba24a79" (UID: "79f81748-b7d5-4ed1-8f28-4386bba24a79"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:38.142243 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.142182    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xk4v\" (UniqueName: \"kubernetes.io/projected/8844ebc7-a83d-4b83-859e-2eb4d1e816c2-kube-api-access-9xk4v\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:38.142243 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.142207    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nprt\" (UniqueName: \"kubernetes.io/projected/79f81748-b7d5-4ed1-8f28-4386bba24a79-kube-api-access-5nprt\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:38.142243 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.142218    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79f81748-b7d5-4ed1-8f28-4386bba24a79-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:38.142243 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.142228    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79f81748-b7d5-4ed1-8f28-4386bba24a79-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:38.142243 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.142238    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79f81748-b7d5-4ed1-8f28-4386bba24a79-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:38.142504 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.142251    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79f81748-b7d5-4ed1-8f28-4386bba24a79-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:38.142504 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.142261    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8844ebc7-a83d-4b83-859e-2eb4d1e816c2-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:38.142504 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.142269    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8844ebc7-a83d-4b83-859e-2eb4d1e816c2-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:38.142504 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.142277    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8844ebc7-a83d-4b83-859e-2eb4d1e816c2-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:38.471444 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.471409    4645 generic.go:356] "Generic (PLEG): container finished" podID="79f81748-b7d5-4ed1-8f28-4386bba24a79" containerID="eb4ad62af9dbd4c98b726505c351c8a7c15c0a288e34167527327263927b3e8b" exitCode=0
Oct 31 19:41:38.471618 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.471469    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz"
Oct 31 19:41:38.471618 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.471494    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz" event={"ID":"79f81748-b7d5-4ed1-8f28-4386bba24a79","Type":"ContainerDied","Data":"eb4ad62af9dbd4c98b726505c351c8a7c15c0a288e34167527327263927b3e8b"}
Oct 31 19:41:38.471618 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.471558    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz" event={"ID":"79f81748-b7d5-4ed1-8f28-4386bba24a79","Type":"ContainerDied","Data":"02fd2bcdada5d507afff7fc4e35fd88cb7c368806f7a4350fc2d48213abc11de"}
Oct 31 19:41:38.471618 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.471589    4645 scope.go:122] "RemoveContainer" containerID="eb4ad62af9dbd4c98b726505c351c8a7c15c0a288e34167527327263927b3e8b"
Oct 31 19:41:38.473317 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.473270    4645 generic.go:356] "Generic (PLEG): container finished" podID="8844ebc7-a83d-4b83-859e-2eb4d1e816c2" containerID="74805777beab59124da8de48f2182a3636ead44b1a52773b0670ac65012c59d1" exitCode=0
Oct 31 19:41:38.473457 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.473347    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc"
Oct 31 19:41:38.473457 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.473426    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc" event={"ID":"8844ebc7-a83d-4b83-859e-2eb4d1e816c2","Type":"ContainerDied","Data":"74805777beab59124da8de48f2182a3636ead44b1a52773b0670ac65012c59d1"}
Oct 31 19:41:38.473561 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.473472    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc" event={"ID":"8844ebc7-a83d-4b83-859e-2eb4d1e816c2","Type":"ContainerDied","Data":"43f8d43cb61a5a2576d7fd746acf91ef1c4652682154d3a048b10b8aa2d3bf04"}
Oct 31 19:41:38.478500 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.478170    4645 generic.go:356] "Generic (PLEG): container finished" podID="585024d3-5768-463e-acdc-81151a7c0461" containerID="69ec18e93e369eb6f8cac06277a33448631ff62e4d2f446fecdc5e4899fdd649" exitCode=0
Oct 31 19:41:38.478500 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.478298    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw" event={"ID":"585024d3-5768-463e-acdc-81151a7c0461","Type":"ContainerDied","Data":"69ec18e93e369eb6f8cac06277a33448631ff62e4d2f446fecdc5e4899fdd649"}
Oct 31 19:41:38.478500 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.478370    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw" event={"ID":"585024d3-5768-463e-acdc-81151a7c0461","Type":"ContainerDied","Data":"36b1612862d08cd8192297d61592f0bb2ca942e01474d1a472abb551f8ecc7a8"}
Oct 31 19:41:38.478500 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.478442    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw"
Oct 31 19:41:38.499966 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.499944    4645 scope.go:122] "RemoveContainer" containerID="eb4ad62af9dbd4c98b726505c351c8a7c15c0a288e34167527327263927b3e8b"
Oct 31 19:41:38.500433 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:41:38.500400    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb4ad62af9dbd4c98b726505c351c8a7c15c0a288e34167527327263927b3e8b\": container with ID starting with eb4ad62af9dbd4c98b726505c351c8a7c15c0a288e34167527327263927b3e8b not found: ID does not exist" containerID="eb4ad62af9dbd4c98b726505c351c8a7c15c0a288e34167527327263927b3e8b"
Oct 31 19:41:38.500561 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.500445    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb4ad62af9dbd4c98b726505c351c8a7c15c0a288e34167527327263927b3e8b"} err="failed to get container status \"eb4ad62af9dbd4c98b726505c351c8a7c15c0a288e34167527327263927b3e8b\": rpc error: code = NotFound desc = could not find container \"eb4ad62af9dbd4c98b726505c351c8a7c15c0a288e34167527327263927b3e8b\": container with ID starting with eb4ad62af9dbd4c98b726505c351c8a7c15c0a288e34167527327263927b3e8b not found: ID does not exist"
Oct 31 19:41:38.500561 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.500471    4645 scope.go:122] "RemoveContainer" containerID="74805777beab59124da8de48f2182a3636ead44b1a52773b0670ac65012c59d1"
Oct 31 19:41:38.508946 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.508922    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz"]
Oct 31 19:41:38.513934 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.513917    4645 scope.go:122] "RemoveContainer" containerID="74805777beab59124da8de48f2182a3636ead44b1a52773b0670ac65012c59d1"
Oct 31 19:41:38.514048 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.514015    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-79qgz"]
Oct 31 19:41:38.514196 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:41:38.514179    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74805777beab59124da8de48f2182a3636ead44b1a52773b0670ac65012c59d1\": container with ID starting with 74805777beab59124da8de48f2182a3636ead44b1a52773b0670ac65012c59d1 not found: ID does not exist" containerID="74805777beab59124da8de48f2182a3636ead44b1a52773b0670ac65012c59d1"
Oct 31 19:41:38.514238 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.514204    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74805777beab59124da8de48f2182a3636ead44b1a52773b0670ac65012c59d1"} err="failed to get container status \"74805777beab59124da8de48f2182a3636ead44b1a52773b0670ac65012c59d1\": rpc error: code = NotFound desc = could not find container \"74805777beab59124da8de48f2182a3636ead44b1a52773b0670ac65012c59d1\": container with ID starting with 74805777beab59124da8de48f2182a3636ead44b1a52773b0670ac65012c59d1 not found: ID does not exist"
Oct 31 19:41:38.514238 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.514220    4645 scope.go:122] "RemoveContainer" containerID="69ec18e93e369eb6f8cac06277a33448631ff62e4d2f446fecdc5e4899fdd649"
Oct 31 19:41:38.524729 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.524709    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc"]
Oct 31 19:41:38.526482 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.526467    4645 scope.go:122] "RemoveContainer" containerID="a6e1edb71ab92a73b91ed16fb9102cc4540988ed9ded70c1dff5c5400f189265"
Oct 31 19:41:38.528957 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.528939    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-fh4rc"]
Oct 31 19:41:38.535853 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.535835    4645 scope.go:122] "RemoveContainer" containerID="69ec18e93e369eb6f8cac06277a33448631ff62e4d2f446fecdc5e4899fdd649"
Oct 31 19:41:38.536136 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:41:38.536117    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69ec18e93e369eb6f8cac06277a33448631ff62e4d2f446fecdc5e4899fdd649\": container with ID starting with 69ec18e93e369eb6f8cac06277a33448631ff62e4d2f446fecdc5e4899fdd649 not found: ID does not exist" containerID="69ec18e93e369eb6f8cac06277a33448631ff62e4d2f446fecdc5e4899fdd649"
Oct 31 19:41:38.536197 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.536142    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69ec18e93e369eb6f8cac06277a33448631ff62e4d2f446fecdc5e4899fdd649"} err="failed to get container status \"69ec18e93e369eb6f8cac06277a33448631ff62e4d2f446fecdc5e4899fdd649\": rpc error: code = NotFound desc = could not find container \"69ec18e93e369eb6f8cac06277a33448631ff62e4d2f446fecdc5e4899fdd649\": container with ID starting with 69ec18e93e369eb6f8cac06277a33448631ff62e4d2f446fecdc5e4899fdd649 not found: ID does not exist"
Oct 31 19:41:38.536197 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.536156    4645 scope.go:122] "RemoveContainer" containerID="a6e1edb71ab92a73b91ed16fb9102cc4540988ed9ded70c1dff5c5400f189265"
Oct 31 19:41:38.536454 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:41:38.536437    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6e1edb71ab92a73b91ed16fb9102cc4540988ed9ded70c1dff5c5400f189265\": container with ID starting with a6e1edb71ab92a73b91ed16fb9102cc4540988ed9ded70c1dff5c5400f189265 not found: ID does not exist" containerID="a6e1edb71ab92a73b91ed16fb9102cc4540988ed9ded70c1dff5c5400f189265"
Oct 31 19:41:38.536509 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.536461    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6e1edb71ab92a73b91ed16fb9102cc4540988ed9ded70c1dff5c5400f189265"} err="failed to get container status \"a6e1edb71ab92a73b91ed16fb9102cc4540988ed9ded70c1dff5c5400f189265\": rpc error: code = NotFound desc = could not find container \"a6e1edb71ab92a73b91ed16fb9102cc4540988ed9ded70c1dff5c5400f189265\": container with ID starting with a6e1edb71ab92a73b91ed16fb9102cc4540988ed9ded70c1dff5c5400f189265 not found: ID does not exist"
Oct 31 19:41:38.537322 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.537304    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw"]
Oct 31 19:41:38.542050 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.542032    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-d4b21c-openshift-default-5f69685d47-ls6dw"]
Oct 31 19:41:38.749228 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:38.749136    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:39.259203 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:39.259166    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:39.371962 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:39.371930    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:39.620232 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:39.620125    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="585024d3-5768-463e-acdc-81151a7c0461" path="/var/lib/kubelet/pods/585024d3-5768-463e-acdc-81151a7c0461/volumes"
Oct 31 19:41:39.620859 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:39.620834    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79f81748-b7d5-4ed1-8f28-4386bba24a79" path="/var/lib/kubelet/pods/79f81748-b7d5-4ed1-8f28-4386bba24a79/volumes"
Oct 31 19:41:39.621323 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:39.621306    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8844ebc7-a83d-4b83-859e-2eb4d1e816c2" path="/var/lib/kubelet/pods/8844ebc7-a83d-4b83-859e-2eb4d1e816c2/volumes"
Oct 31 19:41:39.639712 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:39.639681    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:40.503061 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:40.503033    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:40.584489 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:40.584460    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a4e312-7b78fb8665-b9pn6"]
Oct 31 19:41:40.585159 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:40.585145    4645 cpu_manager.go:414] "containerMap: removing container" podUID="8844ebc7-a83d-4b83-859e-2eb4d1e816c2" containerName="istio-proxy"
Oct 31 19:41:40.585197 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:40.585161    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8844ebc7-a83d-4b83-859e-2eb4d1e816c2" containerName="istio-proxy"
Oct 31 19:41:40.585197 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:40.585183    4645 cpu_manager.go:414] "containerMap: removing container" podUID="585024d3-5768-463e-acdc-81151a7c0461" containerName="istio-proxy"
Oct 31 19:41:40.585197 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:40.585188    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="585024d3-5768-463e-acdc-81151a7c0461" containerName="istio-proxy"
Oct 31 19:41:40.585429 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:40.585204    4645 cpu_manager.go:414] "containerMap: removing container" podUID="4abbcb7a-0390-4ed4-8d3b-468921489dc9" containerName="istio-proxy"
Oct 31 19:41:40.585429 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:40.585209    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4abbcb7a-0390-4ed4-8d3b-468921489dc9" containerName="istio-proxy"
Oct 31 19:41:40.585429 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:40.585215    4645 cpu_manager.go:414] "containerMap: removing container" podUID="fd915c45-10cb-4d0f-a519-44bbe338e43d" containerName="istio-proxy"
Oct 31 19:41:40.585429 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:40.585220    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd915c45-10cb-4d0f-a519-44bbe338e43d" containerName="istio-proxy"
Oct 31 19:41:40.585429 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:40.585241    4645 cpu_manager.go:414] "containerMap: removing container" podUID="9bacc59a-7abc-43b3-81dd-a8829710ad35" containerName="istio-proxy"
Oct 31 19:41:40.585429 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:40.585246    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bacc59a-7abc-43b3-81dd-a8829710ad35" containerName="istio-proxy"
Oct 31 19:41:40.585429 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:40.585256    4645 cpu_manager.go:414] "containerMap: removing container" podUID="79f81748-b7d5-4ed1-8f28-4386bba24a79" containerName="istio-proxy"
Oct 31 19:41:40.585429 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:40.585262    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79f81748-b7d5-4ed1-8f28-4386bba24a79" containerName="istio-proxy"
Oct 31 19:41:40.585429 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:40.585370    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="8844ebc7-a83d-4b83-859e-2eb4d1e816c2" containerName="istio-proxy"
Oct 31 19:41:40.585429 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:40.585382    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="585024d3-5768-463e-acdc-81151a7c0461" containerName="istio-proxy"
Oct 31 19:41:40.585429 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:40.585391    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="4abbcb7a-0390-4ed4-8d3b-468921489dc9" containerName="istio-proxy"
Oct 31 19:41:40.585429 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:40.585412    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd915c45-10cb-4d0f-a519-44bbe338e43d" containerName="istio-proxy"
Oct 31 19:41:40.585429 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:40.585419    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="585024d3-5768-463e-acdc-81151a7c0461" containerName="istio-proxy"
Oct 31 19:41:40.585429 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:40.585428    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bacc59a-7abc-43b3-81dd-a8829710ad35" containerName="istio-proxy"
Oct 31 19:41:40.585429 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:40.585435    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="79f81748-b7d5-4ed1-8f28-4386bba24a79" containerName="istio-proxy"
Oct 31 19:41:40.590669 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:40.590654    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a4e312-7b78fb8665-b9pn6"
Oct 31 19:41:40.597120 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:40.597098    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-a4e312\""
Oct 31 19:41:40.597668 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:40.597649    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-a4e312-dockercfg-g6knc\""
Oct 31 19:41:40.601548 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:40.601528    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a4e312-7b78fb8665-b9pn6"]
Oct 31 19:41:40.622902 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:40.622880    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:40.742301 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:40.737687    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zwm6\" (UniqueName: \"kubernetes.io/projected/ee5d853e-f677-4073-979c-3fe141091e4f-kube-api-access-5zwm6\") pod \"payload-processing-e2e-inf-a-a4e312-7b78fb8665-b9pn6\" (UID: \"ee5d853e-f677-4073-979c-3fe141091e4f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a4e312-7b78fb8665-b9pn6"
Oct 31 19:41:40.742301 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:40.737758    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee5d853e-f677-4073-979c-3fe141091e4f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a4e312-7b78fb8665-b9pn6\" (UID: \"ee5d853e-f677-4073-979c-3fe141091e4f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a4e312-7b78fb8665-b9pn6"
Oct 31 19:41:40.867403 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:40.864783    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zwm6\" (UniqueName: \"kubernetes.io/projected/ee5d853e-f677-4073-979c-3fe141091e4f-kube-api-access-5zwm6\") pod \"payload-processing-e2e-inf-a-a4e312-7b78fb8665-b9pn6\" (UID: \"ee5d853e-f677-4073-979c-3fe141091e4f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a4e312-7b78fb8665-b9pn6"
Oct 31 19:41:40.867403 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:40.865188    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee5d853e-f677-4073-979c-3fe141091e4f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a4e312-7b78fb8665-b9pn6\" (UID: \"ee5d853e-f677-4073-979c-3fe141091e4f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a4e312-7b78fb8665-b9pn6"
Oct 31 19:41:40.890352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:40.890280    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee5d853e-f677-4073-979c-3fe141091e4f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a4e312-7b78fb8665-b9pn6\" (UID: \"ee5d853e-f677-4073-979c-3fe141091e4f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a4e312-7b78fb8665-b9pn6"
Oct 31 19:41:40.930373 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:40.927023    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zwm6\" (UniqueName: \"kubernetes.io/projected/ee5d853e-f677-4073-979c-3fe141091e4f-kube-api-access-5zwm6\") pod \"payload-processing-e2e-inf-a-a4e312-7b78fb8665-b9pn6\" (UID: \"ee5d853e-f677-4073-979c-3fe141091e4f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a4e312-7b78fb8665-b9pn6"
Oct 31 19:41:41.207958 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:41.207928    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a4e312-7b78fb8665-b9pn6"
Oct 31 19:41:41.491086 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:41:41.491058    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee5d853e_f677_4073_979c_3fe141091e4f.slice/crio-307366be07f7b00625b380fe243fd00f25591391d47bb79815c2c03be2cde8db WatchSource:0}: Error finding container 307366be07f7b00625b380fe243fd00f25591391d47bb79815c2c03be2cde8db: Status 404 returned error can't find the container with id 307366be07f7b00625b380fe243fd00f25591391d47bb79815c2c03be2cde8db
Oct 31 19:41:41.492121 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:41.492092    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a4e312-7b78fb8665-b9pn6"]
Oct 31 19:41:41.509744 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:41.509718    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a4e312-7b78fb8665-b9pn6" event={"ID":"ee5d853e-f677-4073-979c-3fe141091e4f","Type":"ContainerStarted","Data":"307366be07f7b00625b380fe243fd00f25591391d47bb79815c2c03be2cde8db"}
Oct 31 19:41:41.554739 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:41.554716    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:42.395976 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:42.395943    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:42.526143 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:42.526099    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a4e312-7b78fb8665-b9pn6" event={"ID":"ee5d853e-f677-4073-979c-3fe141091e4f","Type":"ContainerStarted","Data":"3816993d0f891236223070cc61ac1c7eb7116ad4d7512b347b645dc280b7d861"}
Oct 31 19:41:42.527305 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:42.527283    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a4e312-7b78fb8665-b9pn6"
Oct 31 19:41:42.546983 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:42.546930    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-a4e312-7b78fb8665-b9pn6" podStartSLOduration=2.546913484 podStartE2EDuration="2.546913484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:41:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:41:42.544267276 +0000 UTC m=+1333.117501386" watchObservedRunningTime="2026-10-31 19:41:42.546913484 +0000 UTC m=+1333.120147612"
Oct 31 19:41:43.422959 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:43.422906    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:44.834748 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:44.834717    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:45.136064 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:45.135986    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:45.142993 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:45.142963    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e73256-57b5c8849f-n9qzl"]
Oct 31 19:41:45.243382 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:45.243306    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:45.350062 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:45.350031    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:46.403786 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:46.403753    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:46.590061 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:46.590022    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:46.938485 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:46.938445    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:47.427402 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:47.427283    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:47.968841 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:47.968808    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:48.970379 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:48.970343    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:49.871013 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:49.870977    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr"]
Oct 31 19:41:49.871195 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:49.871174    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr" podUID="fbd78997-222e-4952-84fe-1f33dab54c0c" containerName="istio-proxy" containerID="cri-o://5364b424b2114ec610c9b26810324f27f80f430b71e99baec2911f585a296972" gracePeriod=30
Oct 31 19:41:50.136898 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:50.136814    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:50.143613 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:50.143588    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-e73256-57b5c8849f-n9qzl" podUID="f9857546-78a4-42de-8e5d-896fa0f91d42" containerName="payload-processing" containerID="cri-o://909d0db5de2c0c28812f7e8bc73adbab82c8efd7c25605962cc26a0232c8c023" gracePeriod=25
Oct 31 19:41:50.440682 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:50.440658    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e73256-57b5c8849f-n9qzl"
Oct 31 19:41:50.558542 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:50.558511    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9857546-78a4-42de-8e5d-896fa0f91d42-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9857546-78a4-42de-8e5d-896fa0f91d42-plugins-config-volume\") pod \"f9857546-78a4-42de-8e5d-896fa0f91d42\" (UID: \"f9857546-78a4-42de-8e5d-896fa0f91d42\") "
Oct 31 19:41:50.558705 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:50.558685    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9857546-78a4-42de-8e5d-896fa0f91d42-kube-api-access-h2vvq\" (UniqueName: \"kubernetes.io/projected/f9857546-78a4-42de-8e5d-896fa0f91d42-kube-api-access-h2vvq\") pod \"f9857546-78a4-42de-8e5d-896fa0f91d42\" (UID: \"f9857546-78a4-42de-8e5d-896fa0f91d42\") "
Oct 31 19:41:50.558941 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:50.558920    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9857546-78a4-42de-8e5d-896fa0f91d42-plugins-config-volume" pod "f9857546-78a4-42de-8e5d-896fa0f91d42" (UID: "f9857546-78a4-42de-8e5d-896fa0f91d42"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:41:50.559408 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:50.559357    4645 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9857546-78a4-42de-8e5d-896fa0f91d42-plugins-config-volume\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:50.560647 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:50.560625    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9857546-78a4-42de-8e5d-896fa0f91d42-kube-api-access-h2vvq" pod "f9857546-78a4-42de-8e5d-896fa0f91d42" (UID: "f9857546-78a4-42de-8e5d-896fa0f91d42"). InnerVolumeSpecName "kube-api-access-h2vvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:41:50.590032 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:50.590006    4645 generic.go:356] "Generic (PLEG): container finished" podID="f9857546-78a4-42de-8e5d-896fa0f91d42" containerID="909d0db5de2c0c28812f7e8bc73adbab82c8efd7c25605962cc26a0232c8c023" exitCode=0
Oct 31 19:41:50.590124 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:50.590049    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e73256-57b5c8849f-n9qzl"
Oct 31 19:41:50.590124 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:50.590087    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e73256-57b5c8849f-n9qzl" event={"ID":"f9857546-78a4-42de-8e5d-896fa0f91d42","Type":"ContainerDied","Data":"909d0db5de2c0c28812f7e8bc73adbab82c8efd7c25605962cc26a0232c8c023"}
Oct 31 19:41:50.590201 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:50.590139    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e73256-57b5c8849f-n9qzl" event={"ID":"f9857546-78a4-42de-8e5d-896fa0f91d42","Type":"ContainerDied","Data":"db7787792fe6fb7fbe49cf5370e03045a0a2c0bd9c38e659b82b92fa3342d9c8"}
Oct 31 19:41:50.590201 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:50.590163    4645 scope.go:122] "RemoveContainer" containerID="909d0db5de2c0c28812f7e8bc73adbab82c8efd7c25605962cc26a0232c8c023"
Oct 31 19:41:50.604550 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:50.604527    4645 scope.go:122] "RemoveContainer" containerID="909d0db5de2c0c28812f7e8bc73adbab82c8efd7c25605962cc26a0232c8c023"
Oct 31 19:41:50.604805 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:41:50.604787    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"909d0db5de2c0c28812f7e8bc73adbab82c8efd7c25605962cc26a0232c8c023\": container with ID starting with 909d0db5de2c0c28812f7e8bc73adbab82c8efd7c25605962cc26a0232c8c023 not found: ID does not exist" containerID="909d0db5de2c0c28812f7e8bc73adbab82c8efd7c25605962cc26a0232c8c023"
Oct 31 19:41:50.604870 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:50.604811    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"909d0db5de2c0c28812f7e8bc73adbab82c8efd7c25605962cc26a0232c8c023"} err="failed to get container status \"909d0db5de2c0c28812f7e8bc73adbab82c8efd7c25605962cc26a0232c8c023\": rpc error: code = NotFound desc = could not find container \"909d0db5de2c0c28812f7e8bc73adbab82c8efd7c25605962cc26a0232c8c023\": container with ID starting with 909d0db5de2c0c28812f7e8bc73adbab82c8efd7c25605962cc26a0232c8c023 not found: ID does not exist"
Oct 31 19:41:50.614727 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:50.614683    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e73256-57b5c8849f-n9qzl"]
Oct 31 19:41:50.618640 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:50.618620    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e73256-57b5c8849f-n9qzl"]
Oct 31 19:41:50.660703 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:50.660676    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2vvq\" (UniqueName: \"kubernetes.io/projected/f9857546-78a4-42de-8e5d-896fa0f91d42-kube-api-access-h2vvq\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:50.714018 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:50.713991    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:50.826312 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:50.826283    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:51.461650 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:51.461619    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:51.620000 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:51.619969    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9857546-78a4-42de-8e5d-896fa0f91d42" path="/var/lib/kubelet/pods/f9857546-78a4-42de-8e5d-896fa0f91d42/volumes"
Oct 31 19:41:51.685999 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:51.685970    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:52.088148 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:52.088103    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:52.146222 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:52.146195    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-278c04-86645f6fbc-p5qmg"]
Oct 31 19:41:52.146909 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:52.146894    4645 cpu_manager.go:414] "containerMap: removing container" podUID="f9857546-78a4-42de-8e5d-896fa0f91d42" containerName="payload-processing"
Oct 31 19:41:52.146954 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:52.146910    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9857546-78a4-42de-8e5d-896fa0f91d42" containerName="payload-processing"
Oct 31 19:41:52.146954 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:52.146950    4645 cpu_manager.go:414] "containerMap: removing container" podUID="585024d3-5768-463e-acdc-81151a7c0461" containerName="istio-proxy"
Oct 31 19:41:52.147019 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:52.146956    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="585024d3-5768-463e-acdc-81151a7c0461" containerName="istio-proxy"
Oct 31 19:41:52.147073 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:52.147064    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9857546-78a4-42de-8e5d-896fa0f91d42" containerName="payload-processing"
Oct 31 19:41:52.152063 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:52.152046    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-278c04-86645f6fbc-p5qmg"
Oct 31 19:41:52.159631 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:52.159609    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-278c04-dockercfg-5sz6s\""
Oct 31 19:41:52.159754 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:52.159662    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-278c04\""
Oct 31 19:41:52.167723 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:52.167691    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-278c04-86645f6fbc-p5qmg"]
Oct 31 19:41:52.183366 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:52.183300    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sn98s\" (UniqueName: \"kubernetes.io/projected/12fd3774-779d-49d5-8fe2-6804ed6d50ab-kube-api-access-sn98s\") pod \"payload-pre-processing-e2e-inf-a-278c04-86645f6fbc-p5qmg\" (UID: \"12fd3774-779d-49d5-8fe2-6804ed6d50ab\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-278c04-86645f6fbc-p5qmg"
Oct 31 19:41:52.183495 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:52.183453    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12fd3774-779d-49d5-8fe2-6804ed6d50ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-278c04-86645f6fbc-p5qmg\" (UID: \"12fd3774-779d-49d5-8fe2-6804ed6d50ab\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-278c04-86645f6fbc-p5qmg"
Oct 31 19:41:52.197176 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:52.197148    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-278c04-6969d7fcb9-252tn"]
Oct 31 19:41:52.202579 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:52.202564    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-278c04-6969d7fcb9-252tn"
Oct 31 19:41:52.214195 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:52.214169    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-278c04-6969d7fcb9-252tn"]
Oct 31 19:41:52.226853 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:52.226827    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:52.303371 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:52.302625    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sn98s\" (UniqueName: \"kubernetes.io/projected/12fd3774-779d-49d5-8fe2-6804ed6d50ab-kube-api-access-sn98s\") pod \"payload-pre-processing-e2e-inf-a-278c04-86645f6fbc-p5qmg\" (UID: \"12fd3774-779d-49d5-8fe2-6804ed6d50ab\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-278c04-86645f6fbc-p5qmg"
Oct 31 19:41:52.303371 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:52.302789    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12fd3774-779d-49d5-8fe2-6804ed6d50ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-278c04-86645f6fbc-p5qmg\" (UID: \"12fd3774-779d-49d5-8fe2-6804ed6d50ab\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-278c04-86645f6fbc-p5qmg"
Oct 31 19:41:52.327351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:52.303920    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12fd3774-779d-49d5-8fe2-6804ed6d50ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-278c04-86645f6fbc-p5qmg\" (UID: \"12fd3774-779d-49d5-8fe2-6804ed6d50ab\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-278c04-86645f6fbc-p5qmg"
Oct 31 19:41:52.413410 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:52.363394    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sn98s\" (UniqueName: \"kubernetes.io/projected/12fd3774-779d-49d5-8fe2-6804ed6d50ab-kube-api-access-sn98s\") pod \"payload-pre-processing-e2e-inf-a-278c04-86645f6fbc-p5qmg\" (UID: \"12fd3774-779d-49d5-8fe2-6804ed6d50ab\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-278c04-86645f6fbc-p5qmg"
Oct 31 19:41:52.443377 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:52.424395    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bdc77fe-26fe-4f78-8242-4fed7e6f3c37-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-278c04-6969d7fcb9-252tn\" (UID: \"1bdc77fe-26fe-4f78-8242-4fed7e6f3c37\") " pod="openshift-ingress/payload-processing-e2e-inf-a-278c04-6969d7fcb9-252tn"
Oct 31 19:41:52.443377 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:52.424496    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvgw5\" (UniqueName: \"kubernetes.io/projected/1bdc77fe-26fe-4f78-8242-4fed7e6f3c37-kube-api-access-gvgw5\") pod \"payload-processing-e2e-inf-a-278c04-6969d7fcb9-252tn\" (UID: \"1bdc77fe-26fe-4f78-8242-4fed7e6f3c37\") " pod="openshift-ingress/payload-processing-e2e-inf-a-278c04-6969d7fcb9-252tn"
Oct 31 19:41:52.477354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:52.469918    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-278c04-86645f6fbc-p5qmg"
Oct 31 19:41:52.528078 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:52.527676    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bdc77fe-26fe-4f78-8242-4fed7e6f3c37-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-278c04-6969d7fcb9-252tn\" (UID: \"1bdc77fe-26fe-4f78-8242-4fed7e6f3c37\") " pod="openshift-ingress/payload-processing-e2e-inf-a-278c04-6969d7fcb9-252tn"
Oct 31 19:41:52.530924 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:52.530897    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvgw5\" (UniqueName: \"kubernetes.io/projected/1bdc77fe-26fe-4f78-8242-4fed7e6f3c37-kube-api-access-gvgw5\") pod \"payload-processing-e2e-inf-a-278c04-6969d7fcb9-252tn\" (UID: \"1bdc77fe-26fe-4f78-8242-4fed7e6f3c37\") " pod="openshift-ingress/payload-processing-e2e-inf-a-278c04-6969d7fcb9-252tn"
Oct 31 19:41:52.532078 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:52.530734    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bdc77fe-26fe-4f78-8242-4fed7e6f3c37-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-278c04-6969d7fcb9-252tn\" (UID: \"1bdc77fe-26fe-4f78-8242-4fed7e6f3c37\") " pod="openshift-ingress/payload-processing-e2e-inf-a-278c04-6969d7fcb9-252tn"
Oct 31 19:41:52.551414 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:52.551382    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvgw5\" (UniqueName: \"kubernetes.io/projected/1bdc77fe-26fe-4f78-8242-4fed7e6f3c37-kube-api-access-gvgw5\") pod \"payload-processing-e2e-inf-a-278c04-6969d7fcb9-252tn\" (UID: \"1bdc77fe-26fe-4f78-8242-4fed7e6f3c37\") " pod="openshift-ingress/payload-processing-e2e-inf-a-278c04-6969d7fcb9-252tn"
Oct 31 19:41:52.819346 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:52.819264    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9"]
Oct 31 19:41:52.823832 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:52.823807    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-278c04-6969d7fcb9-252tn"
Oct 31 19:41:52.825453 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:52.825418    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9"
Oct 31 19:41:52.849503 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:52.849475    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-a813e2-openshift-default-dockercfg-vfqwg\""
Oct 31 19:41:52.862313 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:52.862281    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9"]
Oct 31 19:41:52.889874 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:41:52.889846    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12fd3774_779d_49d5_8fe2_6804ed6d50ab.slice/crio-259be388723db631f48f8b4b2767ed7b86cf11de87172aea21a3e08b7839aab5 WatchSource:0}: Error finding container 259be388723db631f48f8b4b2767ed7b86cf11de87172aea21a3e08b7839aab5: Status 404 returned error can't find the container with id 259be388723db631f48f8b4b2767ed7b86cf11de87172aea21a3e08b7839aab5
Oct 31 19:41:52.890255 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:52.890237    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-278c04-86645f6fbc-p5qmg"]
Oct 31 19:41:52.899078 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:52.899057    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:52.953072 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:52.953038    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e5cadc6-cd7f-402d-9550-874d7b3d6376-istio-envoy\") pod \"e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9\" (UID: \"7e5cadc6-cd7f-402d-9550-874d7b3d6376\") " pod="openshift-ingress/e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9"
Oct 31 19:41:52.972603 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:52.972558    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e5cadc6-cd7f-402d-9550-874d7b3d6376-istio-podinfo\") pod \"e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9\" (UID: \"7e5cadc6-cd7f-402d-9550-874d7b3d6376\") " pod="openshift-ingress/e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9"
Oct 31 19:41:52.972758 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:52.972712    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e5cadc6-cd7f-402d-9550-874d7b3d6376-istio-token\") pod \"e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9\" (UID: \"7e5cadc6-cd7f-402d-9550-874d7b3d6376\") " pod="openshift-ingress/e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9"
Oct 31 19:41:52.972758 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:52.972743    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e5cadc6-cd7f-402d-9550-874d7b3d6376-workload-certs\") pod \"e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9\" (UID: \"7e5cadc6-cd7f-402d-9550-874d7b3d6376\") " pod="openshift-ingress/e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9"
Oct 31 19:41:52.972880 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:52.972869    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e5cadc6-cd7f-402d-9550-874d7b3d6376-credential-socket\") pod \"e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9\" (UID: \"7e5cadc6-cd7f-402d-9550-874d7b3d6376\") " pod="openshift-ingress/e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9"
Oct 31 19:41:52.972941 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:52.972906    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e5cadc6-cd7f-402d-9550-874d7b3d6376-istio-data\") pod \"e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9\" (UID: \"7e5cadc6-cd7f-402d-9550-874d7b3d6376\") " pod="openshift-ingress/e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9"
Oct 31 19:41:52.972999 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:52.972978    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khb2x\" (UniqueName: \"kubernetes.io/projected/7e5cadc6-cd7f-402d-9550-874d7b3d6376-kube-api-access-khb2x\") pod \"e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9\" (UID: \"7e5cadc6-cd7f-402d-9550-874d7b3d6376\") " pod="openshift-ingress/e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9"
Oct 31 19:41:52.973151 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:52.973132    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e5cadc6-cd7f-402d-9550-874d7b3d6376-istiod-ca-cert\") pod \"e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9\" (UID: \"7e5cadc6-cd7f-402d-9550-874d7b3d6376\") " pod="openshift-ingress/e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9"
Oct 31 19:41:52.973276 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:52.973254    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e5cadc6-cd7f-402d-9550-874d7b3d6376-workload-socket\") pod \"e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9\" (UID: \"7e5cadc6-cd7f-402d-9550-874d7b3d6376\") " pod="openshift-ingress/e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9"
Oct 31 19:41:53.187736 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:53.159896    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e5cadc6-cd7f-402d-9550-874d7b3d6376-istio-podinfo\") pod \"e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9\" (UID: \"7e5cadc6-cd7f-402d-9550-874d7b3d6376\") " pod="openshift-ingress/e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9"
Oct 31 19:41:53.187736 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:53.159996    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e5cadc6-cd7f-402d-9550-874d7b3d6376-istio-token\") pod \"e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9\" (UID: \"7e5cadc6-cd7f-402d-9550-874d7b3d6376\") " pod="openshift-ingress/e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9"
Oct 31 19:41:53.187736 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:53.160026    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e5cadc6-cd7f-402d-9550-874d7b3d6376-workload-certs\") pod \"e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9\" (UID: \"7e5cadc6-cd7f-402d-9550-874d7b3d6376\") " pod="openshift-ingress/e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9"
Oct 31 19:41:53.187736 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:53.160116    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e5cadc6-cd7f-402d-9550-874d7b3d6376-credential-socket\") pod \"e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9\" (UID: \"7e5cadc6-cd7f-402d-9550-874d7b3d6376\") " pod="openshift-ingress/e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9"
Oct 31 19:41:53.187736 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:53.160150    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e5cadc6-cd7f-402d-9550-874d7b3d6376-istio-data\") pod \"e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9\" (UID: \"7e5cadc6-cd7f-402d-9550-874d7b3d6376\") " pod="openshift-ingress/e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9"
Oct 31 19:41:53.187736 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:53.160240    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khb2x\" (UniqueName: \"kubernetes.io/projected/7e5cadc6-cd7f-402d-9550-874d7b3d6376-kube-api-access-khb2x\") pod \"e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9\" (UID: \"7e5cadc6-cd7f-402d-9550-874d7b3d6376\") " pod="openshift-ingress/e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9"
Oct 31 19:41:53.187736 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:53.160374    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e5cadc6-cd7f-402d-9550-874d7b3d6376-istiod-ca-cert\") pod \"e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9\" (UID: \"7e5cadc6-cd7f-402d-9550-874d7b3d6376\") " pod="openshift-ingress/e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9"
Oct 31 19:41:53.187736 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:53.160476    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e5cadc6-cd7f-402d-9550-874d7b3d6376-workload-socket\") pod \"e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9\" (UID: \"7e5cadc6-cd7f-402d-9550-874d7b3d6376\") " pod="openshift-ingress/e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9"
Oct 31 19:41:53.187736 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:53.160518    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e5cadc6-cd7f-402d-9550-874d7b3d6376-istio-envoy\") pod \"e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9\" (UID: \"7e5cadc6-cd7f-402d-9550-874d7b3d6376\") " pod="openshift-ingress/e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9"
Oct 31 19:41:53.187736 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:53.180740    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e5cadc6-cd7f-402d-9550-874d7b3d6376-credential-socket\") pod \"e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9\" (UID: \"7e5cadc6-cd7f-402d-9550-874d7b3d6376\") " pod="openshift-ingress/e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9"
Oct 31 19:41:53.187736 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:53.181210    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e5cadc6-cd7f-402d-9550-874d7b3d6376-workload-certs\") pod \"e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9\" (UID: \"7e5cadc6-cd7f-402d-9550-874d7b3d6376\") " pod="openshift-ingress/e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9"
Oct 31 19:41:53.190916 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:53.187862    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e5cadc6-cd7f-402d-9550-874d7b3d6376-istio-podinfo\") pod \"e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9\" (UID: \"7e5cadc6-cd7f-402d-9550-874d7b3d6376\") " pod="openshift-ingress/e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9"
Oct 31 19:41:53.190916 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:53.187919    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e5cadc6-cd7f-402d-9550-874d7b3d6376-istio-data\") pod \"e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9\" (UID: \"7e5cadc6-cd7f-402d-9550-874d7b3d6376\") " pod="openshift-ingress/e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9"
Oct 31 19:41:53.190916 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:53.188829    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e5cadc6-cd7f-402d-9550-874d7b3d6376-istiod-ca-cert\") pod \"e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9\" (UID: \"7e5cadc6-cd7f-402d-9550-874d7b3d6376\") " pod="openshift-ingress/e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9"
Oct 31 19:41:53.190916 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:53.188829    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e5cadc6-cd7f-402d-9550-874d7b3d6376-workload-socket\") pod \"e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9\" (UID: \"7e5cadc6-cd7f-402d-9550-874d7b3d6376\") " pod="openshift-ingress/e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9"
Oct 31 19:41:53.193798 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:53.193296    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e5cadc6-cd7f-402d-9550-874d7b3d6376-istio-envoy\") pod \"e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9\" (UID: \"7e5cadc6-cd7f-402d-9550-874d7b3d6376\") " pod="openshift-ingress/e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9"
Oct 31 19:41:53.194781 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:53.194752    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:53.207725 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:53.207675    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khb2x\" (UniqueName: \"kubernetes.io/projected/7e5cadc6-cd7f-402d-9550-874d7b3d6376-kube-api-access-khb2x\") pod \"e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9\" (UID: \"7e5cadc6-cd7f-402d-9550-874d7b3d6376\") " pod="openshift-ingress/e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9"
Oct 31 19:41:53.208582 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:53.208490    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e5cadc6-cd7f-402d-9550-874d7b3d6376-istio-token\") pod \"e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9\" (UID: \"7e5cadc6-cd7f-402d-9550-874d7b3d6376\") " pod="openshift-ingress/e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9"
Oct 31 19:41:53.372818 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:41:53.372789    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bdc77fe_26fe_4f78_8242_4fed7e6f3c37.slice/crio-31c3d412f6e8973b6f1ad22ef0f638e2f59ff82a0b8ad65c5b769b75c43ada9d WatchSource:0}: Error finding container 31c3d412f6e8973b6f1ad22ef0f638e2f59ff82a0b8ad65c5b769b75c43ada9d: Status 404 returned error can't find the container with id 31c3d412f6e8973b6f1ad22ef0f638e2f59ff82a0b8ad65c5b769b75c43ada9d
Oct 31 19:41:53.373137 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:53.373113    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-278c04-6969d7fcb9-252tn"]
Oct 31 19:41:53.481732 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:53.481702    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9"
Oct 31 19:41:53.669829 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:53.669798    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-278c04-86645f6fbc-p5qmg" event={"ID":"12fd3774-779d-49d5-8fe2-6804ed6d50ab","Type":"ContainerStarted","Data":"9ee0280439447d0096853e72d3fea715a0b4d7e8367694f4b17124d87eb9a499"}
Oct 31 19:41:53.670032 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:53.669865    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-278c04-86645f6fbc-p5qmg"
Oct 31 19:41:53.670032 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:53.669889    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-278c04-86645f6fbc-p5qmg" event={"ID":"12fd3774-779d-49d5-8fe2-6804ed6d50ab","Type":"ContainerStarted","Data":"259be388723db631f48f8b4b2767ed7b86cf11de87172aea21a3e08b7839aab5"}
Oct 31 19:41:53.671458 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:53.671434    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-278c04-6969d7fcb9-252tn" event={"ID":"1bdc77fe-26fe-4f78-8242-4fed7e6f3c37","Type":"ContainerStarted","Data":"0737d698d1ae9996feb82c0dc6627c55203dbda8a5e8425fdeabfdc2b56a1dd5"}
Oct 31 19:41:53.671596 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:53.671473    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-278c04-6969d7fcb9-252tn" event={"ID":"1bdc77fe-26fe-4f78-8242-4fed7e6f3c37","Type":"ContainerStarted","Data":"31c3d412f6e8973b6f1ad22ef0f638e2f59ff82a0b8ad65c5b769b75c43ada9d"}
Oct 31 19:41:53.671596 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:53.671506    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-278c04-6969d7fcb9-252tn"
Oct 31 19:41:53.695206 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:53.695157    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-278c04-86645f6fbc-p5qmg" podStartSLOduration=1.6951422809999999 podStartE2EDuration="1.695142281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:41:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:41:53.689518458 +0000 UTC m=+1344.262752615" watchObservedRunningTime="2026-10-31 19:41:53.695142281 +0000 UTC m=+1344.268376410"
Oct 31 19:41:53.711771 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:53.711730    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-278c04-6969d7fcb9-252tn" podStartSLOduration=1.711715836 podStartE2EDuration="1.711715836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:41:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:41:53.709910438 +0000 UTC m=+1344.283144566" watchObservedRunningTime="2026-10-31 19:41:53.711715836 +0000 UTC m=+1344.284949963"
Oct 31 19:41:53.754078 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:53.754055    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:53.792411 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:41:53.792385    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e5cadc6_cd7f_402d_9550_874d7b3d6376.slice/crio-917a571def81aba47cc4a8bee47c0b6ba1bc9897deba983b9648777d673eeb28 WatchSource:0}: Error finding container 917a571def81aba47cc4a8bee47c0b6ba1bc9897deba983b9648777d673eeb28: Status 404 returned error can't find the container with id 917a571def81aba47cc4a8bee47c0b6ba1bc9897deba983b9648777d673eeb28
Oct 31 19:41:53.792980 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:53.792956    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9"]
Oct 31 19:41:53.795583 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:53.795555    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:41:53.795669 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:53.795614    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:41:53.795669 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:53.795644    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:41:54.583354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:54.581671    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a4e312-7b78fb8665-b9pn6"
Oct 31 19:41:54.783029 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:54.782990    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9" event={"ID":"7e5cadc6-cd7f-402d-9550-874d7b3d6376","Type":"ContainerStarted","Data":"2cf907396d4b6896492bbd4c364a1b67a16f35568fb0ef4e23fe0ed52b3a8514"}
Oct 31 19:41:54.783198 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:54.783049    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9" event={"ID":"7e5cadc6-cd7f-402d-9550-874d7b3d6376","Type":"ContainerStarted","Data":"917a571def81aba47cc4a8bee47c0b6ba1bc9897deba983b9648777d673eeb28"}
Oct 31 19:41:54.812052 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:54.811993    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9" podStartSLOduration=2.811979463 podStartE2EDuration="2.811979463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:41:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:41:54.805835296 +0000 UTC m=+1345.379069437" watchObservedRunningTime="2026-10-31 19:41:54.811979463 +0000 UTC m=+1345.385213590"
Oct 31 19:41:55.010942 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:55.010913    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:55.212447 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:55.212422    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr"
Oct 31 19:41:55.315955 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:55.315922    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbd78997-222e-4952-84fe-1f33dab54c0c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbd78997-222e-4952-84fe-1f33dab54c0c-workload-socket\") pod \"fbd78997-222e-4952-84fe-1f33dab54c0c\" (UID: \"fbd78997-222e-4952-84fe-1f33dab54c0c\") "
Oct 31 19:41:55.316103 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:55.315988    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbd78997-222e-4952-84fe-1f33dab54c0c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbd78997-222e-4952-84fe-1f33dab54c0c-credential-socket\") pod \"fbd78997-222e-4952-84fe-1f33dab54c0c\" (UID: \"fbd78997-222e-4952-84fe-1f33dab54c0c\") "
Oct 31 19:41:55.316103 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:55.316082    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbd78997-222e-4952-84fe-1f33dab54c0c-kube-api-access-cm729\" (UniqueName: \"kubernetes.io/projected/fbd78997-222e-4952-84fe-1f33dab54c0c-kube-api-access-cm729\") pod \"fbd78997-222e-4952-84fe-1f33dab54c0c\" (UID: \"fbd78997-222e-4952-84fe-1f33dab54c0c\") "
Oct 31 19:41:55.316215 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:55.316148    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbd78997-222e-4952-84fe-1f33dab54c0c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbd78997-222e-4952-84fe-1f33dab54c0c-istiod-ca-cert\") pod \"fbd78997-222e-4952-84fe-1f33dab54c0c\" (UID: \"fbd78997-222e-4952-84fe-1f33dab54c0c\") "
Oct 31 19:41:55.316215 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:55.316197    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbd78997-222e-4952-84fe-1f33dab54c0c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbd78997-222e-4952-84fe-1f33dab54c0c-istio-data\") pod \"fbd78997-222e-4952-84fe-1f33dab54c0c\" (UID: \"fbd78997-222e-4952-84fe-1f33dab54c0c\") "
Oct 31 19:41:55.316373 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:55.316256    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbd78997-222e-4952-84fe-1f33dab54c0c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbd78997-222e-4952-84fe-1f33dab54c0c-istio-envoy\") pod \"fbd78997-222e-4952-84fe-1f33dab54c0c\" (UID: \"fbd78997-222e-4952-84fe-1f33dab54c0c\") "
Oct 31 19:41:55.316373 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:55.316224    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbd78997-222e-4952-84fe-1f33dab54c0c-workload-socket" pod "fbd78997-222e-4952-84fe-1f33dab54c0c" (UID: "fbd78997-222e-4952-84fe-1f33dab54c0c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:55.316373 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:55.316295    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbd78997-222e-4952-84fe-1f33dab54c0c-credential-socket" pod "fbd78997-222e-4952-84fe-1f33dab54c0c" (UID: "fbd78997-222e-4952-84fe-1f33dab54c0c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:55.316373 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:55.316315    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fbd78997-222e-4952-84fe-1f33dab54c0c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbd78997-222e-4952-84fe-1f33dab54c0c-istio-podinfo\") pod \"fbd78997-222e-4952-84fe-1f33dab54c0c\" (UID: \"fbd78997-222e-4952-84fe-1f33dab54c0c\") "
Oct 31 19:41:55.316577 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:55.316373    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbd78997-222e-4952-84fe-1f33dab54c0c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbd78997-222e-4952-84fe-1f33dab54c0c-workload-certs\") pod \"fbd78997-222e-4952-84fe-1f33dab54c0c\" (UID: \"fbd78997-222e-4952-84fe-1f33dab54c0c\") "
Oct 31 19:41:55.316577 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:55.316469    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbd78997-222e-4952-84fe-1f33dab54c0c-istio-data" pod "fbd78997-222e-4952-84fe-1f33dab54c0c" (UID: "fbd78997-222e-4952-84fe-1f33dab54c0c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:55.316577 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:55.316504    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbd78997-222e-4952-84fe-1f33dab54c0c-istio-token\" (UniqueName: \"kubernetes.io/projected/fbd78997-222e-4952-84fe-1f33dab54c0c-istio-token\") pod \"fbd78997-222e-4952-84fe-1f33dab54c0c\" (UID: \"fbd78997-222e-4952-84fe-1f33dab54c0c\") "
Oct 31 19:41:55.316577 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:55.316535    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbd78997-222e-4952-84fe-1f33dab54c0c-istiod-ca-cert" pod "fbd78997-222e-4952-84fe-1f33dab54c0c" (UID: "fbd78997-222e-4952-84fe-1f33dab54c0c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:41:55.316758 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:55.316674    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbd78997-222e-4952-84fe-1f33dab54c0c-workload-certs" pod "fbd78997-222e-4952-84fe-1f33dab54c0c" (UID: "fbd78997-222e-4952-84fe-1f33dab54c0c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:55.317244 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:55.317217    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbd78997-222e-4952-84fe-1f33dab54c0c-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:55.317415 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:55.317248    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbd78997-222e-4952-84fe-1f33dab54c0c-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:55.317415 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:55.317265    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbd78997-222e-4952-84fe-1f33dab54c0c-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:55.317415 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:55.317280    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbd78997-222e-4952-84fe-1f33dab54c0c-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:55.317415 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:55.317297    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbd78997-222e-4952-84fe-1f33dab54c0c-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:55.318579 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:55.318550    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbd78997-222e-4952-84fe-1f33dab54c0c-kube-api-access-cm729" pod "fbd78997-222e-4952-84fe-1f33dab54c0c" (UID: "fbd78997-222e-4952-84fe-1f33dab54c0c"). InnerVolumeSpecName "kube-api-access-cm729". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:41:55.318668 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:55.318631    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbd78997-222e-4952-84fe-1f33dab54c0c-istio-envoy" pod "fbd78997-222e-4952-84fe-1f33dab54c0c" (UID: "fbd78997-222e-4952-84fe-1f33dab54c0c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:41:55.318668 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:55.318643    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbd78997-222e-4952-84fe-1f33dab54c0c-istio-token" pod "fbd78997-222e-4952-84fe-1f33dab54c0c" (UID: "fbd78997-222e-4952-84fe-1f33dab54c0c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:41:55.318668 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:55.318648    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fbd78997-222e-4952-84fe-1f33dab54c0c-istio-podinfo" pod "fbd78997-222e-4952-84fe-1f33dab54c0c" (UID: "fbd78997-222e-4952-84fe-1f33dab54c0c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:41:55.418498 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:55.418470    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cm729\" (UniqueName: \"kubernetes.io/projected/fbd78997-222e-4952-84fe-1f33dab54c0c-kube-api-access-cm729\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:55.418498 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:55.418496    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbd78997-222e-4952-84fe-1f33dab54c0c-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:55.418498 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:55.418507    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbd78997-222e-4952-84fe-1f33dab54c0c-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:55.418727 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:55.418516    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbd78997-222e-4952-84fe-1f33dab54c0c-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:41:55.482557 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:55.482489    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9"
Oct 31 19:41:55.488469 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:55.488445    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9"
Oct 31 19:41:55.790176 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:55.790095    4645 generic.go:356] "Generic (PLEG): container finished" podID="fbd78997-222e-4952-84fe-1f33dab54c0c" containerID="5364b424b2114ec610c9b26810324f27f80f430b71e99baec2911f585a296972" exitCode=0
Oct 31 19:41:55.790582 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:55.790192    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr"
Oct 31 19:41:55.790582 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:55.790227    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr" event={"ID":"fbd78997-222e-4952-84fe-1f33dab54c0c","Type":"ContainerDied","Data":"5364b424b2114ec610c9b26810324f27f80f430b71e99baec2911f585a296972"}
Oct 31 19:41:55.790582 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:55.790279    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr" event={"ID":"fbd78997-222e-4952-84fe-1f33dab54c0c","Type":"ContainerDied","Data":"5af469c9949eff9bc9e188d683e667898f740e532131e282547f192643320734"}
Oct 31 19:41:55.790582 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:55.790303    4645 scope.go:122] "RemoveContainer" containerID="5364b424b2114ec610c9b26810324f27f80f430b71e99baec2911f585a296972"
Oct 31 19:41:55.790801 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:55.790706    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9"
Oct 31 19:41:55.791838 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:55.791817    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9"
Oct 31 19:41:55.805308 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:55.805279    4645 scope.go:122] "RemoveContainer" containerID="5364b424b2114ec610c9b26810324f27f80f430b71e99baec2911f585a296972"
Oct 31 19:41:55.805761 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:41:55.805740    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5364b424b2114ec610c9b26810324f27f80f430b71e99baec2911f585a296972\": container with ID starting with 5364b424b2114ec610c9b26810324f27f80f430b71e99baec2911f585a296972 not found: ID does not exist" containerID="5364b424b2114ec610c9b26810324f27f80f430b71e99baec2911f585a296972"
Oct 31 19:41:55.805840 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:55.805775    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5364b424b2114ec610c9b26810324f27f80f430b71e99baec2911f585a296972"} err="failed to get container status \"5364b424b2114ec610c9b26810324f27f80f430b71e99baec2911f585a296972\": rpc error: code = NotFound desc = could not find container \"5364b424b2114ec610c9b26810324f27f80f430b71e99baec2911f585a296972\": container with ID starting with 5364b424b2114ec610c9b26810324f27f80f430b71e99baec2911f585a296972 not found: ID does not exist"
Oct 31 19:41:55.856104 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:55.856067    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr"]
Oct 31 19:41:55.864199 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:55.862913    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-s8lwr"]
Oct 31 19:41:55.971406 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:55.971376    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:57.497354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:57.497086    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:57.624915 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:57.624880    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:57.679650 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:57.677076    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbd78997-222e-4952-84fe-1f33dab54c0c" path="/var/lib/kubelet/pods/fbd78997-222e-4952-84fe-1f33dab54c0c/volumes"
Oct 31 19:41:57.895402 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:57.892913    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:58.247388 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:58.235711    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:58.691451 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:58.691401    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n" podUID="dbb97168-db93-4757-9a1b-f9c5eb704fc5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:41:59.210037 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:59.177263    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:59.253351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:59.245313    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669" podUID="9285bee0-bf7c-4fd4-9da7-a1db021cae05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:41:59.359353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:59.334173    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:41:59.403354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:59.402385    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a813e2-747f99fd79-prchw"]
Oct 31 19:41:59.432357 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:59.424229    4645 cpu_manager.go:414] "containerMap: removing container" podUID="fbd78997-222e-4952-84fe-1f33dab54c0c" containerName="istio-proxy"
Oct 31 19:41:59.432357 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:59.424260    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbd78997-222e-4952-84fe-1f33dab54c0c" containerName="istio-proxy"
Oct 31 19:41:59.524557 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:59.523225    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbd78997-222e-4952-84fe-1f33dab54c0c" containerName="istio-proxy"
Oct 31 19:41:59.797410 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:59.796304    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc7pc\" (UniqueName: \"kubernetes.io/projected/38d1e68e-f404-4056-8176-14f7205b2d99-kube-api-access-lc7pc\") pod \"payload-pre-processing-e2e-api-b-a813e2-747f99fd79-prchw\" (UID: \"38d1e68e-f404-4056-8176-14f7205b2d99\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a813e2-747f99fd79-prchw"
Oct 31 19:41:59.797410 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:59.796505    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38d1e68e-f404-4056-8176-14f7205b2d99-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a813e2-747f99fd79-prchw\" (UID: \"38d1e68e-f404-4056-8176-14f7205b2d99\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a813e2-747f99fd79-prchw"
Oct 31 19:41:59.854643 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:59.854614    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a813e2-747f99fd79-prchw"
Oct 31 19:41:59.867849 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:59.867825    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-a813e2-dockercfg-dq87w\""
Oct 31 19:41:59.868170 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:59.868152    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-a813e2\""
Oct 31 19:41:59.909323 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:59.909293    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38d1e68e-f404-4056-8176-14f7205b2d99-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a813e2-747f99fd79-prchw\" (UID: \"38d1e68e-f404-4056-8176-14f7205b2d99\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a813e2-747f99fd79-prchw"
Oct 31 19:41:59.969582 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:59.969552    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc7pc\" (UniqueName: \"kubernetes.io/projected/38d1e68e-f404-4056-8176-14f7205b2d99-kube-api-access-lc7pc\") pod \"payload-pre-processing-e2e-api-b-a813e2-747f99fd79-prchw\" (UID: \"38d1e68e-f404-4056-8176-14f7205b2d99\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a813e2-747f99fd79-prchw"
Oct 31 19:41:59.986370 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:59.982980    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a813e2-dbc7795cc-vqxqm"]
Oct 31 19:41:59.986370 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:41:59.984669    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38d1e68e-f404-4056-8176-14f7205b2d99-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a813e2-747f99fd79-prchw\" (UID: \"38d1e68e-f404-4056-8176-14f7205b2d99\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a813e2-747f99fd79-prchw"
Oct 31 19:42:00.015352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:00.007003    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a813e2-747f99fd79-prchw"]
Oct 31 19:42:00.015352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:00.007228    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a813e2-dbc7795cc-vqxqm"]
Oct 31 19:42:00.015352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:00.007718    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a813e2-dbc7795cc-vqxqm"
Oct 31 19:42:00.039394 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:00.038132    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc7pc\" (UniqueName: \"kubernetes.io/projected/38d1e68e-f404-4056-8176-14f7205b2d99-kube-api-access-lc7pc\") pod \"payload-pre-processing-e2e-api-b-a813e2-747f99fd79-prchw\" (UID: \"38d1e68e-f404-4056-8176-14f7205b2d99\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a813e2-747f99fd79-prchw"
Oct 31 19:42:00.084401 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:00.073895    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac45690b-da7e-4ae0-81dc-6cf69551f665-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a813e2-dbc7795cc-vqxqm\" (UID: \"ac45690b-da7e-4ae0-81dc-6cf69551f665\") " pod="openshift-ingress/payload-processing-e2e-api-b-a813e2-dbc7795cc-vqxqm"
Oct 31 19:42:00.084401 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:00.074198    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m49z\" (UniqueName: \"kubernetes.io/projected/ac45690b-da7e-4ae0-81dc-6cf69551f665-kube-api-access-2m49z\") pod \"payload-processing-e2e-api-b-a813e2-dbc7795cc-vqxqm\" (UID: \"ac45690b-da7e-4ae0-81dc-6cf69551f665\") " pod="openshift-ingress/payload-processing-e2e-api-b-a813e2-dbc7795cc-vqxqm"
Oct 31 19:42:00.161352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:00.160596    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4" podUID="6032a997-b795-4561-805c-110cf67a9d5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:42:00.190351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:00.182849    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x" podUID="a774e286-dcb8-4a2f-8541-26bf231eb08d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:42:00.190351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:00.183141    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw" podUID="1287db54-a587-4c11-9eb6-1606de555894" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": context deadline exceeded"
Oct 31 19:42:00.190351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:00.187041    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac45690b-da7e-4ae0-81dc-6cf69551f665-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a813e2-dbc7795cc-vqxqm\" (UID: \"ac45690b-da7e-4ae0-81dc-6cf69551f665\") " pod="openshift-ingress/payload-processing-e2e-api-b-a813e2-dbc7795cc-vqxqm"
Oct 31 19:42:00.190351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:00.187250    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m49z\" (UniqueName: \"kubernetes.io/projected/ac45690b-da7e-4ae0-81dc-6cf69551f665-kube-api-access-2m49z\") pod \"payload-processing-e2e-api-b-a813e2-dbc7795cc-vqxqm\" (UID: \"ac45690b-da7e-4ae0-81dc-6cf69551f665\") " pod="openshift-ingress/payload-processing-e2e-api-b-a813e2-dbc7795cc-vqxqm"
Oct 31 19:42:00.214352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:00.199151    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac45690b-da7e-4ae0-81dc-6cf69551f665-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a813e2-dbc7795cc-vqxqm\" (UID: \"ac45690b-da7e-4ae0-81dc-6cf69551f665\") " pod="openshift-ingress/payload-processing-e2e-api-b-a813e2-dbc7795cc-vqxqm"
Oct 31 19:42:00.227647 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:00.227083    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m49z\" (UniqueName: \"kubernetes.io/projected/ac45690b-da7e-4ae0-81dc-6cf69551f665-kube-api-access-2m49z\") pod \"payload-processing-e2e-api-b-a813e2-dbc7795cc-vqxqm\" (UID: \"ac45690b-da7e-4ae0-81dc-6cf69551f665\") " pod="openshift-ingress/payload-processing-e2e-api-b-a813e2-dbc7795cc-vqxqm"
Oct 31 19:42:00.251351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:00.237148    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:00.251351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:00.247395    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669" podUID="9285bee0-bf7c-4fd4-9da7-a1db021cae05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 31 19:42:00.251351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:00.247926    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669"
Oct 31 19:42:00.322815 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:00.322573    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a813e2-747f99fd79-prchw"
Oct 31 19:42:00.403195 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:00.368950    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l" podUID="12bea86c-61e5-465e-82fb-b4673a4d1be9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:42:00.409039 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:00.408427    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a813e2-dbc7795cc-vqxqm"
Oct 31 19:42:00.627733 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:00.617598    4645 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669" podUID="9285bee0-bf7c-4fd4-9da7-a1db021cae05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": context deadline exceeded"
Oct 31 19:42:00.714352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:00.700405    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n" podUID="dbb97168-db93-4757-9a1b-f9c5eb704fc5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:42:01.171352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:01.165035    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4" podUID="6032a997-b795-4561-805c-110cf67a9d5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:42:01.171352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:01.165629    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4"
Oct 31 19:42:01.228356 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:01.220912    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw" podUID="1287db54-a587-4c11-9eb6-1606de555894" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:42:01.259353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:01.221203    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x" podUID="a774e286-dcb8-4a2f-8541-26bf231eb08d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 19:42:01.259353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:01.238901    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw"
Oct 31 19:42:01.259353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:01.238949    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x"
Oct 31 19:42:01.259353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:01.253108    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669" podUID="9285bee0-bf7c-4fd4-9da7-a1db021cae05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:42:01.377354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:01.376798    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l" podUID="12bea86c-61e5-465e-82fb-b4673a4d1be9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:42:01.402350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:01.386503    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l"
Oct 31 19:42:01.454406 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:01.453158    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:01.587365 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:01.580455    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-03dfee/test-model-03dfee-kserve-68c5b8bcbf-cwgcs"]
Oct 31 19:42:02.058418 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:02.039561    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-9th7f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:42:02.058418 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:02.039639    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9th7f" podUID="76bc1f3a-74b3-49b3-bde1-cc3b305260d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:42:02.272352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:02.249128    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x" podUID="a774e286-dcb8-4a2f-8541-26bf231eb08d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:42:02.507352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:02.264183    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw" podUID="d504b2d3-c26d-4dcf-a7eb-efe20732ae4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 31 19:42:02.516379 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:02.264348    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw" podUID="1287db54-a587-4c11-9eb6-1606de555894" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": context deadline exceeded"
Oct 31 19:42:02.540409 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:02.539950    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4" podUID="6032a997-b795-4561-805c-110cf67a9d5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 31 19:42:02.561358 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:02.560701    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-6k6cd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:42:02.561358 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:02.560760    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6k6cd" podUID="a8fca631-527d-46c0-9197-5edd7762adb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:42:02.786350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:02.383500    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l" podUID="12bea86c-61e5-465e-82fb-b4673a4d1be9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 31 19:42:03.084406 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:02.859221    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-88sql" podUID="53abb174-56be-4e64-aadc-e871ec9c40f3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:42:03.084406 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:02.859341    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-wntcw" podUID="d6b653ca-0bd2-488f-925b-02a9a8dc509b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded"
Oct 31 19:42:03.198566 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:03.198533    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-03dfee/test-model-03dfee-kserve-68c5b8bcbf-cwgcs"
Oct 31 19:42:03.212387 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:03.206862    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbsgn" podUID="280abee2-f10e-4a96-9735-a3aadde28b69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:42:03.346536 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:03.346458    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-03dfee\"/\"default-dockercfg-tpm9r\""
Oct 31 19:42:03.406429 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:03.383670    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669" podUID="9285bee0-bf7c-4fd4-9da7-a1db021cae05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 31 19:42:03.460360 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:03.450161    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-03dfee\"/\"openshift-service-ca.crt\""
Oct 31 19:42:03.490681 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:03.472943    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-03dfee\"/\"test-model-03dfee-kserve-self-signed-certs\""
Oct 31 19:42:03.515352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:03.511115    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n" podUID="dbb97168-db93-4757-9a1b-f9c5eb704fc5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 19:42:03.547584 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:03.547536    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4" podUID="6032a997-b795-4561-805c-110cf67a9d5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:42:03.561948 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:03.521862    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x" podUID="a774e286-dcb8-4a2f-8541-26bf231eb08d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 19:42:03.568919 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:03.521891    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw" podUID="d504b2d3-c26d-4dcf-a7eb-efe20732ae4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:42:03.823439 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:03.522914    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:03.823439 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:03.527413    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-03dfee\"/\"kube-root-ca.crt\""
Oct 31 19:42:03.823439 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:03.521813    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw" podUID="1287db54-a587-4c11-9eb6-1606de555894" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": context deadline exceeded"
Oct 31 19:42:03.823439 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:03.617133    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e861821-e221-44d2-956a-66f0f656e1b5-tls-certs\") pod \"test-model-03dfee-kserve-68c5b8bcbf-cwgcs\" (UID: \"6e861821-e221-44d2-956a-66f0f656e1b5\") " pod="ai-tenant-e2e-inf-a-03dfee/test-model-03dfee-kserve-68c5b8bcbf-cwgcs"
Oct 31 19:42:03.823439 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:03.819758    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e861821-e221-44d2-956a-66f0f656e1b5-home\") pod \"test-model-03dfee-kserve-68c5b8bcbf-cwgcs\" (UID: \"6e861821-e221-44d2-956a-66f0f656e1b5\") " pod="ai-tenant-e2e-inf-a-03dfee/test-model-03dfee-kserve-68c5b8bcbf-cwgcs"
Oct 31 19:42:03.823439 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:03.819864    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e861821-e221-44d2-956a-66f0f656e1b5-dshm\") pod \"test-model-03dfee-kserve-68c5b8bcbf-cwgcs\" (UID: \"6e861821-e221-44d2-956a-66f0f656e1b5\") " pod="ai-tenant-e2e-inf-a-03dfee/test-model-03dfee-kserve-68c5b8bcbf-cwgcs"
Oct 31 19:42:03.823439 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:03.820167    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gp6z\" (UniqueName: \"kubernetes.io/projected/6e861821-e221-44d2-956a-66f0f656e1b5-kube-api-access-5gp6z\") pod \"test-model-03dfee-kserve-68c5b8bcbf-cwgcs\" (UID: \"6e861821-e221-44d2-956a-66f0f656e1b5\") " pod="ai-tenant-e2e-inf-a-03dfee/test-model-03dfee-kserve-68c5b8bcbf-cwgcs"
Oct 31 19:42:03.823439 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:03.820357    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e861821-e221-44d2-956a-66f0f656e1b5-model-cache\") pod \"test-model-03dfee-kserve-68c5b8bcbf-cwgcs\" (UID: \"6e861821-e221-44d2-956a-66f0f656e1b5\") " pod="ai-tenant-e2e-inf-a-03dfee/test-model-03dfee-kserve-68c5b8bcbf-cwgcs"
Oct 31 19:42:03.823439 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:03.820399    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e861821-e221-44d2-956a-66f0f656e1b5-tmp-dir\") pod \"test-model-03dfee-kserve-68c5b8bcbf-cwgcs\" (UID: \"6e861821-e221-44d2-956a-66f0f656e1b5\") " pod="ai-tenant-e2e-inf-a-03dfee/test-model-03dfee-kserve-68c5b8bcbf-cwgcs"
Oct 31 19:42:03.823439 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:03.702277    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-03dfee/test-model-03dfee-kserve-68c5b8bcbf-cwgcs"]
Oct 31 19:42:03.823439 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:03.820804    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n"
Oct 31 19:42:03.846382 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:03.800427    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l" podUID="12bea86c-61e5-465e-82fb-b4673a4d1be9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:42:04.057131 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:04.057097    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e861821-e221-44d2-956a-66f0f656e1b5-home\") pod \"test-model-03dfee-kserve-68c5b8bcbf-cwgcs\" (UID: \"6e861821-e221-44d2-956a-66f0f656e1b5\") " pod="ai-tenant-e2e-inf-a-03dfee/test-model-03dfee-kserve-68c5b8bcbf-cwgcs"
Oct 31 19:42:04.106879 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:04.092304    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e861821-e221-44d2-956a-66f0f656e1b5-dshm\") pod \"test-model-03dfee-kserve-68c5b8bcbf-cwgcs\" (UID: \"6e861821-e221-44d2-956a-66f0f656e1b5\") " pod="ai-tenant-e2e-inf-a-03dfee/test-model-03dfee-kserve-68c5b8bcbf-cwgcs"
Oct 31 19:42:04.106879 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:04.092603    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gp6z\" (UniqueName: \"kubernetes.io/projected/6e861821-e221-44d2-956a-66f0f656e1b5-kube-api-access-5gp6z\") pod \"test-model-03dfee-kserve-68c5b8bcbf-cwgcs\" (UID: \"6e861821-e221-44d2-956a-66f0f656e1b5\") " pod="ai-tenant-e2e-inf-a-03dfee/test-model-03dfee-kserve-68c5b8bcbf-cwgcs"
Oct 31 19:42:04.106879 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:04.092696    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e861821-e221-44d2-956a-66f0f656e1b5-model-cache\") pod \"test-model-03dfee-kserve-68c5b8bcbf-cwgcs\" (UID: \"6e861821-e221-44d2-956a-66f0f656e1b5\") " pod="ai-tenant-e2e-inf-a-03dfee/test-model-03dfee-kserve-68c5b8bcbf-cwgcs"
Oct 31 19:42:04.106879 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:04.092742    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e861821-e221-44d2-956a-66f0f656e1b5-tmp-dir\") pod \"test-model-03dfee-kserve-68c5b8bcbf-cwgcs\" (UID: \"6e861821-e221-44d2-956a-66f0f656e1b5\") " pod="ai-tenant-e2e-inf-a-03dfee/test-model-03dfee-kserve-68c5b8bcbf-cwgcs"
Oct 31 19:42:04.106879 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:04.092844    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e861821-e221-44d2-956a-66f0f656e1b5-tls-certs\") pod \"test-model-03dfee-kserve-68c5b8bcbf-cwgcs\" (UID: \"6e861821-e221-44d2-956a-66f0f656e1b5\") " pod="ai-tenant-e2e-inf-a-03dfee/test-model-03dfee-kserve-68c5b8bcbf-cwgcs"
Oct 31 19:42:04.106879 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:04.060393    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e861821-e221-44d2-956a-66f0f656e1b5-home\") pod \"test-model-03dfee-kserve-68c5b8bcbf-cwgcs\" (UID: \"6e861821-e221-44d2-956a-66f0f656e1b5\") " pod="ai-tenant-e2e-inf-a-03dfee/test-model-03dfee-kserve-68c5b8bcbf-cwgcs"
Oct 31 19:42:04.106879 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:04.083242    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:04.106879 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:04.103819    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e861821-e221-44d2-956a-66f0f656e1b5-model-cache\") pod \"test-model-03dfee-kserve-68c5b8bcbf-cwgcs\" (UID: \"6e861821-e221-44d2-956a-66f0f656e1b5\") " pod="ai-tenant-e2e-inf-a-03dfee/test-model-03dfee-kserve-68c5b8bcbf-cwgcs"
Oct 31 19:42:04.106879 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:04.104536    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e861821-e221-44d2-956a-66f0f656e1b5-tmp-dir\") pod \"test-model-03dfee-kserve-68c5b8bcbf-cwgcs\" (UID: \"6e861821-e221-44d2-956a-66f0f656e1b5\") " pod="ai-tenant-e2e-inf-a-03dfee/test-model-03dfee-kserve-68c5b8bcbf-cwgcs"
Oct 31 19:42:04.247506 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:04.247196    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gp6z\" (UniqueName: \"kubernetes.io/projected/6e861821-e221-44d2-956a-66f0f656e1b5-kube-api-access-5gp6z\") pod \"test-model-03dfee-kserve-68c5b8bcbf-cwgcs\" (UID: \"6e861821-e221-44d2-956a-66f0f656e1b5\") " pod="ai-tenant-e2e-inf-a-03dfee/test-model-03dfee-kserve-68c5b8bcbf-cwgcs"
Oct 31 19:42:04.372701 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:04.351010    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e861821-e221-44d2-956a-66f0f656e1b5-dshm\") pod \"test-model-03dfee-kserve-68c5b8bcbf-cwgcs\" (UID: \"6e861821-e221-44d2-956a-66f0f656e1b5\") " pod="ai-tenant-e2e-inf-a-03dfee/test-model-03dfee-kserve-68c5b8bcbf-cwgcs"
Oct 31 19:42:04.450464 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:04.450278    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669" podUID="9285bee0-bf7c-4fd4-9da7-a1db021cae05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 31 19:42:04.455629 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:04.450866    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e861821-e221-44d2-956a-66f0f656e1b5-tls-certs\") pod \"test-model-03dfee-kserve-68c5b8bcbf-cwgcs\" (UID: \"6e861821-e221-44d2-956a-66f0f656e1b5\") " pod="ai-tenant-e2e-inf-a-03dfee/test-model-03dfee-kserve-68c5b8bcbf-cwgcs"
Oct 31 19:42:04.528397 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:04.515638    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw"
Oct 31 19:42:04.528397 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:04.515694    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x"
Oct 31 19:42:04.528397 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:04.515725    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n"
Oct 31 19:42:04.528397 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:04.515753    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw"
Oct 31 19:42:04.528397 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:04.515783    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l"
Oct 31 19:42:04.528397 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:04.518404    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw"
Oct 31 19:42:04.528397 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:04.518722    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-03dfee/test-model-03dfee-kserve-68c5b8bcbf-cwgcs"
Oct 31 19:42:04.553852 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:04.548866    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4"
Oct 31 19:42:04.593960 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:04.558375    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a813e2-747f99fd79-prchw"]
Oct 31 19:42:04.627031 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:04.626995    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a813e2-dbc7795cc-vqxqm"]
Oct 31 19:42:04.643845 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:04.643202    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669"
Oct 31 19:42:04.650052 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:04.650019    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7"]
Oct 31 19:42:04.698783 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:04.664474    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7"
Oct 31 19:42:04.792446 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:04.765705    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-e73256-openshift-default-dockercfg-dp7zl\""
Oct 31 19:42:04.796351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:04.794478    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:04.838208 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:04.838175    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca-credential-socket\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7\" (UID: \"c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7"
Oct 31 19:42:04.839366 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:04.839317    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca-workload-certs\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7\" (UID: \"c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7"
Oct 31 19:42:04.840195 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:04.840167    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca-istiod-ca-cert\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7\" (UID: \"c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7"
Oct 31 19:42:04.840374 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:04.840358    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca-istio-data\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7\" (UID: \"c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7"
Oct 31 19:42:04.840530 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:04.840516    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca-istio-podinfo\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7\" (UID: \"c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7"
Oct 31 19:42:04.840689 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:04.840672    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca-istio-token\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7\" (UID: \"c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7"
Oct 31 19:42:04.840814 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:04.840801    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca-workload-socket\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7\" (UID: \"c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7"
Oct 31 19:42:04.841475 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:04.841456    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nxd8\" (UniqueName: \"kubernetes.io/projected/c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca-kube-api-access-6nxd8\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7\" (UID: \"c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7"
Oct 31 19:42:04.843087 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:04.843061    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca-istio-envoy\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7\" (UID: \"c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7"
Oct 31 19:42:04.877555 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:04.849220    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-46875"]
Oct 31 19:42:04.886120 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:04.886081    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-278c04-6969d7fcb9-252tn"
Oct 31 19:42:04.890206 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:04.890183    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-46875"
Oct 31 19:42:04.896535 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:04.896516    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-278c04-86645f6fbc-p5qmg"
Oct 31 19:42:04.935441 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:04.935415    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7"]
Oct 31 19:42:04.950323 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:04.950292    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca-istio-envoy\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7\" (UID: \"c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7"
Oct 31 19:42:04.950911 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:04.950886    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca-credential-socket\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7\" (UID: \"c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7"
Oct 31 19:42:04.951094 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:04.951080    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca-workload-certs\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7\" (UID: \"c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7"
Oct 31 19:42:04.951196 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:04.951183    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca-istiod-ca-cert\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7\" (UID: \"c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7"
Oct 31 19:42:04.951302 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:04.951288    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca-istio-data\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7\" (UID: \"c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7"
Oct 31 19:42:04.951447 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:04.951431    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca-istio-podinfo\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7\" (UID: \"c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7"
Oct 31 19:42:04.951596 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:04.951581    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca-istio-token\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7\" (UID: \"c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7"
Oct 31 19:42:04.951708 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:04.951694    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca-workload-socket\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7\" (UID: \"c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7"
Oct 31 19:42:04.951953 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:04.951938    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nxd8\" (UniqueName: \"kubernetes.io/projected/c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca-kube-api-access-6nxd8\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7\" (UID: \"c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7"
Oct 31 19:42:04.955780 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:04.955758    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca-istio-envoy\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7\" (UID: \"c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7"
Oct 31 19:42:04.956919 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:04.956895    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca-credential-socket\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7\" (UID: \"c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7"
Oct 31 19:42:04.957300 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:04.957279    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca-workload-certs\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7\" (UID: \"c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7"
Oct 31 19:42:04.958364 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:04.958345    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca-istiod-ca-cert\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7\" (UID: \"c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7"
Oct 31 19:42:04.958732 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:04.958712    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca-istio-data\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7\" (UID: \"c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7"
Oct 31 19:42:04.966888 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:04.966866    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-46875"]
Oct 31 19:42:04.972852 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:04.972827    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a813e2-dbc7795cc-vqxqm" event={"ID":"ac45690b-da7e-4ae0-81dc-6cf69551f665","Type":"ContainerStarted","Data":"5b6b37ed591c4ceac1860f13721b1953c61e941111cffd3dcdd97eb1aae2a96d"}
Oct 31 19:42:04.973009 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:04.972993    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a813e2-747f99fd79-prchw"
Oct 31 19:42:04.975979 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:04.975946    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a813e2-747f99fd79-prchw" event={"ID":"38d1e68e-f404-4056-8176-14f7205b2d99","Type":"ContainerStarted","Data":"effefd88c5075c37e2c3ffacde53f5d3461716d84262bd8d2d48d9a933d9bff3"}
Oct 31 19:42:04.976139 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:04.976120    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a813e2-747f99fd79-prchw" event={"ID":"38d1e68e-f404-4056-8176-14f7205b2d99","Type":"ContainerStarted","Data":"7d020be5d14a28d8ac8318d9f41a6c7e292d3e44591996c281bc5dacb179353b"}
Oct 31 19:42:04.976294 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:04.972729    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca-workload-socket\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7\" (UID: \"c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7"
Oct 31 19:42:04.989350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:04.984839    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca-istio-podinfo\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7\" (UID: \"c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7"
Oct 31 19:42:05.014068 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:05.014044    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nxd8\" (UniqueName: \"kubernetes.io/projected/c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca-kube-api-access-6nxd8\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7\" (UID: \"c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7"
Oct 31 19:42:05.014799 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:05.014751    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca-istio-token\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7\" (UID: \"c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7"
Oct 31 19:42:05.053907 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:05.053825    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41a92e12-0b8e-4479-af08-9ef868f8ef83-workload-socket\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-46875\" (UID: \"41a92e12-0b8e-4479-af08-9ef868f8ef83\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-46875"
Oct 31 19:42:05.066728 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:05.057666    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41a92e12-0b8e-4479-af08-9ef868f8ef83-istio-token\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-46875\" (UID: \"41a92e12-0b8e-4479-af08-9ef868f8ef83\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-46875"
Oct 31 19:42:05.066728 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:05.057744    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56bcs\" (UniqueName: \"kubernetes.io/projected/41a92e12-0b8e-4479-af08-9ef868f8ef83-kube-api-access-56bcs\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-46875\" (UID: \"41a92e12-0b8e-4479-af08-9ef868f8ef83\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-46875"
Oct 31 19:42:05.066728 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:05.057852    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41a92e12-0b8e-4479-af08-9ef868f8ef83-workload-certs\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-46875\" (UID: \"41a92e12-0b8e-4479-af08-9ef868f8ef83\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-46875"
Oct 31 19:42:05.066728 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:05.057900    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41a92e12-0b8e-4479-af08-9ef868f8ef83-istio-envoy\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-46875\" (UID: \"41a92e12-0b8e-4479-af08-9ef868f8ef83\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-46875"
Oct 31 19:42:05.066728 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:05.057931    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41a92e12-0b8e-4479-af08-9ef868f8ef83-istio-podinfo\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-46875\" (UID: \"41a92e12-0b8e-4479-af08-9ef868f8ef83\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-46875"
Oct 31 19:42:05.066728 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:05.057991    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41a92e12-0b8e-4479-af08-9ef868f8ef83-istiod-ca-cert\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-46875\" (UID: \"41a92e12-0b8e-4479-af08-9ef868f8ef83\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-46875"
Oct 31 19:42:05.066728 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:05.058090    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41a92e12-0b8e-4479-af08-9ef868f8ef83-credential-socket\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-46875\" (UID: \"41a92e12-0b8e-4479-af08-9ef868f8ef83\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-46875"
Oct 31 19:42:05.066728 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:05.058143    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41a92e12-0b8e-4479-af08-9ef868f8ef83-istio-data\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-46875\" (UID: \"41a92e12-0b8e-4479-af08-9ef868f8ef83\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-46875"
Oct 31 19:42:05.066728 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:05.062956    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a813e2-747f99fd79-prchw" podStartSLOduration=6.062939541 podStartE2EDuration="6.062939541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:41:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:42:05.042765474 +0000 UTC m=+1355.615999603" watchObservedRunningTime="2026-10-31 19:42:05.062939541 +0000 UTC m=+1355.636173670"
Oct 31 19:42:05.202350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:05.194346    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41a92e12-0b8e-4479-af08-9ef868f8ef83-workload-certs\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-46875\" (UID: \"41a92e12-0b8e-4479-af08-9ef868f8ef83\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-46875"
Oct 31 19:42:05.202350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:05.194437    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41a92e12-0b8e-4479-af08-9ef868f8ef83-istio-envoy\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-46875\" (UID: \"41a92e12-0b8e-4479-af08-9ef868f8ef83\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-46875"
Oct 31 19:42:05.202350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:05.194502    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41a92e12-0b8e-4479-af08-9ef868f8ef83-istio-podinfo\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-46875\" (UID: \"41a92e12-0b8e-4479-af08-9ef868f8ef83\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-46875"
Oct 31 19:42:05.202350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:05.194546    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41a92e12-0b8e-4479-af08-9ef868f8ef83-istiod-ca-cert\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-46875\" (UID: \"41a92e12-0b8e-4479-af08-9ef868f8ef83\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-46875"
Oct 31 19:42:05.202350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:05.194634    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41a92e12-0b8e-4479-af08-9ef868f8ef83-credential-socket\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-46875\" (UID: \"41a92e12-0b8e-4479-af08-9ef868f8ef83\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-46875"
Oct 31 19:42:05.202350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:05.194701    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41a92e12-0b8e-4479-af08-9ef868f8ef83-istio-data\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-46875\" (UID: \"41a92e12-0b8e-4479-af08-9ef868f8ef83\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-46875"
Oct 31 19:42:05.202350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:05.195004    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41a92e12-0b8e-4479-af08-9ef868f8ef83-workload-socket\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-46875\" (UID: \"41a92e12-0b8e-4479-af08-9ef868f8ef83\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-46875"
Oct 31 19:42:05.202350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:05.195093    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41a92e12-0b8e-4479-af08-9ef868f8ef83-istio-token\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-46875\" (UID: \"41a92e12-0b8e-4479-af08-9ef868f8ef83\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-46875"
Oct 31 19:42:05.202350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:05.195148    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56bcs\" (UniqueName: \"kubernetes.io/projected/41a92e12-0b8e-4479-af08-9ef868f8ef83-kube-api-access-56bcs\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-46875\" (UID: \"41a92e12-0b8e-4479-af08-9ef868f8ef83\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-46875"
Oct 31 19:42:05.209350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:05.204019    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41a92e12-0b8e-4479-af08-9ef868f8ef83-workload-certs\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-46875\" (UID: \"41a92e12-0b8e-4479-af08-9ef868f8ef83\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-46875"
Oct 31 19:42:05.231176 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:05.212034    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41a92e12-0b8e-4479-af08-9ef868f8ef83-istio-data\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-46875\" (UID: \"41a92e12-0b8e-4479-af08-9ef868f8ef83\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-46875"
Oct 31 19:42:05.233569 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:05.233505    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41a92e12-0b8e-4479-af08-9ef868f8ef83-istiod-ca-cert\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-46875\" (UID: \"41a92e12-0b8e-4479-af08-9ef868f8ef83\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-46875"
Oct 31 19:42:05.234962 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:05.234934    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41a92e12-0b8e-4479-af08-9ef868f8ef83-credential-socket\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-46875\" (UID: \"41a92e12-0b8e-4479-af08-9ef868f8ef83\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-46875"
Oct 31 19:42:05.235612 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:05.235572    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41a92e12-0b8e-4479-af08-9ef868f8ef83-workload-socket\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-46875\" (UID: \"41a92e12-0b8e-4479-af08-9ef868f8ef83\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-46875"
Oct 31 19:42:05.236743 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:05.236557    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7"
Oct 31 19:42:05.248184 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:05.247230    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41a92e12-0b8e-4479-af08-9ef868f8ef83-istio-podinfo\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-46875\" (UID: \"41a92e12-0b8e-4479-af08-9ef868f8ef83\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-46875"
Oct 31 19:42:05.303206 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:05.303098    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-03dfee/test-model-03dfee-kserve-68c5b8bcbf-cwgcs"]
Oct 31 19:42:05.306236 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:05.306138    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41a92e12-0b8e-4479-af08-9ef868f8ef83-istio-token\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-46875\" (UID: \"41a92e12-0b8e-4479-af08-9ef868f8ef83\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-46875"
Oct 31 19:42:05.306733 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:05.306709    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41a92e12-0b8e-4479-af08-9ef868f8ef83-istio-envoy\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-46875\" (UID: \"41a92e12-0b8e-4479-af08-9ef868f8ef83\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-46875"
Oct 31 19:42:05.309431 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:05.309119    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56bcs\" (UniqueName: \"kubernetes.io/projected/41a92e12-0b8e-4479-af08-9ef868f8ef83-kube-api-access-56bcs\") pod \"e2e-api-a-e73256-openshift-default-dfc9897d9-46875\" (UID: \"41a92e12-0b8e-4479-af08-9ef868f8ef83\") " pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-46875"
Oct 31 19:42:05.320951 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:05.320933    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-46875"
Oct 31 19:42:05.637877 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:05.637792    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:05.645625 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:05.643596    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-46875"]
Oct 31 19:42:05.645625 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:42:05.644920    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc960b4c7_b9ab_4c5e_88d8_feaabe41b0ca.slice/crio-c1d19bd5a1ccbf3bc5b060e4cf2257290b0aa203dbe557901aeea33e3c5411a8 WatchSource:0}: Error finding container c1d19bd5a1ccbf3bc5b060e4cf2257290b0aa203dbe557901aeea33e3c5411a8: Status 404 returned error can't find the container with id c1d19bd5a1ccbf3bc5b060e4cf2257290b0aa203dbe557901aeea33e3c5411a8
Oct 31 19:42:05.646919 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:42:05.646893    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41a92e12_0b8e_4479_af08_9ef868f8ef83.slice/crio-911ba69f86a300a6ea928a1e6d0f992e548809d9f664b0d636fb16f878c2e61d WatchSource:0}: Error finding container 911ba69f86a300a6ea928a1e6d0f992e548809d9f664b0d636fb16f878c2e61d: Status 404 returned error can't find the container with id 911ba69f86a300a6ea928a1e6d0f992e548809d9f664b0d636fb16f878c2e61d
Oct 31 19:42:05.648368 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:05.648346    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7"]
Oct 31 19:42:05.648871 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:05.648835    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:42:05.648970 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:05.648919    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:42:05.648970 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:05.648961    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:42:05.649792 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:05.649757    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:42:05.649883 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:05.649831    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:42:05.649883 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:05.649868    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:42:06.007134 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:06.007087    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-03dfee/test-model-03dfee-kserve-68c5b8bcbf-cwgcs" event={"ID":"6e861821-e221-44d2-956a-66f0f656e1b5","Type":"ContainerStarted","Data":"415f7634423a5489e29c5860279bb8e5c3020257b8ed7b665b0fa3936f3e749a"}
Oct 31 19:42:06.007364 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:06.007143    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-03dfee/test-model-03dfee-kserve-68c5b8bcbf-cwgcs" event={"ID":"6e861821-e221-44d2-956a-66f0f656e1b5","Type":"ContainerStarted","Data":"c523a66a71fe9f46498375c8ad4e86cb4818bcc5bbe8922838aa494ed536b625"}
Oct 31 19:42:06.012216 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:06.012074    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-46875" event={"ID":"41a92e12-0b8e-4479-af08-9ef868f8ef83","Type":"ContainerStarted","Data":"ceaebf191fbf94e0ff59dc73985163a6aa49a6433d71fef21408a373b87b2d36"}
Oct 31 19:42:06.012216 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:06.012132    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-46875" event={"ID":"41a92e12-0b8e-4479-af08-9ef868f8ef83","Type":"ContainerStarted","Data":"911ba69f86a300a6ea928a1e6d0f992e548809d9f664b0d636fb16f878c2e61d"}
Oct 31 19:42:06.016931 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:06.015715    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7" event={"ID":"c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca","Type":"ContainerStarted","Data":"83e0163dd2f3808db950854f92a5e080b648e19477398cc92282f9da5e08254b"}
Oct 31 19:42:06.016931 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:06.016890    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7" event={"ID":"c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca","Type":"ContainerStarted","Data":"c1d19bd5a1ccbf3bc5b060e4cf2257290b0aa203dbe557901aeea33e3c5411a8"}
Oct 31 19:42:06.020983 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:06.020951    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a813e2-dbc7795cc-vqxqm" event={"ID":"ac45690b-da7e-4ae0-81dc-6cf69551f665","Type":"ContainerStarted","Data":"e32773b97d5d99ffc8762171ac6ac60ba26ab561c496c3ab56ed1932366744c0"}
Oct 31 19:42:06.021106 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:06.021007    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-a813e2-dbc7795cc-vqxqm"
Oct 31 19:42:06.035220 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:06.035173    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-03dfee/test-model-03dfee-kserve-68c5b8bcbf-cwgcs" podStartSLOduration=5.035157729 podStartE2EDuration="5.035157729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:42:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:42:06.027207374 +0000 UTC m=+1356.600441504" watchObservedRunningTime="2026-10-31 19:42:06.035157729 +0000 UTC m=+1356.608391859"
Oct 31 19:42:06.055245 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:06.054349    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-46875" podStartSLOduration=2.054314101 podStartE2EDuration="2.054314101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:42:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:42:06.049527079 +0000 UTC m=+1356.622761208" watchObservedRunningTime="2026-10-31 19:42:06.054314101 +0000 UTC m=+1356.627548232"
Oct 31 19:42:06.107351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:06.106497    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7" podStartSLOduration=2.106475752 podStartE2EDuration="2.106475752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:42:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:42:06.078763772 +0000 UTC m=+1356.651997902" watchObservedRunningTime="2026-10-31 19:42:06.106475752 +0000 UTC m=+1356.679709904"
Oct 31 19:42:06.113428 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:06.113380    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-a813e2-dbc7795cc-vqxqm" podStartSLOduration=7.113366823 podStartE2EDuration="7.113366823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:41:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:42:06.103757374 +0000 UTC m=+1356.676991503" watchObservedRunningTime="2026-10-31 19:42:06.113366823 +0000 UTC m=+1356.686600952"
Oct 31 19:42:06.271219 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:06.271131    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7"
Oct 31 19:42:06.287732 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:06.272983    4645 patch_prober.go:29] interesting pod/e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Oct 31 19:42:06.287732 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:06.273084    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7" podUID="c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Oct 31 19:42:06.352754 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:06.352666    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-46875"
Oct 31 19:42:06.379479 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:06.356376    4645 patch_prober.go:29] interesting pod/e2e-api-a-e73256-openshift-default-dfc9897d9-46875 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Oct 31 19:42:06.379479 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:06.356431    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-46875" podUID="41a92e12-0b8e-4479-af08-9ef868f8ef83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Oct 31 19:42:06.407375 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:42:06.406798    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 19:42:06.526383 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:06.526264    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-03dfee/test-model-03dfee-kserve-68c5b8bcbf-cwgcs"
Oct 31 19:42:06.527716 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:06.527693    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:06.628509 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:06.628476    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:06.738574 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:06.737797    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:07.242410 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:07.241285    4645 patch_prober.go:29] interesting pod/e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Oct 31 19:42:07.242410 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:07.241402    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7" podUID="c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Oct 31 19:42:07.435560 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:07.435521    4645 patch_prober.go:29] interesting pod/e2e-api-a-e73256-openshift-default-dfc9897d9-46875 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Oct 31 19:42:07.435844 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:07.435813    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-46875" podUID="41a92e12-0b8e-4479-af08-9ef868f8ef83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Oct 31 19:42:07.644837 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:07.644757    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:08.238730 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:08.238363    4645 patch_prober.go:29] interesting pod/e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Oct 31 19:42:08.238730 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:08.238418    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7" podUID="c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Oct 31 19:42:08.321836 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:08.321799    4645 patch_prober.go:29] interesting pod/e2e-api-a-e73256-openshift-default-dfc9897d9-46875 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Oct 31 19:42:08.321986 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:08.321861    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-46875" podUID="41a92e12-0b8e-4479-af08-9ef868f8ef83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Oct 31 19:42:08.535942 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:08.535858    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:08.844573 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:08.844461    4645 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:42:08.844573 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:08.844510    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2" podUID="813ae008-e079-4406-83f7-6eae7acc3ce1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:42:09.240212 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:09.240182    4645 patch_prober.go:29] interesting pod/e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Oct 31 19:42:09.240727 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:09.240700    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7" podUID="c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Oct 31 19:42:09.322195 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:09.322159    4645 patch_prober.go:29] interesting pod/e2e-api-a-e73256-openshift-default-dfc9897d9-46875 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Oct 31 19:42:09.322393 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:09.322224    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-46875" podUID="41a92e12-0b8e-4479-af08-9ef868f8ef83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Oct 31 19:42:09.454833 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:09.454793    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:09.849471 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:09.849428    4645 patch_prober.go:29] interesting pod/e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:42:09.849663 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:09.849484    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm" podUID="773450c2-8daa-4448-a78a-8ede38d60d69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:42:10.237771 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:10.237738    4645 patch_prober.go:29] interesting pod/e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Oct 31 19:42:10.238027 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:10.237794    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7" podUID="c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Oct 31 19:42:10.274550 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:10.273906    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:10.322149 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:10.321672    4645 patch_prober.go:29] interesting pod/e2e-api-a-e73256-openshift-default-dfc9897d9-46875 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Oct 31 19:42:10.322149 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:10.321739    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-46875" podUID="41a92e12-0b8e-4479-af08-9ef868f8ef83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Oct 31 19:42:11.163708 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:11.155427    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:11.250781 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:11.250748    4645 patch_prober.go:29] interesting pod/e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Oct 31 19:42:11.251710 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:11.251243    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7" podUID="c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Oct 31 19:42:11.350409 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:11.326949    4645 patch_prober.go:29] interesting pod/e2e-api-a-e73256-openshift-default-dfc9897d9-46875 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Oct 31 19:42:11.350409 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:11.327023    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-46875" podUID="41a92e12-0b8e-4479-af08-9ef868f8ef83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Oct 31 19:42:11.935549 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:11.935513    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:12.237806 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:12.237737    4645 patch_prober.go:29] interesting pod/e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body=
Oct 31 19:42:12.238001 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:12.237980    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7" podUID="c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused"
Oct 31 19:42:12.322188 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:12.322155    4645 patch_prober.go:29] interesting pod/e2e-api-a-e73256-openshift-default-dfc9897d9-46875 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body=
Oct 31 19:42:12.322403 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:12.322214    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-46875" podUID="41a92e12-0b8e-4479-af08-9ef868f8ef83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused"
Oct 31 19:42:12.835019 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:12.834985    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:13.243457 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:13.243427    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7"
Oct 31 19:42:13.243960 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:13.243937    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7"
Oct 31 19:42:13.244676 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:13.244658    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7"
Oct 31 19:42:13.336825 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:13.336796    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-46875"
Oct 31 19:42:13.338533 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:13.338511    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-46875"
Oct 31 19:42:13.339170 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:13.339149    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-46875"
Oct 31 19:42:13.654209 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:13.654097    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:14.522913 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:14.522878    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-03dfee/test-model-03dfee-kserve-68c5b8bcbf-cwgcs"
Oct 31 19:42:14.543432 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:14.530837    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:14.543432 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:14.536136    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-03dfee/test-model-03dfee-kserve-68c5b8bcbf-cwgcs"
Oct 31 19:42:15.338953 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:15.338913    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:15.357376 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:15.357343    4645 generic.go:356] "Generic (PLEG): container finished" podID="773450c2-8daa-4448-a78a-8ede38d60d69" containerID="523222635d65f76e0c47c008d6856668c03854cd36a652188d27282f7d6a968d" exitCode=0
Oct 31 19:42:15.357376 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:15.357358    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm" event={"ID":"773450c2-8daa-4448-a78a-8ede38d60d69","Type":"ContainerDied","Data":"523222635d65f76e0c47c008d6856668c03854cd36a652188d27282f7d6a968d"}
Oct 31 19:42:15.357816 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:15.357796    4645 scope.go:122] "RemoveContainer" containerID="523222635d65f76e0c47c008d6856668c03854cd36a652188d27282f7d6a968d"
Oct 31 19:42:15.359672 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:15.359647    4645 generic.go:356] "Generic (PLEG): container finished" podID="813ae008-e079-4406-83f7-6eae7acc3ce1" containerID="3c0d8d30d39213108ab50f74714ca089e81860c63ad687013e351e6d86717a2c" exitCode=0
Oct 31 19:42:15.360080 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:15.360046    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:42:15.360199 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:15.360148    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:42:15.360260 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:15.360204    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:42:15.360749 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:15.360725    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2" event={"ID":"813ae008-e079-4406-83f7-6eae7acc3ce1","Type":"ContainerDied","Data":"3c0d8d30d39213108ab50f74714ca089e81860c63ad687013e351e6d86717a2c"}
Oct 31 19:42:15.361128 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:15.361112    4645 scope.go:122] "RemoveContainer" containerID="3c0d8d30d39213108ab50f74714ca089e81860c63ad687013e351e6d86717a2c"
Oct 31 19:42:15.367397 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:15.363018    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:42:15.367397 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:15.363099    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:42:15.367397 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:15.363139    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:42:15.376064 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:15.376045    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-03dfee/test-model-03dfee-kserve-68c5b8bcbf-cwgcs"
Oct 31 19:42:15.471722 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:15.471694    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:15.588348 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:15.583227    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:16.037964 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:16.037930    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a813e2-747f99fd79-prchw"
Oct 31 19:42:16.329466 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:16.329376    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:16.410383 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:16.410340    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2" event={"ID":"813ae008-e079-4406-83f7-6eae7acc3ce1","Type":"ContainerStarted","Data":"5e7a4d56c78c54025c3ad6d7a38648b351dc701cc8fdb9b370bcbd8bd33fda4d"}
Oct 31 19:42:16.415772 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:16.415738    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm" event={"ID":"773450c2-8daa-4448-a78a-8ede38d60d69","Type":"ContainerStarted","Data":"6a77db055a4c872750200ed74919863a2e2716f24fa3ca4f19a69b88f983e37b"}
Oct 31 19:42:16.510352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:16.509693    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm"
Oct 31 19:42:16.510352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:16.509813    4645 patch_prober.go:29] interesting pod/e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 31 19:42:16.510352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:16.509867    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm" podUID="773450c2-8daa-4448-a78a-8ede38d60d69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 31 19:42:16.530809 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:16.530778    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:16.651468 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:16.651364    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:16.838093 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:16.838027    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2"
Oct 31 19:42:16.838567 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:16.838525    4645 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 31 19:42:16.838809 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:16.838767    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2" podUID="813ae008-e079-4406-83f7-6eae7acc3ce1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 31 19:42:16.850469 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:16.850438    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:17.135351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:17.130258    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-a813e2-dbc7795cc-vqxqm"
Oct 31 19:42:17.515405 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:17.514386    4645 patch_prober.go:29] interesting pod/e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 31 19:42:17.515405 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:17.514428    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm" podUID="773450c2-8daa-4448-a78a-8ede38d60d69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 31 19:42:17.520908 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:17.520877    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-88eaf9/test-model-88eaf9-kserve-7f86f56bdf-28gmk"]
Oct 31 19:42:17.527043 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:17.527024    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-88eaf9/test-model-88eaf9-kserve-7f86f56bdf-28gmk"
Oct 31 19:42:17.539152 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:17.539130    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-88eaf9/test-model-88eaf9-kserve-7f86f56bdf-28gmk"]
Oct 31 19:42:17.547802 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:17.547548    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-88eaf9\"/\"kube-root-ca.crt\""
Oct 31 19:42:17.547919 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:17.547887    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-88eaf9\"/\"openshift-service-ca.crt\""
Oct 31 19:42:17.553982 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:17.553964    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:17.609809 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:17.609773    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-88eaf9\"/\"default-dockercfg-zk728\""
Oct 31 19:42:17.609994 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:17.609907    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-88eaf9\"/\"test-model-88eaf9-kserve-self-signed-certs\""
Oct 31 19:42:17.685354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:17.675461    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:17.716351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:17.713919    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/244c8e8b-236e-47cf-99c6-69a0a1fb6af6-home\") pod \"test-model-88eaf9-kserve-7f86f56bdf-28gmk\" (UID: \"244c8e8b-236e-47cf-99c6-69a0a1fb6af6\") " pod="ai-tenant-e2e-inf-b-88eaf9/test-model-88eaf9-kserve-7f86f56bdf-28gmk"
Oct 31 19:42:17.716351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:17.714029    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/244c8e8b-236e-47cf-99c6-69a0a1fb6af6-tmp-dir\") pod \"test-model-88eaf9-kserve-7f86f56bdf-28gmk\" (UID: \"244c8e8b-236e-47cf-99c6-69a0a1fb6af6\") " pod="ai-tenant-e2e-inf-b-88eaf9/test-model-88eaf9-kserve-7f86f56bdf-28gmk"
Oct 31 19:42:17.716351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:17.714150    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/244c8e8b-236e-47cf-99c6-69a0a1fb6af6-dshm\") pod \"test-model-88eaf9-kserve-7f86f56bdf-28gmk\" (UID: \"244c8e8b-236e-47cf-99c6-69a0a1fb6af6\") " pod="ai-tenant-e2e-inf-b-88eaf9/test-model-88eaf9-kserve-7f86f56bdf-28gmk"
Oct 31 19:42:17.716351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:17.714251    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/244c8e8b-236e-47cf-99c6-69a0a1fb6af6-tls-certs\") pod \"test-model-88eaf9-kserve-7f86f56bdf-28gmk\" (UID: \"244c8e8b-236e-47cf-99c6-69a0a1fb6af6\") " pod="ai-tenant-e2e-inf-b-88eaf9/test-model-88eaf9-kserve-7f86f56bdf-28gmk"
Oct 31 19:42:17.716351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:17.714323    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/244c8e8b-236e-47cf-99c6-69a0a1fb6af6-model-cache\") pod \"test-model-88eaf9-kserve-7f86f56bdf-28gmk\" (UID: \"244c8e8b-236e-47cf-99c6-69a0a1fb6af6\") " pod="ai-tenant-e2e-inf-b-88eaf9/test-model-88eaf9-kserve-7f86f56bdf-28gmk"
Oct 31 19:42:17.716351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:17.714905    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfmmj\" (UniqueName: \"kubernetes.io/projected/244c8e8b-236e-47cf-99c6-69a0a1fb6af6-kube-api-access-cfmmj\") pod \"test-model-88eaf9-kserve-7f86f56bdf-28gmk\" (UID: \"244c8e8b-236e-47cf-99c6-69a0a1fb6af6\") " pod="ai-tenant-e2e-inf-b-88eaf9/test-model-88eaf9-kserve-7f86f56bdf-28gmk"
Oct 31 19:42:17.780112 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:17.780021    4645 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 31 19:42:17.780401 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:17.780369    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2" podUID="813ae008-e079-4406-83f7-6eae7acc3ce1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 31 19:42:17.889356 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:17.881489    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/244c8e8b-236e-47cf-99c6-69a0a1fb6af6-tls-certs\") pod \"test-model-88eaf9-kserve-7f86f56bdf-28gmk\" (UID: \"244c8e8b-236e-47cf-99c6-69a0a1fb6af6\") " pod="ai-tenant-e2e-inf-b-88eaf9/test-model-88eaf9-kserve-7f86f56bdf-28gmk"
Oct 31 19:42:17.889356 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:17.881581    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/244c8e8b-236e-47cf-99c6-69a0a1fb6af6-model-cache\") pod \"test-model-88eaf9-kserve-7f86f56bdf-28gmk\" (UID: \"244c8e8b-236e-47cf-99c6-69a0a1fb6af6\") " pod="ai-tenant-e2e-inf-b-88eaf9/test-model-88eaf9-kserve-7f86f56bdf-28gmk"
Oct 31 19:42:17.889356 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:17.881666    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfmmj\" (UniqueName: \"kubernetes.io/projected/244c8e8b-236e-47cf-99c6-69a0a1fb6af6-kube-api-access-cfmmj\") pod \"test-model-88eaf9-kserve-7f86f56bdf-28gmk\" (UID: \"244c8e8b-236e-47cf-99c6-69a0a1fb6af6\") " pod="ai-tenant-e2e-inf-b-88eaf9/test-model-88eaf9-kserve-7f86f56bdf-28gmk"
Oct 31 19:42:17.889356 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:17.881884    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/244c8e8b-236e-47cf-99c6-69a0a1fb6af6-home\") pod \"test-model-88eaf9-kserve-7f86f56bdf-28gmk\" (UID: \"244c8e8b-236e-47cf-99c6-69a0a1fb6af6\") " pod="ai-tenant-e2e-inf-b-88eaf9/test-model-88eaf9-kserve-7f86f56bdf-28gmk"
Oct 31 19:42:17.889356 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:17.881970    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/244c8e8b-236e-47cf-99c6-69a0a1fb6af6-tmp-dir\") pod \"test-model-88eaf9-kserve-7f86f56bdf-28gmk\" (UID: \"244c8e8b-236e-47cf-99c6-69a0a1fb6af6\") " pod="ai-tenant-e2e-inf-b-88eaf9/test-model-88eaf9-kserve-7f86f56bdf-28gmk"
Oct 31 19:42:17.889356 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:17.882074    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/244c8e8b-236e-47cf-99c6-69a0a1fb6af6-dshm\") pod \"test-model-88eaf9-kserve-7f86f56bdf-28gmk\" (UID: \"244c8e8b-236e-47cf-99c6-69a0a1fb6af6\") " pod="ai-tenant-e2e-inf-b-88eaf9/test-model-88eaf9-kserve-7f86f56bdf-28gmk"
Oct 31 19:42:17.889356 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:17.884217    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/244c8e8b-236e-47cf-99c6-69a0a1fb6af6-model-cache\") pod \"test-model-88eaf9-kserve-7f86f56bdf-28gmk\" (UID: \"244c8e8b-236e-47cf-99c6-69a0a1fb6af6\") " pod="ai-tenant-e2e-inf-b-88eaf9/test-model-88eaf9-kserve-7f86f56bdf-28gmk"
Oct 31 19:42:17.889356 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:17.884496    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/244c8e8b-236e-47cf-99c6-69a0a1fb6af6-home\") pod \"test-model-88eaf9-kserve-7f86f56bdf-28gmk\" (UID: \"244c8e8b-236e-47cf-99c6-69a0a1fb6af6\") " pod="ai-tenant-e2e-inf-b-88eaf9/test-model-88eaf9-kserve-7f86f56bdf-28gmk"
Oct 31 19:42:17.889356 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:17.884772    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/244c8e8b-236e-47cf-99c6-69a0a1fb6af6-tmp-dir\") pod \"test-model-88eaf9-kserve-7f86f56bdf-28gmk\" (UID: \"244c8e8b-236e-47cf-99c6-69a0a1fb6af6\") " pod="ai-tenant-e2e-inf-b-88eaf9/test-model-88eaf9-kserve-7f86f56bdf-28gmk"
Oct 31 19:42:17.909558 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:17.909519    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:17.945122 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:17.945046    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/244c8e8b-236e-47cf-99c6-69a0a1fb6af6-tls-certs\") pod \"test-model-88eaf9-kserve-7f86f56bdf-28gmk\" (UID: \"244c8e8b-236e-47cf-99c6-69a0a1fb6af6\") " pod="ai-tenant-e2e-inf-b-88eaf9/test-model-88eaf9-kserve-7f86f56bdf-28gmk"
Oct 31 19:42:17.948751 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:17.948722    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/244c8e8b-236e-47cf-99c6-69a0a1fb6af6-dshm\") pod \"test-model-88eaf9-kserve-7f86f56bdf-28gmk\" (UID: \"244c8e8b-236e-47cf-99c6-69a0a1fb6af6\") " pod="ai-tenant-e2e-inf-b-88eaf9/test-model-88eaf9-kserve-7f86f56bdf-28gmk"
Oct 31 19:42:17.956205 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:17.956165    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfmmj\" (UniqueName: \"kubernetes.io/projected/244c8e8b-236e-47cf-99c6-69a0a1fb6af6-kube-api-access-cfmmj\") pod \"test-model-88eaf9-kserve-7f86f56bdf-28gmk\" (UID: \"244c8e8b-236e-47cf-99c6-69a0a1fb6af6\") " pod="ai-tenant-e2e-inf-b-88eaf9/test-model-88eaf9-kserve-7f86f56bdf-28gmk"
Oct 31 19:42:18.143273 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:18.143192    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-88eaf9/test-model-88eaf9-kserve-7f86f56bdf-28gmk"
Oct 31 19:42:18.508222 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:18.507948    4645 patch_prober.go:29] interesting pod/e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 31 19:42:18.508222 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:18.508008    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm" podUID="773450c2-8daa-4448-a78a-8ede38d60d69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 31 19:42:18.535726 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:42:18.534978    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod244c8e8b_236e_47cf_99c6_69a0a1fb6af6.slice/crio-f109811389ae13edbd72ba99b678c6100f9280133bc50c1c014d814ee7653ab1 WatchSource:0}: Error finding container f109811389ae13edbd72ba99b678c6100f9280133bc50c1c014d814ee7653ab1: Status 404 returned error can't find the container with id f109811389ae13edbd72ba99b678c6100f9280133bc50c1c014d814ee7653ab1
Oct 31 19:42:18.537200 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:18.536855    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-88eaf9/test-model-88eaf9-kserve-7f86f56bdf-28gmk"]
Oct 31 19:42:18.736373 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:18.719227    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:18.790400 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:18.777531    4645 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 31 19:42:18.790400 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:18.777600    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2" podUID="813ae008-e079-4406-83f7-6eae7acc3ce1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 31 19:42:19.450186 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:19.450150    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-88eaf9/test-model-88eaf9-kserve-7f86f56bdf-28gmk" event={"ID":"244c8e8b-236e-47cf-99c6-69a0a1fb6af6","Type":"ContainerStarted","Data":"8bdbf5432e2a53e1ed07dbb3e8b4ee292e895247fa70c95c29bf09d782a7899c"}
Oct 31 19:42:19.450401 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:19.450200    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-88eaf9/test-model-88eaf9-kserve-7f86f56bdf-28gmk" event={"ID":"244c8e8b-236e-47cf-99c6-69a0a1fb6af6","Type":"ContainerStarted","Data":"f109811389ae13edbd72ba99b678c6100f9280133bc50c1c014d814ee7653ab1"}
Oct 31 19:42:19.507867 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:19.507825    4645 patch_prober.go:29] interesting pod/e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 31 19:42:19.508046 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:19.507878    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm" podUID="773450c2-8daa-4448-a78a-8ede38d60d69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 31 19:42:19.542355 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:19.528614    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-88eaf9/test-model-88eaf9-kserve-7f86f56bdf-28gmk" podStartSLOduration=2.5285931980000003 podStartE2EDuration="2.528593198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:42:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:42:19.52725203 +0000 UTC m=+1370.100486160" watchObservedRunningTime="2026-10-31 19:42:19.528593198 +0000 UTC m=+1370.101827328"
Oct 31 19:42:19.547467 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:19.547430    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:19.778079 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:19.777983    4645 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 31 19:42:19.778705 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:19.778672    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2" podUID="813ae008-e079-4406-83f7-6eae7acc3ce1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 31 19:42:20.144175 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:20.144085    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-88eaf9/test-model-88eaf9-kserve-7f86f56bdf-28gmk"
Oct 31 19:42:20.407620 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:20.407413    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:20.509829 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:20.509795    4645 patch_prober.go:29] interesting pod/e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 31 19:42:20.510133 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:20.510101    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm" podUID="773450c2-8daa-4448-a78a-8ede38d60d69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 31 19:42:20.792352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:20.791743    4645 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 31 19:42:20.792352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:20.791807    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2" podUID="813ae008-e079-4406-83f7-6eae7acc3ce1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 31 19:42:21.350345 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:21.350306    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:21.516379 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:21.510923    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:21.524881 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:21.524405    4645 patch_prober.go:29] interesting pod/e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 31 19:42:21.524881 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:21.524480    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm" podUID="773450c2-8daa-4448-a78a-8ede38d60d69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 31 19:42:21.690353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:21.688401    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:21.779501 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:21.776807    4645 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body=
Oct 31 19:42:21.779501 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:21.776873    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2" podUID="813ae008-e079-4406-83f7-6eae7acc3ce1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused"
Oct 31 19:42:22.239351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:22.230567    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:22.377353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:22.376829    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:22.511406 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:22.507789    4645 patch_prober.go:29] interesting pod/e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Oct 31 19:42:22.511406 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:22.507864    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm" podUID="773450c2-8daa-4448-a78a-8ede38d60d69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Oct 31 19:42:22.794578 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:22.794481    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2"
Oct 31 19:42:22.871370 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:22.871340    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2"
Oct 31 19:42:22.952355 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:22.952285    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:23.513959 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:23.513924    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm"
Oct 31 19:42:23.514547 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:23.514440    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm"
Oct 31 19:42:23.515266 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:23.515242    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm"
Oct 31 19:42:23.587021 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:23.586968    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:23.638528 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:23.638501    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2"
Oct 31 19:42:23.728450 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:23.726187    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:24.520027 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:24.519982    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:25.545213 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:25.545178    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:26.357322 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:26.357286    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:27.159005 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:27.158970    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:27.947357 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:27.947312    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:28.143996 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:28.143962    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-88eaf9/test-model-88eaf9-kserve-7f86f56bdf-28gmk"
Oct 31 19:42:28.156952 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:28.156923    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-88eaf9/test-model-88eaf9-kserve-7f86f56bdf-28gmk"
Oct 31 19:42:28.660058 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:28.660023    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:28.741353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:28.715732    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-88eaf9/test-model-88eaf9-kserve-7f86f56bdf-28gmk"
Oct 31 19:42:29.794649 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:29.794602    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:30.274285 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:30.274259    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:30.735884 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:30.735848    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:30.853994 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:30.853911    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:31.182019 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:31.181977    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp"]
Oct 31 19:42:31.189573 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:31.189551    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp"
Oct 31 19:42:31.200114 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:31.198904    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-d45f6c-openshift-default-dockercfg-2clxl\""
Oct 31 19:42:31.204373 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:31.204310    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp"]
Oct 31 19:42:31.249681 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:31.249644    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:31.266013 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:31.265984    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b73e8ddc-c45d-4537-a4c4-f22747469ca8-workload-certs\") pod \"e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp\" (UID: \"b73e8ddc-c45d-4537-a4c4-f22747469ca8\") " pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp"
Oct 31 19:42:31.266180 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:31.266022    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b73e8ddc-c45d-4537-a4c4-f22747469ca8-istiod-ca-cert\") pod \"e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp\" (UID: \"b73e8ddc-c45d-4537-a4c4-f22747469ca8\") " pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp"
Oct 31 19:42:31.266180 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:31.266117    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b73e8ddc-c45d-4537-a4c4-f22747469ca8-istio-podinfo\") pod \"e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp\" (UID: \"b73e8ddc-c45d-4537-a4c4-f22747469ca8\") " pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp"
Oct 31 19:42:31.266180 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:31.266169    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gj68\" (UniqueName: \"kubernetes.io/projected/b73e8ddc-c45d-4537-a4c4-f22747469ca8-kube-api-access-9gj68\") pod \"e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp\" (UID: \"b73e8ddc-c45d-4537-a4c4-f22747469ca8\") " pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp"
Oct 31 19:42:31.266338 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:31.266197    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b73e8ddc-c45d-4537-a4c4-f22747469ca8-workload-socket\") pod \"e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp\" (UID: \"b73e8ddc-c45d-4537-a4c4-f22747469ca8\") " pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp"
Oct 31 19:42:31.266338 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:31.266223    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b73e8ddc-c45d-4537-a4c4-f22747469ca8-istio-envoy\") pod \"e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp\" (UID: \"b73e8ddc-c45d-4537-a4c4-f22747469ca8\") " pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp"
Oct 31 19:42:31.266436 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:31.266423    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b73e8ddc-c45d-4537-a4c4-f22747469ca8-istio-token\") pod \"e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp\" (UID: \"b73e8ddc-c45d-4537-a4c4-f22747469ca8\") " pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp"
Oct 31 19:42:31.266475 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:31.266450    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b73e8ddc-c45d-4537-a4c4-f22747469ca8-credential-socket\") pod \"e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp\" (UID: \"b73e8ddc-c45d-4537-a4c4-f22747469ca8\") " pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp"
Oct 31 19:42:31.266525 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:31.266488    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b73e8ddc-c45d-4537-a4c4-f22747469ca8-istio-data\") pod \"e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp\" (UID: \"b73e8ddc-c45d-4537-a4c4-f22747469ca8\") " pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp"
Oct 31 19:42:31.399875 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:31.399836    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b73e8ddc-c45d-4537-a4c4-f22747469ca8-istio-token\") pod \"e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp\" (UID: \"b73e8ddc-c45d-4537-a4c4-f22747469ca8\") " pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp"
Oct 31 19:42:31.404348 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:31.400502    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b73e8ddc-c45d-4537-a4c4-f22747469ca8-credential-socket\") pod \"e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp\" (UID: \"b73e8ddc-c45d-4537-a4c4-f22747469ca8\") " pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp"
Oct 31 19:42:31.429397 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:31.404580    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b73e8ddc-c45d-4537-a4c4-f22747469ca8-istio-data\") pod \"e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp\" (UID: \"b73e8ddc-c45d-4537-a4c4-f22747469ca8\") " pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp"
Oct 31 19:42:31.430596 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:31.430395    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b73e8ddc-c45d-4537-a4c4-f22747469ca8-workload-certs\") pod \"e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp\" (UID: \"b73e8ddc-c45d-4537-a4c4-f22747469ca8\") " pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp"
Oct 31 19:42:31.431437 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:31.431401    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b73e8ddc-c45d-4537-a4c4-f22747469ca8-istiod-ca-cert\") pod \"e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp\" (UID: \"b73e8ddc-c45d-4537-a4c4-f22747469ca8\") " pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp"
Oct 31 19:42:31.431682 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:31.431663    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b73e8ddc-c45d-4537-a4c4-f22747469ca8-istio-podinfo\") pod \"e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp\" (UID: \"b73e8ddc-c45d-4537-a4c4-f22747469ca8\") " pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp"
Oct 31 19:42:31.439415 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:31.431841    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gj68\" (UniqueName: \"kubernetes.io/projected/b73e8ddc-c45d-4537-a4c4-f22747469ca8-kube-api-access-9gj68\") pod \"e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp\" (UID: \"b73e8ddc-c45d-4537-a4c4-f22747469ca8\") " pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp"
Oct 31 19:42:31.440493 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:31.440386    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b73e8ddc-c45d-4537-a4c4-f22747469ca8-workload-socket\") pod \"e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp\" (UID: \"b73e8ddc-c45d-4537-a4c4-f22747469ca8\") " pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp"
Oct 31 19:42:31.463453 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:31.441438    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b73e8ddc-c45d-4537-a4c4-f22747469ca8-istio-token\") pod \"e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp\" (UID: \"b73e8ddc-c45d-4537-a4c4-f22747469ca8\") " pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp"
Oct 31 19:42:31.463453 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:31.408189    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:31.463453 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:31.431722    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b73e8ddc-c45d-4537-a4c4-f22747469ca8-workload-certs\") pod \"e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp\" (UID: \"b73e8ddc-c45d-4537-a4c4-f22747469ca8\") " pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp"
Oct 31 19:42:31.463453 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:31.408269    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b73e8ddc-c45d-4537-a4c4-f22747469ca8-istio-data\") pod \"e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp\" (UID: \"b73e8ddc-c45d-4537-a4c4-f22747469ca8\") " pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp"
Oct 31 19:42:31.463453 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:31.440960    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b73e8ddc-c45d-4537-a4c4-f22747469ca8-workload-socket\") pod \"e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp\" (UID: \"b73e8ddc-c45d-4537-a4c4-f22747469ca8\") " pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp"
Oct 31 19:42:31.463453 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:31.404089    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b73e8ddc-c45d-4537-a4c4-f22747469ca8-credential-socket\") pod \"e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp\" (UID: \"b73e8ddc-c45d-4537-a4c4-f22747469ca8\") " pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp"
Oct 31 19:42:31.463453 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:31.432369    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b73e8ddc-c45d-4537-a4c4-f22747469ca8-istiod-ca-cert\") pod \"e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp\" (UID: \"b73e8ddc-c45d-4537-a4c4-f22747469ca8\") " pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp"
Oct 31 19:42:31.464968 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:31.441126    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b73e8ddc-c45d-4537-a4c4-f22747469ca8-istio-envoy\") pod \"e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp\" (UID: \"b73e8ddc-c45d-4537-a4c4-f22747469ca8\") " pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp"
Oct 31 19:42:31.509755 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:31.509708    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b73e8ddc-c45d-4537-a4c4-f22747469ca8-istio-envoy\") pod \"e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp\" (UID: \"b73e8ddc-c45d-4537-a4c4-f22747469ca8\") " pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp"
Oct 31 19:42:31.510950 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:31.510916    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b73e8ddc-c45d-4537-a4c4-f22747469ca8-istio-podinfo\") pod \"e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp\" (UID: \"b73e8ddc-c45d-4537-a4c4-f22747469ca8\") " pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp"
Oct 31 19:42:31.511876 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:31.511823    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gj68\" (UniqueName: \"kubernetes.io/projected/b73e8ddc-c45d-4537-a4c4-f22747469ca8-kube-api-access-9gj68\") pod \"e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp\" (UID: \"b73e8ddc-c45d-4537-a4c4-f22747469ca8\") " pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp"
Oct 31 19:42:31.807338 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:31.807249    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp"
Oct 31 19:42:32.109634 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:32.109603    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp"]
Oct 31 19:42:32.110081 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:32.110009    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:32.110344 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:42:32.110232    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb73e8ddc_c45d_4537_a4c4_f22747469ca8.slice/crio-461d2269ee4e677af6a3a40392c9f1e43071098f2c90cb583d66f6f51f356dc0 WatchSource:0}: Error finding container 461d2269ee4e677af6a3a40392c9f1e43071098f2c90cb583d66f6f51f356dc0: Status 404 returned error can't find the container with id 461d2269ee4e677af6a3a40392c9f1e43071098f2c90cb583d66f6f51f356dc0
Oct 31 19:42:32.116213 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:32.116168    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:42:32.116316 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:32.116253    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:42:32.116316 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:32.116283    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:42:32.732869 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:32.732829    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp" event={"ID":"b73e8ddc-c45d-4537-a4c4-f22747469ca8","Type":"ContainerStarted","Data":"dad871fc77704439f1b4795a36263cdb6a1e934a5c0fb333d92e9145bb9b7db1"}
Oct 31 19:42:32.733057 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:32.732892    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp" event={"ID":"b73e8ddc-c45d-4537-a4c4-f22747469ca8","Type":"ContainerStarted","Data":"461d2269ee4e677af6a3a40392c9f1e43071098f2c90cb583d66f6f51f356dc0"}
Oct 31 19:42:32.757687 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:32.757620    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp" podStartSLOduration=1.757603586 podStartE2EDuration="1.757603586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:42:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:42:32.755731451 +0000 UTC m=+1383.328965580" watchObservedRunningTime="2026-10-31 19:42:32.757603586 +0000 UTC m=+1383.330837717"
Oct 31 19:42:32.807355 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:32.807297    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp"
Oct 31 19:42:32.851886 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:32.851851    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:33.636098 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:33.636068    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:33.814438 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:33.814402    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp"
Oct 31 19:42:34.639616 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:34.639578    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:34.764437 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:34.764405    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp"
Oct 31 19:42:34.765248 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:34.765227    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp"
Oct 31 19:42:35.112164 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:35.112131    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:35.252899 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:35.252846    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:35.793019 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:35.792983    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:36.246441 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:42:36.246403    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 19:42:36.346027 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:36.345991    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:36.402109 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:36.402079    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d45f6c-55b6bc85f9-776mg"]
Oct 31 19:42:36.406993 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:36.406972    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d45f6c-55b6bc85f9-776mg"
Oct 31 19:42:36.416042 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:36.416021    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-d45f6c\""
Oct 31 19:42:36.416728 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:36.416690    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-d45f6c-dockercfg-6lzks\""
Oct 31 19:42:36.423141 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:36.423115    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d45f6c-55b6bc85f9-776mg"]
Oct 31 19:42:36.472473 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:36.472445    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:36.634406 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:36.627302    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:36.634406 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:36.630117    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7fbab67-7316-40e5-ab5a-e848cd654c50-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d45f6c-55b6bc85f9-776mg\" (UID: \"d7fbab67-7316-40e5-ab5a-e848cd654c50\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d45f6c-55b6bc85f9-776mg"
Oct 31 19:42:36.651349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:36.640419    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsqdv\" (UniqueName: \"kubernetes.io/projected/d7fbab67-7316-40e5-ab5a-e848cd654c50-kube-api-access-xsqdv\") pod \"payload-pre-processing-e2e-inf-b-d45f6c-55b6bc85f9-776mg\" (UID: \"d7fbab67-7316-40e5-ab5a-e848cd654c50\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d45f6c-55b6bc85f9-776mg"
Oct 31 19:42:36.749575 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:36.749531    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7fbab67-7316-40e5-ab5a-e848cd654c50-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d45f6c-55b6bc85f9-776mg\" (UID: \"d7fbab67-7316-40e5-ab5a-e848cd654c50\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d45f6c-55b6bc85f9-776mg"
Oct 31 19:42:36.751142 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:36.751118    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsqdv\" (UniqueName: \"kubernetes.io/projected/d7fbab67-7316-40e5-ab5a-e848cd654c50-kube-api-access-xsqdv\") pod \"payload-pre-processing-e2e-inf-b-d45f6c-55b6bc85f9-776mg\" (UID: \"d7fbab67-7316-40e5-ab5a-e848cd654c50\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d45f6c-55b6bc85f9-776mg"
Oct 31 19:42:36.751948 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:36.750875    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7fbab67-7316-40e5-ab5a-e848cd654c50-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d45f6c-55b6bc85f9-776mg\" (UID: \"d7fbab67-7316-40e5-ab5a-e848cd654c50\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d45f6c-55b6bc85f9-776mg"
Oct 31 19:42:36.860357 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:36.855025    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsqdv\" (UniqueName: \"kubernetes.io/projected/d7fbab67-7316-40e5-ab5a-e848cd654c50-kube-api-access-xsqdv\") pod \"payload-pre-processing-e2e-inf-b-d45f6c-55b6bc85f9-776mg\" (UID: \"d7fbab67-7316-40e5-ab5a-e848cd654c50\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d45f6c-55b6bc85f9-776mg"
Oct 31 19:42:37.023380 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:37.023314    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d45f6c-55b6bc85f9-776mg"
Oct 31 19:42:37.312890 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:42:37.312851    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7fbab67_7316_40e5_ab5a_e848cd654c50.slice/crio-29f6b7579270e50a5c84056d49a2202a943cf25e307dbf2bd1949c6caad1541b WatchSource:0}: Error finding container 29f6b7579270e50a5c84056d49a2202a943cf25e307dbf2bd1949c6caad1541b: Status 404 returned error can't find the container with id 29f6b7579270e50a5c84056d49a2202a943cf25e307dbf2bd1949c6caad1541b
Oct 31 19:42:37.313835 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:37.313807    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d45f6c-55b6bc85f9-776mg"]
Oct 31 19:42:37.366112 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:37.366076    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e73256-98c5999-c2vqh"]
Oct 31 19:42:37.366371 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:37.366344    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e73256-98c5999-c2vqh" podUID="f3bd9ab0-338b-4718-8489-beb3c99992f1" containerName="maas-api" containerID="cri-o://6c0bb2586938193ce81a5c469f9e3eea69cc86e93ba4f57e0c138d34ef9b6235" gracePeriod=30
Oct 31 19:42:37.461897 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:37.461857    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:37.641873 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:37.641731    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:37.790426 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:37.789232    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:38.022353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:38.017624    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e73256-98c5999-c2vqh"
Oct 31 19:42:38.055357 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:38.038527    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:38.177496 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:38.172928    4645 generic.go:356] "Generic (PLEG): container finished" podID="f3bd9ab0-338b-4718-8489-beb3c99992f1" containerID="6c0bb2586938193ce81a5c469f9e3eea69cc86e93ba4f57e0c138d34ef9b6235" exitCode=0
Oct 31 19:42:38.177496 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:38.173286    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e73256-98c5999-c2vqh" event={"ID":"f3bd9ab0-338b-4718-8489-beb3c99992f1","Type":"ContainerDied","Data":"6c0bb2586938193ce81a5c469f9e3eea69cc86e93ba4f57e0c138d34ef9b6235"}
Oct 31 19:42:38.177496 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:38.174475    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3bd9ab0-338b-4718-8489-beb3c99992f1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3bd9ab0-338b-4718-8489-beb3c99992f1-metrics-certs\") pod \"f3bd9ab0-338b-4718-8489-beb3c99992f1\" (UID: \"f3bd9ab0-338b-4718-8489-beb3c99992f1\") "
Oct 31 19:42:38.177496 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:38.175303    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3bd9ab0-338b-4718-8489-beb3c99992f1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3bd9ab0-338b-4718-8489-beb3c99992f1-maas-api-tls\") pod \"f3bd9ab0-338b-4718-8489-beb3c99992f1\" (UID: \"f3bd9ab0-338b-4718-8489-beb3c99992f1\") "
Oct 31 19:42:38.177496 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:38.175388    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3bd9ab0-338b-4718-8489-beb3c99992f1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3bd9ab0-338b-4718-8489-beb3c99992f1-openshift-service-ca\") pod \"f3bd9ab0-338b-4718-8489-beb3c99992f1\" (UID: \"f3bd9ab0-338b-4718-8489-beb3c99992f1\") "
Oct 31 19:42:38.177496 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:38.175488    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e73256-98c5999-c2vqh"
Oct 31 19:42:38.177496 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:38.175526    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3bd9ab0-338b-4718-8489-beb3c99992f1-kube-api-access-lpqfm\" (UniqueName: \"kubernetes.io/projected/f3bd9ab0-338b-4718-8489-beb3c99992f1-kube-api-access-lpqfm\") pod \"f3bd9ab0-338b-4718-8489-beb3c99992f1\" (UID: \"f3bd9ab0-338b-4718-8489-beb3c99992f1\") "
Oct 31 19:42:38.177496 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:38.175972    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-e73256-98c5999-c2vqh" event={"ID":"f3bd9ab0-338b-4718-8489-beb3c99992f1","Type":"ContainerDied","Data":"a1485bc914b42d67c2a1ef74e11e0a262026e73bbd9c731d7a0044ce8107139b"}
Oct 31 19:42:38.177496 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:38.177181    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3bd9ab0-338b-4718-8489-beb3c99992f1-openshift-service-ca" pod "f3bd9ab0-338b-4718-8489-beb3c99992f1" (UID: "f3bd9ab0-338b-4718-8489-beb3c99992f1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:42:38.178166 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:38.177503    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d45f6c-55b6bc85f9-776mg" event={"ID":"d7fbab67-7316-40e5-ab5a-e848cd654c50","Type":"ContainerStarted","Data":"e7153bfff50994acdd846d9e5fa22f406abf093b81322f4bace69fe5366fe731"}
Oct 31 19:42:38.178166 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:38.177553    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d45f6c-55b6bc85f9-776mg" event={"ID":"d7fbab67-7316-40e5-ab5a-e848cd654c50","Type":"ContainerStarted","Data":"29f6b7579270e50a5c84056d49a2202a943cf25e307dbf2bd1949c6caad1541b"}
Oct 31 19:42:38.187941 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:38.179098    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3bd9ab0-338b-4718-8489-beb3c99992f1-metrics-certs" pod "f3bd9ab0-338b-4718-8489-beb3c99992f1" (UID: "f3bd9ab0-338b-4718-8489-beb3c99992f1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:42:38.187941 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:38.179166    4645 scope.go:122] "RemoveContainer" containerID="6c0bb2586938193ce81a5c469f9e3eea69cc86e93ba4f57e0c138d34ef9b6235"
Oct 31 19:42:38.187941 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:38.183698    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3bd9ab0-338b-4718-8489-beb3c99992f1-maas-api-tls" pod "f3bd9ab0-338b-4718-8489-beb3c99992f1" (UID: "f3bd9ab0-338b-4718-8489-beb3c99992f1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:42:38.187941 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:38.187053    4645 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3bd9ab0-338b-4718-8489-beb3c99992f1-metrics-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:42:38.187941 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:38.187079    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3bd9ab0-338b-4718-8489-beb3c99992f1-maas-api-tls\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:42:38.187941 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:38.187099    4645 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3bd9ab0-338b-4718-8489-beb3c99992f1-openshift-service-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:42:38.196669 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:38.196640    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3bd9ab0-338b-4718-8489-beb3c99992f1-kube-api-access-lpqfm" pod "f3bd9ab0-338b-4718-8489-beb3c99992f1" (UID: "f3bd9ab0-338b-4718-8489-beb3c99992f1"). InnerVolumeSpecName "kube-api-access-lpqfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:42:38.217211 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:38.211181    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d45f6c-55b6bc85f9-776mg" podStartSLOduration=2.211152876 podStartE2EDuration="2.211152876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:42:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:42:38.200672712 +0000 UTC m=+1388.773906842" watchObservedRunningTime="2026-10-31 19:42:38.211152876 +0000 UTC m=+1388.784387006"
Oct 31 19:42:38.238042 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:38.238008    4645 scope.go:122] "RemoveContainer" containerID="6c0bb2586938193ce81a5c469f9e3eea69cc86e93ba4f57e0c138d34ef9b6235"
Oct 31 19:42:38.238571 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:42:38.238539    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c0bb2586938193ce81a5c469f9e3eea69cc86e93ba4f57e0c138d34ef9b6235\": container with ID starting with 6c0bb2586938193ce81a5c469f9e3eea69cc86e93ba4f57e0c138d34ef9b6235 not found: ID does not exist" containerID="6c0bb2586938193ce81a5c469f9e3eea69cc86e93ba4f57e0c138d34ef9b6235"
Oct 31 19:42:38.238710 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:38.238582    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c0bb2586938193ce81a5c469f9e3eea69cc86e93ba4f57e0c138d34ef9b6235"} err="failed to get container status \"6c0bb2586938193ce81a5c469f9e3eea69cc86e93ba4f57e0c138d34ef9b6235\": rpc error: code = NotFound desc = could not find container \"6c0bb2586938193ce81a5c469f9e3eea69cc86e93ba4f57e0c138d34ef9b6235\": container with ID starting with 6c0bb2586938193ce81a5c469f9e3eea69cc86e93ba4f57e0c138d34ef9b6235 not found: ID does not exist"
Oct 31 19:42:38.289616 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:38.289574    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpqfm\" (UniqueName: \"kubernetes.io/projected/f3bd9ab0-338b-4718-8489-beb3c99992f1-kube-api-access-lpqfm\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:42:38.339382 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:38.337873    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:38.503953 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:38.503921    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e73256-98c5999-c2vqh"]
Oct 31 19:42:38.511209 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:38.511184    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-e73256-98c5999-c2vqh"]
Oct 31 19:42:39.042134 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:39.042102    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:39.187821 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:39.187788    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d45f6c-55b6bc85f9-776mg"
Oct 31 19:42:39.621077 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:39.621001    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3bd9ab0-338b-4718-8489-beb3c99992f1" path="/var/lib/kubelet/pods/f3bd9ab0-338b-4718-8489-beb3c99992f1/volumes"
Oct 31 19:42:40.328610 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:40.328579    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:41.436337 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:41.436306    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:41.607834 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:41.607799    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:41.711236 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:41.711133    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:42.068351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:42.068198    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:42.355876 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:42.355794    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:42.487483 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:42.487450    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z"]
Oct 31 19:42:42.495885 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:42.495855    4645 cpu_manager.go:414] "containerMap: removing container" podUID="f3bd9ab0-338b-4718-8489-beb3c99992f1" containerName="maas-api"
Oct 31 19:42:42.495885 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:42.495886    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3bd9ab0-338b-4718-8489-beb3c99992f1" containerName="maas-api"
Oct 31 19:42:42.496129 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:42.496113    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3bd9ab0-338b-4718-8489-beb3c99992f1" containerName="maas-api"
Oct 31 19:42:42.504419 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:42.504382    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z"
Oct 31 19:42:42.506539 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:42.506510    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z"]
Oct 31 19:42:42.514706 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:42.514678    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-9efeae-openshift-default-dockercfg-jgppr\""
Oct 31 19:42:42.552570 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:42.552535    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:42.568847 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:42.568820    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9pw9\" (UniqueName: \"kubernetes.io/projected/8eee5947-9731-4194-aa80-7fbd619e551f-kube-api-access-f9pw9\") pod \"e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z\" (UID: \"8eee5947-9731-4194-aa80-7fbd619e551f\") " pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z"
Oct 31 19:42:42.569048 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:42.569031    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8eee5947-9731-4194-aa80-7fbd619e551f-istio-token\") pod \"e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z\" (UID: \"8eee5947-9731-4194-aa80-7fbd619e551f\") " pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z"
Oct 31 19:42:42.569100 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:42.569065    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8eee5947-9731-4194-aa80-7fbd619e551f-workload-certs\") pod \"e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z\" (UID: \"8eee5947-9731-4194-aa80-7fbd619e551f\") " pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z"
Oct 31 19:42:42.569100 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:42.569085    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8eee5947-9731-4194-aa80-7fbd619e551f-istio-podinfo\") pod \"e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z\" (UID: \"8eee5947-9731-4194-aa80-7fbd619e551f\") " pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z"
Oct 31 19:42:42.569197 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:42.569173    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eee5947-9731-4194-aa80-7fbd619e551f-workload-socket\") pod \"e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z\" (UID: \"8eee5947-9731-4194-aa80-7fbd619e551f\") " pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z"
Oct 31 19:42:42.569239 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:42.569219    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8eee5947-9731-4194-aa80-7fbd619e551f-istiod-ca-cert\") pod \"e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z\" (UID: \"8eee5947-9731-4194-aa80-7fbd619e551f\") " pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z"
Oct 31 19:42:42.569420 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:42.569390    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eee5947-9731-4194-aa80-7fbd619e551f-credential-socket\") pod \"e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z\" (UID: \"8eee5947-9731-4194-aa80-7fbd619e551f\") " pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z"
Oct 31 19:42:42.569513 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:42.569493    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8eee5947-9731-4194-aa80-7fbd619e551f-istio-data\") pod \"e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z\" (UID: \"8eee5947-9731-4194-aa80-7fbd619e551f\") " pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z"
Oct 31 19:42:42.569589 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:42.569572    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8eee5947-9731-4194-aa80-7fbd619e551f-istio-envoy\") pod \"e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z\" (UID: \"8eee5947-9731-4194-aa80-7fbd619e551f\") " pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z"
Oct 31 19:42:42.674372 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:42.671262    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:42.688617 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:42.688575    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eee5947-9731-4194-aa80-7fbd619e551f-credential-socket\") pod \"e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z\" (UID: \"8eee5947-9731-4194-aa80-7fbd619e551f\") " pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z"
Oct 31 19:42:42.689237 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:42.689212    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8eee5947-9731-4194-aa80-7fbd619e551f-istio-data\") pod \"e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z\" (UID: \"8eee5947-9731-4194-aa80-7fbd619e551f\") " pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z"
Oct 31 19:42:42.689477 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:42.689457    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8eee5947-9731-4194-aa80-7fbd619e551f-istio-envoy\") pod \"e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z\" (UID: \"8eee5947-9731-4194-aa80-7fbd619e551f\") " pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z"
Oct 31 19:42:42.689699 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:42.689674    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eee5947-9731-4194-aa80-7fbd619e551f-credential-socket\") pod \"e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z\" (UID: \"8eee5947-9731-4194-aa80-7fbd619e551f\") " pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z"
Oct 31 19:42:42.689838 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:42.689820    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9pw9\" (UniqueName: \"kubernetes.io/projected/8eee5947-9731-4194-aa80-7fbd619e551f-kube-api-access-f9pw9\") pod \"e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z\" (UID: \"8eee5947-9731-4194-aa80-7fbd619e551f\") " pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z"
Oct 31 19:42:42.707354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:42.689958    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8eee5947-9731-4194-aa80-7fbd619e551f-istio-data\") pod \"e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z\" (UID: \"8eee5947-9731-4194-aa80-7fbd619e551f\") " pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z"
Oct 31 19:42:42.707830 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:42.707796    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8eee5947-9731-4194-aa80-7fbd619e551f-istio-token\") pod \"e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z\" (UID: \"8eee5947-9731-4194-aa80-7fbd619e551f\") " pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z"
Oct 31 19:42:42.707999 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:42.707980    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8eee5947-9731-4194-aa80-7fbd619e551f-workload-certs\") pod \"e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z\" (UID: \"8eee5947-9731-4194-aa80-7fbd619e551f\") " pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z"
Oct 31 19:42:42.708104 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:42.708089    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8eee5947-9731-4194-aa80-7fbd619e551f-istio-podinfo\") pod \"e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z\" (UID: \"8eee5947-9731-4194-aa80-7fbd619e551f\") " pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z"
Oct 31 19:42:42.708239 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:42.708223    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eee5947-9731-4194-aa80-7fbd619e551f-workload-socket\") pod \"e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z\" (UID: \"8eee5947-9731-4194-aa80-7fbd619e551f\") " pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z"
Oct 31 19:42:42.708454 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:42.708436    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8eee5947-9731-4194-aa80-7fbd619e551f-istiod-ca-cert\") pod \"e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z\" (UID: \"8eee5947-9731-4194-aa80-7fbd619e551f\") " pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z"
Oct 31 19:42:42.730602 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:42.709833    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eee5947-9731-4194-aa80-7fbd619e551f-workload-socket\") pod \"e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z\" (UID: \"8eee5947-9731-4194-aa80-7fbd619e551f\") " pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z"
Oct 31 19:42:42.730789 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:42.710225    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8eee5947-9731-4194-aa80-7fbd619e551f-workload-certs\") pod \"e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z\" (UID: \"8eee5947-9731-4194-aa80-7fbd619e551f\") " pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z"
Oct 31 19:42:42.731582 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:42.731556    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8eee5947-9731-4194-aa80-7fbd619e551f-istiod-ca-cert\") pod \"e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z\" (UID: \"8eee5947-9731-4194-aa80-7fbd619e551f\") " pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z"
Oct 31 19:42:42.734472 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:42.734447    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8eee5947-9731-4194-aa80-7fbd619e551f-istio-podinfo\") pod \"e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z\" (UID: \"8eee5947-9731-4194-aa80-7fbd619e551f\") " pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z"
Oct 31 19:42:42.815350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:42.806219    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8eee5947-9731-4194-aa80-7fbd619e551f-istio-token\") pod \"e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z\" (UID: \"8eee5947-9731-4194-aa80-7fbd619e551f\") " pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z"
Oct 31 19:42:42.815350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:42.814976    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9pw9\" (UniqueName: \"kubernetes.io/projected/8eee5947-9731-4194-aa80-7fbd619e551f-kube-api-access-f9pw9\") pod \"e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z\" (UID: \"8eee5947-9731-4194-aa80-7fbd619e551f\") " pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z"
Oct 31 19:42:42.815350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:42.815248    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8eee5947-9731-4194-aa80-7fbd619e551f-istio-envoy\") pod \"e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z\" (UID: \"8eee5947-9731-4194-aa80-7fbd619e551f\") " pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z"
Oct 31 19:42:42.838354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:42.829632    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z"
Oct 31 19:42:43.292113 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:43.292082    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z"]
Oct 31 19:42:43.292589 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:42:43.292561    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eee5947_9731_4194_aa80_7fbd619e551f.slice/crio-055feac6c46e5ca11d8579f1e81525390030130b4d4dd7c6446abca5b3361b19 WatchSource:0}: Error finding container 055feac6c46e5ca11d8579f1e81525390030130b4d4dd7c6446abca5b3361b19: Status 404 returned error can't find the container with id 055feac6c46e5ca11d8579f1e81525390030130b4d4dd7c6446abca5b3361b19
Oct 31 19:42:43.295712 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:43.295677    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:42:43.295806 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:43.295759    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:42:43.295875 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:43.295806    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:42:43.947353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:43.946357    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:44.246403 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:44.245719    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z" event={"ID":"8eee5947-9731-4194-aa80-7fbd619e551f","Type":"ContainerStarted","Data":"db0770b5441fe07bb8782b88cf635a09301dc956518832c2b6371caf6bb70dd2"}
Oct 31 19:42:44.246403 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:44.245793    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z" event={"ID":"8eee5947-9731-4194-aa80-7fbd619e551f","Type":"ContainerStarted","Data":"055feac6c46e5ca11d8579f1e81525390030130b4d4dd7c6446abca5b3361b19"}
Oct 31 19:42:44.843352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:44.832110    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z"
Oct 31 19:42:44.939351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:44.938693    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z"
Oct 31 19:42:45.048361 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:45.022523    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z" podStartSLOduration=3.022501791 podStartE2EDuration="3.022501791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:42:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:42:44.303390941 +0000 UTC m=+1394.876625071" watchObservedRunningTime="2026-10-31 19:42:45.022501791 +0000 UTC m=+1395.595735922"
Oct 31 19:42:45.371527 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:45.371496    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z"
Oct 31 19:42:45.411360 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:45.409637    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z"
Oct 31 19:42:45.859354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:45.855406    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l" podUID="12bea86c-61e5-465e-82fb-b4673a4d1be9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:42:45.962355 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:45.958250    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:46.110407 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:46.084146    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:46.243488 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:46.243442    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw" podUID="d504b2d3-c26d-4dcf-a7eb-efe20732ae4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:42:46.301349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:46.300961    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669" podUID="9285bee0-bf7c-4fd4-9da7-a1db021cae05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 31 19:42:46.421351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:46.420077    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-03dfee/test-model-03dfee-kserve-68c5b8bcbf-cwgcs" podUID="6e861821-e221-44d2-956a-66f0f656e1b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.135:8000/ready\": context deadline exceeded"
Oct 31 19:42:46.566355 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:46.495411    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x" podUID="a774e286-dcb8-4a2f-8541-26bf231eb08d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 19:42:46.584466 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:46.584418    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw" podUID="1287db54-a587-4c11-9eb6-1606de555894" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:42:46.627357 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:46.598716    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n" podUID="dbb97168-db93-4757-9a1b-f9c5eb704fc5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 19:42:46.627357 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:46.598955    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4" podUID="6032a997-b795-4561-805c-110cf67a9d5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded"
Oct 31 19:42:46.640521 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:46.640485    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-46875"]
Oct 31 19:42:46.644347 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:46.640964    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-46875" podUID="41a92e12-0b8e-4479-af08-9ef868f8ef83" containerName="istio-proxy" containerID="cri-o://ceaebf191fbf94e0ff59dc73985163a6aa49a6433d71fef21408a373b87b2d36" gracePeriod=30
Oct 31 19:42:46.663921 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:46.663873    4645 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x" podUID="a774e286-dcb8-4a2f-8541-26bf231eb08d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:42:46.777440 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:46.721299    4645 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n" podUID="dbb97168-db93-4757-9a1b-f9c5eb704fc5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:42:46.777440 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:46.743564    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7"]
Oct 31 19:42:46.777440 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:46.743920    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7" podUID="c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca" containerName="istio-proxy" containerID="cri-o://83e0163dd2f3808db950854f92a5e080b648e19477398cc92282f9da5e08254b" gracePeriod=30
Oct 31 19:42:46.777440 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:46.768447    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:46.902751 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:46.889486    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-88eaf9/test-model-88eaf9-kserve-7f86f56bdf-28gmk" podUID="244c8e8b-236e-47cf-99c6-69a0a1fb6af6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.138:8000/ready\": context deadline exceeded"
Oct 31 19:42:46.903483 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:46.903421    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l" podUID="12bea86c-61e5-465e-82fb-b4673a4d1be9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 31 19:42:46.909502 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:46.909477    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l"
Oct 31 19:42:46.955354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:46.933976    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-9th7f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:42:46.955694 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:46.955645    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9th7f" podUID="76bc1f3a-74b3-49b3-bde1-cc3b305260d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:42:47.051739 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:47.037196    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-6k6cd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:42:47.051739 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:47.037254    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6k6cd" podUID="a8fca631-527d-46c0-9197-5edd7762adb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:42:47.341731 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:47.341524    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669" podUID="9285bee0-bf7c-4fd4-9da7-a1db021cae05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:42:47.345033 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:47.343930    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669"
Oct 31 19:42:47.400001 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:47.399964    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-msrfk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:42:47.400263 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:47.400233    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-msrfk" podUID="48b0949c-0c89-4c80-a1dc-64d14839f620" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:42:47.425801 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:47.425751    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-03dfee/test-model-03dfee-kserve-68c5b8bcbf-cwgcs" podUID="6e861821-e221-44d2-956a-66f0f656e1b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.135:8000/ready\": context deadline exceeded"
Oct 31 19:42:47.428419 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:47.428394    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-03dfee/test-model-03dfee-kserve-68c5b8bcbf-cwgcs"
Oct 31 19:42:47.515888 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:47.515840    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x" podUID="a774e286-dcb8-4a2f-8541-26bf231eb08d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:42:47.538635 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:47.538600    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x"
Oct 31 19:42:47.552379 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:47.547124    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:47.833682 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:47.826454    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:47.917353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:47.895451    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-88eaf9/test-model-88eaf9-kserve-7f86f56bdf-28gmk" podUID="244c8e8b-236e-47cf-99c6-69a0a1fb6af6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.138:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:42:47.917353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:47.896292    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw" podUID="1287db54-a587-4c11-9eb6-1606de555894" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": context deadline exceeded"
Oct 31 19:42:47.917353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:47.897965    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n" podUID="dbb97168-db93-4757-9a1b-f9c5eb704fc5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:42:47.917353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:47.901809    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4" podUID="6032a997-b795-4561-805c-110cf67a9d5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:42:47.919918 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:47.919886    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l" podUID="12bea86c-61e5-465e-82fb-b4673a4d1be9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:42:47.968382 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:47.967020    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:48.237389 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:48.232152    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw" podUID="d504b2d3-c26d-4dcf-a7eb-efe20732ae4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 31 19:42:48.385572 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:48.385525    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669" podUID="9285bee0-bf7c-4fd4-9da7-a1db021cae05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:42:48.431473 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:48.431422    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-03dfee/test-model-03dfee-kserve-68c5b8bcbf-cwgcs" podUID="6e861821-e221-44d2-956a-66f0f656e1b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.135:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:42:48.557406 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:48.556397    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x" podUID="a774e286-dcb8-4a2f-8541-26bf231eb08d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 19:42:48.754357 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:48.747308    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:48.896387 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:48.896157    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-88eaf9/test-model-88eaf9-kserve-7f86f56bdf-28gmk" podUID="244c8e8b-236e-47cf-99c6-69a0a1fb6af6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.138:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:42:48.907770 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:48.907711    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4" podUID="6032a997-b795-4561-805c-110cf67a9d5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:42:48.924301 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:48.924255    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l" podUID="12bea86c-61e5-465e-82fb-b4673a4d1be9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded"
Oct 31 19:42:48.945014 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:42:48.935907    4645 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.079s"
Oct 31 19:42:49.296255 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:49.296225    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-88eaf9/test-model-88eaf9-kserve-7f86f56bdf-28gmk"
Oct 31 19:42:49.415350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:49.411704    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n"
Oct 31 19:42:49.415350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:49.411810    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4"
Oct 31 19:42:49.464632 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:49.418503    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw"
Oct 31 19:42:49.464632 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:49.418656    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n"
Oct 31 19:42:49.464632 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:49.418733    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x"
Oct 31 19:42:49.464632 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:49.295969    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw" podUID="d504b2d3-c26d-4dcf-a7eb-efe20732ae4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:42:49.513438 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:49.509692    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw"
Oct 31 19:42:49.529295 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:49.529244    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-03dfee/test-model-03dfee-kserve-68c5b8bcbf-cwgcs" podUID="6e861821-e221-44d2-956a-66f0f656e1b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.135:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:42:49.542536 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:49.542475    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw" podUID="1287db54-a587-4c11-9eb6-1606de555894" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:42:49.623421 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:49.622942    4645 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-b-88eaf9/test-model-88eaf9-kserve-7f86f56bdf-28gmk" podUID="244c8e8b-236e-47cf-99c6-69a0a1fb6af6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.138:8000/health\": context deadline exceeded"
Oct 31 19:42:49.734785 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:49.734447    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:49.937351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:49.912446    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4" podUID="6032a997-b795-4561-805c-110cf67a9d5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:42:50.109350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:50.109125    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-88eaf9/test-model-88eaf9-kserve-7f86f56bdf-28gmk" podUID="244c8e8b-236e-47cf-99c6-69a0a1fb6af6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.138:8000/ready\": context deadline exceeded"
Oct 31 19:42:50.291450 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:50.291260    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l" podUID="12bea86c-61e5-465e-82fb-b4673a4d1be9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:42:50.335601 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:50.335564    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:50.433360 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:50.430517    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw" podUID="d504b2d3-c26d-4dcf-a7eb-efe20732ae4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:42:50.437780 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:50.437753    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669"
Oct 31 19:42:50.438039 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:50.438025    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4"
Oct 31 19:42:50.438178 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:50.438169    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw"
Oct 31 19:42:50.438299 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:50.438289    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-03dfee/test-model-03dfee-kserve-68c5b8bcbf-cwgcs"
Oct 31 19:42:50.452677 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:50.450902    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d45f6c-55b6bc85f9-776mg"
Oct 31 19:42:50.499351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:50.490942    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-88eaf9/test-model-88eaf9-kserve-7f86f56bdf-28gmk"
Oct 31 19:42:50.503004 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:50.502969    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l"
Oct 31 19:42:50.504398 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:50.504376    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw"
Oct 31 19:42:50.716840 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:50.711258    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:50.921370 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:50.921282    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:52.258012 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:52.257977    4645 generic.go:356] "Generic (PLEG): container finished" podID="41a92e12-0b8e-4479-af08-9ef868f8ef83" containerID="ceaebf191fbf94e0ff59dc73985163a6aa49a6433d71fef21408a373b87b2d36" exitCode=0
Oct 31 19:42:52.258808 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:52.258780    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-46875" event={"ID":"41a92e12-0b8e-4479-af08-9ef868f8ef83","Type":"ContainerDied","Data":"ceaebf191fbf94e0ff59dc73985163a6aa49a6433d71fef21408a373b87b2d36"}
Oct 31 19:42:52.260595 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:52.260430    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:52.399550 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:52.399464    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-46875"
Oct 31 19:42:52.431174 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:52.430982    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41a92e12-0b8e-4479-af08-9ef868f8ef83-istio-token\" (UniqueName: \"kubernetes.io/projected/41a92e12-0b8e-4479-af08-9ef868f8ef83-istio-token\") pod \"41a92e12-0b8e-4479-af08-9ef868f8ef83\" (UID: \"41a92e12-0b8e-4479-af08-9ef868f8ef83\") "
Oct 31 19:42:52.431174 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:52.431121    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/41a92e12-0b8e-4479-af08-9ef868f8ef83-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41a92e12-0b8e-4479-af08-9ef868f8ef83-istio-podinfo\") pod \"41a92e12-0b8e-4479-af08-9ef868f8ef83\" (UID: \"41a92e12-0b8e-4479-af08-9ef868f8ef83\") "
Oct 31 19:42:52.431174 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:52.431159    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41a92e12-0b8e-4479-af08-9ef868f8ef83-kube-api-access-56bcs\" (UniqueName: \"kubernetes.io/projected/41a92e12-0b8e-4479-af08-9ef868f8ef83-kube-api-access-56bcs\") pod \"41a92e12-0b8e-4479-af08-9ef868f8ef83\" (UID: \"41a92e12-0b8e-4479-af08-9ef868f8ef83\") "
Oct 31 19:42:52.431593 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:52.431209    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41a92e12-0b8e-4479-af08-9ef868f8ef83-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41a92e12-0b8e-4479-af08-9ef868f8ef83-workload-certs\") pod \"41a92e12-0b8e-4479-af08-9ef868f8ef83\" (UID: \"41a92e12-0b8e-4479-af08-9ef868f8ef83\") "
Oct 31 19:42:52.431593 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:52.431387    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41a92e12-0b8e-4479-af08-9ef868f8ef83-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41a92e12-0b8e-4479-af08-9ef868f8ef83-workload-socket\") pod \"41a92e12-0b8e-4479-af08-9ef868f8ef83\" (UID: \"41a92e12-0b8e-4479-af08-9ef868f8ef83\") "
Oct 31 19:42:52.431593 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:52.431552    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41a92e12-0b8e-4479-af08-9ef868f8ef83-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41a92e12-0b8e-4479-af08-9ef868f8ef83-credential-socket\") pod \"41a92e12-0b8e-4479-af08-9ef868f8ef83\" (UID: \"41a92e12-0b8e-4479-af08-9ef868f8ef83\") "
Oct 31 19:42:52.432010 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:52.431653    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41a92e12-0b8e-4479-af08-9ef868f8ef83-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41a92e12-0b8e-4479-af08-9ef868f8ef83-istio-envoy\") pod \"41a92e12-0b8e-4479-af08-9ef868f8ef83\" (UID: \"41a92e12-0b8e-4479-af08-9ef868f8ef83\") "
Oct 31 19:42:52.441287 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:52.432255    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41a92e12-0b8e-4479-af08-9ef868f8ef83-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41a92e12-0b8e-4479-af08-9ef868f8ef83-istio-data\") pod \"41a92e12-0b8e-4479-af08-9ef868f8ef83\" (UID: \"41a92e12-0b8e-4479-af08-9ef868f8ef83\") "
Oct 31 19:42:52.441287 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:52.432615    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41a92e12-0b8e-4479-af08-9ef868f8ef83-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41a92e12-0b8e-4479-af08-9ef868f8ef83-istiod-ca-cert\") pod \"41a92e12-0b8e-4479-af08-9ef868f8ef83\" (UID: \"41a92e12-0b8e-4479-af08-9ef868f8ef83\") "
Oct 31 19:42:52.441287 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:52.432780    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41a92e12-0b8e-4479-af08-9ef868f8ef83-workload-certs" pod "41a92e12-0b8e-4479-af08-9ef868f8ef83" (UID: "41a92e12-0b8e-4479-af08-9ef868f8ef83"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:42:52.441287 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:52.433650    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41a92e12-0b8e-4479-af08-9ef868f8ef83-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:42:52.441287 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:52.433882    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41a92e12-0b8e-4479-af08-9ef868f8ef83-workload-socket" pod "41a92e12-0b8e-4479-af08-9ef868f8ef83" (UID: "41a92e12-0b8e-4479-af08-9ef868f8ef83"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:42:52.441287 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:52.435351    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41a92e12-0b8e-4479-af08-9ef868f8ef83-credential-socket" pod "41a92e12-0b8e-4479-af08-9ef868f8ef83" (UID: "41a92e12-0b8e-4479-af08-9ef868f8ef83"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:42:52.441287 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:52.435536    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41a92e12-0b8e-4479-af08-9ef868f8ef83-istio-data" pod "41a92e12-0b8e-4479-af08-9ef868f8ef83" (UID: "41a92e12-0b8e-4479-af08-9ef868f8ef83"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:42:52.441287 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:52.435815    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41a92e12-0b8e-4479-af08-9ef868f8ef83-istiod-ca-cert" pod "41a92e12-0b8e-4479-af08-9ef868f8ef83" (UID: "41a92e12-0b8e-4479-af08-9ef868f8ef83"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:42:52.452534 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:52.452056    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41a92e12-0b8e-4479-af08-9ef868f8ef83-istio-envoy" pod "41a92e12-0b8e-4479-af08-9ef868f8ef83" (UID: "41a92e12-0b8e-4479-af08-9ef868f8ef83"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:42:52.454825 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:52.454779    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41a92e12-0b8e-4479-af08-9ef868f8ef83-istio-token" pod "41a92e12-0b8e-4479-af08-9ef868f8ef83" (UID: "41a92e12-0b8e-4479-af08-9ef868f8ef83"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:42:52.475876 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:52.475513    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41a92e12-0b8e-4479-af08-9ef868f8ef83-kube-api-access-56bcs" pod "41a92e12-0b8e-4479-af08-9ef868f8ef83" (UID: "41a92e12-0b8e-4479-af08-9ef868f8ef83"). InnerVolumeSpecName "kube-api-access-56bcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:42:52.487224 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:52.487145    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/41a92e12-0b8e-4479-af08-9ef868f8ef83-istio-podinfo" pod "41a92e12-0b8e-4479-af08-9ef868f8ef83" (UID: "41a92e12-0b8e-4479-af08-9ef868f8ef83"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:42:52.535597 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:52.535571    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41a92e12-0b8e-4479-af08-9ef868f8ef83-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:42:52.535597 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:52.535596    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41a92e12-0b8e-4479-af08-9ef868f8ef83-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:42:52.535762 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:52.535607    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41a92e12-0b8e-4479-af08-9ef868f8ef83-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:42:52.535762 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:52.535617    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41a92e12-0b8e-4479-af08-9ef868f8ef83-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:42:52.535762 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:52.535626    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56bcs\" (UniqueName: \"kubernetes.io/projected/41a92e12-0b8e-4479-af08-9ef868f8ef83-kube-api-access-56bcs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:42:52.535762 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:52.535638    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41a92e12-0b8e-4479-af08-9ef868f8ef83-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:42:52.535762 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:52.535646    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41a92e12-0b8e-4479-af08-9ef868f8ef83-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:42:52.535762 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:52.535654    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41a92e12-0b8e-4479-af08-9ef868f8ef83-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:42:52.540908 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:52.540887    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7"
Oct 31 19:42:52.636558 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:52.636512    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca-credential-socket\") pod \"c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca\" (UID: \"c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca\") "
Oct 31 19:42:52.636738 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:52.636707    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca-istio-token\" (UniqueName: \"kubernetes.io/projected/c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca-istio-token\") pod \"c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca\" (UID: \"c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca\") "
Oct 31 19:42:52.636833 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:52.636806    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca-istiod-ca-cert\") pod \"c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca\" (UID: \"c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca\") "
Oct 31 19:42:52.636901 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:52.636806    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca-credential-socket" pod "c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca" (UID: "c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:42:52.636901 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:52.636864    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca-kube-api-access-6nxd8\" (UniqueName: \"kubernetes.io/projected/c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca-kube-api-access-6nxd8\") pod \"c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca\" (UID: \"c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca\") "
Oct 31 19:42:52.637021 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:52.636899    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca-istio-data\") pod \"c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca\" (UID: \"c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca\") "
Oct 31 19:42:52.637021 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:52.636987    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca-istio-envoy\") pod \"c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca\" (UID: \"c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca\") "
Oct 31 19:42:52.637142 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:52.637049    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca-workload-certs\") pod \"c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca\" (UID: \"c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca\") "
Oct 31 19:42:52.637142 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:52.637122    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca-istio-podinfo\") pod \"c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca\" (UID: \"c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca\") "
Oct 31 19:42:52.637249 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:52.637208    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca-workload-socket\") pod \"c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca\" (UID: \"c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca\") "
Oct 31 19:42:52.637838 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:52.637649    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca-workload-certs" pod "c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca" (UID: "c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:42:52.637962 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:52.637872    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca-workload-socket" pod "c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca" (UID: "c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:42:52.638138 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:52.638103    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca-istio-data" pod "c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca" (UID: "c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:42:52.639072 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:52.638756    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:42:52.639072 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:52.638783    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:42:52.639072 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:52.638796    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:42:52.639072 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:52.638813    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:42:52.639536 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:52.639515    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca-istiod-ca-cert" pod "c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca" (UID: "c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:42:52.639727 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:52.639710    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca-istio-envoy" pod "c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca" (UID: "c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:42:52.639911 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:52.639888    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca-kube-api-access-6nxd8" pod "c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca" (UID: "c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca"). InnerVolumeSpecName "kube-api-access-6nxd8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:42:52.640155 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:52.640132    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca-istio-token" pod "c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca" (UID: "c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:42:52.640224 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:52.640137    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca-istio-podinfo" pod "c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca" (UID: "c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:42:52.740464 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:52.740371    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:42:52.740464 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:52.740405    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:42:52.740464 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:52.740420    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nxd8\" (UniqueName: \"kubernetes.io/projected/c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca-kube-api-access-6nxd8\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:42:52.740464 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:52.740434    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:42:52.740464 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:52.740446    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:42:53.077403 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:53.075431    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:53.433359 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:53.430020    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-46875" event={"ID":"41a92e12-0b8e-4479-af08-9ef868f8ef83","Type":"ContainerDied","Data":"911ba69f86a300a6ea928a1e6d0f992e548809d9f664b0d636fb16f878c2e61d"}
Oct 31 19:42:53.433359 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:53.430099    4645 scope.go:122] "RemoveContainer" containerID="ceaebf191fbf94e0ff59dc73985163a6aa49a6433d71fef21408a373b87b2d36"
Oct 31 19:42:53.457830 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:53.444500    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-46875"
Oct 31 19:42:53.605594 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:53.603875    4645 generic.go:356] "Generic (PLEG): container finished" podID="c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca" containerID="83e0163dd2f3808db950854f92a5e080b648e19477398cc92282f9da5e08254b" exitCode=0
Oct 31 19:42:53.610763 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:53.610715    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7" event={"ID":"c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca","Type":"ContainerDied","Data":"83e0163dd2f3808db950854f92a5e080b648e19477398cc92282f9da5e08254b"}
Oct 31 19:42:53.617901 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:53.611055    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7" event={"ID":"c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca","Type":"ContainerDied","Data":"c1d19bd5a1ccbf3bc5b060e4cf2257290b0aa203dbe557901aeea33e3c5411a8"}
Oct 31 19:42:53.648929 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:53.613648    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7"
Oct 31 19:42:53.648929 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:53.648497    4645 scope.go:122] "RemoveContainer" containerID="83e0163dd2f3808db950854f92a5e080b648e19477398cc92282f9da5e08254b"
Oct 31 19:42:53.677397 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:53.676438    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:53.683107 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:53.683076    4645 scope.go:122] "RemoveContainer" containerID="83e0163dd2f3808db950854f92a5e080b648e19477398cc92282f9da5e08254b"
Oct 31 19:42:53.683858 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:42:53.683801    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83e0163dd2f3808db950854f92a5e080b648e19477398cc92282f9da5e08254b\": container with ID starting with 83e0163dd2f3808db950854f92a5e080b648e19477398cc92282f9da5e08254b not found: ID does not exist" containerID="83e0163dd2f3808db950854f92a5e080b648e19477398cc92282f9da5e08254b"
Oct 31 19:42:53.684029 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:53.684003    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83e0163dd2f3808db950854f92a5e080b648e19477398cc92282f9da5e08254b"} err="failed to get container status \"83e0163dd2f3808db950854f92a5e080b648e19477398cc92282f9da5e08254b\": rpc error: code = NotFound desc = could not find container \"83e0163dd2f3808db950854f92a5e080b648e19477398cc92282f9da5e08254b\": container with ID starting with 83e0163dd2f3808db950854f92a5e080b648e19477398cc92282f9da5e08254b not found: ID does not exist"
Oct 31 19:42:53.835845 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:53.835813    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7"]
Oct 31 19:42:53.846090 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:53.846065    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-xf6w7"]
Oct 31 19:42:53.854376 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:53.854318    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-46875"]
Oct 31 19:42:53.908058 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:53.908026    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-e73256-openshift-default-dfc9897d9-46875"]
Oct 31 19:42:54.552408 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:54.552373    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:55.353491 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:55.353451    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:55.620917 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:55.620837    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41a92e12-0b8e-4479-af08-9ef868f8ef83" path="/var/lib/kubelet/pods/41a92e12-0b8e-4479-af08-9ef868f8ef83/volumes"
Oct 31 19:42:55.621453 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:55.621437    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca" path="/var/lib/kubelet/pods/c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca/volumes"
Oct 31 19:42:56.245796 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:56.245754    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:56.834526 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:56.834483    4645 patch_prober.go:29] interesting pod/e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:42:56.835037 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:56.834547    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9" podUID="7e5cadc6-cd7f-402d-9550-874d7b3d6376" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:42:57.264355 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:57.264304    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:58.117053 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:58.116978    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:58.615919 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:58.615833    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:59.356457 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:59.356424    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:42:59.688498 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:59.688460    4645 generic.go:356] "Generic (PLEG): container finished" podID="7e5cadc6-cd7f-402d-9550-874d7b3d6376" containerID="2cf907396d4b6896492bbd4c364a1b67a16f35568fb0ef4e23fe0ed52b3a8514" exitCode=0
Oct 31 19:42:59.688674 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:59.688529    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9" event={"ID":"7e5cadc6-cd7f-402d-9550-874d7b3d6376","Type":"ContainerDied","Data":"2cf907396d4b6896492bbd4c364a1b67a16f35568fb0ef4e23fe0ed52b3a8514"}
Oct 31 19:42:59.689058 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:59.689040    4645 scope.go:122] "RemoveContainer" containerID="2cf907396d4b6896492bbd4c364a1b67a16f35568fb0ef4e23fe0ed52b3a8514"
Oct 31 19:42:59.691450 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:59.691415    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:42:59.691556 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:59.691489    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:42:59.691556 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:59.691520    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:42:59.898254 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:42:59.898223    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:00.112710 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:00.112623    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:00.327145 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:00.327112    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:00.659393 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:00.657376    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:00.702043 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:00.702011    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9" event={"ID":"7e5cadc6-cd7f-402d-9550-874d7b3d6376","Type":"ContainerStarted","Data":"8ded2c69a5be4f9a7ed997ea434b89bcf73ef6455045ec0c15c45cf4497ef976"}
Oct 31 19:43:01.390049 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:01.390007    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1e8cf0-56bb54564c-t5hz7"]
Oct 31 19:43:01.390965 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:01.390940    4645 cpu_manager.go:414] "containerMap: removing container" podUID="41a92e12-0b8e-4479-af08-9ef868f8ef83" containerName="istio-proxy"
Oct 31 19:43:01.390965 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:01.390963    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41a92e12-0b8e-4479-af08-9ef868f8ef83" containerName="istio-proxy"
Oct 31 19:43:01.391140 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:01.390993    4645 cpu_manager.go:414] "containerMap: removing container" podUID="c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca" containerName="istio-proxy"
Oct 31 19:43:01.391140 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:01.391001    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca" containerName="istio-proxy"
Oct 31 19:43:01.391291 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:01.391276    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="41a92e12-0b8e-4479-af08-9ef868f8ef83" containerName="istio-proxy"
Oct 31 19:43:01.391372 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:01.391298    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="c960b4c7-b9ab-4c5e-88d8-feaabe41b0ca" containerName="istio-proxy"
Oct 31 19:43:01.394937 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:01.394918    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1e8cf0-56bb54564c-t5hz7"
Oct 31 19:43:01.397527 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:01.397498    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1e8cf0-56bb54564c-t5hz7"]
Oct 31 19:43:01.466349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:01.461674    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-1e8cf0-dockercfg-8l2cj\""
Oct 31 19:43:01.466349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:01.461776    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-1e8cf0\""
Oct 31 19:43:01.495350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:01.490084    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9"
Oct 31 19:43:01.495350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:01.490202    4645 patch_prober.go:29] interesting pod/e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body=
Oct 31 19:43:01.495350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:01.490245    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9" podUID="7e5cadc6-cd7f-402d-9550-874d7b3d6376" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused"
Oct 31 19:43:01.584349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:01.578507    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/788ad5fd-ae6a-4953-a113-1357e62f2b0b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1e8cf0-56bb54564c-t5hz7\" (UID: \"788ad5fd-ae6a-4953-a113-1357e62f2b0b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1e8cf0-56bb54564c-t5hz7"
Oct 31 19:43:01.584349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:01.578856    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vxd9\" (UniqueName: \"kubernetes.io/projected/788ad5fd-ae6a-4953-a113-1357e62f2b0b-kube-api-access-2vxd9\") pod \"payload-processing-e2e-inf-a-1e8cf0-56bb54564c-t5hz7\" (UID: \"788ad5fd-ae6a-4953-a113-1357e62f2b0b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1e8cf0-56bb54564c-t5hz7"
Oct 31 19:43:01.700349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:01.694983    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/788ad5fd-ae6a-4953-a113-1357e62f2b0b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1e8cf0-56bb54564c-t5hz7\" (UID: \"788ad5fd-ae6a-4953-a113-1357e62f2b0b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1e8cf0-56bb54564c-t5hz7"
Oct 31 19:43:01.725138 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:01.725104    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/788ad5fd-ae6a-4953-a113-1357e62f2b0b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1e8cf0-56bb54564c-t5hz7\" (UID: \"788ad5fd-ae6a-4953-a113-1357e62f2b0b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1e8cf0-56bb54564c-t5hz7"
Oct 31 19:43:01.743092 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:01.733136    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vxd9\" (UniqueName: \"kubernetes.io/projected/788ad5fd-ae6a-4953-a113-1357e62f2b0b-kube-api-access-2vxd9\") pod \"payload-processing-e2e-inf-a-1e8cf0-56bb54564c-t5hz7\" (UID: \"788ad5fd-ae6a-4953-a113-1357e62f2b0b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1e8cf0-56bb54564c-t5hz7"
Oct 31 19:43:01.907386 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:01.890486    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vxd9\" (UniqueName: \"kubernetes.io/projected/788ad5fd-ae6a-4953-a113-1357e62f2b0b-kube-api-access-2vxd9\") pod \"payload-processing-e2e-inf-a-1e8cf0-56bb54564c-t5hz7\" (UID: \"788ad5fd-ae6a-4953-a113-1357e62f2b0b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1e8cf0-56bb54564c-t5hz7"
Oct 31 19:43:02.072402 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:02.068385    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1e8cf0-56bb54564c-t5hz7"
Oct 31 19:43:02.125367 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:02.125310    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n"]
Oct 31 19:43:02.131265 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:02.131239    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n"
Oct 31 19:43:02.140575 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:02.140547    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-278c04-openshift-default-dockercfg-rsckc\""
Oct 31 19:43:02.146982 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:02.146080    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n"]
Oct 31 19:43:02.282353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:02.262446    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72702097-1cf4-4d92-85b0-3b0b43aa3c51-istio-data\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n\" (UID: \"72702097-1cf4-4d92-85b0-3b0b43aa3c51\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n"
Oct 31 19:43:02.282353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:02.262557    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72702097-1cf4-4d92-85b0-3b0b43aa3c51-workload-certs\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n\" (UID: \"72702097-1cf4-4d92-85b0-3b0b43aa3c51\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n"
Oct 31 19:43:02.282353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:02.262590    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfrwk\" (UniqueName: \"kubernetes.io/projected/72702097-1cf4-4d92-85b0-3b0b43aa3c51-kube-api-access-rfrwk\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n\" (UID: \"72702097-1cf4-4d92-85b0-3b0b43aa3c51\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n"
Oct 31 19:43:02.282353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:02.262765    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72702097-1cf4-4d92-85b0-3b0b43aa3c51-istio-token\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n\" (UID: \"72702097-1cf4-4d92-85b0-3b0b43aa3c51\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n"
Oct 31 19:43:02.282353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:02.262794    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72702097-1cf4-4d92-85b0-3b0b43aa3c51-istio-envoy\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n\" (UID: \"72702097-1cf4-4d92-85b0-3b0b43aa3c51\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n"
Oct 31 19:43:02.282353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:02.262840    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72702097-1cf4-4d92-85b0-3b0b43aa3c51-istio-podinfo\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n\" (UID: \"72702097-1cf4-4d92-85b0-3b0b43aa3c51\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n"
Oct 31 19:43:02.282353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:02.262896    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72702097-1cf4-4d92-85b0-3b0b43aa3c51-workload-socket\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n\" (UID: \"72702097-1cf4-4d92-85b0-3b0b43aa3c51\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n"
Oct 31 19:43:02.282353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:02.262966    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72702097-1cf4-4d92-85b0-3b0b43aa3c51-istiod-ca-cert\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n\" (UID: \"72702097-1cf4-4d92-85b0-3b0b43aa3c51\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n"
Oct 31 19:43:02.282353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:02.263033    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72702097-1cf4-4d92-85b0-3b0b43aa3c51-credential-socket\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n\" (UID: \"72702097-1cf4-4d92-85b0-3b0b43aa3c51\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n"
Oct 31 19:43:02.374407 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:02.368735    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72702097-1cf4-4d92-85b0-3b0b43aa3c51-istio-token\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n\" (UID: \"72702097-1cf4-4d92-85b0-3b0b43aa3c51\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n"
Oct 31 19:43:02.374407 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:02.369143    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72702097-1cf4-4d92-85b0-3b0b43aa3c51-istio-envoy\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n\" (UID: \"72702097-1cf4-4d92-85b0-3b0b43aa3c51\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n"
Oct 31 19:43:02.409886 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:02.406531    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72702097-1cf4-4d92-85b0-3b0b43aa3c51-istio-podinfo\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n\" (UID: \"72702097-1cf4-4d92-85b0-3b0b43aa3c51\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n"
Oct 31 19:43:02.409886 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:02.406623    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72702097-1cf4-4d92-85b0-3b0b43aa3c51-workload-socket\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n\" (UID: \"72702097-1cf4-4d92-85b0-3b0b43aa3c51\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n"
Oct 31 19:43:02.409886 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:02.406712    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72702097-1cf4-4d92-85b0-3b0b43aa3c51-istiod-ca-cert\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n\" (UID: \"72702097-1cf4-4d92-85b0-3b0b43aa3c51\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n"
Oct 31 19:43:02.409886 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:02.406787    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72702097-1cf4-4d92-85b0-3b0b43aa3c51-credential-socket\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n\" (UID: \"72702097-1cf4-4d92-85b0-3b0b43aa3c51\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n"
Oct 31 19:43:02.409886 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:02.406917    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72702097-1cf4-4d92-85b0-3b0b43aa3c51-istio-data\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n\" (UID: \"72702097-1cf4-4d92-85b0-3b0b43aa3c51\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n"
Oct 31 19:43:02.409886 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:02.407034    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72702097-1cf4-4d92-85b0-3b0b43aa3c51-workload-certs\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n\" (UID: \"72702097-1cf4-4d92-85b0-3b0b43aa3c51\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n"
Oct 31 19:43:02.409886 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:02.407076    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfrwk\" (UniqueName: \"kubernetes.io/projected/72702097-1cf4-4d92-85b0-3b0b43aa3c51-kube-api-access-rfrwk\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n\" (UID: \"72702097-1cf4-4d92-85b0-3b0b43aa3c51\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n"
Oct 31 19:43:02.409886 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:02.408485    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72702097-1cf4-4d92-85b0-3b0b43aa3c51-workload-socket\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n\" (UID: \"72702097-1cf4-4d92-85b0-3b0b43aa3c51\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n"
Oct 31 19:43:02.409886 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:02.409135    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72702097-1cf4-4d92-85b0-3b0b43aa3c51-istiod-ca-cert\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n\" (UID: \"72702097-1cf4-4d92-85b0-3b0b43aa3c51\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n"
Oct 31 19:43:02.409886 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:02.409420    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72702097-1cf4-4d92-85b0-3b0b43aa3c51-istio-data\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n\" (UID: \"72702097-1cf4-4d92-85b0-3b0b43aa3c51\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n"
Oct 31 19:43:02.409886 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:02.409642    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72702097-1cf4-4d92-85b0-3b0b43aa3c51-workload-certs\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n\" (UID: \"72702097-1cf4-4d92-85b0-3b0b43aa3c51\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n"
Oct 31 19:43:02.426825 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:02.425282    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:02.426825 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:02.426752    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72702097-1cf4-4d92-85b0-3b0b43aa3c51-istio-podinfo\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n\" (UID: \"72702097-1cf4-4d92-85b0-3b0b43aa3c51\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n"
Oct 31 19:43:02.428457 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:02.427164    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72702097-1cf4-4d92-85b0-3b0b43aa3c51-credential-socket\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n\" (UID: \"72702097-1cf4-4d92-85b0-3b0b43aa3c51\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n"
Oct 31 19:43:02.432117 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:02.431773    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72702097-1cf4-4d92-85b0-3b0b43aa3c51-istio-token\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n\" (UID: \"72702097-1cf4-4d92-85b0-3b0b43aa3c51\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n"
Oct 31 19:43:02.442631 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:02.433057    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfrwk\" (UniqueName: \"kubernetes.io/projected/72702097-1cf4-4d92-85b0-3b0b43aa3c51-kube-api-access-rfrwk\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n\" (UID: \"72702097-1cf4-4d92-85b0-3b0b43aa3c51\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n"
Oct 31 19:43:02.442631 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:02.434368    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72702097-1cf4-4d92-85b0-3b0b43aa3c51-istio-envoy\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n\" (UID: \"72702097-1cf4-4d92-85b0-3b0b43aa3c51\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n"
Oct 31 19:43:02.442631 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:43:02.436544    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod788ad5fd_ae6a_4953_a113_1357e62f2b0b.slice/crio-86e9445306c8e88a8bbb5bfd1c80d849e12382bc3eb62a0131dd84d1a2d887cd WatchSource:0}: Error finding container 86e9445306c8e88a8bbb5bfd1c80d849e12382bc3eb62a0131dd84d1a2d887cd: Status 404 returned error can't find the container with id 86e9445306c8e88a8bbb5bfd1c80d849e12382bc3eb62a0131dd84d1a2d887cd
Oct 31 19:43:02.450016 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:02.449986    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1e8cf0-56bb54564c-t5hz7"]
Oct 31 19:43:02.487583 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:02.485987    4645 patch_prober.go:29] interesting pod/e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body=
Oct 31 19:43:02.487583 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:02.486071    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9" podUID="7e5cadc6-cd7f-402d-9550-874d7b3d6376" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused"
Oct 31 19:43:02.494880 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:02.494836    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n"
Oct 31 19:43:02.529712 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:02.529679    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:02.823936 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:43:02.823903    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72702097_1cf4_4d92_85b0_3b0b43aa3c51.slice/crio-07371aafea10e7d397fca60fff678d5bd031494300b7dcb70a9a4fa042125347 WatchSource:0}: Error finding container 07371aafea10e7d397fca60fff678d5bd031494300b7dcb70a9a4fa042125347: Status 404 returned error can't find the container with id 07371aafea10e7d397fca60fff678d5bd031494300b7dcb70a9a4fa042125347
Oct 31 19:43:02.824981 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:02.824961    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n"]
Oct 31 19:43:02.826963 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:02.826928    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:43:02.827041 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:02.826994    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:43:02.827041 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:02.827026    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:43:02.901310 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:02.901271    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1e8cf0-56bb54564c-t5hz7" event={"ID":"788ad5fd-ae6a-4953-a113-1357e62f2b0b","Type":"ContainerStarted","Data":"930b509ed77244d1252eeab03e45e11aefdee680e2b7680444ab0d49871fd277"}
Oct 31 19:43:02.901462 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:02.901382    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1e8cf0-56bb54564c-t5hz7" event={"ID":"788ad5fd-ae6a-4953-a113-1357e62f2b0b","Type":"ContainerStarted","Data":"86e9445306c8e88a8bbb5bfd1c80d849e12382bc3eb62a0131dd84d1a2d887cd"}
Oct 31 19:43:02.902635 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:02.902614    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-1e8cf0-56bb54564c-t5hz7"
Oct 31 19:43:02.904397 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:02.904370    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n" event={"ID":"72702097-1cf4-4d92-85b0-3b0b43aa3c51","Type":"ContainerStarted","Data":"07371aafea10e7d397fca60fff678d5bd031494300b7dcb70a9a4fa042125347"}
Oct 31 19:43:02.926259 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:02.926212    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-1e8cf0-56bb54564c-t5hz7" podStartSLOduration=1.926196732 podStartE2EDuration="1.926196732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:43:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:43:02.919468239 +0000 UTC m=+1413.492702369" watchObservedRunningTime="2026-10-31 19:43:02.926196732 +0000 UTC m=+1413.499430862"
Oct 31 19:43:03.404079 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:03.404047    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:03.483773 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:03.483740    4645 patch_prober.go:29] interesting pod/e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body=
Oct 31 19:43:03.484048 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:03.484020    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9" podUID="7e5cadc6-cd7f-402d-9550-874d7b3d6376" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused"
Oct 31 19:43:03.923177 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:03.923129    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n" event={"ID":"72702097-1cf4-4d92-85b0-3b0b43aa3c51","Type":"ContainerStarted","Data":"28cc763a8511fedc7186e850d65c1981dbea0ecb650dd4e98440a0670b05689c"}
Oct 31 19:43:03.949011 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:03.948956    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n" podStartSLOduration=1.9489380509999998 podStartE2EDuration="1.948938051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:43:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:43:03.944734641 +0000 UTC m=+1414.517968769" watchObservedRunningTime="2026-10-31 19:43:03.948938051 +0000 UTC m=+1414.522172181"
Oct 31 19:43:04.496361 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:04.495479    4645 patch_prober.go:29] interesting pod/e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body=
Oct 31 19:43:04.496361 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:04.495545    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9" podUID="7e5cadc6-cd7f-402d-9550-874d7b3d6376" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused"
Oct 31 19:43:04.500351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:04.500197    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n"
Oct 31 19:43:04.500525 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:04.500365    4645 patch_prober.go:29] interesting pod/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 31 19:43:04.500525 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:04.500410    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n" podUID="72702097-1cf4-4d92-85b0-3b0b43aa3c51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 31 19:43:04.607888 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:04.607855    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:05.484353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:05.483794    4645 patch_prober.go:29] interesting pod/e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body=
Oct 31 19:43:05.484353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:05.483865    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9" podUID="7e5cadc6-cd7f-402d-9550-874d7b3d6376" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused"
Oct 31 19:43:05.495286 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:05.495250    4645 patch_prober.go:29] interesting pod/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 31 19:43:05.495592 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:05.495560    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n" podUID="72702097-1cf4-4d92-85b0-3b0b43aa3c51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 31 19:43:05.832043 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:05.831966    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:06.251279 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:43:06.249285    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 19:43:06.501421 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:06.501039    4645 patch_prober.go:29] interesting pod/e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body=
Oct 31 19:43:06.501421 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:06.501117    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9" podUID="7e5cadc6-cd7f-402d-9550-874d7b3d6376" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused"
Oct 31 19:43:06.501989 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:06.501582    4645 patch_prober.go:29] interesting pod/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 31 19:43:06.501989 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:06.501616    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n" podUID="72702097-1cf4-4d92-85b0-3b0b43aa3c51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 31 19:43:06.949823 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:06.949789    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:07.064941 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:07.064904    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:07.445608 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:07.445575    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1c0c1d-65fc757fd7-c79ts"]
Oct 31 19:43:07.493871 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:07.493831    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9"
Oct 31 19:43:07.495756 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:07.495726    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9"
Oct 31 19:43:07.497219 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:07.497196    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9"
Oct 31 19:43:07.497414 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:07.497391    4645 patch_prober.go:29] interesting pod/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 31 19:43:07.497513 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:07.497460    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n" podUID="72702097-1cf4-4d92-85b0-3b0b43aa3c51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 31 19:43:07.545912 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:07.545840    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:07.681801 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:07.681766    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:08.122350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:08.119954    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:08.516355 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:08.515993    4645 patch_prober.go:29] interesting pod/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 31 19:43:08.516355 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:08.516053    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n" podUID="72702097-1cf4-4d92-85b0-3b0b43aa3c51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 31 19:43:09.179348 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:09.179295    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:09.496409 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:09.495524    4645 patch_prober.go:29] interesting pod/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Oct 31 19:43:09.496409 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:09.495594    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n" podUID="72702097-1cf4-4d92-85b0-3b0b43aa3c51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Oct 31 19:43:10.139778 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:10.139735    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:10.500915 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:10.500883    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n"
Oct 31 19:43:10.501301 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:10.501183    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n"
Oct 31 19:43:10.502075 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:10.502055    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n"
Oct 31 19:43:11.120256 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:11.120220    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2d319a-7dcc698947-479ft"]
Oct 31 19:43:11.142995 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:11.142946    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2d319a-d95b54579-hzwkl"]
Oct 31 19:43:11.149364 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:11.149316    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:11.283065 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:11.283028    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:12.281907 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:12.281865    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:12.446514 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:12.446473    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1c0c1d-65fc757fd7-c79ts" podUID="bc2c8b81-5e05-4a66-b7b0-23f993cf4f7c" containerName="payload-pre-processing" containerID="cri-o://2d00d04edbf5bd0d000fb9f6bee310837ae5ae214a802d4f79ccea80a42fc416" gracePeriod=25
Oct 31 19:43:12.885199 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:12.885130    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1c0c1d-65fc757fd7-c79ts"
Oct 31 19:43:12.917088 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:12.917059    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc2c8b81-5e05-4a66-b7b0-23f993cf4f7c-kube-api-access-lbsgh\" (UniqueName: \"kubernetes.io/projected/bc2c8b81-5e05-4a66-b7b0-23f993cf4f7c-kube-api-access-lbsgh\") pod \"bc2c8b81-5e05-4a66-b7b0-23f993cf4f7c\" (UID: \"bc2c8b81-5e05-4a66-b7b0-23f993cf4f7c\") "
Oct 31 19:43:12.917243 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:12.917233    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc2c8b81-5e05-4a66-b7b0-23f993cf4f7c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc2c8b81-5e05-4a66-b7b0-23f993cf4f7c-plugins-config-volume\") pod \"bc2c8b81-5e05-4a66-b7b0-23f993cf4f7c\" (UID: \"bc2c8b81-5e05-4a66-b7b0-23f993cf4f7c\") "
Oct 31 19:43:12.917634 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:12.917600    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc2c8b81-5e05-4a66-b7b0-23f993cf4f7c-plugins-config-volume" pod "bc2c8b81-5e05-4a66-b7b0-23f993cf4f7c" (UID: "bc2c8b81-5e05-4a66-b7b0-23f993cf4f7c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:43:12.918028 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:12.918006    4645 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc2c8b81-5e05-4a66-b7b0-23f993cf4f7c-plugins-config-volume\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:43:12.919626 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:12.919598    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc2c8b81-5e05-4a66-b7b0-23f993cf4f7c-kube-api-access-lbsgh" pod "bc2c8b81-5e05-4a66-b7b0-23f993cf4f7c" (UID: "bc2c8b81-5e05-4a66-b7b0-23f993cf4f7c"). InnerVolumeSpecName "kube-api-access-lbsgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:43:13.019355 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:13.019298    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbsgh\" (UniqueName: \"kubernetes.io/projected/bc2c8b81-5e05-4a66-b7b0-23f993cf4f7c-kube-api-access-lbsgh\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:43:13.047682 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:13.047644    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:13.195895 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:13.195861    4645 generic.go:356] "Generic (PLEG): container finished" podID="bc2c8b81-5e05-4a66-b7b0-23f993cf4f7c" containerID="2d00d04edbf5bd0d000fb9f6bee310837ae5ae214a802d4f79ccea80a42fc416" exitCode=0
Oct 31 19:43:13.196078 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:13.195917    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1c0c1d-65fc757fd7-c79ts"
Oct 31 19:43:13.196078 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:13.195937    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1c0c1d-65fc757fd7-c79ts" event={"ID":"bc2c8b81-5e05-4a66-b7b0-23f993cf4f7c","Type":"ContainerDied","Data":"2d00d04edbf5bd0d000fb9f6bee310837ae5ae214a802d4f79ccea80a42fc416"}
Oct 31 19:43:13.196078 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:13.196016    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-1c0c1d-65fc757fd7-c79ts" event={"ID":"bc2c8b81-5e05-4a66-b7b0-23f993cf4f7c","Type":"ContainerDied","Data":"600c4fd605affcc8b27d87af12c86a53e2afb2ea7f7be0ae59eb6edc17005a12"}
Oct 31 19:43:13.196078 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:13.196038    4645 scope.go:122] "RemoveContainer" containerID="2d00d04edbf5bd0d000fb9f6bee310837ae5ae214a802d4f79ccea80a42fc416"
Oct 31 19:43:13.207456 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:13.207439    4645 scope.go:122] "RemoveContainer" containerID="2d00d04edbf5bd0d000fb9f6bee310837ae5ae214a802d4f79ccea80a42fc416"
Oct 31 19:43:13.207704 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:43:13.207687    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d00d04edbf5bd0d000fb9f6bee310837ae5ae214a802d4f79ccea80a42fc416\": container with ID starting with 2d00d04edbf5bd0d000fb9f6bee310837ae5ae214a802d4f79ccea80a42fc416 not found: ID does not exist" containerID="2d00d04edbf5bd0d000fb9f6bee310837ae5ae214a802d4f79ccea80a42fc416"
Oct 31 19:43:13.207754 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:13.207713    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d00d04edbf5bd0d000fb9f6bee310837ae5ae214a802d4f79ccea80a42fc416"} err="failed to get container status \"2d00d04edbf5bd0d000fb9f6bee310837ae5ae214a802d4f79ccea80a42fc416\": rpc error: code = NotFound desc = could not find container \"2d00d04edbf5bd0d000fb9f6bee310837ae5ae214a802d4f79ccea80a42fc416\": container with ID starting with 2d00d04edbf5bd0d000fb9f6bee310837ae5ae214a802d4f79ccea80a42fc416 not found: ID does not exist"
Oct 31 19:43:13.222455 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:13.222431    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1c0c1d-65fc757fd7-c79ts"]
Oct 31 19:43:13.227363 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:13.227342    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-1c0c1d-65fc757fd7-c79ts"]
Oct 31 19:43:13.624131 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:13.624045    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc2c8b81-5e05-4a66-b7b0-23f993cf4f7c" path="/var/lib/kubelet/pods/bc2c8b81-5e05-4a66-b7b0-23f993cf4f7c/volumes"
Oct 31 19:43:14.087938 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:14.087908    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:14.933909 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:14.933878    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-1e8cf0-56bb54564c-t5hz7"
Oct 31 19:43:14.960967 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:14.960936    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:15.807301 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:15.807263    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:16.120792 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:16.120689    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-2d319a-7dcc698947-479ft" podUID="6043e494-bfb6-4b33-82cc-f93a504e36e6" containerName="payload-processing" containerID="cri-o://a753eeba7ef2b69bf03b42b934bddf440f844bb80fa3d42ed7833156460270bf" gracePeriod=25
Oct 31 19:43:16.144585 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:16.144378    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2d319a-d95b54579-hzwkl" podUID="c3957000-2a12-4ca9-9cd5-3464f5b6146e" containerName="payload-pre-processing" containerID="cri-o://209a6a7af0cbae1060246203a66b2194f53c2e9c722b68b440a6593b1e54652c" gracePeriod=25
Oct 31 19:43:16.477003 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:16.476973    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2d319a-7dcc698947-479ft"
Oct 31 19:43:16.565593 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:16.565433    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:16.582794 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:16.582768    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6043e494-bfb6-4b33-82cc-f93a504e36e6-kube-api-access-9768f\" (UniqueName: \"kubernetes.io/projected/6043e494-bfb6-4b33-82cc-f93a504e36e6-kube-api-access-9768f\") pod \"6043e494-bfb6-4b33-82cc-f93a504e36e6\" (UID: \"6043e494-bfb6-4b33-82cc-f93a504e36e6\") "
Oct 31 19:43:16.582933 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:16.582888    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6043e494-bfb6-4b33-82cc-f93a504e36e6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6043e494-bfb6-4b33-82cc-f93a504e36e6-plugins-config-volume\") pod \"6043e494-bfb6-4b33-82cc-f93a504e36e6\" (UID: \"6043e494-bfb6-4b33-82cc-f93a504e36e6\") "
Oct 31 19:43:16.583613 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:16.583581    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6043e494-bfb6-4b33-82cc-f93a504e36e6-plugins-config-volume" pod "6043e494-bfb6-4b33-82cc-f93a504e36e6" (UID: "6043e494-bfb6-4b33-82cc-f93a504e36e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:43:16.585041 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:16.585019    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6043e494-bfb6-4b33-82cc-f93a504e36e6-kube-api-access-9768f" pod "6043e494-bfb6-4b33-82cc-f93a504e36e6" (UID: "6043e494-bfb6-4b33-82cc-f93a504e36e6"). InnerVolumeSpecName "kube-api-access-9768f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:43:16.587983 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:16.587960    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2d319a-d95b54579-hzwkl"
Oct 31 19:43:16.669166 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:16.669129    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:16.685431 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:16.685400    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3957000-2a12-4ca9-9cd5-3464f5b6146e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3957000-2a12-4ca9-9cd5-3464f5b6146e-plugins-config-volume\") pod \"c3957000-2a12-4ca9-9cd5-3464f5b6146e\" (UID: \"c3957000-2a12-4ca9-9cd5-3464f5b6146e\") "
Oct 31 19:43:16.685624 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:16.685609    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3957000-2a12-4ca9-9cd5-3464f5b6146e-kube-api-access-mlc8d\" (UniqueName: \"kubernetes.io/projected/c3957000-2a12-4ca9-9cd5-3464f5b6146e-kube-api-access-mlc8d\") pod \"c3957000-2a12-4ca9-9cd5-3464f5b6146e\" (UID: \"c3957000-2a12-4ca9-9cd5-3464f5b6146e\") "
Oct 31 19:43:16.686348 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:16.686310    4645 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6043e494-bfb6-4b33-82cc-f93a504e36e6-plugins-config-volume\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:43:16.686472 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:16.686371    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9768f\" (UniqueName: \"kubernetes.io/projected/6043e494-bfb6-4b33-82cc-f93a504e36e6-kube-api-access-9768f\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:43:16.687133 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:16.687102    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3957000-2a12-4ca9-9cd5-3464f5b6146e-plugins-config-volume" pod "c3957000-2a12-4ca9-9cd5-3464f5b6146e" (UID: "c3957000-2a12-4ca9-9cd5-3464f5b6146e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:43:16.692990 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:16.692965    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3957000-2a12-4ca9-9cd5-3464f5b6146e-kube-api-access-mlc8d" pod "c3957000-2a12-4ca9-9cd5-3464f5b6146e" (UID: "c3957000-2a12-4ca9-9cd5-3464f5b6146e"). InnerVolumeSpecName "kube-api-access-mlc8d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:43:16.787976 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:16.787941    4645 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3957000-2a12-4ca9-9cd5-3464f5b6146e-plugins-config-volume\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:43:16.787976 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:16.787973    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlc8d\" (UniqueName: \"kubernetes.io/projected/c3957000-2a12-4ca9-9cd5-3464f5b6146e-kube-api-access-mlc8d\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:43:16.804621 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:16.804593    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:17.202766 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:17.201877    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:17.282910 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:17.282874    4645 generic.go:356] "Generic (PLEG): container finished" podID="6043e494-bfb6-4b33-82cc-f93a504e36e6" containerID="a753eeba7ef2b69bf03b42b934bddf440f844bb80fa3d42ed7833156460270bf" exitCode=0
Oct 31 19:43:17.283231 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:17.283201    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2d319a-7dcc698947-479ft" event={"ID":"6043e494-bfb6-4b33-82cc-f93a504e36e6","Type":"ContainerDied","Data":"a753eeba7ef2b69bf03b42b934bddf440f844bb80fa3d42ed7833156460270bf"}
Oct 31 19:43:17.283506 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:17.283481    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2d319a-7dcc698947-479ft" event={"ID":"6043e494-bfb6-4b33-82cc-f93a504e36e6","Type":"ContainerDied","Data":"7317aeb309c1fb0d90eeb3ae53acfba55dc8617df2320a60577bcd9fb6dad159"}
Oct 31 19:43:17.283648 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:17.283601    4645 scope.go:122] "RemoveContainer" containerID="a753eeba7ef2b69bf03b42b934bddf440f844bb80fa3d42ed7833156460270bf"
Oct 31 19:43:17.284365 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:17.283672    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2d319a-7dcc698947-479ft"
Oct 31 19:43:17.289846 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:17.288742    4645 generic.go:356] "Generic (PLEG): container finished" podID="c3957000-2a12-4ca9-9cd5-3464f5b6146e" containerID="209a6a7af0cbae1060246203a66b2194f53c2e9c722b68b440a6593b1e54652c" exitCode=0
Oct 31 19:43:17.289846 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:17.289013    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2d319a-d95b54579-hzwkl" event={"ID":"c3957000-2a12-4ca9-9cd5-3464f5b6146e","Type":"ContainerDied","Data":"209a6a7af0cbae1060246203a66b2194f53c2e9c722b68b440a6593b1e54652c"}
Oct 31 19:43:17.289846 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:17.289061    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2d319a-d95b54579-hzwkl" event={"ID":"c3957000-2a12-4ca9-9cd5-3464f5b6146e","Type":"ContainerDied","Data":"7f01ac7fec817dd16a02bf4f65a15ab2483688d0175611c80ac6abe7940a17c7"}
Oct 31 19:43:17.289846 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:17.289139    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2d319a-d95b54579-hzwkl"
Oct 31 19:43:17.322456 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:17.322431    4645 scope.go:122] "RemoveContainer" containerID="a753eeba7ef2b69bf03b42b934bddf440f844bb80fa3d42ed7833156460270bf"
Oct 31 19:43:17.322793 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:43:17.322769    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a753eeba7ef2b69bf03b42b934bddf440f844bb80fa3d42ed7833156460270bf\": container with ID starting with a753eeba7ef2b69bf03b42b934bddf440f844bb80fa3d42ed7833156460270bf not found: ID does not exist" containerID="a753eeba7ef2b69bf03b42b934bddf440f844bb80fa3d42ed7833156460270bf"
Oct 31 19:43:17.324930 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:17.322812    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a753eeba7ef2b69bf03b42b934bddf440f844bb80fa3d42ed7833156460270bf"} err="failed to get container status \"a753eeba7ef2b69bf03b42b934bddf440f844bb80fa3d42ed7833156460270bf\": rpc error: code = NotFound desc = could not find container \"a753eeba7ef2b69bf03b42b934bddf440f844bb80fa3d42ed7833156460270bf\": container with ID starting with a753eeba7ef2b69bf03b42b934bddf440f844bb80fa3d42ed7833156460270bf not found: ID does not exist"
Oct 31 19:43:17.324930 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:17.322831    4645 scope.go:122] "RemoveContainer" containerID="209a6a7af0cbae1060246203a66b2194f53c2e9c722b68b440a6593b1e54652c"
Oct 31 19:43:17.373525 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:17.373489    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2d319a-d95b54579-hzwkl"]
Oct 31 19:43:17.381352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:17.380053    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2d319a-d95b54579-hzwkl"]
Oct 31 19:43:17.387032 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:17.387008    4645 scope.go:122] "RemoveContainer" containerID="209a6a7af0cbae1060246203a66b2194f53c2e9c722b68b440a6593b1e54652c"
Oct 31 19:43:17.387144 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:17.387130    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2d319a-7dcc698947-479ft"]
Oct 31 19:43:17.387540 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:43:17.387513    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"209a6a7af0cbae1060246203a66b2194f53c2e9c722b68b440a6593b1e54652c\": container with ID starting with 209a6a7af0cbae1060246203a66b2194f53c2e9c722b68b440a6593b1e54652c not found: ID does not exist" containerID="209a6a7af0cbae1060246203a66b2194f53c2e9c722b68b440a6593b1e54652c"
Oct 31 19:43:17.387646 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:17.387545    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"209a6a7af0cbae1060246203a66b2194f53c2e9c722b68b440a6593b1e54652c"} err="failed to get container status \"209a6a7af0cbae1060246203a66b2194f53c2e9c722b68b440a6593b1e54652c\": rpc error: code = NotFound desc = could not find container \"209a6a7af0cbae1060246203a66b2194f53c2e9c722b68b440a6593b1e54652c\": container with ID starting with 209a6a7af0cbae1060246203a66b2194f53c2e9c722b68b440a6593b1e54652c not found: ID does not exist"
Oct 31 19:43:17.391226 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:17.391202    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2d319a-7dcc698947-479ft"]
Oct 31 19:43:17.620314 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:17.620273    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6043e494-bfb6-4b33-82cc-f93a504e36e6" path="/var/lib/kubelet/pods/6043e494-bfb6-4b33-82cc-f93a504e36e6/volumes"
Oct 31 19:43:17.620805 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:17.620782    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3957000-2a12-4ca9-9cd5-3464f5b6146e" path="/var/lib/kubelet/pods/c3957000-2a12-4ca9-9cd5-3464f5b6146e/volumes"
Oct 31 19:43:17.741875 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:17.741845    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:18.821368 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:18.821320    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:19.306484 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:19.306453    4645 generic.go:356] "Generic (PLEG): container finished" podID="e00a761e-b818-439d-b61a-a3ea1c580278" containerID="1d529541c816d13d49dae9bc8cddbdecd570b75a58bcb8147fb30e2c7899cd92" exitCode=0
Oct 31 19:43:19.306484 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:19.306487    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96" event={"ID":"e00a761e-b818-439d-b61a-a3ea1c580278","Type":"ContainerDied","Data":"1d529541c816d13d49dae9bc8cddbdecd570b75a58bcb8147fb30e2c7899cd92"}
Oct 31 19:43:19.306878 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:19.306857    4645 scope.go:122] "RemoveContainer" containerID="1d529541c816d13d49dae9bc8cddbdecd570b75a58bcb8147fb30e2c7899cd92"
Oct 31 19:43:19.308637 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:19.308601    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:43:19.308763 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:19.308691    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:43:19.308763 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:19.308744    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:43:19.848762 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:19.847664    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:20.316657 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:20.316624    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96" event={"ID":"e00a761e-b818-439d-b61a-a3ea1c580278","Type":"ContainerStarted","Data":"5727c8cbde9980e6e36de917aa8c2381d418b424c921c3953d7f12ef258ca8b1"}
Oct 31 19:43:20.401451 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:20.401420    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96"
Oct 31 19:43:20.401752 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:20.401722    4645 patch_prober.go:29] interesting pod/e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 31 19:43:20.401857 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:20.401786    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96" podUID="e00a761e-b818-439d-b61a-a3ea1c580278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 31 19:43:20.583163 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:20.583062    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:21.400948 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:21.400916    4645 patch_prober.go:29] interesting pod/e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 31 19:43:21.401462 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:21.400969    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96" podUID="e00a761e-b818-439d-b61a-a3ea1c580278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 31 19:43:21.443832 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:21.443805    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:22.179352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:22.176199    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:22.400839 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:22.400804    4645 patch_prober.go:29] interesting pod/e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 31 19:43:22.401014 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:22.400860    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96" podUID="e00a761e-b818-439d-b61a-a3ea1c580278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 31 19:43:22.978909 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:22.978874    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:23.279315 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:23.279283    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:23.401708 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:23.401673    4645 patch_prober.go:29] interesting pod/e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 31 19:43:23.402173 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:23.401734    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96" podUID="e00a761e-b818-439d-b61a-a3ea1c580278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 31 19:43:23.737861 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:23.737826    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:23.942707 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:23.942669    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:24.140887 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:24.140800    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2"]
Oct 31 19:43:24.141668 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:24.141644    4645 cpu_manager.go:414] "containerMap: removing container" podUID="bc2c8b81-5e05-4a66-b7b0-23f993cf4f7c" containerName="payload-pre-processing"
Oct 31 19:43:24.141668 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:24.141663    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc2c8b81-5e05-4a66-b7b0-23f993cf4f7c" containerName="payload-pre-processing"
Oct 31 19:43:24.141870 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:24.141678    4645 cpu_manager.go:414] "containerMap: removing container" podUID="6043e494-bfb6-4b33-82cc-f93a504e36e6" containerName="payload-processing"
Oct 31 19:43:24.141870 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:24.141684    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6043e494-bfb6-4b33-82cc-f93a504e36e6" containerName="payload-processing"
Oct 31 19:43:24.141870 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:24.141718    4645 cpu_manager.go:414] "containerMap: removing container" podUID="c3957000-2a12-4ca9-9cd5-3464f5b6146e" containerName="payload-pre-processing"
Oct 31 19:43:24.141870 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:24.141723    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3957000-2a12-4ca9-9cd5-3464f5b6146e" containerName="payload-pre-processing"
Oct 31 19:43:24.141870 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:24.141831    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="6043e494-bfb6-4b33-82cc-f93a504e36e6" containerName="payload-processing"
Oct 31 19:43:24.141870 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:24.141848    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3957000-2a12-4ca9-9cd5-3464f5b6146e" containerName="payload-pre-processing"
Oct 31 19:43:24.141870 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:24.141858    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc2c8b81-5e05-4a66-b7b0-23f993cf4f7c" containerName="payload-pre-processing"
Oct 31 19:43:24.145581 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:24.145559    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2"
Oct 31 19:43:24.170891 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:24.170865    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-f8c3f7-openshift-default-dockercfg-whvcg\""
Oct 31 19:43:24.196777 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:24.196740    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2"]
Oct 31 19:43:24.204794 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:24.204766    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:24.339354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:24.338631    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76b3b657-6b14-43ff-b7ce-1e9e3b615141-istio-envoy\") pod \"e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2\" (UID: \"76b3b657-6b14-43ff-b7ce-1e9e3b615141\") " pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2"
Oct 31 19:43:24.339354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:24.338700    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76b3b657-6b14-43ff-b7ce-1e9e3b615141-istio-token\") pod \"e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2\" (UID: \"76b3b657-6b14-43ff-b7ce-1e9e3b615141\") " pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2"
Oct 31 19:43:24.339354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:24.338744    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qxbj\" (UniqueName: \"kubernetes.io/projected/76b3b657-6b14-43ff-b7ce-1e9e3b615141-kube-api-access-8qxbj\") pod \"e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2\" (UID: \"76b3b657-6b14-43ff-b7ce-1e9e3b615141\") " pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2"
Oct 31 19:43:24.339354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:24.338862    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76b3b657-6b14-43ff-b7ce-1e9e3b615141-istiod-ca-cert\") pod \"e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2\" (UID: \"76b3b657-6b14-43ff-b7ce-1e9e3b615141\") " pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2"
Oct 31 19:43:24.339354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:24.338992    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76b3b657-6b14-43ff-b7ce-1e9e3b615141-istio-podinfo\") pod \"e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2\" (UID: \"76b3b657-6b14-43ff-b7ce-1e9e3b615141\") " pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2"
Oct 31 19:43:24.339354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:24.339029    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76b3b657-6b14-43ff-b7ce-1e9e3b615141-credential-socket\") pod \"e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2\" (UID: \"76b3b657-6b14-43ff-b7ce-1e9e3b615141\") " pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2"
Oct 31 19:43:24.339354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:24.339071    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76b3b657-6b14-43ff-b7ce-1e9e3b615141-workload-socket\") pod \"e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2\" (UID: \"76b3b657-6b14-43ff-b7ce-1e9e3b615141\") " pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2"
Oct 31 19:43:24.339354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:24.339286    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76b3b657-6b14-43ff-b7ce-1e9e3b615141-istio-data\") pod \"e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2\" (UID: \"76b3b657-6b14-43ff-b7ce-1e9e3b615141\") " pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2"
Oct 31 19:43:24.339959 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:24.339318    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76b3b657-6b14-43ff-b7ce-1e9e3b615141-workload-certs\") pod \"e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2\" (UID: \"76b3b657-6b14-43ff-b7ce-1e9e3b615141\") " pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2"
Oct 31 19:43:24.573020 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:24.572983    4645 patch_prober.go:29] interesting pod/e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 31 19:43:24.728390 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:24.727804    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96" podUID="e00a761e-b818-439d-b61a-a3ea1c580278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 31 19:43:24.728390 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:24.670513    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76b3b657-6b14-43ff-b7ce-1e9e3b615141-istio-data\") pod \"e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2\" (UID: \"76b3b657-6b14-43ff-b7ce-1e9e3b615141\") " pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2"
Oct 31 19:43:24.728390 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:24.727990    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76b3b657-6b14-43ff-b7ce-1e9e3b615141-workload-certs\") pod \"e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2\" (UID: \"76b3b657-6b14-43ff-b7ce-1e9e3b615141\") " pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2"
Oct 31 19:43:24.728390 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:24.728140    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76b3b657-6b14-43ff-b7ce-1e9e3b615141-istio-envoy\") pod \"e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2\" (UID: \"76b3b657-6b14-43ff-b7ce-1e9e3b615141\") " pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2"
Oct 31 19:43:24.728390 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:24.728200    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76b3b657-6b14-43ff-b7ce-1e9e3b615141-istio-token\") pod \"e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2\" (UID: \"76b3b657-6b14-43ff-b7ce-1e9e3b615141\") " pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2"
Oct 31 19:43:24.728390 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:24.728237    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qxbj\" (UniqueName: \"kubernetes.io/projected/76b3b657-6b14-43ff-b7ce-1e9e3b615141-kube-api-access-8qxbj\") pod \"e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2\" (UID: \"76b3b657-6b14-43ff-b7ce-1e9e3b615141\") " pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2"
Oct 31 19:43:24.728390 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:24.728397    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76b3b657-6b14-43ff-b7ce-1e9e3b615141-istiod-ca-cert\") pod \"e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2\" (UID: \"76b3b657-6b14-43ff-b7ce-1e9e3b615141\") " pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2"
Oct 31 19:43:24.728945 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:24.728541    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76b3b657-6b14-43ff-b7ce-1e9e3b615141-istio-podinfo\") pod \"e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2\" (UID: \"76b3b657-6b14-43ff-b7ce-1e9e3b615141\") " pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2"
Oct 31 19:43:24.728945 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:24.728582    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76b3b657-6b14-43ff-b7ce-1e9e3b615141-credential-socket\") pod \"e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2\" (UID: \"76b3b657-6b14-43ff-b7ce-1e9e3b615141\") " pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2"
Oct 31 19:43:24.728945 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:24.728622    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76b3b657-6b14-43ff-b7ce-1e9e3b615141-workload-socket\") pod \"e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2\" (UID: \"76b3b657-6b14-43ff-b7ce-1e9e3b615141\") " pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2"
Oct 31 19:43:24.730349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:24.729181    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76b3b657-6b14-43ff-b7ce-1e9e3b615141-workload-socket\") pod \"e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2\" (UID: \"76b3b657-6b14-43ff-b7ce-1e9e3b615141\") " pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2"
Oct 31 19:43:24.730349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:24.604255    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:24.730349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:24.670967    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76b3b657-6b14-43ff-b7ce-1e9e3b615141-istio-data\") pod \"e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2\" (UID: \"76b3b657-6b14-43ff-b7ce-1e9e3b615141\") " pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2"
Oct 31 19:43:24.730349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:24.729968    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76b3b657-6b14-43ff-b7ce-1e9e3b615141-workload-certs\") pod \"e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2\" (UID: \"76b3b657-6b14-43ff-b7ce-1e9e3b615141\") " pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2"
Oct 31 19:43:24.731480 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:24.731449    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76b3b657-6b14-43ff-b7ce-1e9e3b615141-credential-socket\") pod \"e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2\" (UID: \"76b3b657-6b14-43ff-b7ce-1e9e3b615141\") " pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2"
Oct 31 19:43:24.733318 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:24.732500    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76b3b657-6b14-43ff-b7ce-1e9e3b615141-istiod-ca-cert\") pod \"e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2\" (UID: \"76b3b657-6b14-43ff-b7ce-1e9e3b615141\") " pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2"
Oct 31 19:43:24.784195 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:24.780844    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76b3b657-6b14-43ff-b7ce-1e9e3b615141-istio-envoy\") pod \"e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2\" (UID: \"76b3b657-6b14-43ff-b7ce-1e9e3b615141\") " pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2"
Oct 31 19:43:24.787513 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:24.787473    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76b3b657-6b14-43ff-b7ce-1e9e3b615141-istio-podinfo\") pod \"e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2\" (UID: \"76b3b657-6b14-43ff-b7ce-1e9e3b615141\") " pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2"
Oct 31 19:43:24.843388 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:24.843300    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qxbj\" (UniqueName: \"kubernetes.io/projected/76b3b657-6b14-43ff-b7ce-1e9e3b615141-kube-api-access-8qxbj\") pod \"e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2\" (UID: \"76b3b657-6b14-43ff-b7ce-1e9e3b615141\") " pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2"
Oct 31 19:43:24.881732 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:24.881697    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76b3b657-6b14-43ff-b7ce-1e9e3b615141-istio-token\") pod \"e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2\" (UID: \"76b3b657-6b14-43ff-b7ce-1e9e3b615141\") " pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2"
Oct 31 19:43:25.098702 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:25.098604    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2"
Oct 31 19:43:25.136509 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:25.136463    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:25.249737 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:25.249709    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:25.353283 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:25.353202    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:25.404148 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:25.404112    4645 patch_prober.go:29] interesting pod/e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body=
Oct 31 19:43:25.404456 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:25.404425    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96" podUID="e00a761e-b818-439d-b61a-a3ea1c580278" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused"
Oct 31 19:43:25.440263 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:25.440220    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2"]
Oct 31 19:43:25.443120 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:25.443078    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:43:25.443220 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:25.443160    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:43:25.443220 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:25.443202    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:43:25.954353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:25.947962    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2" event={"ID":"76b3b657-6b14-43ff-b7ce-1e9e3b615141","Type":"ContainerStarted","Data":"de1325c6e76563fc630e2eec234bff76121d6e736f691e783aa15922c2597480"}
Oct 31 19:43:25.954353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:25.948053    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2" event={"ID":"76b3b657-6b14-43ff-b7ce-1e9e3b615141","Type":"ContainerStarted","Data":"7ea647d7c238754ee6c2ba9d3a8fc04031ab61763d5327cdaf84b3c90983c87e"}
Oct 31 19:43:25.977584 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:25.976852    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2" podStartSLOduration=1.976835748 podStartE2EDuration="1.976835748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:43:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:43:25.976560893 +0000 UTC m=+1436.549795023" watchObservedRunningTime="2026-10-31 19:43:25.976835748 +0000 UTC m=+1436.550069876"
Oct 31 19:43:26.099134 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:26.099086    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2"
Oct 31 19:43:26.100872 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:26.100843    4645 patch_prober.go:29] interesting pod/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 31 19:43:26.116991 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:26.116953    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2" podUID="76b3b657-6b14-43ff-b7ce-1e9e3b615141" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 31 19:43:26.394454 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:26.394301    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:26.400196 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:26.400166    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96"
Oct 31 19:43:26.405165 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:26.405143    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96"
Oct 31 19:43:26.549095 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:26.549059    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:26.956757 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:26.956732    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96"
Oct 31 19:43:27.104583 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:27.104556    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2"
Oct 31 19:43:27.452702 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:27.452670    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:27.962517 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:27.962485    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2"
Oct 31 19:43:27.963305 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:27.963287    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2"
Oct 31 19:43:28.259787 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:28.259698    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:28.396925 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:28.396890    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:28.508718 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:28.508689    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:28.917500 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:28.917473    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb"]
Oct 31 19:43:28.922109 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:28.922092    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb"
Oct 31 19:43:28.925720 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:28.925700    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-88eaf9-openshift-default-dockercfg-dk2qg\""
Oct 31 19:43:28.944216 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:28.944196    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb"]
Oct 31 19:43:29.044340 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:29.044307    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/892a87a2-b166-43fd-8e00-f5a7edd01857-credential-socket\") pod \"e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb\" (UID: \"892a87a2-b166-43fd-8e00-f5a7edd01857\") " pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb"
Oct 31 19:43:29.044751 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:29.044446    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs9qd\" (UniqueName: \"kubernetes.io/projected/892a87a2-b166-43fd-8e00-f5a7edd01857-kube-api-access-qs9qd\") pod \"e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb\" (UID: \"892a87a2-b166-43fd-8e00-f5a7edd01857\") " pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb"
Oct 31 19:43:29.044751 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:29.044510    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/892a87a2-b166-43fd-8e00-f5a7edd01857-workload-certs\") pod \"e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb\" (UID: \"892a87a2-b166-43fd-8e00-f5a7edd01857\") " pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb"
Oct 31 19:43:29.044751 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:29.044549    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/892a87a2-b166-43fd-8e00-f5a7edd01857-istiod-ca-cert\") pod \"e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb\" (UID: \"892a87a2-b166-43fd-8e00-f5a7edd01857\") " pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb"
Oct 31 19:43:29.044751 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:29.044614    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/892a87a2-b166-43fd-8e00-f5a7edd01857-istio-envoy\") pod \"e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb\" (UID: \"892a87a2-b166-43fd-8e00-f5a7edd01857\") " pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb"
Oct 31 19:43:29.044751 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:29.044653    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/892a87a2-b166-43fd-8e00-f5a7edd01857-istio-token\") pod \"e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb\" (UID: \"892a87a2-b166-43fd-8e00-f5a7edd01857\") " pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb"
Oct 31 19:43:29.044995 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:29.044788    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/892a87a2-b166-43fd-8e00-f5a7edd01857-istio-data\") pod \"e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb\" (UID: \"892a87a2-b166-43fd-8e00-f5a7edd01857\") " pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb"
Oct 31 19:43:29.044995 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:29.044849    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/892a87a2-b166-43fd-8e00-f5a7edd01857-workload-socket\") pod \"e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb\" (UID: \"892a87a2-b166-43fd-8e00-f5a7edd01857\") " pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb"
Oct 31 19:43:29.044995 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:29.044910    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/892a87a2-b166-43fd-8e00-f5a7edd01857-istio-podinfo\") pod \"e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb\" (UID: \"892a87a2-b166-43fd-8e00-f5a7edd01857\") " pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb"
Oct 31 19:43:29.146091 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:29.146063    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/892a87a2-b166-43fd-8e00-f5a7edd01857-istio-data\") pod \"e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb\" (UID: \"892a87a2-b166-43fd-8e00-f5a7edd01857\") " pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb"
Oct 31 19:43:29.146227 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:29.146120    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/892a87a2-b166-43fd-8e00-f5a7edd01857-workload-socket\") pod \"e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb\" (UID: \"892a87a2-b166-43fd-8e00-f5a7edd01857\") " pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb"
Oct 31 19:43:29.146227 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:29.146145    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/892a87a2-b166-43fd-8e00-f5a7edd01857-istio-podinfo\") pod \"e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb\" (UID: \"892a87a2-b166-43fd-8e00-f5a7edd01857\") " pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb"
Oct 31 19:43:29.146318 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:29.146285    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/892a87a2-b166-43fd-8e00-f5a7edd01857-credential-socket\") pod \"e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb\" (UID: \"892a87a2-b166-43fd-8e00-f5a7edd01857\") " pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb"
Oct 31 19:43:29.146425 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:29.146381    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs9qd\" (UniqueName: \"kubernetes.io/projected/892a87a2-b166-43fd-8e00-f5a7edd01857-kube-api-access-qs9qd\") pod \"e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb\" (UID: \"892a87a2-b166-43fd-8e00-f5a7edd01857\") " pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb"
Oct 31 19:43:29.146485 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:29.146433    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/892a87a2-b166-43fd-8e00-f5a7edd01857-workload-certs\") pod \"e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb\" (UID: \"892a87a2-b166-43fd-8e00-f5a7edd01857\") " pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb"
Oct 31 19:43:29.146485 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:29.146473    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/892a87a2-b166-43fd-8e00-f5a7edd01857-istiod-ca-cert\") pod \"e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb\" (UID: \"892a87a2-b166-43fd-8e00-f5a7edd01857\") " pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb"
Oct 31 19:43:29.146645 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:29.146509    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/892a87a2-b166-43fd-8e00-f5a7edd01857-istio-envoy\") pod \"e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb\" (UID: \"892a87a2-b166-43fd-8e00-f5a7edd01857\") " pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb"
Oct 31 19:43:29.146645 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:29.146582    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/892a87a2-b166-43fd-8e00-f5a7edd01857-istio-token\") pod \"e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb\" (UID: \"892a87a2-b166-43fd-8e00-f5a7edd01857\") " pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb"
Oct 31 19:43:29.146645 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:29.146585    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/892a87a2-b166-43fd-8e00-f5a7edd01857-istio-data\") pod \"e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb\" (UID: \"892a87a2-b166-43fd-8e00-f5a7edd01857\") " pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb"
Oct 31 19:43:29.146806 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:29.146706    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/892a87a2-b166-43fd-8e00-f5a7edd01857-workload-socket\") pod \"e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb\" (UID: \"892a87a2-b166-43fd-8e00-f5a7edd01857\") " pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb"
Oct 31 19:43:29.146857 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:29.146801    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/892a87a2-b166-43fd-8e00-f5a7edd01857-credential-socket\") pod \"e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb\" (UID: \"892a87a2-b166-43fd-8e00-f5a7edd01857\") " pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb"
Oct 31 19:43:29.147260 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:29.147222    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/892a87a2-b166-43fd-8e00-f5a7edd01857-workload-certs\") pod \"e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb\" (UID: \"892a87a2-b166-43fd-8e00-f5a7edd01857\") " pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb"
Oct 31 19:43:29.147442 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:29.147420    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/892a87a2-b166-43fd-8e00-f5a7edd01857-istiod-ca-cert\") pod \"e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb\" (UID: \"892a87a2-b166-43fd-8e00-f5a7edd01857\") " pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb"
Oct 31 19:43:29.149135 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:29.149111    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/892a87a2-b166-43fd-8e00-f5a7edd01857-istio-podinfo\") pod \"e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb\" (UID: \"892a87a2-b166-43fd-8e00-f5a7edd01857\") " pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb"
Oct 31 19:43:29.149224 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:29.149166    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/892a87a2-b166-43fd-8e00-f5a7edd01857-istio-envoy\") pod \"e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb\" (UID: \"892a87a2-b166-43fd-8e00-f5a7edd01857\") " pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb"
Oct 31 19:43:29.161936 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:29.161916    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/892a87a2-b166-43fd-8e00-f5a7edd01857-istio-token\") pod \"e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb\" (UID: \"892a87a2-b166-43fd-8e00-f5a7edd01857\") " pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb"
Oct 31 19:43:29.162034 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:29.161936    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs9qd\" (UniqueName: \"kubernetes.io/projected/892a87a2-b166-43fd-8e00-f5a7edd01857-kube-api-access-qs9qd\") pod \"e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb\" (UID: \"892a87a2-b166-43fd-8e00-f5a7edd01857\") " pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb"
Oct 31 19:43:29.238701 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:29.238651    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb"
Oct 31 19:43:29.260402 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:29.260374    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:29.568040 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:29.568005    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb"]
Oct 31 19:43:29.568183 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:43:29.568004    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod892a87a2_b166_43fd_8e00_f5a7edd01857.slice/crio-4a713b1438312e76b495f0932a3c8366c39c0644022d84946b319ee86cb915bd WatchSource:0}: Error finding container 4a713b1438312e76b495f0932a3c8366c39c0644022d84946b319ee86cb915bd: Status 404 returned error can't find the container with id 4a713b1438312e76b495f0932a3c8366c39c0644022d84946b319ee86cb915bd
Oct 31 19:43:29.572202 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:29.572152    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:43:29.572301 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:29.572255    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:43:29.572385 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:29.572298    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:43:29.647802 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:29.647653    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:29.760181 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:29.760139    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f8c3f7-bdc4d8f7-75mhm"]
Oct 31 19:43:29.767576 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:29.767544    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:29.768391 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:29.768366    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f8c3f7-bdc4d8f7-75mhm"
Oct 31 19:43:29.772931 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:29.772254    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-f8c3f7-dockercfg-s8btl\""
Oct 31 19:43:29.780396 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:29.778803    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f8c3f7-bdc4d8f7-75mhm"]
Oct 31 19:43:29.799263 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:29.799210    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-f8c3f7\""
Oct 31 19:43:29.919406 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:29.882837    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33f3e8b8-6af6-4f4d-9ab9-ac00d1bbb9ce-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f8c3f7-bdc4d8f7-75mhm\" (UID: \"33f3e8b8-6af6-4f4d-9ab9-ac00d1bbb9ce\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f8c3f7-bdc4d8f7-75mhm"
Oct 31 19:43:29.919667 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:29.919640    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht62f\" (UniqueName: \"kubernetes.io/projected/33f3e8b8-6af6-4f4d-9ab9-ac00d1bbb9ce-kube-api-access-ht62f\") pod \"payload-processing-e2e-inf-b-f8c3f7-bdc4d8f7-75mhm\" (UID: \"33f3e8b8-6af6-4f4d-9ab9-ac00d1bbb9ce\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f8c3f7-bdc4d8f7-75mhm"
Oct 31 19:43:30.022938 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:30.022890    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb" event={"ID":"892a87a2-b166-43fd-8e00-f5a7edd01857","Type":"ContainerStarted","Data":"a22258fa609d3dc229ac9d2df1c66c4fb141c12f2286cc386573ab78a162dc3a"}
Oct 31 19:43:30.023230 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:30.023187    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb" event={"ID":"892a87a2-b166-43fd-8e00-f5a7edd01857","Type":"ContainerStarted","Data":"4a713b1438312e76b495f0932a3c8366c39c0644022d84946b319ee86cb915bd"}
Oct 31 19:43:30.052349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:30.048649    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb" podStartSLOduration=2.048630481 podStartE2EDuration="2.048630481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:43:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:43:30.047691128 +0000 UTC m=+1440.620925259" watchObservedRunningTime="2026-10-31 19:43:30.048630481 +0000 UTC m=+1440.621864611"
Oct 31 19:43:30.056581 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:30.056537    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33f3e8b8-6af6-4f4d-9ab9-ac00d1bbb9ce-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f8c3f7-bdc4d8f7-75mhm\" (UID: \"33f3e8b8-6af6-4f4d-9ab9-ac00d1bbb9ce\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f8c3f7-bdc4d8f7-75mhm"
Oct 31 19:43:30.057471 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:30.057436    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht62f\" (UniqueName: \"kubernetes.io/projected/33f3e8b8-6af6-4f4d-9ab9-ac00d1bbb9ce-kube-api-access-ht62f\") pod \"payload-processing-e2e-inf-b-f8c3f7-bdc4d8f7-75mhm\" (UID: \"33f3e8b8-6af6-4f4d-9ab9-ac00d1bbb9ce\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f8c3f7-bdc4d8f7-75mhm"
Oct 31 19:43:30.057820 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:30.057801    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33f3e8b8-6af6-4f4d-9ab9-ac00d1bbb9ce-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f8c3f7-bdc4d8f7-75mhm\" (UID: \"33f3e8b8-6af6-4f4d-9ab9-ac00d1bbb9ce\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f8c3f7-bdc4d8f7-75mhm"
Oct 31 19:43:30.140391 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:30.125612    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht62f\" (UniqueName: \"kubernetes.io/projected/33f3e8b8-6af6-4f4d-9ab9-ac00d1bbb9ce-kube-api-access-ht62f\") pod \"payload-processing-e2e-inf-b-f8c3f7-bdc4d8f7-75mhm\" (UID: \"33f3e8b8-6af6-4f4d-9ab9-ac00d1bbb9ce\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f8c3f7-bdc4d8f7-75mhm"
Oct 31 19:43:30.145924 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:30.143722    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f8c3f7-bdc4d8f7-75mhm"
Oct 31 19:43:30.262260 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:30.245727    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb"
Oct 31 19:43:30.262260 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:30.252583    4645 patch_prober.go:29] interesting pod/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body=
Oct 31 19:43:30.262260 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:30.252631    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb" podUID="892a87a2-b166-43fd-8e00-f5a7edd01857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused"
Oct 31 19:43:30.444088 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:43:30.444057    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33f3e8b8_6af6_4f4d_9ab9_ac00d1bbb9ce.slice/crio-5c3b5e9ccdc4571e245f76b92826c2dde0a828bce10f09a3b4279fa2111d8284 WatchSource:0}: Error finding container 5c3b5e9ccdc4571e245f76b92826c2dde0a828bce10f09a3b4279fa2111d8284: Status 404 returned error can't find the container with id 5c3b5e9ccdc4571e245f76b92826c2dde0a828bce10f09a3b4279fa2111d8284
Oct 31 19:43:30.446800 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:30.446772    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f8c3f7-bdc4d8f7-75mhm"]
Oct 31 19:43:30.669152 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:30.668965    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:31.029507 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:31.029467    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f8c3f7-bdc4d8f7-75mhm" event={"ID":"33f3e8b8-6af6-4f4d-9ab9-ac00d1bbb9ce","Type":"ContainerStarted","Data":"cefe3c3880a7157c8a023a3ce173f0524066c6ea79126f747ebea37e1beb1823"}
Oct 31 19:43:31.029683 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:31.029531    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f8c3f7-bdc4d8f7-75mhm" event={"ID":"33f3e8b8-6af6-4f4d-9ab9-ac00d1bbb9ce","Type":"ContainerStarted","Data":"5c3b5e9ccdc4571e245f76b92826c2dde0a828bce10f09a3b4279fa2111d8284"}
Oct 31 19:43:31.029683 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:31.029558    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-f8c3f7-bdc4d8f7-75mhm"
Oct 31 19:43:31.050607 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:31.050560    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-f8c3f7-bdc4d8f7-75mhm" podStartSLOduration=2.050543976 podStartE2EDuration="2.050543976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:43:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:43:31.047032907 +0000 UTC m=+1441.620267037" watchObservedRunningTime="2026-10-31 19:43:31.050543976 +0000 UTC m=+1441.623778105"
Oct 31 19:43:31.240513 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:31.239939    4645 patch_prober.go:29] interesting pod/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body=
Oct 31 19:43:31.240513 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:31.240002    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb" podUID="892a87a2-b166-43fd-8e00-f5a7edd01857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused"
Oct 31 19:43:31.734644 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:31.734599    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:32.239465 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:32.239437    4645 patch_prober.go:29] interesting pod/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body=
Oct 31 19:43:32.239633 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:32.239485    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb" podUID="892a87a2-b166-43fd-8e00-f5a7edd01857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused"
Oct 31 19:43:32.450718 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:32.450684    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:33.269355 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:33.252615    4645 patch_prober.go:29] interesting pod/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body=
Oct 31 19:43:33.269355 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:33.252676    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb" podUID="892a87a2-b166-43fd-8e00-f5a7edd01857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused"
Oct 31 19:43:33.517433 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:33.517392    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:34.239598 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:34.239567    4645 patch_prober.go:29] interesting pod/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body=
Oct 31 19:43:34.239772 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:34.239616    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb" podUID="892a87a2-b166-43fd-8e00-f5a7edd01857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused"
Oct 31 19:43:34.476357 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:34.476153    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:35.239827 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:35.239800    4645 patch_prober.go:29] interesting pod/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body=
Oct 31 19:43:35.240227 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:35.239853    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb" podUID="892a87a2-b166-43fd-8e00-f5a7edd01857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused"
Oct 31 19:43:35.284474 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:35.264268    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:35.981017 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:35.980977    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:36.240092 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:36.240013    4645 patch_prober.go:29] interesting pod/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body=
Oct 31 19:43:36.243571 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:36.241024    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb" podUID="892a87a2-b166-43fd-8e00-f5a7edd01857" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused"
Oct 31 19:43:36.265828 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:43:36.265792    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 19:43:36.486853 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:36.486819    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:36.942761 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:36.942587    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:37.245935 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:37.245835    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb"
Oct 31 19:43:37.246428 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:37.246216    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb"
Oct 31 19:43:37.247714 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:37.247658    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb"
Oct 31 19:43:37.841338 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:37.841296    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:38.507817 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:38.507785    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-a4e312/test-model-a4e312-kserve-f4b656bdc-29rj9"]
Oct 31 19:43:38.514873 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:38.514848    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a4e312/test-model-a4e312-kserve-f4b656bdc-29rj9"
Oct 31 19:43:38.525227 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:38.525200    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a4e312\"/\"kube-root-ca.crt\""
Oct 31 19:43:38.525408 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:38.525303    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a4e312\"/\"default-dockercfg-hcws2\""
Oct 31 19:43:38.525408 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:38.525379    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-a4e312\"/\"openshift-service-ca.crt\""
Oct 31 19:43:38.525589 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:38.525512    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-a4e312\"/\"test-model-a4e312-kserve-self-signed-certs\""
Oct 31 19:43:38.529020 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:38.528997    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a4e312/test-model-a4e312-kserve-f4b656bdc-29rj9"]
Oct 31 19:43:38.568223 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:38.568199    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:38.591110 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:38.591077    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1-model-cache\") pod \"test-model-a4e312-kserve-f4b656bdc-29rj9\" (UID: \"7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1\") " pod="ai-tenant-e2e-inf-a-a4e312/test-model-a4e312-kserve-f4b656bdc-29rj9"
Oct 31 19:43:38.591282 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:38.591266    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1-tls-certs\") pod \"test-model-a4e312-kserve-f4b656bdc-29rj9\" (UID: \"7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1\") " pod="ai-tenant-e2e-inf-a-a4e312/test-model-a4e312-kserve-f4b656bdc-29rj9"
Oct 31 19:43:38.591371 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:38.591341    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v98bh\" (UniqueName: \"kubernetes.io/projected/7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1-kube-api-access-v98bh\") pod \"test-model-a4e312-kserve-f4b656bdc-29rj9\" (UID: \"7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1\") " pod="ai-tenant-e2e-inf-a-a4e312/test-model-a4e312-kserve-f4b656bdc-29rj9"
Oct 31 19:43:38.591488 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:38.591469    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1-home\") pod \"test-model-a4e312-kserve-f4b656bdc-29rj9\" (UID: \"7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1\") " pod="ai-tenant-e2e-inf-a-a4e312/test-model-a4e312-kserve-f4b656bdc-29rj9"
Oct 31 19:43:38.591585 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:38.591572    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1-dshm\") pod \"test-model-a4e312-kserve-f4b656bdc-29rj9\" (UID: \"7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1\") " pod="ai-tenant-e2e-inf-a-a4e312/test-model-a4e312-kserve-f4b656bdc-29rj9"
Oct 31 19:43:38.591628 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:38.591593    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1-tmp-dir\") pod \"test-model-a4e312-kserve-f4b656bdc-29rj9\" (UID: \"7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1\") " pod="ai-tenant-e2e-inf-a-a4e312/test-model-a4e312-kserve-f4b656bdc-29rj9"
Oct 31 19:43:38.687713 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:38.687677    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:38.716657 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:38.716623    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1-dshm\") pod \"test-model-a4e312-kserve-f4b656bdc-29rj9\" (UID: \"7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1\") " pod="ai-tenant-e2e-inf-a-a4e312/test-model-a4e312-kserve-f4b656bdc-29rj9"
Oct 31 19:43:38.717933 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:38.717908    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1-tmp-dir\") pod \"test-model-a4e312-kserve-f4b656bdc-29rj9\" (UID: \"7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1\") " pod="ai-tenant-e2e-inf-a-a4e312/test-model-a4e312-kserve-f4b656bdc-29rj9"
Oct 31 19:43:38.736198 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:38.736162    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1-model-cache\") pod \"test-model-a4e312-kserve-f4b656bdc-29rj9\" (UID: \"7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1\") " pod="ai-tenant-e2e-inf-a-a4e312/test-model-a4e312-kserve-f4b656bdc-29rj9"
Oct 31 19:43:38.737745 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:38.737712    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1-model-cache\") pod \"test-model-a4e312-kserve-f4b656bdc-29rj9\" (UID: \"7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1\") " pod="ai-tenant-e2e-inf-a-a4e312/test-model-a4e312-kserve-f4b656bdc-29rj9"
Oct 31 19:43:38.737963 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:38.718416    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1-tmp-dir\") pod \"test-model-a4e312-kserve-f4b656bdc-29rj9\" (UID: \"7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1\") " pod="ai-tenant-e2e-inf-a-a4e312/test-model-a4e312-kserve-f4b656bdc-29rj9"
Oct 31 19:43:38.738278 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:38.738258    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1-tls-certs\") pod \"test-model-a4e312-kserve-f4b656bdc-29rj9\" (UID: \"7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1\") " pod="ai-tenant-e2e-inf-a-a4e312/test-model-a4e312-kserve-f4b656bdc-29rj9"
Oct 31 19:43:38.738509 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:38.738491    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v98bh\" (UniqueName: \"kubernetes.io/projected/7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1-kube-api-access-v98bh\") pod \"test-model-a4e312-kserve-f4b656bdc-29rj9\" (UID: \"7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1\") " pod="ai-tenant-e2e-inf-a-a4e312/test-model-a4e312-kserve-f4b656bdc-29rj9"
Oct 31 19:43:38.738709 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:38.738693    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1-home\") pod \"test-model-a4e312-kserve-f4b656bdc-29rj9\" (UID: \"7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1\") " pod="ai-tenant-e2e-inf-a-a4e312/test-model-a4e312-kserve-f4b656bdc-29rj9"
Oct 31 19:43:38.752361 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:38.741813    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1-home\") pod \"test-model-a4e312-kserve-f4b656bdc-29rj9\" (UID: \"7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1\") " pod="ai-tenant-e2e-inf-a-a4e312/test-model-a4e312-kserve-f4b656bdc-29rj9"
Oct 31 19:43:38.758317 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:38.758252    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1-dshm\") pod \"test-model-a4e312-kserve-f4b656bdc-29rj9\" (UID: \"7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1\") " pod="ai-tenant-e2e-inf-a-a4e312/test-model-a4e312-kserve-f4b656bdc-29rj9"
Oct 31 19:43:38.780428 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:38.774288    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1-tls-certs\") pod \"test-model-a4e312-kserve-f4b656bdc-29rj9\" (UID: \"7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1\") " pod="ai-tenant-e2e-inf-a-a4e312/test-model-a4e312-kserve-f4b656bdc-29rj9"
Oct 31 19:43:38.791358 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:38.786208    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v98bh\" (UniqueName: \"kubernetes.io/projected/7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1-kube-api-access-v98bh\") pod \"test-model-a4e312-kserve-f4b656bdc-29rj9\" (UID: \"7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1\") " pod="ai-tenant-e2e-inf-a-a4e312/test-model-a4e312-kserve-f4b656bdc-29rj9"
Oct 31 19:43:38.846588 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:38.835504    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a4e312/test-model-a4e312-kserve-f4b656bdc-29rj9"
Oct 31 19:43:39.154902 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:39.154879    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-a4e312/test-model-a4e312-kserve-f4b656bdc-29rj9"]
Oct 31 19:43:39.155743 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:43:39.155709    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b61ab4a_9eb7_48d6_bc3a_7e3b4444a9b1.slice/crio-99efe634b2265a0741d868590d4b0e075b08ff94f2299ec43d51ed8dcaf5e23f WatchSource:0}: Error finding container 99efe634b2265a0741d868590d4b0e075b08ff94f2299ec43d51ed8dcaf5e23f: Status 404 returned error can't find the container with id 99efe634b2265a0741d868590d4b0e075b08ff94f2299ec43d51ed8dcaf5e23f
Oct 31 19:43:39.363569 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:39.363538    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:40.142738 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:40.142703    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:40.152850 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:40.152805    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a4e312/test-model-a4e312-kserve-f4b656bdc-29rj9" event={"ID":"7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1","Type":"ContainerStarted","Data":"cf2f7e92f053205e1c1e3a398ee0045de8bdee5bd3f08695cc94ed51ee666c19"}
Oct 31 19:43:40.153011 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:40.152866    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a4e312/test-model-a4e312-kserve-f4b656bdc-29rj9" event={"ID":"7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1","Type":"ContainerStarted","Data":"99efe634b2265a0741d868590d4b0e075b08ff94f2299ec43d51ed8dcaf5e23f"}
Oct 31 19:43:40.172390 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:40.172312    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-a4e312/test-model-a4e312-kserve-f4b656bdc-29rj9" podStartSLOduration=2.17229583 podStartE2EDuration="2.17229583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:43:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:43:40.170573006 +0000 UTC m=+1450.743807136" watchObservedRunningTime="2026-10-31 19:43:40.17229583 +0000 UTC m=+1450.745529957"
Oct 31 19:43:40.840785 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:40.838394    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-a4e312/test-model-a4e312-kserve-f4b656bdc-29rj9"
Oct 31 19:43:40.985820 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:40.985788    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1c0c1d-fcd798f5d-x5rgq"]
Oct 31 19:43:40.986038 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:40.986001    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c0c1d-fcd798f5d-x5rgq" podUID="accc06a5-6668-4b44-9578-d57b054280e0" containerName="maas-api" containerID="cri-o://c6ba231d8e233ddd10c699593b0fd5ca8c23ec182d89e0b064f7402066de39fc" gracePeriod=30
Oct 31 19:43:41.035681 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:41.035491    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:41.165993 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:41.165955    4645 generic.go:356] "Generic (PLEG): container finished" podID="accc06a5-6668-4b44-9578-d57b054280e0" containerID="c6ba231d8e233ddd10c699593b0fd5ca8c23ec182d89e0b064f7402066de39fc" exitCode=0
Oct 31 19:43:41.167021 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:41.166420    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c0c1d-fcd798f5d-x5rgq" event={"ID":"accc06a5-6668-4b44-9578-d57b054280e0","Type":"ContainerDied","Data":"c6ba231d8e233ddd10c699593b0fd5ca8c23ec182d89e0b064f7402066de39fc"}
Oct 31 19:43:41.340462 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:41.325176    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:41.681140 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:41.632850    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:41.724440 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:41.724414    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c0c1d-fcd798f5d-x5rgq"
Oct 31 19:43:41.766318 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:41.766286    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:41.822120 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:41.822086    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/accc06a5-6668-4b44-9578-d57b054280e0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/accc06a5-6668-4b44-9578-d57b054280e0-maas-api-tls\") pod \"accc06a5-6668-4b44-9578-d57b054280e0\" (UID: \"accc06a5-6668-4b44-9578-d57b054280e0\") "
Oct 31 19:43:41.822525 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:41.822500    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/accc06a5-6668-4b44-9578-d57b054280e0-kube-api-access-297bm\" (UniqueName: \"kubernetes.io/projected/accc06a5-6668-4b44-9578-d57b054280e0-kube-api-access-297bm\") pod \"accc06a5-6668-4b44-9578-d57b054280e0\" (UID: \"accc06a5-6668-4b44-9578-d57b054280e0\") "
Oct 31 19:43:41.822733 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:41.822718    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/accc06a5-6668-4b44-9578-d57b054280e0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/accc06a5-6668-4b44-9578-d57b054280e0-openshift-service-ca\") pod \"accc06a5-6668-4b44-9578-d57b054280e0\" (UID: \"accc06a5-6668-4b44-9578-d57b054280e0\") "
Oct 31 19:43:41.823259 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:41.823236    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/accc06a5-6668-4b44-9578-d57b054280e0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/accc06a5-6668-4b44-9578-d57b054280e0-metrics-certs\") pod \"accc06a5-6668-4b44-9578-d57b054280e0\" (UID: \"accc06a5-6668-4b44-9578-d57b054280e0\") "
Oct 31 19:43:41.839362 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:41.834258    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/accc06a5-6668-4b44-9578-d57b054280e0-openshift-service-ca" pod "accc06a5-6668-4b44-9578-d57b054280e0" (UID: "accc06a5-6668-4b44-9578-d57b054280e0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:43:41.839362 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:41.836968    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/accc06a5-6668-4b44-9578-d57b054280e0-maas-api-tls" pod "accc06a5-6668-4b44-9578-d57b054280e0" (UID: "accc06a5-6668-4b44-9578-d57b054280e0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:43:41.886376 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:41.886286    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/accc06a5-6668-4b44-9578-d57b054280e0-metrics-certs" pod "accc06a5-6668-4b44-9578-d57b054280e0" (UID: "accc06a5-6668-4b44-9578-d57b054280e0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:43:41.894524 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:41.894467    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/accc06a5-6668-4b44-9578-d57b054280e0-kube-api-access-297bm" pod "accc06a5-6668-4b44-9578-d57b054280e0" (UID: "accc06a5-6668-4b44-9578-d57b054280e0"). InnerVolumeSpecName "kube-api-access-297bm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:43:41.934794 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:41.934758    4645 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/accc06a5-6668-4b44-9578-d57b054280e0-metrics-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:43:41.935004 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:41.934990    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/accc06a5-6668-4b44-9578-d57b054280e0-maas-api-tls\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:43:41.935092 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:41.935082    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-297bm\" (UniqueName: \"kubernetes.io/projected/accc06a5-6668-4b44-9578-d57b054280e0-kube-api-access-297bm\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:43:41.935179 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:41.935170    4645 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/accc06a5-6668-4b44-9578-d57b054280e0-openshift-service-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:43:42.039938 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:42.039905    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-f8c3f7-bdc4d8f7-75mhm"
Oct 31 19:43:42.282003 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:42.274793    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:42.355367 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:42.332720    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c0c1d-fcd798f5d-x5rgq"
Oct 31 19:43:42.355954 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:42.355919    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1c0c1d-fcd798f5d-x5rgq" event={"ID":"accc06a5-6668-4b44-9578-d57b054280e0","Type":"ContainerDied","Data":"6684d50fa46481d7d2bb39be7afa89a43cf53a7ffbce88f5e28b9980a110d519"}
Oct 31 19:43:42.356182 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:42.356154    4645 scope.go:122] "RemoveContainer" containerID="c6ba231d8e233ddd10c699593b0fd5ca8c23ec182d89e0b064f7402066de39fc"
Oct 31 19:43:42.536866 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:42.536779    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1c0c1d-fcd798f5d-x5rgq"]
Oct 31 19:43:42.573768 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:42.571226    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1c0c1d-fcd798f5d-x5rgq"]
Oct 31 19:43:43.255397 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:43.252986    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:43.394150 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:43.394115    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:43.639817 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:43.633123    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="accc06a5-6668-4b44-9578-d57b054280e0" path="/var/lib/kubelet/pods/accc06a5-6668-4b44-9578-d57b054280e0/volumes"
Oct 31 19:43:44.287353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:44.286893    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:45.189355 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:45.182818    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:46.152371 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:46.149645    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:46.838945 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:46.838916    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:47.584962 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:47.584925    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:48.724778 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:48.724524    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:48.836481 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:48.836449    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-a4e312/test-model-a4e312-kserve-f4b656bdc-29rj9"
Oct 31 19:43:48.850031 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:48.850004    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-a4e312/test-model-a4e312-kserve-f4b656bdc-29rj9"
Oct 31 19:43:49.473149 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:49.473117    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-a4e312/test-model-a4e312-kserve-f4b656bdc-29rj9"
Oct 31 19:43:49.643598 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:49.643502    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:50.577348 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:50.575900    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:50.766076 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:50.760474    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:50.811202 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:50.811166    4645 patch_prober.go:29] interesting pod/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:43:50.811546 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:50.811500    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp" podUID="b73e8ddc-c45d-4537-a4c4-f22747469ca8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:43:51.010689 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:51.010651    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96"]
Oct 31 19:43:51.027732 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:51.021946    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96" podUID="e00a761e-b818-439d-b61a-a3ea1c580278" containerName="istio-proxy" containerID="cri-o://5727c8cbde9980e6e36de917aa8c2381d418b424c921c3953d7f12ef258ca8b1" gracePeriod=30
Oct 31 19:43:51.046181 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:51.043139    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:51.240216 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:51.240181    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:51.518091 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:51.518057    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:51.884910 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:51.884819    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:52.059623 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:52.059535    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:52.519788 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:52.519508    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:52.627704 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:52.627645    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:52.789484 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:52.789349    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:53.183313 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:53.183189    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:53.333083 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:53.333052    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:53.460634 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:53.460548    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:54.526134 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:54.526099    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:55.647696 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:55.647647    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:56.349071 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:56.349035    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:56.565231 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:56.565196    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:56.577777 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:56.577747    4645 generic.go:356] "Generic (PLEG): container finished" podID="e00a761e-b818-439d-b61a-a3ea1c580278" containerID="5727c8cbde9980e6e36de917aa8c2381d418b424c921c3953d7f12ef258ca8b1" exitCode=0
Oct 31 19:43:56.577941 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:56.577783    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96" event={"ID":"e00a761e-b818-439d-b61a-a3ea1c580278","Type":"ContainerDied","Data":"5727c8cbde9980e6e36de917aa8c2381d418b424c921c3953d7f12ef258ca8b1"}
Oct 31 19:43:56.577941 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:56.577858    4645 scope.go:122] "RemoveContainer" containerID="1d529541c816d13d49dae9bc8cddbdecd570b75a58bcb8147fb30e2c7899cd92"
Oct 31 19:43:56.579627 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:56.579606    4645 generic.go:356] "Generic (PLEG): container finished" podID="b73e8ddc-c45d-4537-a4c4-f22747469ca8" containerID="dad871fc77704439f1b4795a36263cdb6a1e934a5c0fb333d92e9145bb9b7db1" exitCode=0
Oct 31 19:43:56.579737 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:56.579680    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp" event={"ID":"b73e8ddc-c45d-4537-a4c4-f22747469ca8","Type":"ContainerDied","Data":"dad871fc77704439f1b4795a36263cdb6a1e934a5c0fb333d92e9145bb9b7db1"}
Oct 31 19:43:56.580202 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:56.580177    4645 scope.go:122] "RemoveContainer" containerID="dad871fc77704439f1b4795a36263cdb6a1e934a5c0fb333d92e9145bb9b7db1"
Oct 31 19:43:56.582696 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:56.582662    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:43:56.582778 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:56.582754    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:43:56.582822 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:56.582799    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:43:56.946898 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:56.946868    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96"
Oct 31 19:43:57.000620 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:57.000583    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:57.055355 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:57.054164    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e00a761e-b818-439d-b61a-a3ea1c580278-istio-token\" (UniqueName: \"kubernetes.io/projected/e00a761e-b818-439d-b61a-a3ea1c580278-istio-token\") pod \"e00a761e-b818-439d-b61a-a3ea1c580278\" (UID: \"e00a761e-b818-439d-b61a-a3ea1c580278\") "
Oct 31 19:43:57.055355 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:57.054420    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e00a761e-b818-439d-b61a-a3ea1c580278-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e00a761e-b818-439d-b61a-a3ea1c580278-istio-envoy\") pod \"e00a761e-b818-439d-b61a-a3ea1c580278\" (UID: \"e00a761e-b818-439d-b61a-a3ea1c580278\") "
Oct 31 19:43:57.055355 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:57.054470    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e00a761e-b818-439d-b61a-a3ea1c580278-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e00a761e-b818-439d-b61a-a3ea1c580278-credential-socket\") pod \"e00a761e-b818-439d-b61a-a3ea1c580278\" (UID: \"e00a761e-b818-439d-b61a-a3ea1c580278\") "
Oct 31 19:43:57.055355 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:57.054505    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e00a761e-b818-439d-b61a-a3ea1c580278-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e00a761e-b818-439d-b61a-a3ea1c580278-workload-socket\") pod \"e00a761e-b818-439d-b61a-a3ea1c580278\" (UID: \"e00a761e-b818-439d-b61a-a3ea1c580278\") "
Oct 31 19:43:57.084951 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:57.084894    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e00a761e-b818-439d-b61a-a3ea1c580278-workload-socket" pod "e00a761e-b818-439d-b61a-a3ea1c580278" (UID: "e00a761e-b818-439d-b61a-a3ea1c580278"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:43:57.089130 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:57.088815    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e00a761e-b818-439d-b61a-a3ea1c580278-istio-token" pod "e00a761e-b818-439d-b61a-a3ea1c580278" (UID: "e00a761e-b818-439d-b61a-a3ea1c580278"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:43:57.091686 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:57.091463    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e00a761e-b818-439d-b61a-a3ea1c580278-istio-envoy" pod "e00a761e-b818-439d-b61a-a3ea1c580278" (UID: "e00a761e-b818-439d-b61a-a3ea1c580278"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:43:57.093381 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:57.091911    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e00a761e-b818-439d-b61a-a3ea1c580278-credential-socket" pod "e00a761e-b818-439d-b61a-a3ea1c580278" (UID: "e00a761e-b818-439d-b61a-a3ea1c580278"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:43:57.093381 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:57.091986    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e00a761e-b818-439d-b61a-a3ea1c580278-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e00a761e-b818-439d-b61a-a3ea1c580278-istio-podinfo\") pod \"e00a761e-b818-439d-b61a-a3ea1c580278\" (UID: \"e00a761e-b818-439d-b61a-a3ea1c580278\") "
Oct 31 19:43:57.093381 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:57.092107    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e00a761e-b818-439d-b61a-a3ea1c580278-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e00a761e-b818-439d-b61a-a3ea1c580278-istiod-ca-cert\") pod \"e00a761e-b818-439d-b61a-a3ea1c580278\" (UID: \"e00a761e-b818-439d-b61a-a3ea1c580278\") "
Oct 31 19:43:57.093381 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:57.092160    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e00a761e-b818-439d-b61a-a3ea1c580278-kube-api-access-zsbrq\" (UniqueName: \"kubernetes.io/projected/e00a761e-b818-439d-b61a-a3ea1c580278-kube-api-access-zsbrq\") pod \"e00a761e-b818-439d-b61a-a3ea1c580278\" (UID: \"e00a761e-b818-439d-b61a-a3ea1c580278\") "
Oct 31 19:43:57.093381 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:57.092261    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e00a761e-b818-439d-b61a-a3ea1c580278-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e00a761e-b818-439d-b61a-a3ea1c580278-istio-data\") pod \"e00a761e-b818-439d-b61a-a3ea1c580278\" (UID: \"e00a761e-b818-439d-b61a-a3ea1c580278\") "
Oct 31 19:43:57.093381 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:57.092292    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e00a761e-b818-439d-b61a-a3ea1c580278-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e00a761e-b818-439d-b61a-a3ea1c580278-workload-certs\") pod \"e00a761e-b818-439d-b61a-a3ea1c580278\" (UID: \"e00a761e-b818-439d-b61a-a3ea1c580278\") "
Oct 31 19:43:57.094543 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:57.094171    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e00a761e-b818-439d-b61a-a3ea1c580278-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:43:57.094543 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:57.094201    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e00a761e-b818-439d-b61a-a3ea1c580278-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:43:57.094543 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:57.094215    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e00a761e-b818-439d-b61a-a3ea1c580278-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:43:57.094543 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:57.094229    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e00a761e-b818-439d-b61a-a3ea1c580278-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:43:57.096568 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:57.096246    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e00a761e-b818-439d-b61a-a3ea1c580278-workload-certs" pod "e00a761e-b818-439d-b61a-a3ea1c580278" (UID: "e00a761e-b818-439d-b61a-a3ea1c580278"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:43:57.098367 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:57.098320    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e00a761e-b818-439d-b61a-a3ea1c580278-istio-data" pod "e00a761e-b818-439d-b61a-a3ea1c580278" (UID: "e00a761e-b818-439d-b61a-a3ea1c580278"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:43:57.098951 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:57.098908    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e00a761e-b818-439d-b61a-a3ea1c580278-istiod-ca-cert" pod "e00a761e-b818-439d-b61a-a3ea1c580278" (UID: "e00a761e-b818-439d-b61a-a3ea1c580278"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:43:57.107433 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:57.107384    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e00a761e-b818-439d-b61a-a3ea1c580278-kube-api-access-zsbrq" pod "e00a761e-b818-439d-b61a-a3ea1c580278" (UID: "e00a761e-b818-439d-b61a-a3ea1c580278"). InnerVolumeSpecName "kube-api-access-zsbrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:43:57.109422 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:57.109383    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e00a761e-b818-439d-b61a-a3ea1c580278-istio-podinfo" pod "e00a761e-b818-439d-b61a-a3ea1c580278" (UID: "e00a761e-b818-439d-b61a-a3ea1c580278"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:43:57.196490 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:57.196451    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e00a761e-b818-439d-b61a-a3ea1c580278-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:43:57.196490 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:57.196488    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsbrq\" (UniqueName: \"kubernetes.io/projected/e00a761e-b818-439d-b61a-a3ea1c580278-kube-api-access-zsbrq\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:43:57.196763 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:57.196511    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e00a761e-b818-439d-b61a-a3ea1c580278-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:43:57.196763 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:57.196530    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e00a761e-b818-439d-b61a-a3ea1c580278-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:43:57.196763 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:57.196545    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e00a761e-b818-439d-b61a-a3ea1c580278-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:43:57.247989 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:57.247944    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:57.366828 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:57.366781    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:57.601000 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:57.600967    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96" event={"ID":"e00a761e-b818-439d-b61a-a3ea1c580278","Type":"ContainerDied","Data":"50800259c1dffc6adfe5f7f539e4b48f1189a5e8b7b07457addb071afaf8d90f"}
Oct 31 19:43:58.231435 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:57.601049    4645 scope.go:122] "RemoveContainer" containerID="5727c8cbde9980e6e36de917aa8c2381d418b424c921c3953d7f12ef258ca8b1"
Oct 31 19:43:58.231435 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:57.601202    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96"
Oct 31 19:43:58.231435 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:57.627435    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:58.231435 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:57.637067    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp" event={"ID":"b73e8ddc-c45d-4537-a4c4-f22747469ca8","Type":"ContainerStarted","Data":"c5b432470850c5b9ec1435a151d8d255b18704cc2e8d732ec21f426793c16b00"}
Oct 31 19:43:58.231435 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:57.698496    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8dcfa8-68fb9b44f5-6dmlv"]
Oct 31 19:43:58.231435 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:57.699652    4645 cpu_manager.go:414] "containerMap: removing container" podUID="accc06a5-6668-4b44-9578-d57b054280e0" containerName="maas-api"
Oct 31 19:43:58.231435 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:57.699673    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="accc06a5-6668-4b44-9578-d57b054280e0" containerName="maas-api"
Oct 31 19:43:58.231435 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:57.699713    4645 cpu_manager.go:414] "containerMap: removing container" podUID="e00a761e-b818-439d-b61a-a3ea1c580278" containerName="istio-proxy"
Oct 31 19:43:58.231435 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:57.699722    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e00a761e-b818-439d-b61a-a3ea1c580278" containerName="istio-proxy"
Oct 31 19:43:58.231435 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:57.699735    4645 cpu_manager.go:414] "containerMap: removing container" podUID="e00a761e-b818-439d-b61a-a3ea1c580278" containerName="istio-proxy"
Oct 31 19:43:58.231435 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:57.699744    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e00a761e-b818-439d-b61a-a3ea1c580278" containerName="istio-proxy"
Oct 31 19:43:58.231435 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:57.699952    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="e00a761e-b818-439d-b61a-a3ea1c580278" containerName="istio-proxy"
Oct 31 19:43:58.231435 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:57.699973    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="accc06a5-6668-4b44-9578-d57b054280e0" containerName="maas-api"
Oct 31 19:43:58.231435 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:57.712182    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96"]
Oct 31 19:43:58.231435 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:57.712272    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8dcfa8-68fb9b44f5-6dmlv"
Oct 31 19:43:58.231435 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:57.713216    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-1c0c1d-openshift-default-857c68594c-pzw96"]
Oct 31 19:43:58.231435 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:57.717795    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-8dcfa8-dockercfg-b7tb4\""
Oct 31 19:43:58.231435 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:57.717908    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-8dcfa8\""
Oct 31 19:43:58.231435 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:57.725937    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8dcfa8-68fb9b44f5-6dmlv"]
Oct 31 19:43:58.231435 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:57.732952    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8dcfa8-86d859f49f-j5f54"]
Oct 31 19:43:58.231435 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:57.734307    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="e00a761e-b818-439d-b61a-a3ea1c580278" containerName="istio-proxy"
Oct 31 19:43:58.231435 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:57.745952    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8dcfa8-86d859f49f-j5f54"
Oct 31 19:43:58.231435 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:57.793435    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8dcfa8-86d859f49f-j5f54"]
Oct 31 19:43:58.231435 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:57.794004    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:58.231435 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:57.814065    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ec69924-e842-432a-a9fa-05db6c0d0fbf-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-8dcfa8-68fb9b44f5-6dmlv\" (UID: \"0ec69924-e842-432a-a9fa-05db6c0d0fbf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8dcfa8-68fb9b44f5-6dmlv"
Oct 31 19:43:58.231435 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:57.814375    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfmjk\" (UniqueName: \"kubernetes.io/projected/0ec69924-e842-432a-a9fa-05db6c0d0fbf-kube-api-access-bfmjk\") pod \"payload-pre-processing-e2e-inf-b-8dcfa8-68fb9b44f5-6dmlv\" (UID: \"0ec69924-e842-432a-a9fa-05db6c0d0fbf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8dcfa8-68fb9b44f5-6dmlv"
Oct 31 19:43:58.231435 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:57.816035    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp"
Oct 31 19:43:58.231435 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:57.816149    4645 patch_prober.go:29] interesting pod/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Oct 31 19:43:58.231435 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:57.816200    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp" podUID="b73e8ddc-c45d-4537-a4c4-f22747469ca8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Oct 31 19:43:58.231435 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:57.926730    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lplc9\" (UniqueName: \"kubernetes.io/projected/d8315528-0430-4e79-9f94-82df4e73befa-kube-api-access-lplc9\") pod \"payload-processing-e2e-inf-b-8dcfa8-86d859f49f-j5f54\" (UID: \"d8315528-0430-4e79-9f94-82df4e73befa\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8dcfa8-86d859f49f-j5f54"
Oct 31 19:43:58.231435 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:57.926861    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfmjk\" (UniqueName: \"kubernetes.io/projected/0ec69924-e842-432a-a9fa-05db6c0d0fbf-kube-api-access-bfmjk\") pod \"payload-pre-processing-e2e-inf-b-8dcfa8-68fb9b44f5-6dmlv\" (UID: \"0ec69924-e842-432a-a9fa-05db6c0d0fbf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8dcfa8-68fb9b44f5-6dmlv"
Oct 31 19:43:58.231435 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:57.927099    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ec69924-e842-432a-a9fa-05db6c0d0fbf-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-8dcfa8-68fb9b44f5-6dmlv\" (UID: \"0ec69924-e842-432a-a9fa-05db6c0d0fbf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8dcfa8-68fb9b44f5-6dmlv"
Oct 31 19:43:58.231435 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:57.927131    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8315528-0430-4e79-9f94-82df4e73befa-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-8dcfa8-86d859f49f-j5f54\" (UID: \"d8315528-0430-4e79-9f94-82df4e73befa\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8dcfa8-86d859f49f-j5f54"
Oct 31 19:43:58.231435 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:57.960685    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ec69924-e842-432a-a9fa-05db6c0d0fbf-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-8dcfa8-68fb9b44f5-6dmlv\" (UID: \"0ec69924-e842-432a-a9fa-05db6c0d0fbf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8dcfa8-68fb9b44f5-6dmlv"
Oct 31 19:43:58.231435 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:58.230920    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8315528-0430-4e79-9f94-82df4e73befa-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-8dcfa8-86d859f49f-j5f54\" (UID: \"d8315528-0430-4e79-9f94-82df4e73befa\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8dcfa8-86d859f49f-j5f54"
Oct 31 19:43:58.231435 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:58.231074    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lplc9\" (UniqueName: \"kubernetes.io/projected/d8315528-0430-4e79-9f94-82df4e73befa-kube-api-access-lplc9\") pod \"payload-processing-e2e-inf-b-8dcfa8-86d859f49f-j5f54\" (UID: \"d8315528-0430-4e79-9f94-82df4e73befa\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8dcfa8-86d859f49f-j5f54"
Oct 31 19:43:58.233606 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:58.233261    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8315528-0430-4e79-9f94-82df4e73befa-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-8dcfa8-86d859f49f-j5f54\" (UID: \"d8315528-0430-4e79-9f94-82df4e73befa\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8dcfa8-86d859f49f-j5f54"
Oct 31 19:43:58.237107 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:58.237085    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfmjk\" (UniqueName: \"kubernetes.io/projected/0ec69924-e842-432a-a9fa-05db6c0d0fbf-kube-api-access-bfmjk\") pod \"payload-pre-processing-e2e-inf-b-8dcfa8-68fb9b44f5-6dmlv\" (UID: \"0ec69924-e842-432a-a9fa-05db6c0d0fbf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8dcfa8-68fb9b44f5-6dmlv"
Oct 31 19:43:58.242887 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:58.242835    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lplc9\" (UniqueName: \"kubernetes.io/projected/d8315528-0430-4e79-9f94-82df4e73befa-kube-api-access-lplc9\") pod \"payload-processing-e2e-inf-b-8dcfa8-86d859f49f-j5f54\" (UID: \"d8315528-0430-4e79-9f94-82df4e73befa\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8dcfa8-86d859f49f-j5f54"
Oct 31 19:43:58.338010 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:58.337979    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8dcfa8-68fb9b44f5-6dmlv"
Oct 31 19:43:58.402549 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:58.398742    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8dcfa8-86d859f49f-j5f54"
Oct 31 19:43:58.408099 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:58.404434    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-d45f6c/test-model-d45f6c-kserve-784b7b884c-7mgtp"]
Oct 31 19:43:58.411405 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:58.410287    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d45f6c/test-model-d45f6c-kserve-784b7b884c-7mgtp"
Oct 31 19:43:58.411858 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:58.411830    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-d45f6c/test-model-d45f6c-kserve-784b7b884c-7mgtp"]
Oct 31 19:43:58.419813 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:58.419718    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-d45f6c\"/\"default-dockercfg-9jn2q\""
Oct 31 19:43:58.419813 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:58.419784    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-d45f6c\"/\"kube-root-ca.crt\""
Oct 31 19:43:58.419813 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:58.419801    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-d45f6c\"/\"openshift-service-ca.crt\""
Oct 31 19:43:58.420232 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:58.420213    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-d45f6c\"/\"test-model-d45f6c-kserve-self-signed-certs\""
Oct 31 19:43:58.438215 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:58.437433    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16133210-4877-4f71-9d0d-fe90c29cca43-home\") pod \"test-model-d45f6c-kserve-784b7b884c-7mgtp\" (UID: \"16133210-4877-4f71-9d0d-fe90c29cca43\") " pod="ai-tenant-e2e-inf-b-d45f6c/test-model-d45f6c-kserve-784b7b884c-7mgtp"
Oct 31 19:43:58.438215 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:58.437488    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16133210-4877-4f71-9d0d-fe90c29cca43-model-cache\") pod \"test-model-d45f6c-kserve-784b7b884c-7mgtp\" (UID: \"16133210-4877-4f71-9d0d-fe90c29cca43\") " pod="ai-tenant-e2e-inf-b-d45f6c/test-model-d45f6c-kserve-784b7b884c-7mgtp"
Oct 31 19:43:58.438215 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:58.437619    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16133210-4877-4f71-9d0d-fe90c29cca43-dshm\") pod \"test-model-d45f6c-kserve-784b7b884c-7mgtp\" (UID: \"16133210-4877-4f71-9d0d-fe90c29cca43\") " pod="ai-tenant-e2e-inf-b-d45f6c/test-model-d45f6c-kserve-784b7b884c-7mgtp"
Oct 31 19:43:58.438215 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:58.437894    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16133210-4877-4f71-9d0d-fe90c29cca43-tls-certs\") pod \"test-model-d45f6c-kserve-784b7b884c-7mgtp\" (UID: \"16133210-4877-4f71-9d0d-fe90c29cca43\") " pod="ai-tenant-e2e-inf-b-d45f6c/test-model-d45f6c-kserve-784b7b884c-7mgtp"
Oct 31 19:43:58.438215 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:58.437928    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16133210-4877-4f71-9d0d-fe90c29cca43-tmp-dir\") pod \"test-model-d45f6c-kserve-784b7b884c-7mgtp\" (UID: \"16133210-4877-4f71-9d0d-fe90c29cca43\") " pod="ai-tenant-e2e-inf-b-d45f6c/test-model-d45f6c-kserve-784b7b884c-7mgtp"
Oct 31 19:43:58.438215 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:58.437953    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llwmq\" (UniqueName: \"kubernetes.io/projected/16133210-4877-4f71-9d0d-fe90c29cca43-kube-api-access-llwmq\") pod \"test-model-d45f6c-kserve-784b7b884c-7mgtp\" (UID: \"16133210-4877-4f71-9d0d-fe90c29cca43\") " pod="ai-tenant-e2e-inf-b-d45f6c/test-model-d45f6c-kserve-784b7b884c-7mgtp"
Oct 31 19:43:58.452459 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:58.451816    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:58.540316 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:58.540259    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16133210-4877-4f71-9d0d-fe90c29cca43-home\") pod \"test-model-d45f6c-kserve-784b7b884c-7mgtp\" (UID: \"16133210-4877-4f71-9d0d-fe90c29cca43\") " pod="ai-tenant-e2e-inf-b-d45f6c/test-model-d45f6c-kserve-784b7b884c-7mgtp"
Oct 31 19:43:58.540459 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:58.540311    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16133210-4877-4f71-9d0d-fe90c29cca43-model-cache\") pod \"test-model-d45f6c-kserve-784b7b884c-7mgtp\" (UID: \"16133210-4877-4f71-9d0d-fe90c29cca43\") " pod="ai-tenant-e2e-inf-b-d45f6c/test-model-d45f6c-kserve-784b7b884c-7mgtp"
Oct 31 19:43:58.540459 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:58.540430    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16133210-4877-4f71-9d0d-fe90c29cca43-dshm\") pod \"test-model-d45f6c-kserve-784b7b884c-7mgtp\" (UID: \"16133210-4877-4f71-9d0d-fe90c29cca43\") " pod="ai-tenant-e2e-inf-b-d45f6c/test-model-d45f6c-kserve-784b7b884c-7mgtp"
Oct 31 19:43:58.540611 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:58.540593    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16133210-4877-4f71-9d0d-fe90c29cca43-tls-certs\") pod \"test-model-d45f6c-kserve-784b7b884c-7mgtp\" (UID: \"16133210-4877-4f71-9d0d-fe90c29cca43\") " pod="ai-tenant-e2e-inf-b-d45f6c/test-model-d45f6c-kserve-784b7b884c-7mgtp"
Oct 31 19:43:58.540662 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:58.540628    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16133210-4877-4f71-9d0d-fe90c29cca43-tmp-dir\") pod \"test-model-d45f6c-kserve-784b7b884c-7mgtp\" (UID: \"16133210-4877-4f71-9d0d-fe90c29cca43\") " pod="ai-tenant-e2e-inf-b-d45f6c/test-model-d45f6c-kserve-784b7b884c-7mgtp"
Oct 31 19:43:58.540662 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:58.540652    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llwmq\" (UniqueName: \"kubernetes.io/projected/16133210-4877-4f71-9d0d-fe90c29cca43-kube-api-access-llwmq\") pod \"test-model-d45f6c-kserve-784b7b884c-7mgtp\" (UID: \"16133210-4877-4f71-9d0d-fe90c29cca43\") " pod="ai-tenant-e2e-inf-b-d45f6c/test-model-d45f6c-kserve-784b7b884c-7mgtp"
Oct 31 19:43:58.541501 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:58.541476    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16133210-4877-4f71-9d0d-fe90c29cca43-home\") pod \"test-model-d45f6c-kserve-784b7b884c-7mgtp\" (UID: \"16133210-4877-4f71-9d0d-fe90c29cca43\") " pod="ai-tenant-e2e-inf-b-d45f6c/test-model-d45f6c-kserve-784b7b884c-7mgtp"
Oct 31 19:43:58.541791 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:58.541759    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16133210-4877-4f71-9d0d-fe90c29cca43-model-cache\") pod \"test-model-d45f6c-kserve-784b7b884c-7mgtp\" (UID: \"16133210-4877-4f71-9d0d-fe90c29cca43\") " pod="ai-tenant-e2e-inf-b-d45f6c/test-model-d45f6c-kserve-784b7b884c-7mgtp"
Oct 31 19:43:58.542512 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:58.542464    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16133210-4877-4f71-9d0d-fe90c29cca43-tmp-dir\") pod \"test-model-d45f6c-kserve-784b7b884c-7mgtp\" (UID: \"16133210-4877-4f71-9d0d-fe90c29cca43\") " pod="ai-tenant-e2e-inf-b-d45f6c/test-model-d45f6c-kserve-784b7b884c-7mgtp"
Oct 31 19:43:58.544496 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:58.544473    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16133210-4877-4f71-9d0d-fe90c29cca43-dshm\") pod \"test-model-d45f6c-kserve-784b7b884c-7mgtp\" (UID: \"16133210-4877-4f71-9d0d-fe90c29cca43\") " pod="ai-tenant-e2e-inf-b-d45f6c/test-model-d45f6c-kserve-784b7b884c-7mgtp"
Oct 31 19:43:58.545350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:58.545309    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16133210-4877-4f71-9d0d-fe90c29cca43-tls-certs\") pod \"test-model-d45f6c-kserve-784b7b884c-7mgtp\" (UID: \"16133210-4877-4f71-9d0d-fe90c29cca43\") " pod="ai-tenant-e2e-inf-b-d45f6c/test-model-d45f6c-kserve-784b7b884c-7mgtp"
Oct 31 19:43:58.553946 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:58.553922    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llwmq\" (UniqueName: \"kubernetes.io/projected/16133210-4877-4f71-9d0d-fe90c29cca43-kube-api-access-llwmq\") pod \"test-model-d45f6c-kserve-784b7b884c-7mgtp\" (UID: \"16133210-4877-4f71-9d0d-fe90c29cca43\") " pod="ai-tenant-e2e-inf-b-d45f6c/test-model-d45f6c-kserve-784b7b884c-7mgtp"
Oct 31 19:43:58.647818 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:58.647780    4645 generic.go:356] "Generic (PLEG): container finished" podID="8eee5947-9731-4194-aa80-7fbd619e551f" containerID="db0770b5441fe07bb8782b88cf635a09301dc956518832c2b6371caf6bb70dd2" exitCode=0
Oct 31 19:43:58.647989 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:58.647844    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z" event={"ID":"8eee5947-9731-4194-aa80-7fbd619e551f","Type":"ContainerDied","Data":"db0770b5441fe07bb8782b88cf635a09301dc956518832c2b6371caf6bb70dd2"}
Oct 31 19:43:58.648304 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:58.648281    4645 scope.go:122] "RemoveContainer" containerID="db0770b5441fe07bb8782b88cf635a09301dc956518832c2b6371caf6bb70dd2"
Oct 31 19:43:58.652276 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:58.650550    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:43:58.652276 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:58.650636    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:43:58.652276 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:58.650682    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:43:58.741097 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:58.716990    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:58.822432 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:58.822381    4645 patch_prober.go:29] interesting pod/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Oct 31 19:43:58.822585 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:58.822480    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp" podUID="b73e8ddc-c45d-4537-a4c4-f22747469ca8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Oct 31 19:43:58.874051 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:58.862831    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d45f6c/test-model-d45f6c-kserve-784b7b884c-7mgtp"
Oct 31 19:43:58.912352 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:43:58.895540    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ec69924_e842_432a_a9fa_05db6c0d0fbf.slice/crio-79bc132b69b670f0cc12c147ae27cb8ab0025eba0d3467cc0aec5c39deaab2ea WatchSource:0}: Error finding container 79bc132b69b670f0cc12c147ae27cb8ab0025eba0d3467cc0aec5c39deaab2ea: Status 404 returned error can't find the container with id 79bc132b69b670f0cc12c147ae27cb8ab0025eba0d3467cc0aec5c39deaab2ea
Oct 31 19:43:59.001832 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:58.970479    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8dcfa8-68fb9b44f5-6dmlv"]
Oct 31 19:43:59.012911 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:59.012878    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8dcfa8-86d859f49f-j5f54"]
Oct 31 19:43:59.015942 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:43:59.015917    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8315528_0430_4e79_9f94_82df4e73befa.slice/crio-1694cafc1245e3393ed2f21ff5a86d4c421227794817cd0e0473f42df2b0b06f WatchSource:0}: Error finding container 1694cafc1245e3393ed2f21ff5a86d4c421227794817cd0e0473f42df2b0b06f: Status 404 returned error can't find the container with id 1694cafc1245e3393ed2f21ff5a86d4c421227794817cd0e0473f42df2b0b06f
Oct 31 19:43:59.333418 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:59.333384    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-d45f6c/test-model-d45f6c-kserve-784b7b884c-7mgtp"]
Oct 31 19:43:59.333895 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:43:59.333874    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod16133210_4877_4f71_9d0d_fe90c29cca43.slice/crio-4555c0a45ece39f69018c79ce8cde0b77ace386759f150150437933fb0d4d9f0 WatchSource:0}: Error finding container 4555c0a45ece39f69018c79ce8cde0b77ace386759f150150437933fb0d4d9f0: Status 404 returned error can't find the container with id 4555c0a45ece39f69018c79ce8cde0b77ace386759f150150437933fb0d4d9f0
Oct 31 19:43:59.591149 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:59.590901    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:59.622747 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:59.622531    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e00a761e-b818-439d-b61a-a3ea1c580278" path="/var/lib/kubelet/pods/e00a761e-b818-439d-b61a-a3ea1c580278/volumes"
Oct 31 19:43:59.633536 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:59.633513    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-785d26-589cf5bf5f-fmrnc"]
Oct 31 19:43:59.638246 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:59.638231    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-785d26-589cf5bf5f-fmrnc"
Oct 31 19:43:59.647687 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:59.647668    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-785d26\""
Oct 31 19:43:59.650845 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:59.650826    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-785d26-dockercfg-jzb7m\""
Oct 31 19:43:59.654275 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:59.654246    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-785d26-589cf5bf5f-fmrnc"]
Oct 31 19:43:59.700748 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:59.700232    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d45f6c/test-model-d45f6c-kserve-784b7b884c-7mgtp" event={"ID":"16133210-4877-4f71-9d0d-fe90c29cca43","Type":"ContainerStarted","Data":"927fee25bffc85ab95557c76ecacea961b89d56f08061c909db500df96cbf623"}
Oct 31 19:43:59.700748 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:59.700283    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d45f6c/test-model-d45f6c-kserve-784b7b884c-7mgtp" event={"ID":"16133210-4877-4f71-9d0d-fe90c29cca43","Type":"ContainerStarted","Data":"4555c0a45ece39f69018c79ce8cde0b77ace386759f150150437933fb0d4d9f0"}
Oct 31 19:43:59.701296 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:59.701268    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-785d26-6467945f49-6xsr7"]
Oct 31 19:43:59.704124 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:59.704100    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbfl5\" (UniqueName: \"kubernetes.io/projected/cdd7678b-849b-4a59-8cf5-227831aa2037-kube-api-access-nbfl5\") pod \"payload-pre-processing-e2e-inf-a-785d26-589cf5bf5f-fmrnc\" (UID: \"cdd7678b-849b-4a59-8cf5-227831aa2037\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-785d26-589cf5bf5f-fmrnc"
Oct 31 19:43:59.704476 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:59.704452    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdd7678b-849b-4a59-8cf5-227831aa2037-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-785d26-589cf5bf5f-fmrnc\" (UID: \"cdd7678b-849b-4a59-8cf5-227831aa2037\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-785d26-589cf5bf5f-fmrnc"
Oct 31 19:43:59.721607 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:59.707888    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8dcfa8-68fb9b44f5-6dmlv" event={"ID":"0ec69924-e842-432a-a9fa-05db6c0d0fbf","Type":"ContainerStarted","Data":"cda7edf66edde4444772acf54632eeefb09f20a62d103ac4ab89940c0abfa530"}
Oct 31 19:43:59.721607 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:59.707943    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8dcfa8-68fb9b44f5-6dmlv" event={"ID":"0ec69924-e842-432a-a9fa-05db6c0d0fbf","Type":"ContainerStarted","Data":"79bc132b69b670f0cc12c147ae27cb8ab0025eba0d3467cc0aec5c39deaab2ea"}
Oct 31 19:43:59.721607 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:59.707973    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8dcfa8-68fb9b44f5-6dmlv"
Oct 31 19:43:59.721607 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:59.708085    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-785d26-6467945f49-6xsr7"
Oct 31 19:43:59.721607 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:59.717443    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8dcfa8-86d859f49f-j5f54" event={"ID":"d8315528-0430-4e79-9f94-82df4e73befa","Type":"ContainerStarted","Data":"13afe4d98f3967936393bbdcb7595c7f58c5d1ebd79c33561aa881c8147d4a77"}
Oct 31 19:43:59.721607 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:59.717484    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8dcfa8-86d859f49f-j5f54" event={"ID":"d8315528-0430-4e79-9f94-82df4e73befa","Type":"ContainerStarted","Data":"1694cafc1245e3393ed2f21ff5a86d4c421227794817cd0e0473f42df2b0b06f"}
Oct 31 19:43:59.721607 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:59.717957    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-8dcfa8-86d859f49f-j5f54"
Oct 31 19:43:59.721607 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:59.720186    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-785d26-6467945f49-6xsr7"]
Oct 31 19:43:59.722184 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:59.722142    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z" event={"ID":"8eee5947-9731-4194-aa80-7fbd619e551f","Type":"ContainerStarted","Data":"ff05706b16b2a500578c373d429e7fb217fc2519b8ee19ec35c709f2d08a75f5"}
Oct 31 19:43:59.732198 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:59.732147    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-d45f6c/test-model-d45f6c-kserve-784b7b884c-7mgtp" podStartSLOduration=1.732133944 podStartE2EDuration="1.732133944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:43:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:43:59.723642412 +0000 UTC m=+1470.296876542" watchObservedRunningTime="2026-10-31 19:43:59.732133944 +0000 UTC m=+1470.305368095"
Oct 31 19:43:59.750429 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:59.750406    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:59.750690 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:59.750502    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-8dcfa8-86d859f49f-j5f54" podStartSLOduration=2.750489096 podStartE2EDuration="2.750489096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:43:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:43:59.747487293 +0000 UTC m=+1470.320721422" watchObservedRunningTime="2026-10-31 19:43:59.750489096 +0000 UTC m=+1470.323723225"
Oct 31 19:43:59.805956 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:59.805926    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdd7678b-849b-4a59-8cf5-227831aa2037-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-785d26-589cf5bf5f-fmrnc\" (UID: \"cdd7678b-849b-4a59-8cf5-227831aa2037\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-785d26-589cf5bf5f-fmrnc"
Oct 31 19:43:59.806122 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:59.806016    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbfl5\" (UniqueName: \"kubernetes.io/projected/cdd7678b-849b-4a59-8cf5-227831aa2037-kube-api-access-nbfl5\") pod \"payload-pre-processing-e2e-inf-a-785d26-589cf5bf5f-fmrnc\" (UID: \"cdd7678b-849b-4a59-8cf5-227831aa2037\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-785d26-589cf5bf5f-fmrnc"
Oct 31 19:43:59.807124 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:59.807096    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdd7678b-849b-4a59-8cf5-227831aa2037-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-785d26-589cf5bf5f-fmrnc\" (UID: \"cdd7678b-849b-4a59-8cf5-227831aa2037\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-785d26-589cf5bf5f-fmrnc"
Oct 31 19:43:59.808415 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:59.808390    4645 patch_prober.go:29] interesting pod/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Oct 31 19:43:59.809232 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:59.809194    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp" podUID="b73e8ddc-c45d-4537-a4c4-f22747469ca8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Oct 31 19:43:59.809970 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:59.809931    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8dcfa8-68fb9b44f5-6dmlv" podStartSLOduration=2.809920371 podStartE2EDuration="2.809920371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:43:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:43:59.807594956 +0000 UTC m=+1470.380829085" watchObservedRunningTime="2026-10-31 19:43:59.809920371 +0000 UTC m=+1470.383154499"
Oct 31 19:43:59.819970 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:59.819852    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbfl5\" (UniqueName: \"kubernetes.io/projected/cdd7678b-849b-4a59-8cf5-227831aa2037-kube-api-access-nbfl5\") pod \"payload-pre-processing-e2e-inf-a-785d26-589cf5bf5f-fmrnc\" (UID: \"cdd7678b-849b-4a59-8cf5-227831aa2037\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-785d26-589cf5bf5f-fmrnc"
Oct 31 19:43:59.836623 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:59.833466    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z"
Oct 31 19:43:59.876780 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:59.872438    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-d45f6c/test-model-d45f6c-kserve-784b7b884c-7mgtp"
Oct 31 19:43:59.894303 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:59.894269    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:43:59.909666 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:59.907700    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8rmh\" (UniqueName: \"kubernetes.io/projected/dfb5bd91-de17-4a3b-afd7-08cb9d538c50-kube-api-access-n8rmh\") pod \"payload-processing-e2e-inf-a-785d26-6467945f49-6xsr7\" (UID: \"dfb5bd91-de17-4a3b-afd7-08cb9d538c50\") " pod="openshift-ingress/payload-processing-e2e-inf-a-785d26-6467945f49-6xsr7"
Oct 31 19:43:59.909666 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:59.908234    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfb5bd91-de17-4a3b-afd7-08cb9d538c50-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-785d26-6467945f49-6xsr7\" (UID: \"dfb5bd91-de17-4a3b-afd7-08cb9d538c50\") " pod="openshift-ingress/payload-processing-e2e-inf-a-785d26-6467945f49-6xsr7"
Oct 31 19:43:59.979353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:43:59.978816    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-785d26-589cf5bf5f-fmrnc"
Oct 31 19:44:00.011383 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:00.009565    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfb5bd91-de17-4a3b-afd7-08cb9d538c50-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-785d26-6467945f49-6xsr7\" (UID: \"dfb5bd91-de17-4a3b-afd7-08cb9d538c50\") " pod="openshift-ingress/payload-processing-e2e-inf-a-785d26-6467945f49-6xsr7"
Oct 31 19:44:00.011383 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:00.009778    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8rmh\" (UniqueName: \"kubernetes.io/projected/dfb5bd91-de17-4a3b-afd7-08cb9d538c50-kube-api-access-n8rmh\") pod \"payload-processing-e2e-inf-a-785d26-6467945f49-6xsr7\" (UID: \"dfb5bd91-de17-4a3b-afd7-08cb9d538c50\") " pod="openshift-ingress/payload-processing-e2e-inf-a-785d26-6467945f49-6xsr7"
Oct 31 19:44:00.011383 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:00.011145    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfb5bd91-de17-4a3b-afd7-08cb9d538c50-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-785d26-6467945f49-6xsr7\" (UID: \"dfb5bd91-de17-4a3b-afd7-08cb9d538c50\") " pod="openshift-ingress/payload-processing-e2e-inf-a-785d26-6467945f49-6xsr7"
Oct 31 19:44:00.040353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:00.038805    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8rmh\" (UniqueName: \"kubernetes.io/projected/dfb5bd91-de17-4a3b-afd7-08cb9d538c50-kube-api-access-n8rmh\") pod \"payload-processing-e2e-inf-a-785d26-6467945f49-6xsr7\" (UID: \"dfb5bd91-de17-4a3b-afd7-08cb9d538c50\") " pod="openshift-ingress/payload-processing-e2e-inf-a-785d26-6467945f49-6xsr7"
Oct 31 19:44:00.341353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:00.331073    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-785d26-6467945f49-6xsr7"
Oct 31 19:44:00.378799 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:00.373690    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z"
Oct 31 19:44:00.811528 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:00.811475    4645 patch_prober.go:29] interesting pod/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Oct 31 19:44:00.811702 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:00.811575    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp" podUID="b73e8ddc-c45d-4537-a4c4-f22747469ca8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Oct 31 19:44:00.838033 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:00.837998    4645 patch_prober.go:29] interesting pod/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 31 19:44:00.838427 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:00.838394    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z" podUID="8eee5947-9731-4194-aa80-7fbd619e551f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 31 19:44:00.919620 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:44:00.919585    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfb5bd91_de17_4a3b_afd7_08cb9d538c50.slice/crio-ac85ea6f908322580eb05763371f92a079c762b66c4c418b927fe60d5c18f9fe WatchSource:0}: Error finding container ac85ea6f908322580eb05763371f92a079c762b66c4c418b927fe60d5c18f9fe: Status 404 returned error can't find the container with id ac85ea6f908322580eb05763371f92a079c762b66c4c418b927fe60d5c18f9fe
Oct 31 19:44:00.922463 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:00.922434    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-785d26-6467945f49-6xsr7"]
Oct 31 19:44:01.015113 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:01.015082    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-785d26-589cf5bf5f-fmrnc"]
Oct 31 19:44:01.018514 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:44:01.018481    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdd7678b_849b_4a59_8cf5_227831aa2037.slice/crio-061e4fe8dbd0fd77df88024ad36d50e06df20f8250204301188c6c975a9b1d98 WatchSource:0}: Error finding container 061e4fe8dbd0fd77df88024ad36d50e06df20f8250204301188c6c975a9b1d98: Status 404 returned error can't find the container with id 061e4fe8dbd0fd77df88024ad36d50e06df20f8250204301188c6c975a9b1d98
Oct 31 19:44:01.152096 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:01.152064    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:01.730267 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:01.729781    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-278c04/test-model-278c04-kserve-6b6f77c967-2f8mt"]
Oct 31 19:44:01.737193 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:01.737164    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-278c04/test-model-278c04-kserve-6b6f77c967-2f8mt"
Oct 31 19:44:01.740762 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:01.740733    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-278c04/test-model-278c04-kserve-6b6f77c967-2f8mt"]
Oct 31 19:44:01.748466 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:01.748420    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-278c04\"/\"openshift-service-ca.crt\""
Oct 31 19:44:01.749232 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:01.749075    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-278c04\"/\"default-dockercfg-2fq8d\""
Oct 31 19:44:01.749232 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:01.749157    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-278c04\"/\"kube-root-ca.crt\""
Oct 31 19:44:01.749681 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:01.749662    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-278c04\"/\"test-model-278c04-kserve-self-signed-certs\""
Oct 31 19:44:01.752128 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:01.752097    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-785d26-6467945f49-6xsr7" event={"ID":"dfb5bd91-de17-4a3b-afd7-08cb9d538c50","Type":"ContainerStarted","Data":"d64c55bd9ce7ec704d4cc217aed7dfa483df19e03ec11f1ff5ac94c22cf0099f"}
Oct 31 19:44:01.752235 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:01.752218    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-785d26-6467945f49-6xsr7"
Oct 31 19:44:01.752304 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:01.752252    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-785d26-6467945f49-6xsr7" event={"ID":"dfb5bd91-de17-4a3b-afd7-08cb9d538c50","Type":"ContainerStarted","Data":"ac85ea6f908322580eb05763371f92a079c762b66c4c418b927fe60d5c18f9fe"}
Oct 31 19:44:01.753977 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:01.753949    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:01.759390 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:01.755461    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-785d26-589cf5bf5f-fmrnc" event={"ID":"cdd7678b-849b-4a59-8cf5-227831aa2037","Type":"ContainerStarted","Data":"4a81ca6acd86ce80ca101ee87b197875a7b6133afca7c8d1c4e42e459e1275af"}
Oct 31 19:44:01.759390 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:01.755510    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-785d26-589cf5bf5f-fmrnc" event={"ID":"cdd7678b-849b-4a59-8cf5-227831aa2037","Type":"ContainerStarted","Data":"061e4fe8dbd0fd77df88024ad36d50e06df20f8250204301188c6c975a9b1d98"}
Oct 31 19:44:01.814559 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:01.814529    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-785d26-589cf5bf5f-fmrnc"
Oct 31 19:44:01.814864 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:01.814839    4645 patch_prober.go:29] interesting pod/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Oct 31 19:44:01.814944 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:01.814881    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp" podUID="b73e8ddc-c45d-4537-a4c4-f22747469ca8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Oct 31 19:44:01.830903 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:01.830881    4645 patch_prober.go:29] interesting pod/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 31 19:44:01.831028 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:01.830919    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z" podUID="8eee5947-9731-4194-aa80-7fbd619e551f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 31 19:44:01.832625 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:01.832601    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/beb01fd0-e706-4d84-b1a6-c41614630f39-dshm\") pod \"test-model-278c04-kserve-6b6f77c967-2f8mt\" (UID: \"beb01fd0-e706-4d84-b1a6-c41614630f39\") " pod="ai-tenant-e2e-inf-a-278c04/test-model-278c04-kserve-6b6f77c967-2f8mt"
Oct 31 19:44:01.832728 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:01.832673    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/beb01fd0-e706-4d84-b1a6-c41614630f39-home\") pod \"test-model-278c04-kserve-6b6f77c967-2f8mt\" (UID: \"beb01fd0-e706-4d84-b1a6-c41614630f39\") " pod="ai-tenant-e2e-inf-a-278c04/test-model-278c04-kserve-6b6f77c967-2f8mt"
Oct 31 19:44:01.832785 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:01.832724    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/beb01fd0-e706-4d84-b1a6-c41614630f39-tmp-dir\") pod \"test-model-278c04-kserve-6b6f77c967-2f8mt\" (UID: \"beb01fd0-e706-4d84-b1a6-c41614630f39\") " pod="ai-tenant-e2e-inf-a-278c04/test-model-278c04-kserve-6b6f77c967-2f8mt"
Oct 31 19:44:01.832851 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:01.832799    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/beb01fd0-e706-4d84-b1a6-c41614630f39-tls-certs\") pod \"test-model-278c04-kserve-6b6f77c967-2f8mt\" (UID: \"beb01fd0-e706-4d84-b1a6-c41614630f39\") " pod="ai-tenant-e2e-inf-a-278c04/test-model-278c04-kserve-6b6f77c967-2f8mt"
Oct 31 19:44:01.832913 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:01.832864    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/beb01fd0-e706-4d84-b1a6-c41614630f39-model-cache\") pod \"test-model-278c04-kserve-6b6f77c967-2f8mt\" (UID: \"beb01fd0-e706-4d84-b1a6-c41614630f39\") " pod="ai-tenant-e2e-inf-a-278c04/test-model-278c04-kserve-6b6f77c967-2f8mt"
Oct 31 19:44:01.833224 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:01.833193    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzdwj\" (UniqueName: \"kubernetes.io/projected/beb01fd0-e706-4d84-b1a6-c41614630f39-kube-api-access-wzdwj\") pod \"test-model-278c04-kserve-6b6f77c967-2f8mt\" (UID: \"beb01fd0-e706-4d84-b1a6-c41614630f39\") " pod="ai-tenant-e2e-inf-a-278c04/test-model-278c04-kserve-6b6f77c967-2f8mt"
Oct 31 19:44:01.837369 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:01.837298    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-785d26-6467945f49-6xsr7" podStartSLOduration=2.837286971 podStartE2EDuration="2.837286971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:43:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:44:01.833402669 +0000 UTC m=+1472.406636797" watchObservedRunningTime="2026-10-31 19:44:01.837286971 +0000 UTC m=+1472.410521122"
Oct 31 19:44:01.854303 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:01.854251    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-785d26-589cf5bf5f-fmrnc" podStartSLOduration=2.854234828 podStartE2EDuration="2.854234828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:43:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:44:01.851198652 +0000 UTC m=+1472.424432782" watchObservedRunningTime="2026-10-31 19:44:01.854234828 +0000 UTC m=+1472.427468957"
Oct 31 19:44:01.920919 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:01.920851    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:01.925885 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:01.925856    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2d319a-bdff7977-7zzh4"]
Oct 31 19:44:01.926115 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:01.926088    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2d319a-bdff7977-7zzh4" podUID="92811d3a-a4be-49a8-a815-d987ff5ccc1a" containerName="maas-api" containerID="cri-o://41cd9734f12733f63edc7d14cd795477eddfbed3f86a8989f786f0fc29e7dbc7" gracePeriod=30
Oct 31 19:44:01.935209 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:01.935178    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/beb01fd0-e706-4d84-b1a6-c41614630f39-tls-certs\") pod \"test-model-278c04-kserve-6b6f77c967-2f8mt\" (UID: \"beb01fd0-e706-4d84-b1a6-c41614630f39\") " pod="ai-tenant-e2e-inf-a-278c04/test-model-278c04-kserve-6b6f77c967-2f8mt"
Oct 31 19:44:01.935532 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:01.935506    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/beb01fd0-e706-4d84-b1a6-c41614630f39-model-cache\") pod \"test-model-278c04-kserve-6b6f77c967-2f8mt\" (UID: \"beb01fd0-e706-4d84-b1a6-c41614630f39\") " pod="ai-tenant-e2e-inf-a-278c04/test-model-278c04-kserve-6b6f77c967-2f8mt"
Oct 31 19:44:01.935819 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:01.935801    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzdwj\" (UniqueName: \"kubernetes.io/projected/beb01fd0-e706-4d84-b1a6-c41614630f39-kube-api-access-wzdwj\") pod \"test-model-278c04-kserve-6b6f77c967-2f8mt\" (UID: \"beb01fd0-e706-4d84-b1a6-c41614630f39\") " pod="ai-tenant-e2e-inf-a-278c04/test-model-278c04-kserve-6b6f77c967-2f8mt"
Oct 31 19:44:01.935970 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:01.935954    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/beb01fd0-e706-4d84-b1a6-c41614630f39-dshm\") pod \"test-model-278c04-kserve-6b6f77c967-2f8mt\" (UID: \"beb01fd0-e706-4d84-b1a6-c41614630f39\") " pod="ai-tenant-e2e-inf-a-278c04/test-model-278c04-kserve-6b6f77c967-2f8mt"
Oct 31 19:44:01.936056 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:01.936042    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/beb01fd0-e706-4d84-b1a6-c41614630f39-home\") pod \"test-model-278c04-kserve-6b6f77c967-2f8mt\" (UID: \"beb01fd0-e706-4d84-b1a6-c41614630f39\") " pod="ai-tenant-e2e-inf-a-278c04/test-model-278c04-kserve-6b6f77c967-2f8mt"
Oct 31 19:44:01.936144 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:01.936127    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/beb01fd0-e706-4d84-b1a6-c41614630f39-tmp-dir\") pod \"test-model-278c04-kserve-6b6f77c967-2f8mt\" (UID: \"beb01fd0-e706-4d84-b1a6-c41614630f39\") " pod="ai-tenant-e2e-inf-a-278c04/test-model-278c04-kserve-6b6f77c967-2f8mt"
Oct 31 19:44:01.937545 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:01.937318    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/beb01fd0-e706-4d84-b1a6-c41614630f39-model-cache\") pod \"test-model-278c04-kserve-6b6f77c967-2f8mt\" (UID: \"beb01fd0-e706-4d84-b1a6-c41614630f39\") " pod="ai-tenant-e2e-inf-a-278c04/test-model-278c04-kserve-6b6f77c967-2f8mt"
Oct 31 19:44:01.938227 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:01.938199    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/beb01fd0-e706-4d84-b1a6-c41614630f39-home\") pod \"test-model-278c04-kserve-6b6f77c967-2f8mt\" (UID: \"beb01fd0-e706-4d84-b1a6-c41614630f39\") " pod="ai-tenant-e2e-inf-a-278c04/test-model-278c04-kserve-6b6f77c967-2f8mt"
Oct 31 19:44:01.938866 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:01.938843    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/beb01fd0-e706-4d84-b1a6-c41614630f39-tmp-dir\") pod \"test-model-278c04-kserve-6b6f77c967-2f8mt\" (UID: \"beb01fd0-e706-4d84-b1a6-c41614630f39\") " pod="ai-tenant-e2e-inf-a-278c04/test-model-278c04-kserve-6b6f77c967-2f8mt"
Oct 31 19:44:01.940341 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:01.940305    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/beb01fd0-e706-4d84-b1a6-c41614630f39-tls-certs\") pod \"test-model-278c04-kserve-6b6f77c967-2f8mt\" (UID: \"beb01fd0-e706-4d84-b1a6-c41614630f39\") " pod="ai-tenant-e2e-inf-a-278c04/test-model-278c04-kserve-6b6f77c967-2f8mt"
Oct 31 19:44:01.940915 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:01.940897    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/beb01fd0-e706-4d84-b1a6-c41614630f39-dshm\") pod \"test-model-278c04-kserve-6b6f77c967-2f8mt\" (UID: \"beb01fd0-e706-4d84-b1a6-c41614630f39\") " pod="ai-tenant-e2e-inf-a-278c04/test-model-278c04-kserve-6b6f77c967-2f8mt"
Oct 31 19:44:01.949424 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:01.949392    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzdwj\" (UniqueName: \"kubernetes.io/projected/beb01fd0-e706-4d84-b1a6-c41614630f39-kube-api-access-wzdwj\") pod \"test-model-278c04-kserve-6b6f77c967-2f8mt\" (UID: \"beb01fd0-e706-4d84-b1a6-c41614630f39\") " pod="ai-tenant-e2e-inf-a-278c04/test-model-278c04-kserve-6b6f77c967-2f8mt"
Oct 31 19:44:02.076353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:02.059559    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-278c04/test-model-278c04-kserve-6b6f77c967-2f8mt"
Oct 31 19:44:02.154468 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:02.130833    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:02.449360 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:02.439270    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:02.841533 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:02.829390    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:02.859691 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:02.848715    4645 patch_prober.go:29] interesting pod/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 31 19:44:02.859691 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:02.848777    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z" podUID="8eee5947-9731-4194-aa80-7fbd619e551f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 31 19:44:02.943608 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:02.917456    4645 patch_prober.go:29] interesting pod/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Oct 31 19:44:02.943608 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:02.917521    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp" podUID="b73e8ddc-c45d-4537-a4c4-f22747469ca8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Oct 31 19:44:03.020451 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:03.020364    4645 generic.go:356] "Generic (PLEG): container finished" podID="92811d3a-a4be-49a8-a815-d987ff5ccc1a" containerID="41cd9734f12733f63edc7d14cd795477eddfbed3f86a8989f786f0fc29e7dbc7" exitCode=0
Oct 31 19:44:03.021830 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:03.021785    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2d319a-bdff7977-7zzh4" event={"ID":"92811d3a-a4be-49a8-a815-d987ff5ccc1a","Type":"ContainerDied","Data":"41cd9734f12733f63edc7d14cd795477eddfbed3f86a8989f786f0fc29e7dbc7"}
Oct 31 19:44:03.109404 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:03.108546    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:03.225321 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:03.225295    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2d319a-bdff7977-7zzh4"
Oct 31 19:44:03.240822 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:44:03.240798    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbeb01fd0_e706_4d84_b1a6_c41614630f39.slice/crio-22bd3c8c0e4f06b877c905f60a1823febcc79934f392ca08e80c7cba92554e42 WatchSource:0}: Error finding container 22bd3c8c0e4f06b877c905f60a1823febcc79934f392ca08e80c7cba92554e42: Status 404 returned error can't find the container with id 22bd3c8c0e4f06b877c905f60a1823febcc79934f392ca08e80c7cba92554e42
Oct 31 19:44:03.241993 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:03.241969    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-278c04/test-model-278c04-kserve-6b6f77c967-2f8mt"]
Oct 31 19:44:03.339538 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:03.339495    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92811d3a-a4be-49a8-a815-d987ff5ccc1a-kube-api-access-6r4dr\" (UniqueName: \"kubernetes.io/projected/92811d3a-a4be-49a8-a815-d987ff5ccc1a-kube-api-access-6r4dr\") pod \"92811d3a-a4be-49a8-a815-d987ff5ccc1a\" (UID: \"92811d3a-a4be-49a8-a815-d987ff5ccc1a\") "
Oct 31 19:44:03.339831 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:03.339805    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92811d3a-a4be-49a8-a815-d987ff5ccc1a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92811d3a-a4be-49a8-a815-d987ff5ccc1a-openshift-service-ca\") pod \"92811d3a-a4be-49a8-a815-d987ff5ccc1a\" (UID: \"92811d3a-a4be-49a8-a815-d987ff5ccc1a\") "
Oct 31 19:44:03.339931 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:03.339861    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92811d3a-a4be-49a8-a815-d987ff5ccc1a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92811d3a-a4be-49a8-a815-d987ff5ccc1a-maas-api-tls\") pod \"92811d3a-a4be-49a8-a815-d987ff5ccc1a\" (UID: \"92811d3a-a4be-49a8-a815-d987ff5ccc1a\") "
Oct 31 19:44:03.340008 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:03.339990    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92811d3a-a4be-49a8-a815-d987ff5ccc1a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/92811d3a-a4be-49a8-a815-d987ff5ccc1a-metrics-certs\") pod \"92811d3a-a4be-49a8-a815-d987ff5ccc1a\" (UID: \"92811d3a-a4be-49a8-a815-d987ff5ccc1a\") "
Oct 31 19:44:03.341178 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:03.341145    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92811d3a-a4be-49a8-a815-d987ff5ccc1a-openshift-service-ca" pod "92811d3a-a4be-49a8-a815-d987ff5ccc1a" (UID: "92811d3a-a4be-49a8-a815-d987ff5ccc1a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:44:03.341649 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:03.341628    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92811d3a-a4be-49a8-a815-d987ff5ccc1a-kube-api-access-6r4dr" pod "92811d3a-a4be-49a8-a815-d987ff5ccc1a" (UID: "92811d3a-a4be-49a8-a815-d987ff5ccc1a"). InnerVolumeSpecName "kube-api-access-6r4dr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:44:03.342164 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:03.342149    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92811d3a-a4be-49a8-a815-d987ff5ccc1a-maas-api-tls" pod "92811d3a-a4be-49a8-a815-d987ff5ccc1a" (UID: "92811d3a-a4be-49a8-a815-d987ff5ccc1a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:44:03.342660 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:03.342640    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92811d3a-a4be-49a8-a815-d987ff5ccc1a-metrics-certs" pod "92811d3a-a4be-49a8-a815-d987ff5ccc1a" (UID: "92811d3a-a4be-49a8-a815-d987ff5ccc1a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:44:03.443280 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:03.443246    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6r4dr\" (UniqueName: \"kubernetes.io/projected/92811d3a-a4be-49a8-a815-d987ff5ccc1a-kube-api-access-6r4dr\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:44:03.443515 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:03.443498    4645 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92811d3a-a4be-49a8-a815-d987ff5ccc1a-openshift-service-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:44:03.443608 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:03.443595    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92811d3a-a4be-49a8-a815-d987ff5ccc1a-maas-api-tls\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:44:03.443795 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:03.443784    4645 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92811d3a-a4be-49a8-a815-d987ff5ccc1a-metrics-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:44:03.807933 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:03.807891    4645 patch_prober.go:29] interesting pod/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Oct 31 19:44:03.808112 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:03.807953    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp" podUID="b73e8ddc-c45d-4537-a4c4-f22747469ca8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Oct 31 19:44:03.825123 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:03.825090    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:03.830818 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:03.830788    4645 patch_prober.go:29] interesting pod/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 31 19:44:03.830978 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:03.830838    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z" podUID="8eee5947-9731-4194-aa80-7fbd619e551f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 31 19:44:04.032198 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:04.032058    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-278c04/test-model-278c04-kserve-6b6f77c967-2f8mt" event={"ID":"beb01fd0-e706-4d84-b1a6-c41614630f39","Type":"ContainerStarted","Data":"7f9af928a1f875f399ff0791a342c36e0e1ab6fcf3ab396edf3cabc54d2fb54b"}
Oct 31 19:44:04.032198 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:04.032117    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-278c04/test-model-278c04-kserve-6b6f77c967-2f8mt" event={"ID":"beb01fd0-e706-4d84-b1a6-c41614630f39","Type":"ContainerStarted","Data":"22bd3c8c0e4f06b877c905f60a1823febcc79934f392ca08e80c7cba92554e42"}
Oct 31 19:44:04.033934 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:04.033825    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2d319a-bdff7977-7zzh4"
Oct 31 19:44:04.033934 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:04.033884    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2d319a-bdff7977-7zzh4" event={"ID":"92811d3a-a4be-49a8-a815-d987ff5ccc1a","Type":"ContainerDied","Data":"ad85df88458b605a304d06e26eebf070f26f07ca9d58382d92252301d2c8487d"}
Oct 31 19:44:04.034091 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:04.033939    4645 scope.go:122] "RemoveContainer" containerID="41cd9734f12733f63edc7d14cd795477eddfbed3f86a8989f786f0fc29e7dbc7"
Oct 31 19:44:04.069348 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:04.060393    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-278c04/test-model-278c04-kserve-6b6f77c967-2f8mt"
Oct 31 19:44:04.084977 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:04.084900    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-278c04/test-model-278c04-kserve-6b6f77c967-2f8mt" podStartSLOduration=3.084879614 podStartE2EDuration="3.084879614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:44:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:44:04.072767816 +0000 UTC m=+1474.646001945" watchObservedRunningTime="2026-10-31 19:44:04.084879614 +0000 UTC m=+1474.658113747"
Oct 31 19:44:04.098452 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:04.098414    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2d319a-bdff7977-7zzh4"]
Oct 31 19:44:04.102700 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:04.102677    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2d319a-bdff7977-7zzh4"]
Oct 31 19:44:04.131010 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:04.130960    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:04.776355 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:04.774698    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp"
Oct 31 19:44:04.803475 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:04.803445    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:04.812490 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:04.812464    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp"
Oct 31 19:44:04.830394 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:04.830309    4645 patch_prober.go:29] interesting pod/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 31 19:44:04.830522 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:04.830397    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z" podUID="8eee5947-9731-4194-aa80-7fbd619e551f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 31 19:44:05.049127 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:05.049058    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp"
Oct 31 19:44:05.620483 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:05.620454    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92811d3a-a4be-49a8-a815-d987ff5ccc1a" path="/var/lib/kubelet/pods/92811d3a-a4be-49a8-a815-d987ff5ccc1a/volumes"
Oct 31 19:44:05.646983 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:05.646953    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-9efeae/test-model-9efeae-kserve-7d97f79d8f-2nnrl"]
Oct 31 19:44:05.647922 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:05.647900    4645 cpu_manager.go:414] "containerMap: removing container" podUID="92811d3a-a4be-49a8-a815-d987ff5ccc1a" containerName="maas-api"
Oct 31 19:44:05.647922 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:05.647922    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92811d3a-a4be-49a8-a815-d987ff5ccc1a" containerName="maas-api"
Oct 31 19:44:05.648140 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:05.648126    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="92811d3a-a4be-49a8-a815-d987ff5ccc1a" containerName="maas-api"
Oct 31 19:44:05.653755 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:05.653733    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9efeae/test-model-9efeae-kserve-7d97f79d8f-2nnrl"
Oct 31 19:44:05.689368 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:05.687760    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-9efeae\"/\"openshift-service-ca.crt\""
Oct 31 19:44:05.689368 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:05.687789    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-9efeae\"/\"test-model-9efeae-kserve-self-signed-certs\""
Oct 31 19:44:05.689368 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:05.687877    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-9efeae\"/\"default-dockercfg-z62lg\""
Oct 31 19:44:05.689368 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:05.688432    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-9efeae\"/\"kube-root-ca.crt\""
Oct 31 19:44:05.692730 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:05.692624    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-9efeae/test-model-9efeae-kserve-7d97f79d8f-2nnrl"]
Oct 31 19:44:05.705142 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:05.705119    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72f178d8-b759-4bf3-90fa-1b0a1a8cb51e-tmp-dir\") pod \"test-model-9efeae-kserve-7d97f79d8f-2nnrl\" (UID: \"72f178d8-b759-4bf3-90fa-1b0a1a8cb51e\") " pod="ai-tenant-e2e-inf-b-9efeae/test-model-9efeae-kserve-7d97f79d8f-2nnrl"
Oct 31 19:44:05.705248 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:05.705149    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pztm7\" (UniqueName: \"kubernetes.io/projected/72f178d8-b759-4bf3-90fa-1b0a1a8cb51e-kube-api-access-pztm7\") pod \"test-model-9efeae-kserve-7d97f79d8f-2nnrl\" (UID: \"72f178d8-b759-4bf3-90fa-1b0a1a8cb51e\") " pod="ai-tenant-e2e-inf-b-9efeae/test-model-9efeae-kserve-7d97f79d8f-2nnrl"
Oct 31 19:44:05.705248 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:05.705193    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72f178d8-b759-4bf3-90fa-1b0a1a8cb51e-model-cache\") pod \"test-model-9efeae-kserve-7d97f79d8f-2nnrl\" (UID: \"72f178d8-b759-4bf3-90fa-1b0a1a8cb51e\") " pod="ai-tenant-e2e-inf-b-9efeae/test-model-9efeae-kserve-7d97f79d8f-2nnrl"
Oct 31 19:44:05.705248 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:05.705210    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72f178d8-b759-4bf3-90fa-1b0a1a8cb51e-home\") pod \"test-model-9efeae-kserve-7d97f79d8f-2nnrl\" (UID: \"72f178d8-b759-4bf3-90fa-1b0a1a8cb51e\") " pod="ai-tenant-e2e-inf-b-9efeae/test-model-9efeae-kserve-7d97f79d8f-2nnrl"
Oct 31 19:44:05.705248 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:05.705233    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72f178d8-b759-4bf3-90fa-1b0a1a8cb51e-tls-certs\") pod \"test-model-9efeae-kserve-7d97f79d8f-2nnrl\" (UID: \"72f178d8-b759-4bf3-90fa-1b0a1a8cb51e\") " pod="ai-tenant-e2e-inf-b-9efeae/test-model-9efeae-kserve-7d97f79d8f-2nnrl"
Oct 31 19:44:05.705403 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:05.705250    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72f178d8-b759-4bf3-90fa-1b0a1a8cb51e-dshm\") pod \"test-model-9efeae-kserve-7d97f79d8f-2nnrl\" (UID: \"72f178d8-b759-4bf3-90fa-1b0a1a8cb51e\") " pod="ai-tenant-e2e-inf-b-9efeae/test-model-9efeae-kserve-7d97f79d8f-2nnrl"
Oct 31 19:44:05.781083 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:05.781054    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:05.806306 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:05.806279    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72f178d8-b759-4bf3-90fa-1b0a1a8cb51e-tmp-dir\") pod \"test-model-9efeae-kserve-7d97f79d8f-2nnrl\" (UID: \"72f178d8-b759-4bf3-90fa-1b0a1a8cb51e\") " pod="ai-tenant-e2e-inf-b-9efeae/test-model-9efeae-kserve-7d97f79d8f-2nnrl"
Oct 31 19:44:05.806437 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:05.806311    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pztm7\" (UniqueName: \"kubernetes.io/projected/72f178d8-b759-4bf3-90fa-1b0a1a8cb51e-kube-api-access-pztm7\") pod \"test-model-9efeae-kserve-7d97f79d8f-2nnrl\" (UID: \"72f178d8-b759-4bf3-90fa-1b0a1a8cb51e\") " pod="ai-tenant-e2e-inf-b-9efeae/test-model-9efeae-kserve-7d97f79d8f-2nnrl"
Oct 31 19:44:05.806437 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:05.806387    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72f178d8-b759-4bf3-90fa-1b0a1a8cb51e-model-cache\") pod \"test-model-9efeae-kserve-7d97f79d8f-2nnrl\" (UID: \"72f178d8-b759-4bf3-90fa-1b0a1a8cb51e\") " pod="ai-tenant-e2e-inf-b-9efeae/test-model-9efeae-kserve-7d97f79d8f-2nnrl"
Oct 31 19:44:05.806437 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:05.806405    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72f178d8-b759-4bf3-90fa-1b0a1a8cb51e-home\") pod \"test-model-9efeae-kserve-7d97f79d8f-2nnrl\" (UID: \"72f178d8-b759-4bf3-90fa-1b0a1a8cb51e\") " pod="ai-tenant-e2e-inf-b-9efeae/test-model-9efeae-kserve-7d97f79d8f-2nnrl"
Oct 31 19:44:05.806437 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:05.806433    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72f178d8-b759-4bf3-90fa-1b0a1a8cb51e-tls-certs\") pod \"test-model-9efeae-kserve-7d97f79d8f-2nnrl\" (UID: \"72f178d8-b759-4bf3-90fa-1b0a1a8cb51e\") " pod="ai-tenant-e2e-inf-b-9efeae/test-model-9efeae-kserve-7d97f79d8f-2nnrl"
Oct 31 19:44:05.806661 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:05.806452    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72f178d8-b759-4bf3-90fa-1b0a1a8cb51e-dshm\") pod \"test-model-9efeae-kserve-7d97f79d8f-2nnrl\" (UID: \"72f178d8-b759-4bf3-90fa-1b0a1a8cb51e\") " pod="ai-tenant-e2e-inf-b-9efeae/test-model-9efeae-kserve-7d97f79d8f-2nnrl"
Oct 31 19:44:05.806747 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:05.806714    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72f178d8-b759-4bf3-90fa-1b0a1a8cb51e-tmp-dir\") pod \"test-model-9efeae-kserve-7d97f79d8f-2nnrl\" (UID: \"72f178d8-b759-4bf3-90fa-1b0a1a8cb51e\") " pod="ai-tenant-e2e-inf-b-9efeae/test-model-9efeae-kserve-7d97f79d8f-2nnrl"
Oct 31 19:44:05.806823 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:05.806794    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72f178d8-b759-4bf3-90fa-1b0a1a8cb51e-model-cache\") pod \"test-model-9efeae-kserve-7d97f79d8f-2nnrl\" (UID: \"72f178d8-b759-4bf3-90fa-1b0a1a8cb51e\") " pod="ai-tenant-e2e-inf-b-9efeae/test-model-9efeae-kserve-7d97f79d8f-2nnrl"
Oct 31 19:44:05.806933 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:05.806848    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72f178d8-b759-4bf3-90fa-1b0a1a8cb51e-home\") pod \"test-model-9efeae-kserve-7d97f79d8f-2nnrl\" (UID: \"72f178d8-b759-4bf3-90fa-1b0a1a8cb51e\") " pod="ai-tenant-e2e-inf-b-9efeae/test-model-9efeae-kserve-7d97f79d8f-2nnrl"
Oct 31 19:44:05.809215 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:05.809194    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72f178d8-b759-4bf3-90fa-1b0a1a8cb51e-dshm\") pod \"test-model-9efeae-kserve-7d97f79d8f-2nnrl\" (UID: \"72f178d8-b759-4bf3-90fa-1b0a1a8cb51e\") " pod="ai-tenant-e2e-inf-b-9efeae/test-model-9efeae-kserve-7d97f79d8f-2nnrl"
Oct 31 19:44:05.809295 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:05.809214    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72f178d8-b759-4bf3-90fa-1b0a1a8cb51e-tls-certs\") pod \"test-model-9efeae-kserve-7d97f79d8f-2nnrl\" (UID: \"72f178d8-b759-4bf3-90fa-1b0a1a8cb51e\") " pod="ai-tenant-e2e-inf-b-9efeae/test-model-9efeae-kserve-7d97f79d8f-2nnrl"
Oct 31 19:44:05.830427 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:05.830402    4645 patch_prober.go:29] interesting pod/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 31 19:44:05.830505 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:05.830444    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z" podUID="8eee5947-9731-4194-aa80-7fbd619e551f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 31 19:44:05.834695 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:05.834674    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pztm7\" (UniqueName: \"kubernetes.io/projected/72f178d8-b759-4bf3-90fa-1b0a1a8cb51e-kube-api-access-pztm7\") pod \"test-model-9efeae-kserve-7d97f79d8f-2nnrl\" (UID: \"72f178d8-b759-4bf3-90fa-1b0a1a8cb51e\") " pod="ai-tenant-e2e-inf-b-9efeae/test-model-9efeae-kserve-7d97f79d8f-2nnrl"
Oct 31 19:44:06.003175 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:06.003116    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9efeae/test-model-9efeae-kserve-7d97f79d8f-2nnrl"
Oct 31 19:44:06.258149 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:44:06.258053    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 19:44:06.372361 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:44:06.372287    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod72f178d8_b759_4bf3_90fa_1b0a1a8cb51e.slice/crio-9e9320bc3f9c210608621b8bae4399ce5066c541f30c75e7bde9c0aa3abaa328 WatchSource:0}: Error finding container 9e9320bc3f9c210608621b8bae4399ce5066c541f30c75e7bde9c0aa3abaa328: Status 404 returned error can't find the container with id 9e9320bc3f9c210608621b8bae4399ce5066c541f30c75e7bde9c0aa3abaa328
Oct 31 19:44:06.372871 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:06.372802    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-9efeae/test-model-9efeae-kserve-7d97f79d8f-2nnrl"]
Oct 31 19:44:06.835632 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:06.835602    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z"
Oct 31 19:44:06.836828 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:06.836805    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z"
Oct 31 19:44:06.897873 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:06.897839    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:07.089848 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:07.089758    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9efeae/test-model-9efeae-kserve-7d97f79d8f-2nnrl" event={"ID":"72f178d8-b759-4bf3-90fa-1b0a1a8cb51e","Type":"ContainerStarted","Data":"4982e92d7210d288d06cf8ec93fd057a6a03d5d70a801ab9afd8397eb88c0560"}
Oct 31 19:44:07.089848 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:07.089811    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9efeae/test-model-9efeae-kserve-7d97f79d8f-2nnrl" event={"ID":"72f178d8-b759-4bf3-90fa-1b0a1a8cb51e","Type":"ContainerStarted","Data":"9e9320bc3f9c210608621b8bae4399ce5066c541f30c75e7bde9c0aa3abaa328"}
Oct 31 19:44:07.114640 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:07.114595    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-9efeae/test-model-9efeae-kserve-7d97f79d8f-2nnrl" podStartSLOduration=2.114582125 podStartE2EDuration="2.114582125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:44:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:44:07.110441862 +0000 UTC m=+1477.683675990" watchObservedRunningTime="2026-10-31 19:44:07.114582125 +0000 UTC m=+1477.687816293"
Oct 31 19:44:07.638481 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:07.638445    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:08.003696 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:08.003666    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-9efeae/test-model-9efeae-kserve-7d97f79d8f-2nnrl"
Oct 31 19:44:08.703837 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:08.703799    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:08.866500 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:08.866470    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-d45f6c/test-model-d45f6c-kserve-784b7b884c-7mgtp"
Oct 31 19:44:08.878776 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:08.878754    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-d45f6c/test-model-d45f6c-kserve-784b7b884c-7mgtp"
Oct 31 19:44:09.115708 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:09.115634    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-d45f6c/test-model-d45f6c-kserve-784b7b884c-7mgtp"
Oct 31 19:44:09.536182 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:09.536142    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:09.681979 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:09.681944    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:09.796045 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:09.795953    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:10.002312 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:10.002275    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm"]
Oct 31 19:44:10.002610 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:10.002582    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm" podUID="773450c2-8daa-4448-a78a-8ede38d60d69" containerName="istio-proxy" containerID="cri-o://6a77db055a4c872750200ed74919863a2e2716f24fa3ca4f19a69b88f983e37b" gracePeriod=30
Oct 31 19:44:10.036309 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:10.036280    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:10.739042 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:10.739013    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8dcfa8-68fb9b44f5-6dmlv"
Oct 31 19:44:10.964776 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:10.964742    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:11.816783 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:11.816754    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-8dcfa8-86d859f49f-j5f54"
Oct 31 19:44:11.817267 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:11.817242    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:12.060316 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:12.060245    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-278c04/test-model-278c04-kserve-6b6f77c967-2f8mt"
Oct 31 19:44:12.072661 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:12.072598    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-278c04/test-model-278c04-kserve-6b6f77c967-2f8mt"
Oct 31 19:44:12.216079 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:12.216054    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-278c04/test-model-278c04-kserve-6b6f77c967-2f8mt"
Oct 31 19:44:12.790350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:12.790189    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:13.026523 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:13.026496    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-785d26-6467945f49-6xsr7"
Oct 31 19:44:13.829015 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:13.828987    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:14.037736 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:14.037710    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-785d26-589cf5bf5f-fmrnc"
Oct 31 19:44:14.673997 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:14.673962    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:15.240151 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:15.240122    4645 generic.go:356] "Generic (PLEG): container finished" podID="773450c2-8daa-4448-a78a-8ede38d60d69" containerID="6a77db055a4c872750200ed74919863a2e2716f24fa3ca4f19a69b88f983e37b" exitCode=0
Oct 31 19:44:15.240638 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:15.240203    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm" event={"ID":"773450c2-8daa-4448-a78a-8ede38d60d69","Type":"ContainerDied","Data":"6a77db055a4c872750200ed74919863a2e2716f24fa3ca4f19a69b88f983e37b"}
Oct 31 19:44:15.240638 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:15.240278    4645 scope.go:122] "RemoveContainer" containerID="523222635d65f76e0c47c008d6856668c03854cd36a652188d27282f7d6a968d"
Oct 31 19:44:15.294821 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:15.294797    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm"
Oct 31 19:44:15.349427 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:15.349400    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/773450c2-8daa-4448-a78a-8ede38d60d69-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/773450c2-8daa-4448-a78a-8ede38d60d69-workload-socket\") pod \"773450c2-8daa-4448-a78a-8ede38d60d69\" (UID: \"773450c2-8daa-4448-a78a-8ede38d60d69\") "
Oct 31 19:44:15.349572 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:15.349456    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/773450c2-8daa-4448-a78a-8ede38d60d69-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/773450c2-8daa-4448-a78a-8ede38d60d69-istiod-ca-cert\") pod \"773450c2-8daa-4448-a78a-8ede38d60d69\" (UID: \"773450c2-8daa-4448-a78a-8ede38d60d69\") "
Oct 31 19:44:15.349572 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:15.349489    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/773450c2-8daa-4448-a78a-8ede38d60d69-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/773450c2-8daa-4448-a78a-8ede38d60d69-istio-data\") pod \"773450c2-8daa-4448-a78a-8ede38d60d69\" (UID: \"773450c2-8daa-4448-a78a-8ede38d60d69\") "
Oct 31 19:44:15.349572 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:15.349545    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/773450c2-8daa-4448-a78a-8ede38d60d69-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/773450c2-8daa-4448-a78a-8ede38d60d69-workload-certs\") pod \"773450c2-8daa-4448-a78a-8ede38d60d69\" (UID: \"773450c2-8daa-4448-a78a-8ede38d60d69\") "
Oct 31 19:44:15.349743 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:15.349631    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/773450c2-8daa-4448-a78a-8ede38d60d69-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/773450c2-8daa-4448-a78a-8ede38d60d69-istio-envoy\") pod \"773450c2-8daa-4448-a78a-8ede38d60d69\" (UID: \"773450c2-8daa-4448-a78a-8ede38d60d69\") "
Oct 31 19:44:15.349743 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:15.349673    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/773450c2-8daa-4448-a78a-8ede38d60d69-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/773450c2-8daa-4448-a78a-8ede38d60d69-credential-socket\") pod \"773450c2-8daa-4448-a78a-8ede38d60d69\" (UID: \"773450c2-8daa-4448-a78a-8ede38d60d69\") "
Oct 31 19:44:15.349743 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:15.349686    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/773450c2-8daa-4448-a78a-8ede38d60d69-workload-socket" pod "773450c2-8daa-4448-a78a-8ede38d60d69" (UID: "773450c2-8daa-4448-a78a-8ede38d60d69"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:44:15.349743 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:15.349733    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/773450c2-8daa-4448-a78a-8ede38d60d69-istio-data" pod "773450c2-8daa-4448-a78a-8ede38d60d69" (UID: "773450c2-8daa-4448-a78a-8ede38d60d69"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:44:15.349948 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:15.349888    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/773450c2-8daa-4448-a78a-8ede38d60d69-istio-token\" (UniqueName: \"kubernetes.io/projected/773450c2-8daa-4448-a78a-8ede38d60d69-istio-token\") pod \"773450c2-8daa-4448-a78a-8ede38d60d69\" (UID: \"773450c2-8daa-4448-a78a-8ede38d60d69\") "
Oct 31 19:44:15.350000 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:15.349943    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/773450c2-8daa-4448-a78a-8ede38d60d69-credential-socket" pod "773450c2-8daa-4448-a78a-8ede38d60d69" (UID: "773450c2-8daa-4448-a78a-8ede38d60d69"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:44:15.350000 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:15.349981    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/773450c2-8daa-4448-a78a-8ede38d60d69-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/773450c2-8daa-4448-a78a-8ede38d60d69-istio-podinfo\") pod \"773450c2-8daa-4448-a78a-8ede38d60d69\" (UID: \"773450c2-8daa-4448-a78a-8ede38d60d69\") "
Oct 31 19:44:15.350084 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:15.350048    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/773450c2-8daa-4448-a78a-8ede38d60d69-kube-api-access-8bgh8\" (UniqueName: \"kubernetes.io/projected/773450c2-8daa-4448-a78a-8ede38d60d69-kube-api-access-8bgh8\") pod \"773450c2-8daa-4448-a78a-8ede38d60d69\" (UID: \"773450c2-8daa-4448-a78a-8ede38d60d69\") "
Oct 31 19:44:15.350134 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:15.350079    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/773450c2-8daa-4448-a78a-8ede38d60d69-workload-certs" pod "773450c2-8daa-4448-a78a-8ede38d60d69" (UID: "773450c2-8daa-4448-a78a-8ede38d60d69"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:44:15.350194 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:15.350152    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/773450c2-8daa-4448-a78a-8ede38d60d69-istiod-ca-cert" pod "773450c2-8daa-4448-a78a-8ede38d60d69" (UID: "773450c2-8daa-4448-a78a-8ede38d60d69"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:44:15.351077 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:15.350956    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/773450c2-8daa-4448-a78a-8ede38d60d69-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:44:15.351077 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:15.350987    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/773450c2-8daa-4448-a78a-8ede38d60d69-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:44:15.351077 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:15.351001    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/773450c2-8daa-4448-a78a-8ede38d60d69-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:44:15.351077 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:15.351015    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/773450c2-8daa-4448-a78a-8ede38d60d69-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:44:15.351077 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:15.351028    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/773450c2-8daa-4448-a78a-8ede38d60d69-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:44:15.352420 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:15.352393    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/773450c2-8daa-4448-a78a-8ede38d60d69-istio-envoy" pod "773450c2-8daa-4448-a78a-8ede38d60d69" (UID: "773450c2-8daa-4448-a78a-8ede38d60d69"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:44:15.352526 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:15.352507    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/773450c2-8daa-4448-a78a-8ede38d60d69-istio-podinfo" pod "773450c2-8daa-4448-a78a-8ede38d60d69" (UID: "773450c2-8daa-4448-a78a-8ede38d60d69"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:44:15.352825 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:15.352794    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/773450c2-8daa-4448-a78a-8ede38d60d69-kube-api-access-8bgh8" pod "773450c2-8daa-4448-a78a-8ede38d60d69" (UID: "773450c2-8daa-4448-a78a-8ede38d60d69"). InnerVolumeSpecName "kube-api-access-8bgh8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:44:15.353763 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:15.353736    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/773450c2-8daa-4448-a78a-8ede38d60d69-istio-token" pod "773450c2-8daa-4448-a78a-8ede38d60d69" (UID: "773450c2-8daa-4448-a78a-8ede38d60d69"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:44:15.452139 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:15.452113    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/773450c2-8daa-4448-a78a-8ede38d60d69-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:44:15.452139 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:15.452137    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/773450c2-8daa-4448-a78a-8ede38d60d69-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:44:15.452289 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:15.452148    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/773450c2-8daa-4448-a78a-8ede38d60d69-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:44:15.452289 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:15.452162    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bgh8\" (UniqueName: \"kubernetes.io/projected/773450c2-8daa-4448-a78a-8ede38d60d69-kube-api-access-8bgh8\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:44:15.537870 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:15.537805    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:16.004512 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:16.004479    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-9efeae/test-model-9efeae-kserve-7d97f79d8f-2nnrl"
Oct 31 19:44:16.016865 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:16.016842    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-9efeae/test-model-9efeae-kserve-7d97f79d8f-2nnrl"
Oct 31 19:44:16.249218 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:16.249195    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm"
Oct 31 19:44:16.249676 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:16.249242    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm" event={"ID":"773450c2-8daa-4448-a78a-8ede38d60d69","Type":"ContainerDied","Data":"f8eb68858bd1fa87910f43ec36038e85e96ebbc1ab4f5d24ded9790d8356c4ef"}
Oct 31 19:44:16.249676 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:16.249296    4645 scope.go:122] "RemoveContainer" containerID="6a77db055a4c872750200ed74919863a2e2716f24fa3ca4f19a69b88f983e37b"
Oct 31 19:44:16.261101 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:16.261081    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-9efeae/test-model-9efeae-kserve-7d97f79d8f-2nnrl"
Oct 31 19:44:16.267421 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:16.267386    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:16.270392 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:16.270370    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm"]
Oct 31 19:44:16.276771 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:16.276749    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-2d319a-openshift-default-64d669fc9c-wjjhm"]
Oct 31 19:44:16.588648 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:16.588569    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:17.185638 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:17.185599    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:17.301143 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:17.301111    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:17.620549 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:17.620461    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="773450c2-8daa-4448-a78a-8ede38d60d69" path="/var/lib/kubelet/pods/773450c2-8daa-4448-a78a-8ede38d60d69/volumes"
Oct 31 19:44:18.035525 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:18.035495    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:18.830825 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:18.830790    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:19.631985 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:19.631950    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:20.434245 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:20.434207    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:20.766124 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:20.766044    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a813e2-dbc7795cc-vqxqm"]
Oct 31 19:44:20.782394 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:20.782363    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a813e2-747f99fd79-prchw"]
Oct 31 19:44:20.857397 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:20.857366    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:20.969215 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:20.969185    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:21.503943 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:21.503898    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:22.398828 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:22.398794    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8dcfa8-6c7cb4c95f-cws7q"]
Oct 31 19:44:22.399638 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:22.399621    4645 cpu_manager.go:414] "containerMap: removing container" podUID="773450c2-8daa-4448-a78a-8ede38d60d69" containerName="istio-proxy"
Oct 31 19:44:22.399693 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:22.399639    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="773450c2-8daa-4448-a78a-8ede38d60d69" containerName="istio-proxy"
Oct 31 19:44:22.399693 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:22.399665    4645 cpu_manager.go:414] "containerMap: removing container" podUID="773450c2-8daa-4448-a78a-8ede38d60d69" containerName="istio-proxy"
Oct 31 19:44:22.399693 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:22.399671    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="773450c2-8daa-4448-a78a-8ede38d60d69" containerName="istio-proxy"
Oct 31 19:44:22.399805 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:22.399796    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="773450c2-8daa-4448-a78a-8ede38d60d69" containerName="istio-proxy"
Oct 31 19:44:22.403628 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:22.403611    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8dcfa8-6c7cb4c95f-cws7q"
Oct 31 19:44:22.409512 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:22.409490    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-8dcfa8-serving-cert\""
Oct 31 19:44:22.416042 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:22.416021    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8dcfa8-6c7cb4c95f-cws7q"]
Oct 31 19:44:22.488692 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:22.488663    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be41f707-30c3-49f1-91c6-bea2e72c172a-maas-api-tls\") pod \"maas-api-e2e-inf-b-8dcfa8-6c7cb4c95f-cws7q\" (UID: \"be41f707-30c3-49f1-91c6-bea2e72c172a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8dcfa8-6c7cb4c95f-cws7q"
Oct 31 19:44:22.488852 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:22.488719    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be41f707-30c3-49f1-91c6-bea2e72c172a-metrics-certs\") pod \"maas-api-e2e-inf-b-8dcfa8-6c7cb4c95f-cws7q\" (UID: \"be41f707-30c3-49f1-91c6-bea2e72c172a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8dcfa8-6c7cb4c95f-cws7q"
Oct 31 19:44:22.488852 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:22.488827    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54dwh\" (UniqueName: \"kubernetes.io/projected/be41f707-30c3-49f1-91c6-bea2e72c172a-kube-api-access-54dwh\") pod \"maas-api-e2e-inf-b-8dcfa8-6c7cb4c95f-cws7q\" (UID: \"be41f707-30c3-49f1-91c6-bea2e72c172a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8dcfa8-6c7cb4c95f-cws7q"
Oct 31 19:44:22.489053 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:22.489035    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be41f707-30c3-49f1-91c6-bea2e72c172a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8dcfa8-6c7cb4c95f-cws7q\" (UID: \"be41f707-30c3-49f1-91c6-bea2e72c172a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8dcfa8-6c7cb4c95f-cws7q"
Oct 31 19:44:22.498631 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:22.498598    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:22.590446 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:22.590418    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be41f707-30c3-49f1-91c6-bea2e72c172a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8dcfa8-6c7cb4c95f-cws7q\" (UID: \"be41f707-30c3-49f1-91c6-bea2e72c172a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8dcfa8-6c7cb4c95f-cws7q"
Oct 31 19:44:22.590809 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:22.590623    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be41f707-30c3-49f1-91c6-bea2e72c172a-maas-api-tls\") pod \"maas-api-e2e-inf-b-8dcfa8-6c7cb4c95f-cws7q\" (UID: \"be41f707-30c3-49f1-91c6-bea2e72c172a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8dcfa8-6c7cb4c95f-cws7q"
Oct 31 19:44:22.590809 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:22.590686    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be41f707-30c3-49f1-91c6-bea2e72c172a-metrics-certs\") pod \"maas-api-e2e-inf-b-8dcfa8-6c7cb4c95f-cws7q\" (UID: \"be41f707-30c3-49f1-91c6-bea2e72c172a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8dcfa8-6c7cb4c95f-cws7q"
Oct 31 19:44:22.590809 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:22.590738    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54dwh\" (UniqueName: \"kubernetes.io/projected/be41f707-30c3-49f1-91c6-bea2e72c172a-kube-api-access-54dwh\") pod \"maas-api-e2e-inf-b-8dcfa8-6c7cb4c95f-cws7q\" (UID: \"be41f707-30c3-49f1-91c6-bea2e72c172a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8dcfa8-6c7cb4c95f-cws7q"
Oct 31 19:44:22.591248 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:22.591224    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be41f707-30c3-49f1-91c6-bea2e72c172a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8dcfa8-6c7cb4c95f-cws7q\" (UID: \"be41f707-30c3-49f1-91c6-bea2e72c172a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8dcfa8-6c7cb4c95f-cws7q"
Oct 31 19:44:22.593156 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:22.593131    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be41f707-30c3-49f1-91c6-bea2e72c172a-maas-api-tls\") pod \"maas-api-e2e-inf-b-8dcfa8-6c7cb4c95f-cws7q\" (UID: \"be41f707-30c3-49f1-91c6-bea2e72c172a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8dcfa8-6c7cb4c95f-cws7q"
Oct 31 19:44:22.593236 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:22.593135    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be41f707-30c3-49f1-91c6-bea2e72c172a-metrics-certs\") pod \"maas-api-e2e-inf-b-8dcfa8-6c7cb4c95f-cws7q\" (UID: \"be41f707-30c3-49f1-91c6-bea2e72c172a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8dcfa8-6c7cb4c95f-cws7q"
Oct 31 19:44:22.602854 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:22.602834    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54dwh\" (UniqueName: \"kubernetes.io/projected/be41f707-30c3-49f1-91c6-bea2e72c172a-kube-api-access-54dwh\") pod \"maas-api-e2e-inf-b-8dcfa8-6c7cb4c95f-cws7q\" (UID: \"be41f707-30c3-49f1-91c6-bea2e72c172a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8dcfa8-6c7cb4c95f-cws7q"
Oct 31 19:44:22.606739 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:22.606700    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:22.714629 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:22.714594    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:22.720646 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:22.720612    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8dcfa8-6c7cb4c95f-cws7q"
Oct 31 19:44:23.042939 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:44:23.042908    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe41f707_30c3_49f1_91c6_bea2e72c172a.slice/crio-701ac01b7d0826ad77372bb75eea56acf3fe2f9466544c5df01486dd79bcb11f WatchSource:0}: Error finding container 701ac01b7d0826ad77372bb75eea56acf3fe2f9466544c5df01486dd79bcb11f: Status 404 returned error can't find the container with id 701ac01b7d0826ad77372bb75eea56acf3fe2f9466544c5df01486dd79bcb11f
Oct 31 19:44:23.043951 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:23.043924    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8dcfa8-6c7cb4c95f-cws7q"]
Oct 31 19:44:23.044607 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:23.044583    4645 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 19:44:23.384367 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:23.384267    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8dcfa8-6c7cb4c95f-cws7q" event={"ID":"be41f707-30c3-49f1-91c6-bea2e72c172a","Type":"ContainerStarted","Data":"701ac01b7d0826ad77372bb75eea56acf3fe2f9466544c5df01486dd79bcb11f"}
Oct 31 19:44:23.571712 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:23.571689    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:24.407907 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:24.407867    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8dcfa8-6c7cb4c95f-cws7q" event={"ID":"be41f707-30c3-49f1-91c6-bea2e72c172a","Type":"ContainerStarted","Data":"97254e810ef08368367cb3f4cb56afcd1c2227c7a7a5cf37093c12a9228dd64c"}
Oct 31 19:44:24.409119 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:24.409096    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8dcfa8-6c7cb4c95f-cws7q"
Oct 31 19:44:24.443062 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:24.443009    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8dcfa8-6c7cb4c95f-cws7q" podStartSLOduration=1.982932449 podStartE2EDuration="2.442990183s" totalImagesPullingTime="460.057734ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:44:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:44:24.436922179 +0000 UTC m=+1495.010156308" watchObservedRunningTime="2026-10-31 19:44:24.442990183 +0000 UTC m=+1495.016224313"
Oct 31 19:44:24.493344 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:24.493305    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:24.669026 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:24.668993    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:25.766992 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:25.766920    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-a813e2-dbc7795cc-vqxqm" podUID="ac45690b-da7e-4ae0-81dc-6cf69551f665" containerName="payload-processing" containerID="cri-o://e32773b97d5d99ffc8762171ac6ac60ba26ab561c496c3ab56ed1932366744c0" gracePeriod=25
Oct 31 19:44:25.782668 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:25.782608    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a813e2-747f99fd79-prchw" podUID="38d1e68e-f404-4056-8176-14f7205b2d99" containerName="payload-pre-processing" containerID="cri-o://effefd88c5075c37e2c3ffacde53f5d3461716d84262bd8d2d48d9a933d9bff3" gracePeriod=25
Oct 31 19:44:25.784403 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:25.784314    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:26.024086 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:44:26.023159    4645 logging.go:55] [core] [Channel #1412 SubChannel #1413]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.133:50052", ServerName: "10.133.0.133:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.133:50052: connect: connection refused"
Oct 31 19:44:26.114485 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:26.114459    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a813e2-dbc7795cc-vqxqm"
Oct 31 19:44:26.154726 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:26.154701    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a813e2-747f99fd79-prchw"
Oct 31 19:44:26.218377 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:26.218314    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac45690b-da7e-4ae0-81dc-6cf69551f665-kube-api-access-2m49z\" (UniqueName: \"kubernetes.io/projected/ac45690b-da7e-4ae0-81dc-6cf69551f665-kube-api-access-2m49z\") pod \"ac45690b-da7e-4ae0-81dc-6cf69551f665\" (UID: \"ac45690b-da7e-4ae0-81dc-6cf69551f665\") "
Oct 31 19:44:26.218529 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:26.218511    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38d1e68e-f404-4056-8176-14f7205b2d99-kube-api-access-lc7pc\" (UniqueName: \"kubernetes.io/projected/38d1e68e-f404-4056-8176-14f7205b2d99-kube-api-access-lc7pc\") pod \"38d1e68e-f404-4056-8176-14f7205b2d99\" (UID: \"38d1e68e-f404-4056-8176-14f7205b2d99\") "
Oct 31 19:44:26.218616 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:26.218590    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38d1e68e-f404-4056-8176-14f7205b2d99-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38d1e68e-f404-4056-8176-14f7205b2d99-plugins-config-volume\") pod \"38d1e68e-f404-4056-8176-14f7205b2d99\" (UID: \"38d1e68e-f404-4056-8176-14f7205b2d99\") "
Oct 31 19:44:26.218768 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:26.218744    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac45690b-da7e-4ae0-81dc-6cf69551f665-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac45690b-da7e-4ae0-81dc-6cf69551f665-plugins-config-volume\") pod \"ac45690b-da7e-4ae0-81dc-6cf69551f665\" (UID: \"ac45690b-da7e-4ae0-81dc-6cf69551f665\") "
Oct 31 19:44:26.219255 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:26.219028    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38d1e68e-f404-4056-8176-14f7205b2d99-plugins-config-volume" pod "38d1e68e-f404-4056-8176-14f7205b2d99" (UID: "38d1e68e-f404-4056-8176-14f7205b2d99"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:44:26.219393 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:26.219276    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac45690b-da7e-4ae0-81dc-6cf69551f665-plugins-config-volume" pod "ac45690b-da7e-4ae0-81dc-6cf69551f665" (UID: "ac45690b-da7e-4ae0-81dc-6cf69551f665"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:44:26.219603 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:26.219579    4645 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38d1e68e-f404-4056-8176-14f7205b2d99-plugins-config-volume\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:44:26.219699 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:26.219610    4645 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac45690b-da7e-4ae0-81dc-6cf69551f665-plugins-config-volume\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:44:26.220686 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:26.220654    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac45690b-da7e-4ae0-81dc-6cf69551f665-kube-api-access-2m49z" pod "ac45690b-da7e-4ae0-81dc-6cf69551f665" (UID: "ac45690b-da7e-4ae0-81dc-6cf69551f665"). InnerVolumeSpecName "kube-api-access-2m49z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:44:26.220757 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:26.220698    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38d1e68e-f404-4056-8176-14f7205b2d99-kube-api-access-lc7pc" pod "38d1e68e-f404-4056-8176-14f7205b2d99" (UID: "38d1e68e-f404-4056-8176-14f7205b2d99"). InnerVolumeSpecName "kube-api-access-lc7pc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:44:26.320824 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:26.320774    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2m49z\" (UniqueName: \"kubernetes.io/projected/ac45690b-da7e-4ae0-81dc-6cf69551f665-kube-api-access-2m49z\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:44:26.320824 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:26.320796    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lc7pc\" (UniqueName: \"kubernetes.io/projected/38d1e68e-f404-4056-8176-14f7205b2d99-kube-api-access-lc7pc\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:44:26.425056 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:26.425027    4645 generic.go:356] "Generic (PLEG): container finished" podID="ac45690b-da7e-4ae0-81dc-6cf69551f665" containerID="e32773b97d5d99ffc8762171ac6ac60ba26ab561c496c3ab56ed1932366744c0" exitCode=0
Oct 31 19:44:26.425188 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:26.425079    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a813e2-dbc7795cc-vqxqm"
Oct 31 19:44:26.425188 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:26.425110    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a813e2-dbc7795cc-vqxqm" event={"ID":"ac45690b-da7e-4ae0-81dc-6cf69551f665","Type":"ContainerDied","Data":"e32773b97d5d99ffc8762171ac6ac60ba26ab561c496c3ab56ed1932366744c0"}
Oct 31 19:44:26.425188 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:26.425160    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a813e2-dbc7795cc-vqxqm" event={"ID":"ac45690b-da7e-4ae0-81dc-6cf69551f665","Type":"ContainerDied","Data":"5b6b37ed591c4ceac1860f13721b1953c61e941111cffd3dcdd97eb1aae2a96d"}
Oct 31 19:44:26.425188 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:26.425182    4645 scope.go:122] "RemoveContainer" containerID="e32773b97d5d99ffc8762171ac6ac60ba26ab561c496c3ab56ed1932366744c0"
Oct 31 19:44:26.426830 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:26.426804    4645 generic.go:356] "Generic (PLEG): container finished" podID="38d1e68e-f404-4056-8176-14f7205b2d99" containerID="effefd88c5075c37e2c3ffacde53f5d3461716d84262bd8d2d48d9a933d9bff3" exitCode=0
Oct 31 19:44:26.426944 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:26.426838    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a813e2-747f99fd79-prchw"
Oct 31 19:44:26.426944 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:26.426874    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a813e2-747f99fd79-prchw" event={"ID":"38d1e68e-f404-4056-8176-14f7205b2d99","Type":"ContainerDied","Data":"effefd88c5075c37e2c3ffacde53f5d3461716d84262bd8d2d48d9a933d9bff3"}
Oct 31 19:44:26.426944 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:26.426916    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a813e2-747f99fd79-prchw" event={"ID":"38d1e68e-f404-4056-8176-14f7205b2d99","Type":"ContainerDied","Data":"7d020be5d14a28d8ac8318d9f41a6c7e292d3e44591996c281bc5dacb179353b"}
Oct 31 19:44:26.437677 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:26.437622    4645 scope.go:122] "RemoveContainer" containerID="e32773b97d5d99ffc8762171ac6ac60ba26ab561c496c3ab56ed1932366744c0"
Oct 31 19:44:26.438043 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:44:26.438012    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e32773b97d5d99ffc8762171ac6ac60ba26ab561c496c3ab56ed1932366744c0\": container with ID starting with e32773b97d5d99ffc8762171ac6ac60ba26ab561c496c3ab56ed1932366744c0 not found: ID does not exist" containerID="e32773b97d5d99ffc8762171ac6ac60ba26ab561c496c3ab56ed1932366744c0"
Oct 31 19:44:26.438154 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:26.438046    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e32773b97d5d99ffc8762171ac6ac60ba26ab561c496c3ab56ed1932366744c0"} err="failed to get container status \"e32773b97d5d99ffc8762171ac6ac60ba26ab561c496c3ab56ed1932366744c0\": rpc error: code = NotFound desc = could not find container \"e32773b97d5d99ffc8762171ac6ac60ba26ab561c496c3ab56ed1932366744c0\": container with ID starting with e32773b97d5d99ffc8762171ac6ac60ba26ab561c496c3ab56ed1932366744c0 not found: ID does not exist"
Oct 31 19:44:26.438154 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:26.438068    4645 scope.go:122] "RemoveContainer" containerID="effefd88c5075c37e2c3ffacde53f5d3461716d84262bd8d2d48d9a933d9bff3"
Oct 31 19:44:26.451082 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:26.451064    4645 scope.go:122] "RemoveContainer" containerID="effefd88c5075c37e2c3ffacde53f5d3461716d84262bd8d2d48d9a933d9bff3"
Oct 31 19:44:26.451457 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:44:26.451395    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"effefd88c5075c37e2c3ffacde53f5d3461716d84262bd8d2d48d9a933d9bff3\": container with ID starting with effefd88c5075c37e2c3ffacde53f5d3461716d84262bd8d2d48d9a933d9bff3 not found: ID does not exist" containerID="effefd88c5075c37e2c3ffacde53f5d3461716d84262bd8d2d48d9a933d9bff3"
Oct 31 19:44:26.451457 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:26.451426    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"effefd88c5075c37e2c3ffacde53f5d3461716d84262bd8d2d48d9a933d9bff3"} err="failed to get container status \"effefd88c5075c37e2c3ffacde53f5d3461716d84262bd8d2d48d9a933d9bff3\": rpc error: code = NotFound desc = could not find container \"effefd88c5075c37e2c3ffacde53f5d3461716d84262bd8d2d48d9a933d9bff3\": container with ID starting with effefd88c5075c37e2c3ffacde53f5d3461716d84262bd8d2d48d9a933d9bff3 not found: ID does not exist"
Oct 31 19:44:26.460431 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:26.460410    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a813e2-dbc7795cc-vqxqm"]
Oct 31 19:44:26.464745 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:26.464714    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a813e2-dbc7795cc-vqxqm"]
Oct 31 19:44:26.481178 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:26.481156    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a813e2-747f99fd79-prchw"]
Oct 31 19:44:26.486095 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:26.486073    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a813e2-747f99fd79-prchw"]
Oct 31 19:44:26.521192 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:26.521163    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:26.762119 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:26.762086    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-b1fae7/test-model-b1fae7-kserve-7c67554bff-4dwfc"]
Oct 31 19:44:26.763376 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:26.763354    4645 cpu_manager.go:414] "containerMap: removing container" podUID="ac45690b-da7e-4ae0-81dc-6cf69551f665" containerName="payload-processing"
Oct 31 19:44:26.763444 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:26.763378    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac45690b-da7e-4ae0-81dc-6cf69551f665" containerName="payload-processing"
Oct 31 19:44:26.763481 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:26.763474    4645 cpu_manager.go:414] "containerMap: removing container" podUID="38d1e68e-f404-4056-8176-14f7205b2d99" containerName="payload-pre-processing"
Oct 31 19:44:26.763534 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:26.763483    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38d1e68e-f404-4056-8176-14f7205b2d99" containerName="payload-pre-processing"
Oct 31 19:44:26.763712 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:26.763696    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="773450c2-8daa-4448-a78a-8ede38d60d69" containerName="istio-proxy"
Oct 31 19:44:26.763750 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:26.763718    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="38d1e68e-f404-4056-8176-14f7205b2d99" containerName="payload-pre-processing"
Oct 31 19:44:26.763750 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:26.763736    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac45690b-da7e-4ae0-81dc-6cf69551f665" containerName="payload-processing"
Oct 31 19:44:26.769509 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:26.769487    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b1fae7/test-model-b1fae7-kserve-7c67554bff-4dwfc"
Oct 31 19:44:26.776737 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:26.776640    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-b1fae7\"/\"kube-root-ca.crt\""
Oct 31 19:44:26.776737 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:26.776701    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-b1fae7\"/\"openshift-service-ca.crt\""
Oct 31 19:44:26.778493 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:26.777452    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-b1fae7\"/\"test-model-b1fae7-kserve-self-signed-certs\""
Oct 31 19:44:26.778493 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:26.777532    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-b1fae7\"/\"default-dockercfg-bgztl\""
Oct 31 19:44:26.784512 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:26.784490    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-b1fae7/test-model-b1fae7-kserve-7c67554bff-4dwfc"]
Oct 31 19:44:26.827029 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:26.826991    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:26.830524 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:26.830501    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e692773b-183e-4265-a650-edab04468a46-tls-certs\") pod \"test-model-b1fae7-kserve-7c67554bff-4dwfc\" (UID: \"e692773b-183e-4265-a650-edab04468a46\") " pod="ai-tenant-e2e-inf-a-b1fae7/test-model-b1fae7-kserve-7c67554bff-4dwfc"
Oct 31 19:44:26.830789 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:26.830766    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e692773b-183e-4265-a650-edab04468a46-tmp-dir\") pod \"test-model-b1fae7-kserve-7c67554bff-4dwfc\" (UID: \"e692773b-183e-4265-a650-edab04468a46\") " pod="ai-tenant-e2e-inf-a-b1fae7/test-model-b1fae7-kserve-7c67554bff-4dwfc"
Oct 31 19:44:26.830919 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:26.830892    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qhf2\" (UniqueName: \"kubernetes.io/projected/e692773b-183e-4265-a650-edab04468a46-kube-api-access-7qhf2\") pod \"test-model-b1fae7-kserve-7c67554bff-4dwfc\" (UID: \"e692773b-183e-4265-a650-edab04468a46\") " pod="ai-tenant-e2e-inf-a-b1fae7/test-model-b1fae7-kserve-7c67554bff-4dwfc"
Oct 31 19:44:26.831043 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:26.830943    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e692773b-183e-4265-a650-edab04468a46-home\") pod \"test-model-b1fae7-kserve-7c67554bff-4dwfc\" (UID: \"e692773b-183e-4265-a650-edab04468a46\") " pod="ai-tenant-e2e-inf-a-b1fae7/test-model-b1fae7-kserve-7c67554bff-4dwfc"
Oct 31 19:44:26.831043 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:26.830964    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e692773b-183e-4265-a650-edab04468a46-dshm\") pod \"test-model-b1fae7-kserve-7c67554bff-4dwfc\" (UID: \"e692773b-183e-4265-a650-edab04468a46\") " pod="ai-tenant-e2e-inf-a-b1fae7/test-model-b1fae7-kserve-7c67554bff-4dwfc"
Oct 31 19:44:26.831172 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:26.831059    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e692773b-183e-4265-a650-edab04468a46-model-cache\") pod \"test-model-b1fae7-kserve-7c67554bff-4dwfc\" (UID: \"e692773b-183e-4265-a650-edab04468a46\") " pod="ai-tenant-e2e-inf-a-b1fae7/test-model-b1fae7-kserve-7c67554bff-4dwfc"
Oct 31 19:44:26.932608 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:26.932579    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e692773b-183e-4265-a650-edab04468a46-tls-certs\") pod \"test-model-b1fae7-kserve-7c67554bff-4dwfc\" (UID: \"e692773b-183e-4265-a650-edab04468a46\") " pod="ai-tenant-e2e-inf-a-b1fae7/test-model-b1fae7-kserve-7c67554bff-4dwfc"
Oct 31 19:44:26.932758 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:26.932743    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e692773b-183e-4265-a650-edab04468a46-tmp-dir\") pod \"test-model-b1fae7-kserve-7c67554bff-4dwfc\" (UID: \"e692773b-183e-4265-a650-edab04468a46\") " pod="ai-tenant-e2e-inf-a-b1fae7/test-model-b1fae7-kserve-7c67554bff-4dwfc"
Oct 31 19:44:26.932846 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:26.932829    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qhf2\" (UniqueName: \"kubernetes.io/projected/e692773b-183e-4265-a650-edab04468a46-kube-api-access-7qhf2\") pod \"test-model-b1fae7-kserve-7c67554bff-4dwfc\" (UID: \"e692773b-183e-4265-a650-edab04468a46\") " pod="ai-tenant-e2e-inf-a-b1fae7/test-model-b1fae7-kserve-7c67554bff-4dwfc"
Oct 31 19:44:26.932894 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:26.932880    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e692773b-183e-4265-a650-edab04468a46-home\") pod \"test-model-b1fae7-kserve-7c67554bff-4dwfc\" (UID: \"e692773b-183e-4265-a650-edab04468a46\") " pod="ai-tenant-e2e-inf-a-b1fae7/test-model-b1fae7-kserve-7c67554bff-4dwfc"
Oct 31 19:44:26.933031 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:26.932998    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e692773b-183e-4265-a650-edab04468a46-dshm\") pod \"test-model-b1fae7-kserve-7c67554bff-4dwfc\" (UID: \"e692773b-183e-4265-a650-edab04468a46\") " pod="ai-tenant-e2e-inf-a-b1fae7/test-model-b1fae7-kserve-7c67554bff-4dwfc"
Oct 31 19:44:26.933136 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:26.933058    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e692773b-183e-4265-a650-edab04468a46-model-cache\") pod \"test-model-b1fae7-kserve-7c67554bff-4dwfc\" (UID: \"e692773b-183e-4265-a650-edab04468a46\") " pod="ai-tenant-e2e-inf-a-b1fae7/test-model-b1fae7-kserve-7c67554bff-4dwfc"
Oct 31 19:44:26.933199 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:26.933172    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e692773b-183e-4265-a650-edab04468a46-tmp-dir\") pod \"test-model-b1fae7-kserve-7c67554bff-4dwfc\" (UID: \"e692773b-183e-4265-a650-edab04468a46\") " pod="ai-tenant-e2e-inf-a-b1fae7/test-model-b1fae7-kserve-7c67554bff-4dwfc"
Oct 31 19:44:26.933252 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:26.933221    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e692773b-183e-4265-a650-edab04468a46-home\") pod \"test-model-b1fae7-kserve-7c67554bff-4dwfc\" (UID: \"e692773b-183e-4265-a650-edab04468a46\") " pod="ai-tenant-e2e-inf-a-b1fae7/test-model-b1fae7-kserve-7c67554bff-4dwfc"
Oct 31 19:44:26.933410 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:26.933389    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e692773b-183e-4265-a650-edab04468a46-model-cache\") pod \"test-model-b1fae7-kserve-7c67554bff-4dwfc\" (UID: \"e692773b-183e-4265-a650-edab04468a46\") " pod="ai-tenant-e2e-inf-a-b1fae7/test-model-b1fae7-kserve-7c67554bff-4dwfc"
Oct 31 19:44:26.935245 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:26.935225    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e692773b-183e-4265-a650-edab04468a46-dshm\") pod \"test-model-b1fae7-kserve-7c67554bff-4dwfc\" (UID: \"e692773b-183e-4265-a650-edab04468a46\") " pod="ai-tenant-e2e-inf-a-b1fae7/test-model-b1fae7-kserve-7c67554bff-4dwfc"
Oct 31 19:44:26.935423 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:26.935404    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e692773b-183e-4265-a650-edab04468a46-tls-certs\") pod \"test-model-b1fae7-kserve-7c67554bff-4dwfc\" (UID: \"e692773b-183e-4265-a650-edab04468a46\") " pod="ai-tenant-e2e-inf-a-b1fae7/test-model-b1fae7-kserve-7c67554bff-4dwfc"
Oct 31 19:44:26.998351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:26.997031    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qhf2\" (UniqueName: \"kubernetes.io/projected/e692773b-183e-4265-a650-edab04468a46-kube-api-access-7qhf2\") pod \"test-model-b1fae7-kserve-7c67554bff-4dwfc\" (UID: \"e692773b-183e-4265-a650-edab04468a46\") " pod="ai-tenant-e2e-inf-a-b1fae7/test-model-b1fae7-kserve-7c67554bff-4dwfc"
Oct 31 19:44:27.023671 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:27.023580    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a813e2-747f99fd79-prchw" podUID="38d1e68e-f404-4056-8176-14f7205b2d99" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.133:50052\" within 1s: context deadline exceeded"
Oct 31 19:44:27.111411 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:27.111278    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b1fae7/test-model-b1fae7-kserve-7c67554bff-4dwfc"
Oct 31 19:44:27.427768 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:44:27.427732    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode692773b_183e_4265_a650_edab04468a46.slice/crio-e9c054d1c89802997af8a1028e49cbf954f573685ffb7bb88ca6dbc5f02d0c9a WatchSource:0}: Error finding container e9c054d1c89802997af8a1028e49cbf954f573685ffb7bb88ca6dbc5f02d0c9a: Status 404 returned error can't find the container with id e9c054d1c89802997af8a1028e49cbf954f573685ffb7bb88ca6dbc5f02d0c9a
Oct 31 19:44:27.427932 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:27.427910    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-b1fae7/test-model-b1fae7-kserve-7c67554bff-4dwfc"]
Oct 31 19:44:27.522965 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:27.522936    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:27.628268 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:27.628233    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38d1e68e-f404-4056-8176-14f7205b2d99" path="/var/lib/kubelet/pods/38d1e68e-f404-4056-8176-14f7205b2d99/volumes"
Oct 31 19:44:27.628820 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:27.628804    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac45690b-da7e-4ae0-81dc-6cf69551f665" path="/var/lib/kubelet/pods/ac45690b-da7e-4ae0-81dc-6cf69551f665/volumes"
Oct 31 19:44:28.449310 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:28.449275    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b1fae7/test-model-b1fae7-kserve-7c67554bff-4dwfc" event={"ID":"e692773b-183e-4265-a650-edab04468a46","Type":"ContainerStarted","Data":"80baa86f6f84f41f15fbf8e9e6569c2e9cfa1ad1e6dee063ae0d1fa9229fa792"}
Oct 31 19:44:28.449734 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:28.449321    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b1fae7/test-model-b1fae7-kserve-7c67554bff-4dwfc" event={"ID":"e692773b-183e-4265-a650-edab04468a46","Type":"ContainerStarted","Data":"e9c054d1c89802997af8a1028e49cbf954f573685ffb7bb88ca6dbc5f02d0c9a"}
Oct 31 19:44:28.455324 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:28.455282    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:28.473906 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:28.473856    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-b1fae7/test-model-b1fae7-kserve-7c67554bff-4dwfc" podStartSLOduration=2.4738435770000002 podStartE2EDuration="2.473843577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:44:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:44:28.470137862 +0000 UTC m=+1499.043371993" watchObservedRunningTime="2026-10-31 19:44:28.473843577 +0000 UTC m=+1499.047077706"
Oct 31 19:44:29.111860 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:29.111831    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-b1fae7/test-model-b1fae7-kserve-7c67554bff-4dwfc"
Oct 31 19:44:29.285376 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:29.285312    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:30.038124 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:30.038085    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:30.892213 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:30.892163    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:31.433041 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:31.433010    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8dcfa8-6c7cb4c95f-cws7q"
Oct 31 19:44:31.649787 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:31.649751    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:32.457900 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:32.457864    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:33.433816 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:33.433765    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:34.142398 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:34.142364    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:35.112994 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:35.112956    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:35.878958 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:35.878910    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:36.247627 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:44:36.247587    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 19:44:36.632248 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:36.632156    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:37.111789 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:37.111760    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-b1fae7/test-model-b1fae7-kserve-7c67554bff-4dwfc"
Oct 31 19:44:37.124240 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:37.124218    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-b1fae7/test-model-b1fae7-kserve-7c67554bff-4dwfc"
Oct 31 19:44:37.552546 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:37.552519    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-b1fae7/test-model-b1fae7-kserve-7c67554bff-4dwfc"
Oct 31 19:44:37.652874 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:37.652845    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:38.340738 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:38.340708    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:39.535149 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:39.535102    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:40.981357 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:40.968342    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:41.228419 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:41.228380    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-f8c3f7/test-model-f8c3f7-kserve-6fccbd6557-ts9sg"]
Oct 31 19:44:41.337411 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:41.301781    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f8c3f7/test-model-f8c3f7-kserve-6fccbd6557-ts9sg"
Oct 31 19:44:41.364222 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:41.337653    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:41.379388 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:41.377942    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-f8c3f7\"/\"default-dockercfg-4rbzx\""
Oct 31 19:44:41.379388 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:41.378484    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-f8c3f7\"/\"test-model-f8c3f7-kserve-self-signed-certs\""
Oct 31 19:44:41.379388 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:41.379229    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-f8c3f7\"/\"openshift-service-ca.crt\""
Oct 31 19:44:41.397598 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:41.381658    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-f8c3f7\"/\"kube-root-ca.crt\""
Oct 31 19:44:41.397598 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:41.392287    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e6b6da3-9983-463c-a814-e07d9a22ed1f-tmp-dir\") pod \"test-model-f8c3f7-kserve-6fccbd6557-ts9sg\" (UID: \"2e6b6da3-9983-463c-a814-e07d9a22ed1f\") " pod="ai-tenant-e2e-inf-b-f8c3f7/test-model-f8c3f7-kserve-6fccbd6557-ts9sg"
Oct 31 19:44:41.397598 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:41.392688    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e6b6da3-9983-463c-a814-e07d9a22ed1f-home\") pod \"test-model-f8c3f7-kserve-6fccbd6557-ts9sg\" (UID: \"2e6b6da3-9983-463c-a814-e07d9a22ed1f\") " pod="ai-tenant-e2e-inf-b-f8c3f7/test-model-f8c3f7-kserve-6fccbd6557-ts9sg"
Oct 31 19:44:41.397598 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:41.392818    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5fsp\" (UniqueName: \"kubernetes.io/projected/2e6b6da3-9983-463c-a814-e07d9a22ed1f-kube-api-access-m5fsp\") pod \"test-model-f8c3f7-kserve-6fccbd6557-ts9sg\" (UID: \"2e6b6da3-9983-463c-a814-e07d9a22ed1f\") " pod="ai-tenant-e2e-inf-b-f8c3f7/test-model-f8c3f7-kserve-6fccbd6557-ts9sg"
Oct 31 19:44:41.397598 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:41.393031    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e6b6da3-9983-463c-a814-e07d9a22ed1f-model-cache\") pod \"test-model-f8c3f7-kserve-6fccbd6557-ts9sg\" (UID: \"2e6b6da3-9983-463c-a814-e07d9a22ed1f\") " pod="ai-tenant-e2e-inf-b-f8c3f7/test-model-f8c3f7-kserve-6fccbd6557-ts9sg"
Oct 31 19:44:41.397598 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:41.393142    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e6b6da3-9983-463c-a814-e07d9a22ed1f-dshm\") pod \"test-model-f8c3f7-kserve-6fccbd6557-ts9sg\" (UID: \"2e6b6da3-9983-463c-a814-e07d9a22ed1f\") " pod="ai-tenant-e2e-inf-b-f8c3f7/test-model-f8c3f7-kserve-6fccbd6557-ts9sg"
Oct 31 19:44:41.397598 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:41.393265    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e6b6da3-9983-463c-a814-e07d9a22ed1f-tls-certs\") pod \"test-model-f8c3f7-kserve-6fccbd6557-ts9sg\" (UID: \"2e6b6da3-9983-463c-a814-e07d9a22ed1f\") " pod="ai-tenant-e2e-inf-b-f8c3f7/test-model-f8c3f7-kserve-6fccbd6557-ts9sg"
Oct 31 19:44:41.509357 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:41.505222    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e6b6da3-9983-463c-a814-e07d9a22ed1f-dshm\") pod \"test-model-f8c3f7-kserve-6fccbd6557-ts9sg\" (UID: \"2e6b6da3-9983-463c-a814-e07d9a22ed1f\") " pod="ai-tenant-e2e-inf-b-f8c3f7/test-model-f8c3f7-kserve-6fccbd6557-ts9sg"
Oct 31 19:44:41.509357 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:41.505303    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e6b6da3-9983-463c-a814-e07d9a22ed1f-tls-certs\") pod \"test-model-f8c3f7-kserve-6fccbd6557-ts9sg\" (UID: \"2e6b6da3-9983-463c-a814-e07d9a22ed1f\") " pod="ai-tenant-e2e-inf-b-f8c3f7/test-model-f8c3f7-kserve-6fccbd6557-ts9sg"
Oct 31 19:44:41.509357 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:41.505447    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e6b6da3-9983-463c-a814-e07d9a22ed1f-tmp-dir\") pod \"test-model-f8c3f7-kserve-6fccbd6557-ts9sg\" (UID: \"2e6b6da3-9983-463c-a814-e07d9a22ed1f\") " pod="ai-tenant-e2e-inf-b-f8c3f7/test-model-f8c3f7-kserve-6fccbd6557-ts9sg"
Oct 31 19:44:41.509357 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:41.505735    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e6b6da3-9983-463c-a814-e07d9a22ed1f-home\") pod \"test-model-f8c3f7-kserve-6fccbd6557-ts9sg\" (UID: \"2e6b6da3-9983-463c-a814-e07d9a22ed1f\") " pod="ai-tenant-e2e-inf-b-f8c3f7/test-model-f8c3f7-kserve-6fccbd6557-ts9sg"
Oct 31 19:44:41.509357 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:41.505806    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5fsp\" (UniqueName: \"kubernetes.io/projected/2e6b6da3-9983-463c-a814-e07d9a22ed1f-kube-api-access-m5fsp\") pod \"test-model-f8c3f7-kserve-6fccbd6557-ts9sg\" (UID: \"2e6b6da3-9983-463c-a814-e07d9a22ed1f\") " pod="ai-tenant-e2e-inf-b-f8c3f7/test-model-f8c3f7-kserve-6fccbd6557-ts9sg"
Oct 31 19:44:41.526863 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:41.526829    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e6b6da3-9983-463c-a814-e07d9a22ed1f-tmp-dir\") pod \"test-model-f8c3f7-kserve-6fccbd6557-ts9sg\" (UID: \"2e6b6da3-9983-463c-a814-e07d9a22ed1f\") " pod="ai-tenant-e2e-inf-b-f8c3f7/test-model-f8c3f7-kserve-6fccbd6557-ts9sg"
Oct 31 19:44:41.580924 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:41.573488    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e6b6da3-9983-463c-a814-e07d9a22ed1f-home\") pod \"test-model-f8c3f7-kserve-6fccbd6557-ts9sg\" (UID: \"2e6b6da3-9983-463c-a814-e07d9a22ed1f\") " pod="ai-tenant-e2e-inf-b-f8c3f7/test-model-f8c3f7-kserve-6fccbd6557-ts9sg"
Oct 31 19:44:41.650273 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:41.650194    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e6b6da3-9983-463c-a814-e07d9a22ed1f-dshm\") pod \"test-model-f8c3f7-kserve-6fccbd6557-ts9sg\" (UID: \"2e6b6da3-9983-463c-a814-e07d9a22ed1f\") " pod="ai-tenant-e2e-inf-b-f8c3f7/test-model-f8c3f7-kserve-6fccbd6557-ts9sg"
Oct 31 19:44:41.700444 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:41.700402    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5fsp\" (UniqueName: \"kubernetes.io/projected/2e6b6da3-9983-463c-a814-e07d9a22ed1f-kube-api-access-m5fsp\") pod \"test-model-f8c3f7-kserve-6fccbd6557-ts9sg\" (UID: \"2e6b6da3-9983-463c-a814-e07d9a22ed1f\") " pod="ai-tenant-e2e-inf-b-f8c3f7/test-model-f8c3f7-kserve-6fccbd6557-ts9sg"
Oct 31 19:44:41.754914 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:41.754867    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e6b6da3-9983-463c-a814-e07d9a22ed1f-model-cache\") pod \"test-model-f8c3f7-kserve-6fccbd6557-ts9sg\" (UID: \"2e6b6da3-9983-463c-a814-e07d9a22ed1f\") " pod="ai-tenant-e2e-inf-b-f8c3f7/test-model-f8c3f7-kserve-6fccbd6557-ts9sg"
Oct 31 19:44:41.756008 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:41.755966    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e6b6da3-9983-463c-a814-e07d9a22ed1f-model-cache\") pod \"test-model-f8c3f7-kserve-6fccbd6557-ts9sg\" (UID: \"2e6b6da3-9983-463c-a814-e07d9a22ed1f\") " pod="ai-tenant-e2e-inf-b-f8c3f7/test-model-f8c3f7-kserve-6fccbd6557-ts9sg"
Oct 31 19:44:41.821167 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:41.821106    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e6b6da3-9983-463c-a814-e07d9a22ed1f-tls-certs\") pod \"test-model-f8c3f7-kserve-6fccbd6557-ts9sg\" (UID: \"2e6b6da3-9983-463c-a814-e07d9a22ed1f\") " pod="ai-tenant-e2e-inf-b-f8c3f7/test-model-f8c3f7-kserve-6fccbd6557-ts9sg"
Oct 31 19:44:41.854353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:41.849606    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-f8c3f7/test-model-f8c3f7-kserve-6fccbd6557-ts9sg"]
Oct 31 19:44:42.047350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:42.046393    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f8c3f7/test-model-f8c3f7-kserve-6fccbd6557-ts9sg"
Oct 31 19:44:42.280356 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:42.273529    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-9efeae/test-model-9efeae-kserve-7d97f79d8f-2nnrl" podUID="72f178d8-b759-4bf3-90fa-1b0a1a8cb51e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.154:8000/ready\": context deadline exceeded"
Oct 31 19:44:42.532401 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:42.531686    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669" podUID="9285bee0-bf7c-4fd4-9da7-a1db021cae05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:44:42.557645 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:42.557606    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw" podUID="1287db54-a587-4c11-9eb6-1606de555894" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:44:43.074353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:43.069199    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l" podUID="12bea86c-61e5-465e-82fb-b4673a4d1be9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:44:43.115380 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:43.104935    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:43.115380 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:44:43.105596    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2e6b6da3_9983_463c_a814_e07d9a22ed1f.slice/crio-c3045f81894e4e24d257d936c668e52b3a319e50f52178e1a3bc19fd538ac3d7 WatchSource:0}: Error finding container c3045f81894e4e24d257d936c668e52b3a319e50f52178e1a3bc19fd538ac3d7: Status 404 returned error can't find the container with id c3045f81894e4e24d257d936c668e52b3a319e50f52178e1a3bc19fd538ac3d7
Oct 31 19:44:43.151862 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:43.123449    4645 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-278c04/test-model-278c04-kserve-6b6f77c967-2f8mt" podUID="beb01fd0-e706-4d84-b1a6-c41614630f39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.153:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:44:43.181588 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:43.181552    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-d45f6c/test-model-d45f6c-kserve-784b7b884c-7mgtp" podUID="16133210-4877-4f71-9d0d-fe90c29cca43" containerName="main" probeResult="failure" output="Get \"https://10.133.0.150:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:44:43.220348 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:43.203596    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-f8c3f7/test-model-f8c3f7-kserve-6fccbd6557-ts9sg"]
Oct 31 19:44:43.691636 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:43.691505    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:44.175721 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:44.164391    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f8c3f7/test-model-f8c3f7-kserve-6fccbd6557-ts9sg" event={"ID":"2e6b6da3-9983-463c-a814-e07d9a22ed1f","Type":"ContainerStarted","Data":"a47060f93eb207e0243ec596833623fd5c66cea327a8f338c70d52f0ab6fef78"}
Oct 31 19:44:44.175721 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:44.164444    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f8c3f7/test-model-f8c3f7-kserve-6fccbd6557-ts9sg" event={"ID":"2e6b6da3-9983-463c-a814-e07d9a22ed1f","Type":"ContainerStarted","Data":"c3045f81894e4e24d257d936c668e52b3a319e50f52178e1a3bc19fd538ac3d7"}
Oct 31 19:44:44.204266 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:44.204216    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-f8c3f7/test-model-f8c3f7-kserve-6fccbd6557-ts9sg" podStartSLOduration=3.20420182 podStartE2EDuration="3.20420182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:44:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:44:44.198623921 +0000 UTC m=+1514.771858052" watchObservedRunningTime="2026-10-31 19:44:44.20420182 +0000 UTC m=+1514.777435949"
Oct 31 19:44:44.845660 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:44.845624    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:45.047018 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:45.046991    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-f8c3f7/test-model-f8c3f7-kserve-6fccbd6557-ts9sg"
Oct 31 19:44:45.548692 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:45.548658    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:46.353989 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:46.353923    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:47.330663 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:47.330622    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:48.159232 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:48.159192    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:49.127129 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:49.127044    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:50.235659 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:50.235626    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:51.205864 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:51.205816    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:51.400076 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:51.400037    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:52.047293 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:52.047191    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-f8c3f7/test-model-f8c3f7-kserve-6fccbd6557-ts9sg"
Oct 31 19:44:52.060412 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:52.060363    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-f8c3f7/test-model-f8c3f7-kserve-6fccbd6557-ts9sg"
Oct 31 19:44:52.234960 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:52.234928    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:52.332179 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:52.332083    4645 generic.go:356] "Generic (PLEG): container finished" podID="76b3b657-6b14-43ff-b7ce-1e9e3b615141" containerID="de1325c6e76563fc630e2eec234bff76121d6e736f691e783aa15922c2597480" exitCode=0
Oct 31 19:44:52.332348 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:52.332152    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2" event={"ID":"76b3b657-6b14-43ff-b7ce-1e9e3b615141","Type":"ContainerDied","Data":"de1325c6e76563fc630e2eec234bff76121d6e736f691e783aa15922c2597480"}
Oct 31 19:44:52.332717 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:52.332693    4645 scope.go:122] "RemoveContainer" containerID="de1325c6e76563fc630e2eec234bff76121d6e736f691e783aa15922c2597480"
Oct 31 19:44:52.334861 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:52.334828    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:44:52.334959 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:52.334907    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:44:52.334959 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:52.334943    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:44:52.344628 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:52.344596    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-f8c3f7/test-model-f8c3f7-kserve-6fccbd6557-ts9sg"
Oct 31 19:44:52.469264 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:52.469233    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:53.370357 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:53.364604    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2" event={"ID":"76b3b657-6b14-43ff-b7ce-1e9e3b615141","Type":"ContainerStarted","Data":"fbec275cf9320e13f885a9f0ff45095d5e7c5b2aac8ba10f9d7c72adbf5e195a"}
Oct 31 19:44:53.427864 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:53.427812    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:53.535375 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:53.535289    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:53.922741 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:53.922702    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:54.099716 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:54.099678    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2"
Oct 31 19:44:54.101151 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:54.101119    4645 patch_prober.go:29] interesting pod/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 31 19:44:54.101319 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:54.101176    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2" podUID="76b3b657-6b14-43ff-b7ce-1e9e3b615141" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 31 19:44:54.196379 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:54.196256    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-1e8cf0/test-model-1e8cf0-kserve-74747d94df-4gd76"]
Oct 31 19:44:54.204686 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:54.204645    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1e8cf0/test-model-1e8cf0-kserve-74747d94df-4gd76"
Oct 31 19:44:54.210497 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:54.210462    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-1e8cf0/test-model-1e8cf0-kserve-74747d94df-4gd76"]
Oct 31 19:44:54.221166 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:54.221137    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-1e8cf0\"/\"openshift-service-ca.crt\""
Oct 31 19:44:54.221705 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:54.221274    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-1e8cf0\"/\"kube-root-ca.crt\""
Oct 31 19:44:54.221705 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:54.221494    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-1e8cf0\"/\"default-dockercfg-bpskx\""
Oct 31 19:44:54.221705 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:54.221576    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-1e8cf0\"/\"test-model-1e8cf0-kserve-self-signed-certs\""
Oct 31 19:44:54.284413 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:54.283626    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:54.305584 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:54.305528    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b673f934-bc6d-4ae8-8adf-4acedf4a8df6-home\") pod \"test-model-1e8cf0-kserve-74747d94df-4gd76\" (UID: \"b673f934-bc6d-4ae8-8adf-4acedf4a8df6\") " pod="ai-tenant-e2e-inf-a-1e8cf0/test-model-1e8cf0-kserve-74747d94df-4gd76"
Oct 31 19:44:54.305785 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:54.305678    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45gc7\" (UniqueName: \"kubernetes.io/projected/b673f934-bc6d-4ae8-8adf-4acedf4a8df6-kube-api-access-45gc7\") pod \"test-model-1e8cf0-kserve-74747d94df-4gd76\" (UID: \"b673f934-bc6d-4ae8-8adf-4acedf4a8df6\") " pod="ai-tenant-e2e-inf-a-1e8cf0/test-model-1e8cf0-kserve-74747d94df-4gd76"
Oct 31 19:44:54.305785 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:54.305722    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b673f934-bc6d-4ae8-8adf-4acedf4a8df6-dshm\") pod \"test-model-1e8cf0-kserve-74747d94df-4gd76\" (UID: \"b673f934-bc6d-4ae8-8adf-4acedf4a8df6\") " pod="ai-tenant-e2e-inf-a-1e8cf0/test-model-1e8cf0-kserve-74747d94df-4gd76"
Oct 31 19:44:54.305881 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:54.305817    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b673f934-bc6d-4ae8-8adf-4acedf4a8df6-model-cache\") pod \"test-model-1e8cf0-kserve-74747d94df-4gd76\" (UID: \"b673f934-bc6d-4ae8-8adf-4acedf4a8df6\") " pod="ai-tenant-e2e-inf-a-1e8cf0/test-model-1e8cf0-kserve-74747d94df-4gd76"
Oct 31 19:44:54.305881 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:54.305860    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b673f934-bc6d-4ae8-8adf-4acedf4a8df6-tls-certs\") pod \"test-model-1e8cf0-kserve-74747d94df-4gd76\" (UID: \"b673f934-bc6d-4ae8-8adf-4acedf4a8df6\") " pod="ai-tenant-e2e-inf-a-1e8cf0/test-model-1e8cf0-kserve-74747d94df-4gd76"
Oct 31 19:44:54.305979 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:54.305903    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b673f934-bc6d-4ae8-8adf-4acedf4a8df6-tmp-dir\") pod \"test-model-1e8cf0-kserve-74747d94df-4gd76\" (UID: \"b673f934-bc6d-4ae8-8adf-4acedf4a8df6\") " pod="ai-tenant-e2e-inf-a-1e8cf0/test-model-1e8cf0-kserve-74747d94df-4gd76"
Oct 31 19:44:54.408437 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:54.408386    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45gc7\" (UniqueName: \"kubernetes.io/projected/b673f934-bc6d-4ae8-8adf-4acedf4a8df6-kube-api-access-45gc7\") pod \"test-model-1e8cf0-kserve-74747d94df-4gd76\" (UID: \"b673f934-bc6d-4ae8-8adf-4acedf4a8df6\") " pod="ai-tenant-e2e-inf-a-1e8cf0/test-model-1e8cf0-kserve-74747d94df-4gd76"
Oct 31 19:44:54.408663 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:54.408461    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b673f934-bc6d-4ae8-8adf-4acedf4a8df6-dshm\") pod \"test-model-1e8cf0-kserve-74747d94df-4gd76\" (UID: \"b673f934-bc6d-4ae8-8adf-4acedf4a8df6\") " pod="ai-tenant-e2e-inf-a-1e8cf0/test-model-1e8cf0-kserve-74747d94df-4gd76"
Oct 31 19:44:54.408663 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:54.408550    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b673f934-bc6d-4ae8-8adf-4acedf4a8df6-model-cache\") pod \"test-model-1e8cf0-kserve-74747d94df-4gd76\" (UID: \"b673f934-bc6d-4ae8-8adf-4acedf4a8df6\") " pod="ai-tenant-e2e-inf-a-1e8cf0/test-model-1e8cf0-kserve-74747d94df-4gd76"
Oct 31 19:44:54.408663 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:54.408604    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b673f934-bc6d-4ae8-8adf-4acedf4a8df6-tls-certs\") pod \"test-model-1e8cf0-kserve-74747d94df-4gd76\" (UID: \"b673f934-bc6d-4ae8-8adf-4acedf4a8df6\") " pod="ai-tenant-e2e-inf-a-1e8cf0/test-model-1e8cf0-kserve-74747d94df-4gd76"
Oct 31 19:44:54.408861 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:54.408667    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b673f934-bc6d-4ae8-8adf-4acedf4a8df6-tmp-dir\") pod \"test-model-1e8cf0-kserve-74747d94df-4gd76\" (UID: \"b673f934-bc6d-4ae8-8adf-4acedf4a8df6\") " pod="ai-tenant-e2e-inf-a-1e8cf0/test-model-1e8cf0-kserve-74747d94df-4gd76"
Oct 31 19:44:54.409134 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:54.409102    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b673f934-bc6d-4ae8-8adf-4acedf4a8df6-home\") pod \"test-model-1e8cf0-kserve-74747d94df-4gd76\" (UID: \"b673f934-bc6d-4ae8-8adf-4acedf4a8df6\") " pod="ai-tenant-e2e-inf-a-1e8cf0/test-model-1e8cf0-kserve-74747d94df-4gd76"
Oct 31 19:44:54.409987 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:54.409930    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b673f934-bc6d-4ae8-8adf-4acedf4a8df6-home\") pod \"test-model-1e8cf0-kserve-74747d94df-4gd76\" (UID: \"b673f934-bc6d-4ae8-8adf-4acedf4a8df6\") " pod="ai-tenant-e2e-inf-a-1e8cf0/test-model-1e8cf0-kserve-74747d94df-4gd76"
Oct 31 19:44:54.410248 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:54.410224    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b673f934-bc6d-4ae8-8adf-4acedf4a8df6-model-cache\") pod \"test-model-1e8cf0-kserve-74747d94df-4gd76\" (UID: \"b673f934-bc6d-4ae8-8adf-4acedf4a8df6\") " pod="ai-tenant-e2e-inf-a-1e8cf0/test-model-1e8cf0-kserve-74747d94df-4gd76"
Oct 31 19:44:54.410559 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:54.410539    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b673f934-bc6d-4ae8-8adf-4acedf4a8df6-tmp-dir\") pod \"test-model-1e8cf0-kserve-74747d94df-4gd76\" (UID: \"b673f934-bc6d-4ae8-8adf-4acedf4a8df6\") " pod="ai-tenant-e2e-inf-a-1e8cf0/test-model-1e8cf0-kserve-74747d94df-4gd76"
Oct 31 19:44:54.412960 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:54.412931    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b673f934-bc6d-4ae8-8adf-4acedf4a8df6-dshm\") pod \"test-model-1e8cf0-kserve-74747d94df-4gd76\" (UID: \"b673f934-bc6d-4ae8-8adf-4acedf4a8df6\") " pod="ai-tenant-e2e-inf-a-1e8cf0/test-model-1e8cf0-kserve-74747d94df-4gd76"
Oct 31 19:44:54.415398 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:54.415373    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b673f934-bc6d-4ae8-8adf-4acedf4a8df6-tls-certs\") pod \"test-model-1e8cf0-kserve-74747d94df-4gd76\" (UID: \"b673f934-bc6d-4ae8-8adf-4acedf4a8df6\") " pod="ai-tenant-e2e-inf-a-1e8cf0/test-model-1e8cf0-kserve-74747d94df-4gd76"
Oct 31 19:44:54.424457 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:54.424423    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45gc7\" (UniqueName: \"kubernetes.io/projected/b673f934-bc6d-4ae8-8adf-4acedf4a8df6-kube-api-access-45gc7\") pod \"test-model-1e8cf0-kserve-74747d94df-4gd76\" (UID: \"b673f934-bc6d-4ae8-8adf-4acedf4a8df6\") " pod="ai-tenant-e2e-inf-a-1e8cf0/test-model-1e8cf0-kserve-74747d94df-4gd76"
Oct 31 19:44:54.448657 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:54.448570    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:54.562935 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:54.562885    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1e8cf0/test-model-1e8cf0-kserve-74747d94df-4gd76"
Oct 31 19:44:54.948464 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:54.934477    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:55.056562 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:55.047307    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:55.102278 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:55.102240    4645 patch_prober.go:29] interesting pod/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 31 19:44:55.102798 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:55.102293    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2" podUID="76b3b657-6b14-43ff-b7ce-1e9e3b615141" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 31 19:44:55.419302 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:55.411223    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-1e8cf0/test-model-1e8cf0-kserve-74747d94df-4gd76"]
Oct 31 19:44:55.468359 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:55.467039    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:56.099574 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:56.099540    4645 patch_prober.go:29] interesting pod/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 31 19:44:56.100038 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:56.100005    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2" podUID="76b3b657-6b14-43ff-b7ce-1e9e3b615141" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 31 19:44:56.424182 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:56.424136    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1e8cf0/test-model-1e8cf0-kserve-74747d94df-4gd76" event={"ID":"b673f934-bc6d-4ae8-8adf-4acedf4a8df6","Type":"ContainerStarted","Data":"2853a7d35e514bd396357bfe4083025dd17e252065609eab05e30dd0eaacb8b2"}
Oct 31 19:44:56.424491 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:56.424467    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1e8cf0/test-model-1e8cf0-kserve-74747d94df-4gd76" event={"ID":"b673f934-bc6d-4ae8-8adf-4acedf4a8df6","Type":"ContainerStarted","Data":"5042eddc43348d7c3ccefa1c22cb551754b3c15aa3a7d9ec57f4641fa1b89d6f"}
Oct 31 19:44:56.448298 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:56.448234    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-1e8cf0/test-model-1e8cf0-kserve-74747d94df-4gd76" podStartSLOduration=2.448216636 podStartE2EDuration="2.448216636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:44:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:44:56.445970085 +0000 UTC m=+1527.019204215" watchObservedRunningTime="2026-10-31 19:44:56.448216636 +0000 UTC m=+1527.021450793"
Oct 31 19:44:56.550778 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:56.550740    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:56.580355 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:56.576406    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-1e8cf0/test-model-1e8cf0-kserve-74747d94df-4gd76"
Oct 31 19:44:57.099118 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:57.099077    4645 patch_prober.go:29] interesting pod/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 31 19:44:57.099360 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:57.099136    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2" podUID="76b3b657-6b14-43ff-b7ce-1e9e3b615141" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 31 19:44:57.640551 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:57.640514    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:57.970354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:57.966407    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2"
Oct 31 19:44:58.099387 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:58.099319    4645 patch_prober.go:29] interesting pod/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 31 19:44:58.099570 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:58.099409    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2" podUID="76b3b657-6b14-43ff-b7ce-1e9e3b615141" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 31 19:44:58.412855 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:58.412825    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:58.677403 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:58.675731    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:58.815407 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:58.815369    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:59.099647 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:59.099557    4645 patch_prober.go:29] interesting pod/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 31 19:44:59.099647 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:59.099620    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2" podUID="76b3b657-6b14-43ff-b7ce-1e9e3b615141" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 31 19:44:59.438365 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:59.438318    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:44:59.923456 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:44:59.923425    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:00.011483 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.011446    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d8c398-69fd677f-s95lc"]
Oct 31 19:45:00.016421 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.016400    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d8c398-69fd677f-s95lc"
Oct 31 19:45:00.022024 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.021997    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-d8c398-dockercfg-zv2hq\""
Oct 31 19:45:00.022502 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.022486    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-d8c398\""
Oct 31 19:45:00.028544 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.028521    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d8c398-69fd677f-s95lc"]
Oct 31 19:45:00.054696 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.054381    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:00.249467 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.244373    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcztf\" (UniqueName: \"kubernetes.io/projected/475d73a5-f2c4-4cc8-b573-b42497e3e7a9-kube-api-access-kcztf\") pod \"payload-pre-processing-e2e-inf-b-d8c398-69fd677f-s95lc\" (UID: \"475d73a5-f2c4-4cc8-b573-b42497e3e7a9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d8c398-69fd677f-s95lc"
Oct 31 19:45:00.255095 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.254640    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/475d73a5-f2c4-4cc8-b573-b42497e3e7a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d8c398-69fd677f-s95lc\" (UID: \"475d73a5-f2c4-4cc8-b573-b42497e3e7a9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d8c398-69fd677f-s95lc"
Oct 31 19:45:00.411353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.410344    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcztf\" (UniqueName: \"kubernetes.io/projected/475d73a5-f2c4-4cc8-b573-b42497e3e7a9-kube-api-access-kcztf\") pod \"payload-pre-processing-e2e-inf-b-d8c398-69fd677f-s95lc\" (UID: \"475d73a5-f2c4-4cc8-b573-b42497e3e7a9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d8c398-69fd677f-s95lc"
Oct 31 19:45:00.411353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.410703    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/475d73a5-f2c4-4cc8-b573-b42497e3e7a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d8c398-69fd677f-s95lc\" (UID: \"475d73a5-f2c4-4cc8-b573-b42497e3e7a9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d8c398-69fd677f-s95lc"
Oct 31 19:45:00.433354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.411957    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/475d73a5-f2c4-4cc8-b573-b42497e3e7a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d8c398-69fd677f-s95lc\" (UID: \"475d73a5-f2c4-4cc8-b573-b42497e3e7a9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d8c398-69fd677f-s95lc"
Oct 31 19:45:00.433354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.427115    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-435728-29891265-tfbvw"]
Oct 31 19:45:00.473350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.465647    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a891ce-29891265-md69g"]
Oct 31 19:45:00.473350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.467524    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-435728-29891265-tfbvw"
Oct 31 19:45:00.546032 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.545941    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 19:45:00.577433 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.551562    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcztf\" (UniqueName: \"kubernetes.io/projected/475d73a5-f2c4-4cc8-b573-b42497e3e7a9-kube-api-access-kcztf\") pod \"payload-pre-processing-e2e-inf-b-d8c398-69fd677f-s95lc\" (UID: \"475d73a5-f2c4-4cc8-b573-b42497e3e7a9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d8c398-69fd677f-s95lc"
Oct 31 19:45:00.590353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.583062    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-435728-29891265-tfbvw"]
Oct 31 19:45:00.590353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.583113    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a891ce-29891265-md69g"]
Oct 31 19:45:00.590353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.583234    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2"
Oct 31 19:45:00.590353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.583265    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-fd1c91-29891265-rcrqk"]
Oct 31 19:45:00.590353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.583397    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a891ce-29891265-md69g"
Oct 31 19:45:00.598838 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.598806    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53b785a3-a308-43ec-92b9-ccd7d6a8bcad-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-435728-29891265-tfbvw\" (UID: \"53b785a3-a308-43ec-92b9-ccd7d6a8bcad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-435728-29891265-tfbvw"
Oct 31 19:45:00.598981 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.598903    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28z8w\" (UniqueName: \"kubernetes.io/projected/53b785a3-a308-43ec-92b9-ccd7d6a8bcad-kube-api-access-28z8w\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-435728-29891265-tfbvw\" (UID: \"53b785a3-a308-43ec-92b9-ccd7d6a8bcad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-435728-29891265-tfbvw"
Oct 31 19:45:00.604058 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.604033    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2"
Oct 31 19:45:00.604205 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.604131    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-fd1c91-29891265-rcrqk"
Oct 31 19:45:00.613199 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.613178    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a813e2-29891265-dh4wq"]
Oct 31 19:45:00.618690 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.618670    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a813e2-29891265-dh4wq"
Oct 31 19:45:00.619243 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.619226    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891265-9vwg9"]
Oct 31 19:45:00.624125 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.624105    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-23843f-29891265-52rjc"]
Oct 31 19:45:00.624226 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.624204    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891265-9vwg9"
Oct 31 19:45:00.629198 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.629180    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891265-r6vvt"]
Oct 31 19:45:00.629312 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.629295    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-23843f-29891265-52rjc"
Oct 31 19:45:00.632964 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.632949    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d8c398-69fd677f-s95lc"
Oct 31 19:45:00.634948 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.634929    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-fd1c91-29891265-rcrqk"]
Oct 31 19:45:00.635052 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.635012    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891265-r6vvt"
Oct 31 19:45:00.637750 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.637729    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a813e2-29891265-dh4wq"]
Oct 31 19:45:00.644735 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.644716    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1e8cf0-29891265-mbhwx"]
Oct 31 19:45:00.654084 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.653572    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-785d26-29891265-8cn6g"]
Oct 31 19:45:00.657246 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.656450    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1e8cf0-29891265-mbhwx"
Oct 31 19:45:00.667556 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.667533    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8dcfa8-29891265-9lkmt"]
Oct 31 19:45:00.668396 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.668373    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-785d26-29891265-8cn6g"
Oct 31 19:45:00.691159 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.677758    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-03dfee-29891265-njm8t"]
Oct 31 19:45:00.691159 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.677865    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8dcfa8-29891265-9lkmt"
Oct 31 19:45:00.691159 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.686670    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f5c1ab-29891265-96ksc"]
Oct 31 19:45:00.691159 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.686765    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-03dfee-29891265-njm8t"
Oct 31 19:45:00.700358 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.700310    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jfnb\" (UniqueName: \"kubernetes.io/projected/220956a9-b7c5-48b8-a598-61326c85691a-kube-api-access-8jfnb\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-a891ce-29891265-md69g\" (UID: \"220956a9-b7c5-48b8-a598-61326c85691a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a891ce-29891265-md69g"
Oct 31 19:45:00.700511 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.700393    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jsfh\" (UniqueName: \"kubernetes.io/projected/769ee829-1015-45b1-9dd7-d9cfe372fa9c-kube-api-access-8jsfh\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-fd1c91-29891265-rcrqk\" (UID: \"769ee829-1015-45b1-9dd7-d9cfe372fa9c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-fd1c91-29891265-rcrqk"
Oct 31 19:45:00.700511 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.700417    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6f57036-6c45-4def-a9c1-0132ab845b3e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891265-9vwg9\" (UID: \"e6f57036-6c45-4def-a9c1-0132ab845b3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891265-9vwg9"
Oct 31 19:45:00.700511 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.700464    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/615544a2-1b13-4bce-aeb6-7b3b32e6a509-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-23843f-29891265-52rjc\" (UID: \"615544a2-1b13-4bce-aeb6-7b3b32e6a509\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-23843f-29891265-52rjc"
Oct 31 19:45:00.700511 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.700488    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66szc\" (UniqueName: \"kubernetes.io/projected/57977c65-f0c2-46d1-a05c-20bd8ed579c3-kube-api-access-66szc\") pod \"maas-api-key-cleanup-e2e-inf-a-1e8cf0-29891265-mbhwx\" (UID: \"57977c65-f0c2-46d1-a05c-20bd8ed579c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1e8cf0-29891265-mbhwx"
Oct 31 19:45:00.700759 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.700543    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/220956a9-b7c5-48b8-a598-61326c85691a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-a891ce-29891265-md69g\" (UID: \"220956a9-b7c5-48b8-a598-61326c85691a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a891ce-29891265-md69g"
Oct 31 19:45:00.700759 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.700615    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7198524-cd1a-4038-83c6-55f9a5391970-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-8730d5-29891265-r6vvt\" (UID: \"e7198524-cd1a-4038-83c6-55f9a5391970\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891265-r6vvt"
Oct 31 19:45:00.700759 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.700702    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97e726ce-2b00-4703-8c9e-277ea9bb4c4a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-a813e2-29891265-dh4wq\" (UID: \"97e726ce-2b00-4703-8c9e-277ea9bb4c4a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a813e2-29891265-dh4wq"
Oct 31 19:45:00.700759 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.700749    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxm98\" (UniqueName: \"kubernetes.io/projected/e6f57036-6c45-4def-a9c1-0132ab845b3e-kube-api-access-lxm98\") pod \"maas-api-key-cleanup-29891265-9vwg9\" (UID: \"e6f57036-6c45-4def-a9c1-0132ab845b3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891265-9vwg9"
Oct 31 19:45:00.700999 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.700786    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gknw9\" (UniqueName: \"kubernetes.io/projected/e7198524-cd1a-4038-83c6-55f9a5391970-kube-api-access-gknw9\") pod \"maas-api-key-cleanup-e2e-worker-w6-8730d5-29891265-r6vvt\" (UID: \"e7198524-cd1a-4038-83c6-55f9a5391970\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891265-r6vvt"
Oct 31 19:45:00.700999 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.700852    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh449\" (UniqueName: \"kubernetes.io/projected/615544a2-1b13-4bce-aeb6-7b3b32e6a509-kube-api-access-qh449\") pod \"maas-api-key-cleanup-e2e-worker-w0-23843f-29891265-52rjc\" (UID: \"615544a2-1b13-4bce-aeb6-7b3b32e6a509\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-23843f-29891265-52rjc"
Oct 31 19:45:00.700999 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.700875    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57977c65-f0c2-46d1-a05c-20bd8ed579c3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-1e8cf0-29891265-mbhwx\" (UID: \"57977c65-f0c2-46d1-a05c-20bd8ed579c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1e8cf0-29891265-mbhwx"
Oct 31 19:45:00.700999 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.700915    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53b785a3-a308-43ec-92b9-ccd7d6a8bcad-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-435728-29891265-tfbvw\" (UID: \"53b785a3-a308-43ec-92b9-ccd7d6a8bcad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-435728-29891265-tfbvw"
Oct 31 19:45:00.700999 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.700936    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/769ee829-1015-45b1-9dd7-d9cfe372fa9c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-fd1c91-29891265-rcrqk\" (UID: \"769ee829-1015-45b1-9dd7-d9cfe372fa9c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-fd1c91-29891265-rcrqk"
Oct 31 19:45:00.700999 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.700961    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdzb5\" (UniqueName: \"kubernetes.io/projected/97e726ce-2b00-4703-8c9e-277ea9bb4c4a-kube-api-access-rdzb5\") pod \"maas-api-key-cleanup-e2e-api-b-a813e2-29891265-dh4wq\" (UID: \"97e726ce-2b00-4703-8c9e-277ea9bb4c4a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a813e2-29891265-dh4wq"
Oct 31 19:45:00.701394 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.701040    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28z8w\" (UniqueName: \"kubernetes.io/projected/53b785a3-a308-43ec-92b9-ccd7d6a8bcad-kube-api-access-28z8w\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-435728-29891265-tfbvw\" (UID: \"53b785a3-a308-43ec-92b9-ccd7d6a8bcad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-435728-29891265-tfbvw"
Oct 31 19:45:00.709546 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.709076    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53b785a3-a308-43ec-92b9-ccd7d6a8bcad-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-435728-29891265-tfbvw\" (UID: \"53b785a3-a308-43ec-92b9-ccd7d6a8bcad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-435728-29891265-tfbvw"
Oct 31 19:45:00.712765 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.712675    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-88eaf9-29891265-zttxn"]
Oct 31 19:45:00.719818 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.719433    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f5c1ab-29891265-96ksc"
Oct 31 19:45:00.730271 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.730233    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28z8w\" (UniqueName: \"kubernetes.io/projected/53b785a3-a308-43ec-92b9-ccd7d6a8bcad-kube-api-access-28z8w\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-435728-29891265-tfbvw\" (UID: \"53b785a3-a308-43ec-92b9-ccd7d6a8bcad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-435728-29891265-tfbvw"
Oct 31 19:45:00.732642 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.732608    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-23843f-29891265-52rjc"]
Oct 31 19:45:00.732740 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.732657    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ae2e60-29891265-5s5f8"]
Oct 31 19:45:00.732740 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.732720    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-88eaf9-29891265-zttxn"
Oct 31 19:45:00.744708 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.744684    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891265-r6vvt"]
Oct 31 19:45:00.744799 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.744737    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891265-9vwg9"]
Oct 31 19:45:00.744799 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.744769    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1e8cf0-29891265-mbhwx"]
Oct 31 19:45:00.744917 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.744861    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ae2e60-29891265-5s5f8"
Oct 31 19:45:00.751431 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.751365    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-785d26-29891265-8cn6g"]
Oct 31 19:45:00.766542 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.760085    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8dcfa8-29891265-9lkmt"]
Oct 31 19:45:00.778834 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.778027    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-03dfee-29891265-njm8t"]
Oct 31 19:45:00.785117 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.783867    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f5c1ab-29891265-96ksc"]
Oct 31 19:45:00.791765 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.791742    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ae2e60-29891265-5s5f8"]
Oct 31 19:45:00.797562 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.797066    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-88eaf9-29891265-zttxn"]
Oct 31 19:45:00.803804 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.803774    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/615544a2-1b13-4bce-aeb6-7b3b32e6a509-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-23843f-29891265-52rjc\" (UID: \"615544a2-1b13-4bce-aeb6-7b3b32e6a509\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-23843f-29891265-52rjc"
Oct 31 19:45:00.803926 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.803822    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ae53805-07ba-4f39-bcbf-f19b67f0ad16-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-f5c1ab-29891265-96ksc\" (UID: \"4ae53805-07ba-4f39-bcbf-f19b67f0ad16\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f5c1ab-29891265-96ksc"
Oct 31 19:45:00.803926 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.803872    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66szc\" (UniqueName: \"kubernetes.io/projected/57977c65-f0c2-46d1-a05c-20bd8ed579c3-kube-api-access-66szc\") pod \"maas-api-key-cleanup-e2e-inf-a-1e8cf0-29891265-mbhwx\" (UID: \"57977c65-f0c2-46d1-a05c-20bd8ed579c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1e8cf0-29891265-mbhwx"
Oct 31 19:45:00.804051 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.803957    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/220956a9-b7c5-48b8-a598-61326c85691a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-a891ce-29891265-md69g\" (UID: \"220956a9-b7c5-48b8-a598-61326c85691a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a891ce-29891265-md69g"
Oct 31 19:45:00.804051 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.804027    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7198524-cd1a-4038-83c6-55f9a5391970-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-8730d5-29891265-r6vvt\" (UID: \"e7198524-cd1a-4038-83c6-55f9a5391970\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891265-r6vvt"
Oct 31 19:45:00.804163 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.804065    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3620ed03-6b03-4c6f-83be-484c41686458-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-ae2e60-29891265-5s5f8\" (UID: \"3620ed03-6b03-4c6f-83be-484c41686458\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ae2e60-29891265-5s5f8"
Oct 31 19:45:00.804163 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.804104    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbd4246c-22ba-4116-8e31-b53d3bdc2fbb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-8dcfa8-29891265-9lkmt\" (UID: \"bbd4246c-22ba-4116-8e31-b53d3bdc2fbb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8dcfa8-29891265-9lkmt"
Oct 31 19:45:00.804163 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.804135    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5j5s\" (UniqueName: \"kubernetes.io/projected/25370d12-c485-47b9-986a-a775c3fbd2b7-kube-api-access-g5j5s\") pod \"maas-api-key-cleanup-e2e-inf-b-88eaf9-29891265-zttxn\" (UID: \"25370d12-c485-47b9-986a-a775c3fbd2b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-88eaf9-29891265-zttxn"
Oct 31 19:45:00.804378 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.804163    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8nc4\" (UniqueName: \"kubernetes.io/projected/2e3605c3-ec3b-4fb7-be2a-40fe32c5adea-kube-api-access-b8nc4\") pod \"maas-api-key-cleanup-e2e-inf-a-785d26-29891265-8cn6g\" (UID: \"2e3605c3-ec3b-4fb7-be2a-40fe32c5adea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-785d26-29891265-8cn6g"
Oct 31 19:45:00.804378 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.804256    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97e726ce-2b00-4703-8c9e-277ea9bb4c4a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-a813e2-29891265-dh4wq\" (UID: \"97e726ce-2b00-4703-8c9e-277ea9bb4c4a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a813e2-29891265-dh4wq"
Oct 31 19:45:00.804378 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.804345    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxm98\" (UniqueName: \"kubernetes.io/projected/e6f57036-6c45-4def-a9c1-0132ab845b3e-kube-api-access-lxm98\") pod \"maas-api-key-cleanup-29891265-9vwg9\" (UID: \"e6f57036-6c45-4def-a9c1-0132ab845b3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891265-9vwg9"
Oct 31 19:45:00.805786 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.804374    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntwl4\" (UniqueName: \"kubernetes.io/projected/4ae53805-07ba-4f39-bcbf-f19b67f0ad16-kube-api-access-ntwl4\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-f5c1ab-29891265-96ksc\" (UID: \"4ae53805-07ba-4f39-bcbf-f19b67f0ad16\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f5c1ab-29891265-96ksc"
Oct 31 19:45:00.805786 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.804425    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gknw9\" (UniqueName: \"kubernetes.io/projected/e7198524-cd1a-4038-83c6-55f9a5391970-kube-api-access-gknw9\") pod \"maas-api-key-cleanup-e2e-worker-w6-8730d5-29891265-r6vvt\" (UID: \"e7198524-cd1a-4038-83c6-55f9a5391970\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891265-r6vvt"
Oct 31 19:45:00.805786 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.804474    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khhgd\" (UniqueName: \"kubernetes.io/projected/bbd4246c-22ba-4116-8e31-b53d3bdc2fbb-kube-api-access-khhgd\") pod \"maas-api-key-cleanup-e2e-inf-b-8dcfa8-29891265-9lkmt\" (UID: \"bbd4246c-22ba-4116-8e31-b53d3bdc2fbb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8dcfa8-29891265-9lkmt"
Oct 31 19:45:00.805786 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.804547    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh449\" (UniqueName: \"kubernetes.io/projected/615544a2-1b13-4bce-aeb6-7b3b32e6a509-kube-api-access-qh449\") pod \"maas-api-key-cleanup-e2e-worker-w0-23843f-29891265-52rjc\" (UID: \"615544a2-1b13-4bce-aeb6-7b3b32e6a509\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-23843f-29891265-52rjc"
Oct 31 19:45:00.805786 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.804574    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57977c65-f0c2-46d1-a05c-20bd8ed579c3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-1e8cf0-29891265-mbhwx\" (UID: \"57977c65-f0c2-46d1-a05c-20bd8ed579c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1e8cf0-29891265-mbhwx"
Oct 31 19:45:00.805786 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.804599    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e3605c3-ec3b-4fb7-be2a-40fe32c5adea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-785d26-29891265-8cn6g\" (UID: \"2e3605c3-ec3b-4fb7-be2a-40fe32c5adea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-785d26-29891265-8cn6g"
Oct 31 19:45:00.805786 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.804661    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/769ee829-1015-45b1-9dd7-d9cfe372fa9c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-fd1c91-29891265-rcrqk\" (UID: \"769ee829-1015-45b1-9dd7-d9cfe372fa9c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-fd1c91-29891265-rcrqk"
Oct 31 19:45:00.805786 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.804702    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdzb5\" (UniqueName: \"kubernetes.io/projected/97e726ce-2b00-4703-8c9e-277ea9bb4c4a-kube-api-access-rdzb5\") pod \"maas-api-key-cleanup-e2e-api-b-a813e2-29891265-dh4wq\" (UID: \"97e726ce-2b00-4703-8c9e-277ea9bb4c4a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a813e2-29891265-dh4wq"
Oct 31 19:45:00.805786 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.804759    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25370d12-c485-47b9-986a-a775c3fbd2b7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-88eaf9-29891265-zttxn\" (UID: \"25370d12-c485-47b9-986a-a775c3fbd2b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-88eaf9-29891265-zttxn"
Oct 31 19:45:00.805786 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.804883    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jfnb\" (UniqueName: \"kubernetes.io/projected/220956a9-b7c5-48b8-a598-61326c85691a-kube-api-access-8jfnb\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-a891ce-29891265-md69g\" (UID: \"220956a9-b7c5-48b8-a598-61326c85691a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a891ce-29891265-md69g"
Oct 31 19:45:00.805786 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.804956    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jsfh\" (UniqueName: \"kubernetes.io/projected/769ee829-1015-45b1-9dd7-d9cfe372fa9c-kube-api-access-8jsfh\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-fd1c91-29891265-rcrqk\" (UID: \"769ee829-1015-45b1-9dd7-d9cfe372fa9c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-fd1c91-29891265-rcrqk"
Oct 31 19:45:00.805786 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.804992    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6f57036-6c45-4def-a9c1-0132ab845b3e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891265-9vwg9\" (UID: \"e6f57036-6c45-4def-a9c1-0132ab845b3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891265-9vwg9"
Oct 31 19:45:00.805786 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.805019    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3961f45f-4b0d-4880-b8d8-e55798d4abe8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-03dfee-29891265-njm8t\" (UID: \"3961f45f-4b0d-4880-b8d8-e55798d4abe8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-03dfee-29891265-njm8t"
Oct 31 19:45:00.805786 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.805047    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/615544a2-1b13-4bce-aeb6-7b3b32e6a509-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-23843f-29891265-52rjc\" (UID: \"615544a2-1b13-4bce-aeb6-7b3b32e6a509\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-23843f-29891265-52rjc"
Oct 31 19:45:00.805786 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.805055    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9sfr\" (UniqueName: \"kubernetes.io/projected/3961f45f-4b0d-4880-b8d8-e55798d4abe8-kube-api-access-m9sfr\") pod \"maas-api-key-cleanup-e2e-inf-a-03dfee-29891265-njm8t\" (UID: \"3961f45f-4b0d-4880-b8d8-e55798d4abe8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-03dfee-29891265-njm8t"
Oct 31 19:45:00.806517 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.805815    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/220956a9-b7c5-48b8-a598-61326c85691a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-a891ce-29891265-md69g\" (UID: \"220956a9-b7c5-48b8-a598-61326c85691a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a891ce-29891265-md69g"
Oct 31 19:45:00.806517 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.806275    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57977c65-f0c2-46d1-a05c-20bd8ed579c3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-1e8cf0-29891265-mbhwx\" (UID: \"57977c65-f0c2-46d1-a05c-20bd8ed579c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1e8cf0-29891265-mbhwx"
Oct 31 19:45:00.806675 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.806648    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7198524-cd1a-4038-83c6-55f9a5391970-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-8730d5-29891265-r6vvt\" (UID: \"e7198524-cd1a-4038-83c6-55f9a5391970\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891265-r6vvt"
Oct 31 19:45:00.807065 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.807039    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97e726ce-2b00-4703-8c9e-277ea9bb4c4a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-a813e2-29891265-dh4wq\" (UID: \"97e726ce-2b00-4703-8c9e-277ea9bb4c4a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a813e2-29891265-dh4wq"
Oct 31 19:45:00.807444 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.807411    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/769ee829-1015-45b1-9dd7-d9cfe372fa9c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-fd1c91-29891265-rcrqk\" (UID: \"769ee829-1015-45b1-9dd7-d9cfe372fa9c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-fd1c91-29891265-rcrqk"
Oct 31 19:45:00.808268 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.808233    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqxnr\" (UniqueName: \"kubernetes.io/projected/3620ed03-6b03-4c6f-83be-484c41686458-kube-api-access-cqxnr\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-ae2e60-29891265-5s5f8\" (UID: \"3620ed03-6b03-4c6f-83be-484c41686458\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ae2e60-29891265-5s5f8"
Oct 31 19:45:00.809041 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.808744    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6f57036-6c45-4def-a9c1-0132ab845b3e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891265-9vwg9\" (UID: \"e6f57036-6c45-4def-a9c1-0132ab845b3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891265-9vwg9"
Oct 31 19:45:00.844259 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.844102    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jsfh\" (UniqueName: \"kubernetes.io/projected/769ee829-1015-45b1-9dd7-d9cfe372fa9c-kube-api-access-8jsfh\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-fd1c91-29891265-rcrqk\" (UID: \"769ee829-1015-45b1-9dd7-d9cfe372fa9c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-fd1c91-29891265-rcrqk"
Oct 31 19:45:00.844259 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.844189    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gknw9\" (UniqueName: \"kubernetes.io/projected/e7198524-cd1a-4038-83c6-55f9a5391970-kube-api-access-gknw9\") pod \"maas-api-key-cleanup-e2e-worker-w6-8730d5-29891265-r6vvt\" (UID: \"e7198524-cd1a-4038-83c6-55f9a5391970\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891265-r6vvt"
Oct 31 19:45:00.846135 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.845219    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66szc\" (UniqueName: \"kubernetes.io/projected/57977c65-f0c2-46d1-a05c-20bd8ed579c3-kube-api-access-66szc\") pod \"maas-api-key-cleanup-e2e-inf-a-1e8cf0-29891265-mbhwx\" (UID: \"57977c65-f0c2-46d1-a05c-20bd8ed579c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1e8cf0-29891265-mbhwx"
Oct 31 19:45:00.846736 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.846688    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdzb5\" (UniqueName: \"kubernetes.io/projected/97e726ce-2b00-4703-8c9e-277ea9bb4c4a-kube-api-access-rdzb5\") pod \"maas-api-key-cleanup-e2e-api-b-a813e2-29891265-dh4wq\" (UID: \"97e726ce-2b00-4703-8c9e-277ea9bb4c4a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a813e2-29891265-dh4wq"
Oct 31 19:45:00.847466 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.847442    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxm98\" (UniqueName: \"kubernetes.io/projected/e6f57036-6c45-4def-a9c1-0132ab845b3e-kube-api-access-lxm98\") pod \"maas-api-key-cleanup-29891265-9vwg9\" (UID: \"e6f57036-6c45-4def-a9c1-0132ab845b3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891265-9vwg9"
Oct 31 19:45:00.847654 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.847642    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh449\" (UniqueName: \"kubernetes.io/projected/615544a2-1b13-4bce-aeb6-7b3b32e6a509-kube-api-access-qh449\") pod \"maas-api-key-cleanup-e2e-worker-w0-23843f-29891265-52rjc\" (UID: \"615544a2-1b13-4bce-aeb6-7b3b32e6a509\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-23843f-29891265-52rjc"
Oct 31 19:45:00.848156 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.848137    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jfnb\" (UniqueName: \"kubernetes.io/projected/220956a9-b7c5-48b8-a598-61326c85691a-kube-api-access-8jfnb\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-a891ce-29891265-md69g\" (UID: \"220956a9-b7c5-48b8-a598-61326c85691a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a891ce-29891265-md69g"
Oct 31 19:45:00.908164 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.908136    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-435728-29891265-tfbvw"
Oct 31 19:45:00.910019 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.909991    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3961f45f-4b0d-4880-b8d8-e55798d4abe8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-03dfee-29891265-njm8t\" (UID: \"3961f45f-4b0d-4880-b8d8-e55798d4abe8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-03dfee-29891265-njm8t"
Oct 31 19:45:00.910148 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.910035    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9sfr\" (UniqueName: \"kubernetes.io/projected/3961f45f-4b0d-4880-b8d8-e55798d4abe8-kube-api-access-m9sfr\") pod \"maas-api-key-cleanup-e2e-inf-a-03dfee-29891265-njm8t\" (UID: \"3961f45f-4b0d-4880-b8d8-e55798d4abe8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-03dfee-29891265-njm8t"
Oct 31 19:45:00.910148 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.910070    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqxnr\" (UniqueName: \"kubernetes.io/projected/3620ed03-6b03-4c6f-83be-484c41686458-kube-api-access-cqxnr\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-ae2e60-29891265-5s5f8\" (UID: \"3620ed03-6b03-4c6f-83be-484c41686458\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ae2e60-29891265-5s5f8"
Oct 31 19:45:00.910305 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.910117    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ae53805-07ba-4f39-bcbf-f19b67f0ad16-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-f5c1ab-29891265-96ksc\" (UID: \"4ae53805-07ba-4f39-bcbf-f19b67f0ad16\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f5c1ab-29891265-96ksc"
Oct 31 19:45:00.910394 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.910375    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3620ed03-6b03-4c6f-83be-484c41686458-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-ae2e60-29891265-5s5f8\" (UID: \"3620ed03-6b03-4c6f-83be-484c41686458\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ae2e60-29891265-5s5f8"
Oct 31 19:45:00.910455 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.910420    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbd4246c-22ba-4116-8e31-b53d3bdc2fbb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-8dcfa8-29891265-9lkmt\" (UID: \"bbd4246c-22ba-4116-8e31-b53d3bdc2fbb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8dcfa8-29891265-9lkmt"
Oct 31 19:45:00.910570 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.910452    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5j5s\" (UniqueName: \"kubernetes.io/projected/25370d12-c485-47b9-986a-a775c3fbd2b7-kube-api-access-g5j5s\") pod \"maas-api-key-cleanup-e2e-inf-b-88eaf9-29891265-zttxn\" (UID: \"25370d12-c485-47b9-986a-a775c3fbd2b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-88eaf9-29891265-zttxn"
Oct 31 19:45:00.910570 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.910479    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8nc4\" (UniqueName: \"kubernetes.io/projected/2e3605c3-ec3b-4fb7-be2a-40fe32c5adea-kube-api-access-b8nc4\") pod \"maas-api-key-cleanup-e2e-inf-a-785d26-29891265-8cn6g\" (UID: \"2e3605c3-ec3b-4fb7-be2a-40fe32c5adea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-785d26-29891265-8cn6g"
Oct 31 19:45:00.910675 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.910602    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntwl4\" (UniqueName: \"kubernetes.io/projected/4ae53805-07ba-4f39-bcbf-f19b67f0ad16-kube-api-access-ntwl4\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-f5c1ab-29891265-96ksc\" (UID: \"4ae53805-07ba-4f39-bcbf-f19b67f0ad16\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f5c1ab-29891265-96ksc"
Oct 31 19:45:00.910735 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.910680    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khhgd\" (UniqueName: \"kubernetes.io/projected/bbd4246c-22ba-4116-8e31-b53d3bdc2fbb-kube-api-access-khhgd\") pod \"maas-api-key-cleanup-e2e-inf-b-8dcfa8-29891265-9lkmt\" (UID: \"bbd4246c-22ba-4116-8e31-b53d3bdc2fbb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8dcfa8-29891265-9lkmt"
Oct 31 19:45:00.910829 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.910805    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3961f45f-4b0d-4880-b8d8-e55798d4abe8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-03dfee-29891265-njm8t\" (UID: \"3961f45f-4b0d-4880-b8d8-e55798d4abe8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-03dfee-29891265-njm8t"
Oct 31 19:45:00.910959 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.910940    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e3605c3-ec3b-4fb7-be2a-40fe32c5adea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-785d26-29891265-8cn6g\" (UID: \"2e3605c3-ec3b-4fb7-be2a-40fe32c5adea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-785d26-29891265-8cn6g"
Oct 31 19:45:00.911067 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.911038    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25370d12-c485-47b9-986a-a775c3fbd2b7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-88eaf9-29891265-zttxn\" (UID: \"25370d12-c485-47b9-986a-a775c3fbd2b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-88eaf9-29891265-zttxn"
Oct 31 19:45:00.911721 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.911700    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e3605c3-ec3b-4fb7-be2a-40fe32c5adea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-785d26-29891265-8cn6g\" (UID: \"2e3605c3-ec3b-4fb7-be2a-40fe32c5adea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-785d26-29891265-8cn6g"
Oct 31 19:45:00.912577 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.912063    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ae53805-07ba-4f39-bcbf-f19b67f0ad16-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-f5c1ab-29891265-96ksc\" (UID: \"4ae53805-07ba-4f39-bcbf-f19b67f0ad16\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f5c1ab-29891265-96ksc"
Oct 31 19:45:00.912577 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.912091    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25370d12-c485-47b9-986a-a775c3fbd2b7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-88eaf9-29891265-zttxn\" (UID: \"25370d12-c485-47b9-986a-a775c3fbd2b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-88eaf9-29891265-zttxn"
Oct 31 19:45:00.913678 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.913656    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3620ed03-6b03-4c6f-83be-484c41686458-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-ae2e60-29891265-5s5f8\" (UID: \"3620ed03-6b03-4c6f-83be-484c41686458\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ae2e60-29891265-5s5f8"
Oct 31 19:45:00.914229 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.914206    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbd4246c-22ba-4116-8e31-b53d3bdc2fbb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-8dcfa8-29891265-9lkmt\" (UID: \"bbd4246c-22ba-4116-8e31-b53d3bdc2fbb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8dcfa8-29891265-9lkmt"
Oct 31 19:45:00.920932 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.920906    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a891ce-29891265-md69g"
Oct 31 19:45:00.923268 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.923248    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khhgd\" (UniqueName: \"kubernetes.io/projected/bbd4246c-22ba-4116-8e31-b53d3bdc2fbb-kube-api-access-khhgd\") pod \"maas-api-key-cleanup-e2e-inf-b-8dcfa8-29891265-9lkmt\" (UID: \"bbd4246c-22ba-4116-8e31-b53d3bdc2fbb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8dcfa8-29891265-9lkmt"
Oct 31 19:45:00.923434 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.923412    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9sfr\" (UniqueName: \"kubernetes.io/projected/3961f45f-4b0d-4880-b8d8-e55798d4abe8-kube-api-access-m9sfr\") pod \"maas-api-key-cleanup-e2e-inf-a-03dfee-29891265-njm8t\" (UID: \"3961f45f-4b0d-4880-b8d8-e55798d4abe8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-03dfee-29891265-njm8t"
Oct 31 19:45:00.925095 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.925046    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5j5s\" (UniqueName: \"kubernetes.io/projected/25370d12-c485-47b9-986a-a775c3fbd2b7-kube-api-access-g5j5s\") pod \"maas-api-key-cleanup-e2e-inf-b-88eaf9-29891265-zttxn\" (UID: \"25370d12-c485-47b9-986a-a775c3fbd2b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-88eaf9-29891265-zttxn"
Oct 31 19:45:00.927275 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.927250    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8nc4\" (UniqueName: \"kubernetes.io/projected/2e3605c3-ec3b-4fb7-be2a-40fe32c5adea-kube-api-access-b8nc4\") pod \"maas-api-key-cleanup-e2e-inf-a-785d26-29891265-8cn6g\" (UID: \"2e3605c3-ec3b-4fb7-be2a-40fe32c5adea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-785d26-29891265-8cn6g"
Oct 31 19:45:00.928702 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.928680    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntwl4\" (UniqueName: \"kubernetes.io/projected/4ae53805-07ba-4f39-bcbf-f19b67f0ad16-kube-api-access-ntwl4\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-f5c1ab-29891265-96ksc\" (UID: \"4ae53805-07ba-4f39-bcbf-f19b67f0ad16\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f5c1ab-29891265-96ksc"
Oct 31 19:45:00.928808 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.928743    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqxnr\" (UniqueName: \"kubernetes.io/projected/3620ed03-6b03-4c6f-83be-484c41686458-kube-api-access-cqxnr\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-ae2e60-29891265-5s5f8\" (UID: \"3620ed03-6b03-4c6f-83be-484c41686458\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ae2e60-29891265-5s5f8"
Oct 31 19:45:00.930403 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.930381    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-fd1c91-29891265-rcrqk"
Oct 31 19:45:00.939950 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.939923    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a813e2-29891265-dh4wq"
Oct 31 19:45:00.960135 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.960108    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891265-9vwg9"
Oct 31 19:45:00.983601 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:45:00.983554    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod475d73a5_f2c4_4cc8_b573_b42497e3e7a9.slice/crio-0f9e1644e9ff0c542349acd9e6be1c29c5c20163c0bf3c93103df9e6d1b90615 WatchSource:0}: Error finding container 0f9e1644e9ff0c542349acd9e6be1c29c5c20163c0bf3c93103df9e6d1b90615: Status 404 returned error can't find the container with id 0f9e1644e9ff0c542349acd9e6be1c29c5c20163c0bf3c93103df9e6d1b90615
Oct 31 19:45:00.996653 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:00.996537    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d8c398-69fd677f-s95lc"]
Oct 31 19:45:01.035421 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:01.035323    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:01.039637 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:01.035719    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-23843f-29891265-52rjc"
Oct 31 19:45:01.090968 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:01.090477    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891265-r6vvt"
Oct 31 19:45:01.090968 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:01.090919    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1e8cf0-29891265-mbhwx"
Oct 31 19:45:01.144241 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:01.144210    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-785d26-29891265-8cn6g"
Oct 31 19:45:01.152520 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:01.152237    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8dcfa8-29891265-9lkmt"
Oct 31 19:45:01.189622 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:01.189314    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-03dfee-29891265-njm8t"
Oct 31 19:45:01.197708 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:01.197393    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f5c1ab-29891265-96ksc"
Oct 31 19:45:01.219300 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:01.211280    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-88eaf9-29891265-zttxn"
Oct 31 19:45:01.219300 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:01.215498    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ae2e60-29891265-5s5f8"
Oct 31 19:45:01.432481 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:01.432440    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-435728-29891265-tfbvw"]
Oct 31 19:45:01.441499 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:45:01.441472    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53b785a3_a308_43ec_92b9_ccd7d6a8bcad.slice/crio-b8a91d43e980aad7e6b95cea72cad226fef6924a3bcb6267bd378029c389c06d WatchSource:0}: Error finding container b8a91d43e980aad7e6b95cea72cad226fef6924a3bcb6267bd378029c389c06d: Status 404 returned error can't find the container with id b8a91d43e980aad7e6b95cea72cad226fef6924a3bcb6267bd378029c389c06d
Oct 31 19:45:01.485653 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:01.485624    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq"]
Oct 31 19:45:01.493380 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:01.493348    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq"
Oct 31 19:45:01.516379 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:01.516347    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq"]
Oct 31 19:45:01.560691 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:01.560658    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a813e2-29891265-dh4wq"]
Oct 31 19:45:01.571425 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:01.571402    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891265-9vwg9"]
Oct 31 19:45:01.583308 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:01.583279    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37f24530-822f-4186-acfa-a7912c710c56-istio-envoy\") pod \"e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq\" (UID: \"37f24530-822f-4186-acfa-a7912c710c56\") " pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq"
Oct 31 19:45:01.583441 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:01.583375    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37f24530-822f-4186-acfa-a7912c710c56-istio-token\") pod \"e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq\" (UID: \"37f24530-822f-4186-acfa-a7912c710c56\") " pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq"
Oct 31 19:45:01.583513 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:01.583461    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37f24530-822f-4186-acfa-a7912c710c56-workload-certs\") pod \"e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq\" (UID: \"37f24530-822f-4186-acfa-a7912c710c56\") " pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq"
Oct 31 19:45:01.583513 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:01.583503    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37f24530-822f-4186-acfa-a7912c710c56-workload-socket\") pod \"e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq\" (UID: \"37f24530-822f-4186-acfa-a7912c710c56\") " pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq"
Oct 31 19:45:01.583634 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:01.583589    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37f24530-822f-4186-acfa-a7912c710c56-istiod-ca-cert\") pod \"e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq\" (UID: \"37f24530-822f-4186-acfa-a7912c710c56\") " pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq"
Oct 31 19:45:01.583634 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:01.583620    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37f24530-822f-4186-acfa-a7912c710c56-credential-socket\") pod \"e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq\" (UID: \"37f24530-822f-4186-acfa-a7912c710c56\") " pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq"
Oct 31 19:45:01.583766 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:01.583749    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37f24530-822f-4186-acfa-a7912c710c56-istio-data\") pod \"e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq\" (UID: \"37f24530-822f-4186-acfa-a7912c710c56\") " pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq"
Oct 31 19:45:01.583913 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:01.583893    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37f24530-822f-4186-acfa-a7912c710c56-istio-podinfo\") pod \"e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq\" (UID: \"37f24530-822f-4186-acfa-a7912c710c56\") " pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq"
Oct 31 19:45:01.584020 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:01.583963    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jjrm\" (UniqueName: \"kubernetes.io/projected/37f24530-822f-4186-acfa-a7912c710c56-kube-api-access-9jjrm\") pod \"e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq\" (UID: \"37f24530-822f-4186-acfa-a7912c710c56\") " pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq"
Oct 31 19:45:01.589998 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:01.589979    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-23843f-29891265-52rjc"]
Oct 31 19:45:01.592501 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:45:01.592448    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod615544a2_1b13_4bce_aeb6_7b3b32e6a509.slice/crio-e5e95bae79d9a6eac9ffb1eb124b7f223785facb8d11d443408b4abd750761e5 WatchSource:0}: Error finding container e5e95bae79d9a6eac9ffb1eb124b7f223785facb8d11d443408b4abd750761e5: Status 404 returned error can't find the container with id e5e95bae79d9a6eac9ffb1eb124b7f223785facb8d11d443408b4abd750761e5
Oct 31 19:45:01.629923 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:01.629882    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-23843f-29891265-52rjc" event={"ID":"615544a2-1b13-4bce-aeb6-7b3b32e6a509","Type":"ContainerStarted","Data":"e5e95bae79d9a6eac9ffb1eb124b7f223785facb8d11d443408b4abd750761e5"}
Oct 31 19:45:01.630190 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:01.630165    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891265-9vwg9" event={"ID":"e6f57036-6c45-4def-a9c1-0132ab845b3e","Type":"ContainerStarted","Data":"57535395ac7e22369d5701eeddeb3f9608aa2db7ecfca66eee33fc79b36f7ee8"}
Oct 31 19:45:01.631672 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:01.631653    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-435728-29891265-tfbvw" event={"ID":"53b785a3-a308-43ec-92b9-ccd7d6a8bcad","Type":"ContainerStarted","Data":"b8a91d43e980aad7e6b95cea72cad226fef6924a3bcb6267bd378029c389c06d"}
Oct 31 19:45:01.632964 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:01.632944    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a813e2-29891265-dh4wq" event={"ID":"97e726ce-2b00-4703-8c9e-277ea9bb4c4a","Type":"ContainerStarted","Data":"0a37feb215e107ccd7a60b6134cfea14faa61b10b2092298d41dabca60252103"}
Oct 31 19:45:01.634389 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:01.634362    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d8c398-69fd677f-s95lc" event={"ID":"475d73a5-f2c4-4cc8-b573-b42497e3e7a9","Type":"ContainerStarted","Data":"5eabd87c48f5fd10f070fa880b377084b28a4d3bd90e8104dc7ef49145100750"}
Oct 31 19:45:01.634490 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:01.634406    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d8c398-69fd677f-s95lc" event={"ID":"475d73a5-f2c4-4cc8-b573-b42497e3e7a9","Type":"ContainerStarted","Data":"0f9e1644e9ff0c542349acd9e6be1c29c5c20163c0bf3c93103df9e6d1b90615"}
Oct 31 19:45:01.634634 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:01.634621    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d8c398-69fd677f-s95lc"
Oct 31 19:45:01.655853 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:01.655809    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d8c398-69fd677f-s95lc" podStartSLOduration=2.655794597 podStartE2EDuration="2.655794597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:44:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:45:01.654260568 +0000 UTC m=+1532.227494699" watchObservedRunningTime="2026-10-31 19:45:01.655794597 +0000 UTC m=+1532.229028729"
Oct 31 19:45:01.685730 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:01.685699    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37f24530-822f-4186-acfa-a7912c710c56-istio-data\") pod \"e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq\" (UID: \"37f24530-822f-4186-acfa-a7912c710c56\") " pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq"
Oct 31 19:45:01.685923 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:01.685900    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37f24530-822f-4186-acfa-a7912c710c56-istio-podinfo\") pod \"e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq\" (UID: \"37f24530-822f-4186-acfa-a7912c710c56\") " pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq"
Oct 31 19:45:01.686028 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:01.686011    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jjrm\" (UniqueName: \"kubernetes.io/projected/37f24530-822f-4186-acfa-a7912c710c56-kube-api-access-9jjrm\") pod \"e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq\" (UID: \"37f24530-822f-4186-acfa-a7912c710c56\") " pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq"
Oct 31 19:45:01.686113 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:01.686078    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37f24530-822f-4186-acfa-a7912c710c56-istio-envoy\") pod \"e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq\" (UID: \"37f24530-822f-4186-acfa-a7912c710c56\") " pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq"
Oct 31 19:45:01.686241 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:01.686114    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37f24530-822f-4186-acfa-a7912c710c56-istio-data\") pod \"e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq\" (UID: \"37f24530-822f-4186-acfa-a7912c710c56\") " pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq"
Oct 31 19:45:01.686241 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:01.686129    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37f24530-822f-4186-acfa-a7912c710c56-istio-token\") pod \"e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq\" (UID: \"37f24530-822f-4186-acfa-a7912c710c56\") " pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq"
Oct 31 19:45:01.686241 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:01.686201    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37f24530-822f-4186-acfa-a7912c710c56-workload-certs\") pod \"e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq\" (UID: \"37f24530-822f-4186-acfa-a7912c710c56\") " pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq"
Oct 31 19:45:01.686463 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:01.686247    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37f24530-822f-4186-acfa-a7912c710c56-workload-socket\") pod \"e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq\" (UID: \"37f24530-822f-4186-acfa-a7912c710c56\") " pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq"
Oct 31 19:45:01.686463 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:01.686361    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37f24530-822f-4186-acfa-a7912c710c56-istiod-ca-cert\") pod \"e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq\" (UID: \"37f24530-822f-4186-acfa-a7912c710c56\") " pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq"
Oct 31 19:45:01.686463 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:01.686402    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37f24530-822f-4186-acfa-a7912c710c56-credential-socket\") pod \"e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq\" (UID: \"37f24530-822f-4186-acfa-a7912c710c56\") " pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq"
Oct 31 19:45:01.686826 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:01.686802    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37f24530-822f-4186-acfa-a7912c710c56-credential-socket\") pod \"e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq\" (UID: \"37f24530-822f-4186-acfa-a7912c710c56\") " pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq"
Oct 31 19:45:01.687618 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:01.687567    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37f24530-822f-4186-acfa-a7912c710c56-workload-socket\") pod \"e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq\" (UID: \"37f24530-822f-4186-acfa-a7912c710c56\") " pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq"
Oct 31 19:45:01.688291 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:01.688247    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37f24530-822f-4186-acfa-a7912c710c56-istiod-ca-cert\") pod \"e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq\" (UID: \"37f24530-822f-4186-acfa-a7912c710c56\") " pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq"
Oct 31 19:45:01.688861 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:01.688838    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37f24530-822f-4186-acfa-a7912c710c56-workload-certs\") pod \"e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq\" (UID: \"37f24530-822f-4186-acfa-a7912c710c56\") " pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq"
Oct 31 19:45:01.691374 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:01.691345    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37f24530-822f-4186-acfa-a7912c710c56-istio-podinfo\") pod \"e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq\" (UID: \"37f24530-822f-4186-acfa-a7912c710c56\") " pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq"
Oct 31 19:45:01.691724 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:01.691701    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37f24530-822f-4186-acfa-a7912c710c56-istio-envoy\") pod \"e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq\" (UID: \"37f24530-822f-4186-acfa-a7912c710c56\") " pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq"
Oct 31 19:45:01.698458 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:01.698437    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a891ce-29891265-md69g"]
Oct 31 19:45:01.701492 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:45:01.701464    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod220956a9_b7c5_48b8_a598_61326c85691a.slice/crio-415b0234203ca2ef2c91e78b763cda02736ca4317b283d6bdfd3d848f074de18 WatchSource:0}: Error finding container 415b0234203ca2ef2c91e78b763cda02736ca4317b283d6bdfd3d848f074de18: Status 404 returned error can't find the container with id 415b0234203ca2ef2c91e78b763cda02736ca4317b283d6bdfd3d848f074de18
Oct 31 19:45:01.704181 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:01.704156    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jjrm\" (UniqueName: \"kubernetes.io/projected/37f24530-822f-4186-acfa-a7912c710c56-kube-api-access-9jjrm\") pod \"e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq\" (UID: \"37f24530-822f-4186-acfa-a7912c710c56\") " pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq"
Oct 31 19:45:01.708505 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:01.708483    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37f24530-822f-4186-acfa-a7912c710c56-istio-token\") pod \"e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq\" (UID: \"37f24530-822f-4186-acfa-a7912c710c56\") " pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq"
Oct 31 19:45:01.719085 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:01.719061    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1e8cf0-29891265-mbhwx"]
Oct 31 19:45:01.722492 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:45:01.722468    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57977c65_f0c2_46d1_a05c_20bd8ed579c3.slice/crio-e332333b2d18f171988ecc34257cfe03302668981cd9e783e345ab283e898628 WatchSource:0}: Error finding container e332333b2d18f171988ecc34257cfe03302668981cd9e783e345ab283e898628: Status 404 returned error can't find the container with id e332333b2d18f171988ecc34257cfe03302668981cd9e783e345ab283e898628
Oct 31 19:45:01.739650 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:45:01.739624    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e3605c3_ec3b_4fb7_be2a_40fe32c5adea.slice/crio-8ea0aa59bcc8af817bd845b6f8a2ca428d43cb3e13eb74e71d5e47a56035bb59 WatchSource:0}: Error finding container 8ea0aa59bcc8af817bd845b6f8a2ca428d43cb3e13eb74e71d5e47a56035bb59: Status 404 returned error can't find the container with id 8ea0aa59bcc8af817bd845b6f8a2ca428d43cb3e13eb74e71d5e47a56035bb59
Oct 31 19:45:01.740917 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:01.740900    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-785d26-29891265-8cn6g"]
Oct 31 19:45:01.763898 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:45:01.763876    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbd4246c_22ba_4116_8e31_b53d3bdc2fbb.slice/crio-8475caf4e62a63a4ea6e546b905fe67af9c5a6da5cb440515abfe21cc4fcf9ea WatchSource:0}: Error finding container 8475caf4e62a63a4ea6e546b905fe67af9c5a6da5cb440515abfe21cc4fcf9ea: Status 404 returned error can't find the container with id 8475caf4e62a63a4ea6e546b905fe67af9c5a6da5cb440515abfe21cc4fcf9ea
Oct 31 19:45:01.764802 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:01.764783    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8dcfa8-29891265-9lkmt"]
Oct 31 19:45:01.821282 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:45:01.821250    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3961f45f_4b0d_4880_b8d8_e55798d4abe8.slice/crio-982f45c6f66cd2868373d25b98ff84111951da289dc0e241ed1172f6e1252b81 WatchSource:0}: Error finding container 982f45c6f66cd2868373d25b98ff84111951da289dc0e241ed1172f6e1252b81: Status 404 returned error can't find the container with id 982f45c6f66cd2868373d25b98ff84111951da289dc0e241ed1172f6e1252b81
Oct 31 19:45:01.834598 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:01.834574    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-03dfee-29891265-njm8t"]
Oct 31 19:45:01.853495 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:01.853470    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-fd1c91-29891265-rcrqk"]
Oct 31 19:45:01.859422 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:45:01.859401    4645 kuberuntime_manager.go:1707] "Unhandled Error" err="container cleanup start failed in pod maas-api-key-cleanup-e2e-inf-b-88eaf9-29891265-zttxn_odh-ai-gateway-infra(25370d12-c485-47b9-986a-a775c3fbd2b7): ErrImagePull: pull QPS exceeded" logger="UnhandledError"
Oct 31 19:45:01.859520 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:45:01.859433    4645 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with ErrImagePull: \"pull QPS exceeded\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-88eaf9-29891265-zttxn" podUID="25370d12-c485-47b9-986a-a775c3fbd2b7"
Oct 31 19:45:01.859520 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:01.859503    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f5c1ab-29891265-96ksc"]
Oct 31 19:45:01.864869 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:01.864680    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ae2e60-29891265-5s5f8"]
Oct 31 19:45:01.869914 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:01.869894    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-88eaf9-29891265-zttxn"]
Oct 31 19:45:01.878051 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:45:01.878016    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7198524_cd1a_4038_83c6_55f9a5391970.slice/crio-0fc69bfd6fba82b0e962a2ed3e1d978e2a12b839dd4bc1510ad0ca65e14d648a WatchSource:0}: Error finding container 0fc69bfd6fba82b0e962a2ed3e1d978e2a12b839dd4bc1510ad0ca65e14d648a: Status 404 returned error can't find the container with id 0fc69bfd6fba82b0e962a2ed3e1d978e2a12b839dd4bc1510ad0ca65e14d648a
Oct 31 19:45:01.880366 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:01.880319    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891265-r6vvt"]
Oct 31 19:45:01.880469 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:45:01.880451    4645 kuberuntime_manager.go:1707] "Unhandled Error" err="container cleanup start failed in pod maas-api-key-cleanup-e2e-worker-w6-8730d5-29891265-r6vvt_odh-ai-gateway-infra(e7198524-cd1a-4038-83c6-55f9a5391970): ErrImagePull: pull QPS exceeded" logger="UnhandledError"
Oct 31 19:45:01.880506 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:45:01.880483    4645 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with ErrImagePull: \"pull QPS exceeded\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891265-r6vvt" podUID="e7198524-cd1a-4038-83c6-55f9a5391970"
Oct 31 19:45:01.928544 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:01.928516    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq"
Oct 31 19:45:02.126169 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:02.126088    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:02.235753 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:02.235722    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq"]
Oct 31 19:45:02.236119 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:02.236054    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:45:02.236285 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:02.236260    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:45:02.236446 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:02.236420    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:45:02.646393 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:02.646290    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f5c1ab-29891265-96ksc" event={"ID":"4ae53805-07ba-4f39-bcbf-f19b67f0ad16","Type":"ContainerStarted","Data":"96510bec294712e6004b8f9047034d741f54d4dbac52192642b186eb42f282ee"}
Oct 31 19:45:02.647967 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:02.647930    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8dcfa8-29891265-9lkmt" event={"ID":"bbd4246c-22ba-4116-8e31-b53d3bdc2fbb","Type":"ContainerStarted","Data":"8475caf4e62a63a4ea6e546b905fe67af9c5a6da5cb440515abfe21cc4fcf9ea"}
Oct 31 19:45:02.649922 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:02.649889    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-785d26-29891265-8cn6g" event={"ID":"2e3605c3-ec3b-4fb7-be2a-40fe32c5adea","Type":"ContainerStarted","Data":"8ea0aa59bcc8af817bd845b6f8a2ca428d43cb3e13eb74e71d5e47a56035bb59"}
Oct 31 19:45:02.651338 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:02.651295    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1e8cf0-29891265-mbhwx" event={"ID":"57977c65-f0c2-46d1-a05c-20bd8ed579c3","Type":"ContainerStarted","Data":"e332333b2d18f171988ecc34257cfe03302668981cd9e783e345ab283e898628"}
Oct 31 19:45:02.652865 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:02.652844    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ae2e60-29891265-5s5f8" event={"ID":"3620ed03-6b03-4c6f-83be-484c41686458","Type":"ContainerStarted","Data":"e6678a4880ee9d086d1889bfb8b7446ef49b17f48b47be8ed21483b9b09b0dfd"}
Oct 31 19:45:02.654174 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:02.654151    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-03dfee-29891265-njm8t" event={"ID":"3961f45f-4b0d-4880-b8d8-e55798d4abe8","Type":"ContainerStarted","Data":"982f45c6f66cd2868373d25b98ff84111951da289dc0e241ed1172f6e1252b81"}
Oct 31 19:45:02.655375 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:02.655349    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-fd1c91-29891265-rcrqk" event={"ID":"769ee829-1015-45b1-9dd7-d9cfe372fa9c","Type":"ContainerStarted","Data":"f903486bf893265c23d3cd19a8290df2a8b3226ec37289fa53fb7397054b1c4a"}
Oct 31 19:45:02.656744 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:02.656718    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-88eaf9-29891265-zttxn" event={"ID":"25370d12-c485-47b9-986a-a775c3fbd2b7","Type":"ContainerStarted","Data":"1d03dfce658b08348f7cad84a0ad834f2cf7c7f1b11f6b5ec2a0e2d8b83ef824"}
Oct 31 19:45:02.658204 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:02.658175    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a891ce-29891265-md69g" event={"ID":"220956a9-b7c5-48b8-a598-61326c85691a","Type":"ContainerStarted","Data":"415b0234203ca2ef2c91e78b763cda02736ca4317b283d6bdfd3d848f074de18"}
Oct 31 19:45:02.658640 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:45:02.658616    4645 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with ImagePullBackOff: \"Back-off pulling image \\\"registry.redhat.io/ubi9/ubi-minimal:9.7\\\": ErrImagePull: pull QPS exceeded\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-88eaf9-29891265-zttxn" podUID="25370d12-c485-47b9-986a-a775c3fbd2b7"
Oct 31 19:45:02.659918 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:02.659895    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq" event={"ID":"37f24530-822f-4186-acfa-a7912c710c56","Type":"ContainerStarted","Data":"5fa2823567167438a072ca570e805fcb42cf724e81caee7226a2805694a07e70"}
Oct 31 19:45:02.660001 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:02.659936    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq" event={"ID":"37f24530-822f-4186-acfa-a7912c710c56","Type":"ContainerStarted","Data":"556a80c364930ecf78a02105780d7b7ee1883f649979a2a96eb7e2f5bbad38a1"}
Oct 31 19:45:02.661489 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:02.661467    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891265-r6vvt" event={"ID":"e7198524-cd1a-4038-83c6-55f9a5391970","Type":"ContainerStarted","Data":"0fc69bfd6fba82b0e962a2ed3e1d978e2a12b839dd4bc1510ad0ca65e14d648a"}
Oct 31 19:45:02.662917 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:45:02.662898    4645 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with ImagePullBackOff: \"Back-off pulling image \\\"registry.redhat.io/ubi9/ubi-minimal:9.7\\\": ErrImagePull: pull QPS exceeded\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891265-r6vvt" podUID="e7198524-cd1a-4038-83c6-55f9a5391970"
Oct 31 19:45:02.712163 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:02.712115    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq" podStartSLOduration=1.712099468 podStartE2EDuration="1.712099468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:45:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:45:02.709899453 +0000 UTC m=+1533.283133581" watchObservedRunningTime="2026-10-31 19:45:02.712099468 +0000 UTC m=+1533.285333597"
Oct 31 19:45:02.929253 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:02.929220    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq"
Oct 31 19:45:03.046870 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:03.046834    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:03.724018 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:45:03.721567    4645 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with ImagePullBackOff: \"Back-off pulling image \\\"registry.redhat.io/ubi9/ubi-minimal:9.7\\\": ErrImagePull: pull QPS exceeded\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-88eaf9-29891265-zttxn" podUID="25370d12-c485-47b9-986a-a775c3fbd2b7"
Oct 31 19:45:03.724018 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:45:03.722549    4645 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with ImagePullBackOff: \"Back-off pulling image \\\"registry.redhat.io/ubi9/ubi-minimal:9.7\\\": ErrImagePull: pull QPS exceeded\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891265-r6vvt" podUID="e7198524-cd1a-4038-83c6-55f9a5391970"
Oct 31 19:45:03.931571 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:03.931253    4645 patch_prober.go:29] interesting pod/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Oct 31 19:45:03.931571 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:03.931312    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq" podUID="37f24530-822f-4186-acfa-a7912c710c56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Oct 31 19:45:04.088955 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:04.088808    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:04.586360 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:04.584111    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-1e8cf0/test-model-1e8cf0-kserve-74747d94df-4gd76"
Oct 31 19:45:04.608880 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:04.608848    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-1e8cf0/test-model-1e8cf0-kserve-74747d94df-4gd76"
Oct 31 19:45:04.766725 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:04.766697    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-1e8cf0/test-model-1e8cf0-kserve-74747d94df-4gd76"
Oct 31 19:45:04.929003 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:04.928972    4645 patch_prober.go:29] interesting pod/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Oct 31 19:45:04.930252 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:04.930194    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq" podUID="37f24530-822f-4186-acfa-a7912c710c56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Oct 31 19:45:05.181485 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:05.181157    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:05.935654 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:05.935133    4645 patch_prober.go:29] interesting pod/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Oct 31 19:45:05.935654 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:05.935197    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq" podUID="37f24530-822f-4186-acfa-a7912c710c56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Oct 31 19:45:06.090382 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:06.077342    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:06.292604 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:45:06.292482    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 19:45:06.892268 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:06.892229    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:06.945587 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:06.941064    4645 patch_prober.go:29] interesting pod/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Oct 31 19:45:06.945587 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:06.941127    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq" podUID="37f24530-822f-4186-acfa-a7912c710c56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Oct 31 19:45:07.859352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:07.853310    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:07.942024 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:07.941988    4645 patch_prober.go:29] interesting pod/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Oct 31 19:45:07.942215 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:07.942052    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq" podUID="37f24530-822f-4186-acfa-a7912c710c56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Oct 31 19:45:08.679465 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:08.679135    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:09.182633 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:09.182218    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq"
Oct 31 19:45:09.189481 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:09.189317    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq"
Oct 31 19:45:09.191845 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:09.191675    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq"
Oct 31 19:45:09.322305 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:09.322268    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a813e2-29891265-dh4wq"]
Oct 31 19:45:09.328610 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:09.326183    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:09.594407 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:09.594088    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:09.927166 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:09.927132    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:10.094410 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:10.094368    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:10.398270 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:10.398187    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:10.816369 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:10.815908    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:11.435350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:11.435234    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:12.211128 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:12.211095    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:12.667143 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:12.665957    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d8c398-69fd677f-s95lc"
Oct 31 19:45:13.261020 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:13.260985    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:14.350445 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:14.350416    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:14.963697 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:14.963649    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:15.074839 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:15.074811    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:15.187065 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:15.187034    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:15.338871 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:15.338834    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:16.092250 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:16.092223    4645 generic.go:356] "Generic (PLEG): container finished" podID="3961f45f-4b0d-4880-b8d8-e55798d4abe8" containerID="2261a89930816ad78c75d11a93572652df4bca4d9d44c393d54261803b842edc" exitCode=60
Oct 31 19:45:16.092678 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:16.092308    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-03dfee-29891265-njm8t" event={"ID":"3961f45f-4b0d-4880-b8d8-e55798d4abe8","Type":"ContainerDied","Data":"2261a89930816ad78c75d11a93572652df4bca4d9d44c393d54261803b842edc"}
Oct 31 19:45:16.092746 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:16.092685    4645 scope.go:122] "RemoveContainer" containerID="2261a89930816ad78c75d11a93572652df4bca4d9d44c393d54261803b842edc"
Oct 31 19:45:16.094161 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:16.094139    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-fd1c91-29891265-rcrqk" event={"ID":"769ee829-1015-45b1-9dd7-d9cfe372fa9c","Type":"ContainerStarted","Data":"057510dd4613f5f5a98fd4ebbcd592be69d7302dcb127dc576167e795f1bb7c0"}
Oct 31 19:45:16.095799 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:16.095771    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-23843f-29891265-52rjc" event={"ID":"615544a2-1b13-4bce-aeb6-7b3b32e6a509","Type":"ContainerStarted","Data":"a1fb6962cafa58d9d92cb5bff0154fb44a21e803bd43c734aab516bb11e1ad14"}
Oct 31 19:45:16.097745 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:16.097719    4645 generic.go:356] "Generic (PLEG): container finished" podID="220956a9-b7c5-48b8-a598-61326c85691a" containerID="2fd1091566eec6cf0b0ffae72b713ff067b544449c88bf4f1eef6a563fe290db" exitCode=60
Oct 31 19:45:16.097869 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:16.097783    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a891ce-29891265-md69g" event={"ID":"220956a9-b7c5-48b8-a598-61326c85691a","Type":"ContainerDied","Data":"2fd1091566eec6cf0b0ffae72b713ff067b544449c88bf4f1eef6a563fe290db"}
Oct 31 19:45:16.098063 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:16.098022    4645 scope.go:122] "RemoveContainer" containerID="2fd1091566eec6cf0b0ffae72b713ff067b544449c88bf4f1eef6a563fe290db"
Oct 31 19:45:16.099735 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:16.099716    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891265-9vwg9" event={"ID":"e6f57036-6c45-4def-a9c1-0132ab845b3e","Type":"ContainerStarted","Data":"82fa4c9d5bede9a62e9c308b1c560f4d6a3d37aef819bb6ed06d4fe361fca00f"}
Oct 31 19:45:16.101465 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:16.101443    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f5c1ab-29891265-96ksc" event={"ID":"4ae53805-07ba-4f39-bcbf-f19b67f0ad16","Type":"ContainerStarted","Data":"f81944db505f31ccbbe57eb0b51acf00499f56387d5893e64f9ec4da02f30c4e"}
Oct 31 19:45:16.102926 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:16.102904    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8dcfa8-29891265-9lkmt" event={"ID":"bbd4246c-22ba-4116-8e31-b53d3bdc2fbb","Type":"ContainerStarted","Data":"7eb6a73923070b715ba0af509fa6361f97b6650d7b82d8f436a6f4be3f3abe4a"}
Oct 31 19:45:16.104518 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:16.104496    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-785d26-29891265-8cn6g" event={"ID":"2e3605c3-ec3b-4fb7-be2a-40fe32c5adea","Type":"ContainerStarted","Data":"f885fb780b2aca10240f93f5720cb25732566a400b1dec8b889809cafcae3717"}
Oct 31 19:45:16.106162 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:16.106137    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1e8cf0-29891265-mbhwx" event={"ID":"57977c65-f0c2-46d1-a05c-20bd8ed579c3","Type":"ContainerStarted","Data":"009e16e2b4966db62f56b1ca9470e473315d3672073405432d27f31535f3b4a1"}
Oct 31 19:45:16.107846 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:16.107820    4645 generic.go:356] "Generic (PLEG): container finished" podID="3620ed03-6b03-4c6f-83be-484c41686458" containerID="3fbcb75ce7baf277ccb221417f957e80a68381dcdc63f3aead7a5880705fae03" exitCode=60
Oct 31 19:45:16.107959 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:16.107906    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ae2e60-29891265-5s5f8" event={"ID":"3620ed03-6b03-4c6f-83be-484c41686458","Type":"ContainerDied","Data":"3fbcb75ce7baf277ccb221417f957e80a68381dcdc63f3aead7a5880705fae03"}
Oct 31 19:45:16.108153 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:16.108088    4645 scope.go:122] "RemoveContainer" containerID="3fbcb75ce7baf277ccb221417f957e80a68381dcdc63f3aead7a5880705fae03"
Oct 31 19:45:16.112093 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:16.112070    4645 generic.go:356] "Generic (PLEG): container finished" podID="53b785a3-a308-43ec-92b9-ccd7d6a8bcad" containerID="007d65adbc3ee976859a894db4253a5813fbef193ee7e9946195a3178dfd721e" exitCode=60
Oct 31 19:45:16.112404 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:16.112381    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-435728-29891265-tfbvw" event={"ID":"53b785a3-a308-43ec-92b9-ccd7d6a8bcad","Type":"ContainerDied","Data":"007d65adbc3ee976859a894db4253a5813fbef193ee7e9946195a3178dfd721e"}
Oct 31 19:45:16.112730 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:16.112680    4645 scope.go:122] "RemoveContainer" containerID="007d65adbc3ee976859a894db4253a5813fbef193ee7e9946195a3178dfd721e"
Oct 31 19:45:16.121264 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:16.121235    4645 generic.go:356] "Generic (PLEG): container finished" podID="97e726ce-2b00-4703-8c9e-277ea9bb4c4a" containerID="cce52a8a1396c6c13285253cd47253e38e10409fb4611fab522b347b32f9f9c9" exitCode=6
Oct 31 19:45:16.121361 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:16.121286    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a813e2-29891265-dh4wq" event={"ID":"97e726ce-2b00-4703-8c9e-277ea9bb4c4a","Type":"ContainerDied","Data":"cce52a8a1396c6c13285253cd47253e38e10409fb4611fab522b347b32f9f9c9"}
Oct 31 19:45:16.143857 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:16.143819    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-23843f-29891265-52rjc" podStartSLOduration=2.759159096 podStartE2EDuration="16.143804408s" totalImagesPullingTime="13.384645312s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:45:16.135390567 +0000 UTC m=+1546.708624692" watchObservedRunningTime="2026-10-31 19:45:16.143804408 +0000 UTC m=+1546.717038531"
Oct 31 19:45:16.164751 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:16.160793    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:16.164751 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:16.162493    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-fd1c91-29891265-rcrqk" podStartSLOduration=3.22418833 podStartE2EDuration="16.16247621s" totalImagesPullingTime="12.93828788s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:45:16.153412824 +0000 UTC m=+1546.726646956" watchObservedRunningTime="2026-10-31 19:45:16.16247621 +0000 UTC m=+1546.735710340"
Oct 31 19:45:16.207944 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:16.207883    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891265-9vwg9" podStartSLOduration=3.010756569 podStartE2EDuration="16.207864273s" totalImagesPullingTime="13.197107704s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:45:16.198836802 +0000 UTC m=+1546.772070930" watchObservedRunningTime="2026-10-31 19:45:16.207864273 +0000 UTC m=+1546.781098402"
Oct 31 19:45:16.281930 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:16.281879    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8dcfa8-29891265-9lkmt" podStartSLOduration=3.283714726 podStartE2EDuration="16.28186139s" totalImagesPullingTime="12.998146664s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:45:16.272568091 +0000 UTC m=+1546.845802224" watchObservedRunningTime="2026-10-31 19:45:16.28186139 +0000 UTC m=+1546.855095519"
Oct 31 19:45:16.326756 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:16.326696    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-785d26-29891265-8cn6g" podStartSLOduration=3.262878383 podStartE2EDuration="16.326677639s" totalImagesPullingTime="13.063799256s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:45:16.32444017 +0000 UTC m=+1546.897674311" watchObservedRunningTime="2026-10-31 19:45:16.326677639 +0000 UTC m=+1546.899911768"
Oct 31 19:45:16.350669 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:16.350618    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1e8cf0-29891265-mbhwx" podStartSLOduration=3.108486876 podStartE2EDuration="16.350600529s" totalImagesPullingTime="13.242113653s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:45:16.341610147 +0000 UTC m=+1546.914844276" watchObservedRunningTime="2026-10-31 19:45:16.350600529 +0000 UTC m=+1546.923834655"
Oct 31 19:45:16.383921 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:16.383875    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f5c1ab-29891265-96ksc" podStartSLOduration=3.259669338 podStartE2EDuration="16.38386239s" totalImagesPullingTime="13.124193052s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:45:16.381523311 +0000 UTC m=+1546.954757439" watchObservedRunningTime="2026-10-31 19:45:16.38386239 +0000 UTC m=+1546.957096517"
Oct 31 19:45:16.644532 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:16.644508    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:16.813548 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:16.813526    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a813e2-29891265-dh4wq"
Oct 31 19:45:16.979460 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:16.979438    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97e726ce-2b00-4703-8c9e-277ea9bb4c4a-kube-api-access-rdzb5\" (UniqueName: \"kubernetes.io/projected/97e726ce-2b00-4703-8c9e-277ea9bb4c4a-kube-api-access-rdzb5\") pod \"97e726ce-2b00-4703-8c9e-277ea9bb4c4a\" (UID: \"97e726ce-2b00-4703-8c9e-277ea9bb4c4a\") "
Oct 31 19:45:16.979635 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:16.979539    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97e726ce-2b00-4703-8c9e-277ea9bb4c4a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97e726ce-2b00-4703-8c9e-277ea9bb4c4a-maas-api-service-ca\") pod \"97e726ce-2b00-4703-8c9e-277ea9bb4c4a\" (UID: \"97e726ce-2b00-4703-8c9e-277ea9bb4c4a\") "
Oct 31 19:45:16.979887 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:16.979859    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97e726ce-2b00-4703-8c9e-277ea9bb4c4a-maas-api-service-ca" pod "97e726ce-2b00-4703-8c9e-277ea9bb4c4a" (UID: "97e726ce-2b00-4703-8c9e-277ea9bb4c4a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:45:16.980114 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:16.980095    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/97e726ce-2b00-4703-8c9e-277ea9bb4c4a-maas-api-service-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:16.981692 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:16.981673    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97e726ce-2b00-4703-8c9e-277ea9bb4c4a-kube-api-access-rdzb5" pod "97e726ce-2b00-4703-8c9e-277ea9bb4c4a" (UID: "97e726ce-2b00-4703-8c9e-277ea9bb4c4a"). InnerVolumeSpecName "kube-api-access-rdzb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:45:17.081527 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:17.081503    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdzb5\" (UniqueName: \"kubernetes.io/projected/97e726ce-2b00-4703-8c9e-277ea9bb4c4a-kube-api-access-rdzb5\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:17.130910 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:17.130882    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-435728-29891265-tfbvw" event={"ID":"53b785a3-a308-43ec-92b9-ccd7d6a8bcad","Type":"ContainerStarted","Data":"e263899d5c3d156e483a87f30c8d878ae4a89e6fdfb67c0ee69a861404237fd1"}
Oct 31 19:45:17.132252 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:17.132231    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a813e2-29891265-dh4wq"
Oct 31 19:45:17.132359 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:17.132247    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a813e2-29891265-dh4wq" event={"ID":"97e726ce-2b00-4703-8c9e-277ea9bb4c4a","Type":"ContainerDied","Data":"0a37feb215e107ccd7a60b6134cfea14faa61b10b2092298d41dabca60252103"}
Oct 31 19:45:17.132359 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:17.132301    4645 scope.go:122] "RemoveContainer" containerID="cce52a8a1396c6c13285253cd47253e38e10409fb4611fab522b347b32f9f9c9"
Oct 31 19:45:17.134769 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:17.134743    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-03dfee-29891265-njm8t" event={"ID":"3961f45f-4b0d-4880-b8d8-e55798d4abe8","Type":"ContainerStarted","Data":"cd74e21b378df60e66d0039caadb0de062a9db8891dd59bed8e5863b5134ffe7"}
Oct 31 19:45:17.136442 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:17.136417    4645 generic.go:356] "Generic (PLEG): container finished" podID="769ee829-1015-45b1-9dd7-d9cfe372fa9c" containerID="057510dd4613f5f5a98fd4ebbcd592be69d7302dcb127dc576167e795f1bb7c0" exitCode=60
Oct 31 19:45:17.136561 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:17.136537    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-fd1c91-29891265-rcrqk" event={"ID":"769ee829-1015-45b1-9dd7-d9cfe372fa9c","Type":"ContainerDied","Data":"057510dd4613f5f5a98fd4ebbcd592be69d7302dcb127dc576167e795f1bb7c0"}
Oct 31 19:45:17.136860 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:17.136837    4645 scope.go:122] "RemoveContainer" containerID="057510dd4613f5f5a98fd4ebbcd592be69d7302dcb127dc576167e795f1bb7c0"
Oct 31 19:45:17.138396 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:17.138369    4645 generic.go:356] "Generic (PLEG): container finished" podID="615544a2-1b13-4bce-aeb6-7b3b32e6a509" containerID="a1fb6962cafa58d9d92cb5bff0154fb44a21e803bd43c734aab516bb11e1ad14" exitCode=60
Oct 31 19:45:17.138485 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:17.138447    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-23843f-29891265-52rjc" event={"ID":"615544a2-1b13-4bce-aeb6-7b3b32e6a509","Type":"ContainerDied","Data":"a1fb6962cafa58d9d92cb5bff0154fb44a21e803bd43c734aab516bb11e1ad14"}
Oct 31 19:45:17.138791 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:17.138774    4645 scope.go:122] "RemoveContainer" containerID="a1fb6962cafa58d9d92cb5bff0154fb44a21e803bd43c734aab516bb11e1ad14"
Oct 31 19:45:17.140625 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:17.140601    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a891ce-29891265-md69g" event={"ID":"220956a9-b7c5-48b8-a598-61326c85691a","Type":"ContainerStarted","Data":"1874b70e23d4508fcc45132c388072af9fed705d09a94acd6034702e51696a68"}
Oct 31 19:45:17.142426 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:17.142404    4645 generic.go:356] "Generic (PLEG): container finished" podID="e6f57036-6c45-4def-a9c1-0132ab845b3e" containerID="82fa4c9d5bede9a62e9c308b1c560f4d6a3d37aef819bb6ed06d4fe361fca00f" exitCode=60
Oct 31 19:45:17.142532 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:17.142459    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891265-9vwg9" event={"ID":"e6f57036-6c45-4def-a9c1-0132ab845b3e","Type":"ContainerDied","Data":"82fa4c9d5bede9a62e9c308b1c560f4d6a3d37aef819bb6ed06d4fe361fca00f"}
Oct 31 19:45:17.142723 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:17.142706    4645 scope.go:122] "RemoveContainer" containerID="82fa4c9d5bede9a62e9c308b1c560f4d6a3d37aef819bb6ed06d4fe361fca00f"
Oct 31 19:45:17.144069 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:17.144053    4645 generic.go:356] "Generic (PLEG): container finished" podID="e7198524-cd1a-4038-83c6-55f9a5391970" containerID="106c430a2500395ee4a8d208081034cea2a821cb49bd5a3bc7ec14844193f8df" exitCode=60
Oct 31 19:45:17.144158 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:17.144120    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891265-r6vvt" event={"ID":"e7198524-cd1a-4038-83c6-55f9a5391970","Type":"ContainerDied","Data":"106c430a2500395ee4a8d208081034cea2a821cb49bd5a3bc7ec14844193f8df"}
Oct 31 19:45:17.144533 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:17.144515    4645 scope.go:122] "RemoveContainer" containerID="106c430a2500395ee4a8d208081034cea2a821cb49bd5a3bc7ec14844193f8df"
Oct 31 19:45:17.146198 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:17.146177    4645 generic.go:356] "Generic (PLEG): container finished" podID="4ae53805-07ba-4f39-bcbf-f19b67f0ad16" containerID="f81944db505f31ccbbe57eb0b51acf00499f56387d5893e64f9ec4da02f30c4e" exitCode=60
Oct 31 19:45:17.146288 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:17.146245    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f5c1ab-29891265-96ksc" event={"ID":"4ae53805-07ba-4f39-bcbf-f19b67f0ad16","Type":"ContainerDied","Data":"f81944db505f31ccbbe57eb0b51acf00499f56387d5893e64f9ec4da02f30c4e"}
Oct 31 19:45:17.146492 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:17.146474    4645 scope.go:122] "RemoveContainer" containerID="f81944db505f31ccbbe57eb0b51acf00499f56387d5893e64f9ec4da02f30c4e"
Oct 31 19:45:17.150054 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:17.150035    4645 generic.go:356] "Generic (PLEG): container finished" podID="bbd4246c-22ba-4116-8e31-b53d3bdc2fbb" containerID="7eb6a73923070b715ba0af509fa6361f97b6650d7b82d8f436a6f4be3f3abe4a" exitCode=60
Oct 31 19:45:17.150423 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:17.150104    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8dcfa8-29891265-9lkmt" event={"ID":"bbd4246c-22ba-4116-8e31-b53d3bdc2fbb","Type":"ContainerDied","Data":"7eb6a73923070b715ba0af509fa6361f97b6650d7b82d8f436a6f4be3f3abe4a"}
Oct 31 19:45:17.150423 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:17.150315    4645 scope.go:122] "RemoveContainer" containerID="7eb6a73923070b715ba0af509fa6361f97b6650d7b82d8f436a6f4be3f3abe4a"
Oct 31 19:45:17.153302 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:17.152223    4645 generic.go:356] "Generic (PLEG): container finished" podID="2e3605c3-ec3b-4fb7-be2a-40fe32c5adea" containerID="f885fb780b2aca10240f93f5720cb25732566a400b1dec8b889809cafcae3717" exitCode=60
Oct 31 19:45:17.153302 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:17.152300    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-785d26-29891265-8cn6g" event={"ID":"2e3605c3-ec3b-4fb7-be2a-40fe32c5adea","Type":"ContainerDied","Data":"f885fb780b2aca10240f93f5720cb25732566a400b1dec8b889809cafcae3717"}
Oct 31 19:45:17.153302 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:17.152578    4645 scope.go:122] "RemoveContainer" containerID="f885fb780b2aca10240f93f5720cb25732566a400b1dec8b889809cafcae3717"
Oct 31 19:45:17.154142 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:17.154125    4645 generic.go:356] "Generic (PLEG): container finished" podID="57977c65-f0c2-46d1-a05c-20bd8ed579c3" containerID="009e16e2b4966db62f56b1ca9470e473315d3672073405432d27f31535f3b4a1" exitCode=60
Oct 31 19:45:17.154209 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:17.154190    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1e8cf0-29891265-mbhwx" event={"ID":"57977c65-f0c2-46d1-a05c-20bd8ed579c3","Type":"ContainerDied","Data":"009e16e2b4966db62f56b1ca9470e473315d3672073405432d27f31535f3b4a1"}
Oct 31 19:45:17.154401 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:17.154379    4645 scope.go:122] "RemoveContainer" containerID="009e16e2b4966db62f56b1ca9470e473315d3672073405432d27f31535f3b4a1"
Oct 31 19:45:17.156291 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:17.156270    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ae2e60-29891265-5s5f8" event={"ID":"3620ed03-6b03-4c6f-83be-484c41686458","Type":"ContainerStarted","Data":"613208704e41a1fa0120083831fb7a8cb1c6d08bef79521d76b03c00865bb5c6"}
Oct 31 19:45:17.162269 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:17.162231    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-435728-29891265-tfbvw" podStartSLOduration=3.934251522 podStartE2EDuration="17.16221951s" totalImagesPullingTime="13.227967988s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:45:17.157944062 +0000 UTC m=+1547.731178192" watchObservedRunningTime="2026-10-31 19:45:17.16221951 +0000 UTC m=+1547.735453637"
Oct 31 19:45:17.224782 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:17.224659    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:17.241377 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:17.234445    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a813e2-29891265-dh4wq"]
Oct 31 19:45:17.275111 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:17.272045    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a813e2-29891265-dh4wq"]
Oct 31 19:45:17.333427 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:17.333362    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ae2e60-29891265-5s5f8" podStartSLOduration=4.422568245 podStartE2EDuration="17.333324463s" totalImagesPullingTime="12.910756218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:45:17.330628844 +0000 UTC m=+1547.903862973" watchObservedRunningTime="2026-10-31 19:45:17.333324463 +0000 UTC m=+1547.906558593"
Oct 31 19:45:17.500039 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:17.499974    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:17.521957 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:17.521915    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-03dfee-29891265-njm8t" podStartSLOduration=5.022701829 podStartE2EDuration="17.521901569s" totalImagesPullingTime="12.49919974s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:45:17.518532882 +0000 UTC m=+1548.091767010" watchObservedRunningTime="2026-10-31 19:45:17.521901569 +0000 UTC m=+1548.095135697"
Oct 31 19:45:17.543144 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:17.543101    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a891ce-29891265-md69g" podStartSLOduration=4.496976457 podStartE2EDuration="17.543089582s" totalImagesPullingTime="13.046113125s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:45:17.540061045 +0000 UTC m=+1548.113295174" watchObservedRunningTime="2026-10-31 19:45:17.543089582 +0000 UTC m=+1548.116323709"
Oct 31 19:45:17.627141 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:17.627116    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97e726ce-2b00-4703-8c9e-277ea9bb4c4a" path="/var/lib/kubelet/pods/97e726ce-2b00-4703-8c9e-277ea9bb4c4a/volumes"
Oct 31 19:45:18.175849 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:18.175119    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-23843f-29891265-52rjc" event={"ID":"615544a2-1b13-4bce-aeb6-7b3b32e6a509","Type":"ContainerStarted","Data":"b7c6416e274fb6e0164de111d20cd44a8684834f78955789f14321b26b853dbe"}
Oct 31 19:45:18.177409 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:18.177377    4645 generic.go:356] "Generic (PLEG): container finished" podID="220956a9-b7c5-48b8-a598-61326c85691a" containerID="1874b70e23d4508fcc45132c388072af9fed705d09a94acd6034702e51696a68" exitCode=60
Oct 31 19:45:18.177636 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:18.177418    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a891ce-29891265-md69g" event={"ID":"220956a9-b7c5-48b8-a598-61326c85691a","Type":"ContainerDied","Data":"1874b70e23d4508fcc45132c388072af9fed705d09a94acd6034702e51696a68"}
Oct 31 19:45:18.177745 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:18.177679    4645 scope.go:122] "RemoveContainer" containerID="2fd1091566eec6cf0b0ffae72b713ff067b544449c88bf4f1eef6a563fe290db"
Oct 31 19:45:18.177817 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:18.177753    4645 scope.go:122] "RemoveContainer" containerID="1874b70e23d4508fcc45132c388072af9fed705d09a94acd6034702e51696a68"
Oct 31 19:45:18.178184 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:45:18.178159    4645 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-a891ce-29891265-md69g_odh-ai-gateway-infra(220956a9-b7c5-48b8-a598-61326c85691a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a891ce-29891265-md69g" podUID="220956a9-b7c5-48b8-a598-61326c85691a"
Oct 31 19:45:18.194893 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:18.194865    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891265-9vwg9" event={"ID":"e6f57036-6c45-4def-a9c1-0132ab845b3e","Type":"ContainerStarted","Data":"9e22ba3b319379c0fce668b1e733ca36e551a79dd8980e5b28c87a22f90b958a"}
Oct 31 19:45:18.197919 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:18.197892    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891265-r6vvt" event={"ID":"e7198524-cd1a-4038-83c6-55f9a5391970","Type":"ContainerStarted","Data":"32dbade5037b8291ce61e8fcb623c66b9b4cf1fb48db6b83f724a52fb9bf2271"}
Oct 31 19:45:18.201393 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:18.201366    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f5c1ab-29891265-96ksc" event={"ID":"4ae53805-07ba-4f39-bcbf-f19b67f0ad16","Type":"ContainerStarted","Data":"3d8e0da5a7cf672372700edd09b8cf8ab9a8ddc4a125e03fade92d9508b0dfb7"}
Oct 31 19:45:18.204880 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:18.204854    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8dcfa8-29891265-9lkmt" event={"ID":"bbd4246c-22ba-4116-8e31-b53d3bdc2fbb","Type":"ContainerStarted","Data":"bedfa461ebbc34af1c8ee00169ea95fe31be59b33f410fe57163b88cc6b86e1a"}
Oct 31 19:45:18.209655 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:18.209626    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-785d26-29891265-8cn6g" event={"ID":"2e3605c3-ec3b-4fb7-be2a-40fe32c5adea","Type":"ContainerStarted","Data":"c16ea1602ad97fb5a566344e8f7cf5f40c8d9b639d55ae61eae4f88dfde4d22e"}
Oct 31 19:45:18.211311 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:18.211288    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:18.213419 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:18.213394    4645 generic.go:356] "Generic (PLEG): container finished" podID="3620ed03-6b03-4c6f-83be-484c41686458" containerID="613208704e41a1fa0120083831fb7a8cb1c6d08bef79521d76b03c00865bb5c6" exitCode=60
Oct 31 19:45:18.213519 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:18.213451    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ae2e60-29891265-5s5f8" event={"ID":"3620ed03-6b03-4c6f-83be-484c41686458","Type":"ContainerDied","Data":"613208704e41a1fa0120083831fb7a8cb1c6d08bef79521d76b03c00865bb5c6"}
Oct 31 19:45:18.213519 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:18.213504    4645 scope.go:122] "RemoveContainer" containerID="3fbcb75ce7baf277ccb221417f957e80a68381dcdc63f3aead7a5880705fae03"
Oct 31 19:45:18.213800 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:18.213771    4645 scope.go:122] "RemoveContainer" containerID="613208704e41a1fa0120083831fb7a8cb1c6d08bef79521d76b03c00865bb5c6"
Oct 31 19:45:18.214111 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:45:18.214085    4645 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-ae2e60-29891265-5s5f8_odh-ai-gateway-infra(3620ed03-6b03-4c6f-83be-484c41686458)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ae2e60-29891265-5s5f8" podUID="3620ed03-6b03-4c6f-83be-484c41686458"
Oct 31 19:45:18.216107 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:18.216086    4645 generic.go:356] "Generic (PLEG): container finished" podID="53b785a3-a308-43ec-92b9-ccd7d6a8bcad" containerID="e263899d5c3d156e483a87f30c8d878ae4a89e6fdfb67c0ee69a861404237fd1" exitCode=60
Oct 31 19:45:18.216228 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:18.216186    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-435728-29891265-tfbvw" event={"ID":"53b785a3-a308-43ec-92b9-ccd7d6a8bcad","Type":"ContainerDied","Data":"e263899d5c3d156e483a87f30c8d878ae4a89e6fdfb67c0ee69a861404237fd1"}
Oct 31 19:45:18.216640 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:18.216531    4645 scope.go:122] "RemoveContainer" containerID="e263899d5c3d156e483a87f30c8d878ae4a89e6fdfb67c0ee69a861404237fd1"
Oct 31 19:45:18.216869 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:45:18.216821    4645 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-435728-29891265-tfbvw_odh-ai-gateway-infra(53b785a3-a308-43ec-92b9-ccd7d6a8bcad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-435728-29891265-tfbvw" podUID="53b785a3-a308-43ec-92b9-ccd7d6a8bcad"
Oct 31 19:45:18.222739 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:18.222717    4645 generic.go:356] "Generic (PLEG): container finished" podID="3961f45f-4b0d-4880-b8d8-e55798d4abe8" containerID="cd74e21b378df60e66d0039caadb0de062a9db8891dd59bed8e5863b5134ffe7" exitCode=60
Oct 31 19:45:18.222844 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:18.222791    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-03dfee-29891265-njm8t" event={"ID":"3961f45f-4b0d-4880-b8d8-e55798d4abe8","Type":"ContainerDied","Data":"cd74e21b378df60e66d0039caadb0de062a9db8891dd59bed8e5863b5134ffe7"}
Oct 31 19:45:18.225502 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:18.225475    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-fd1c91-29891265-rcrqk" event={"ID":"769ee829-1015-45b1-9dd7-d9cfe372fa9c","Type":"ContainerStarted","Data":"7b43a4d11228fc8314c3e91b60a4b756c64af2b7ac42fbcc21986e6d0cba2771"}
Oct 31 19:45:18.226180 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:18.226162    4645 scope.go:122] "RemoveContainer" containerID="cd74e21b378df60e66d0039caadb0de062a9db8891dd59bed8e5863b5134ffe7"
Oct 31 19:45:18.226487 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:45:18.226464    4645 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-03dfee-29891265-njm8t_odh-ai-gateway-infra(3961f45f-4b0d-4880-b8d8-e55798d4abe8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-03dfee-29891265-njm8t" podUID="3961f45f-4b0d-4880-b8d8-e55798d4abe8"
Oct 31 19:45:18.232781 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:18.232764    4645 scope.go:122] "RemoveContainer" containerID="007d65adbc3ee976859a894db4253a5813fbef193ee7e9946195a3178dfd721e"
Oct 31 19:45:18.258545 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:18.258223    4645 scope.go:122] "RemoveContainer" containerID="2261a89930816ad78c75d11a93572652df4bca4d9d44c393d54261803b842edc"
Oct 31 19:45:18.334426 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:18.334293    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891265-r6vvt" podStartSLOduration=4.591169674 podStartE2EDuration="18.334274089s" totalImagesPullingTime="13.743104415s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:45:18.329526595 +0000 UTC m=+1548.902760725" watchObservedRunningTime="2026-10-31 19:45:18.334274089 +0000 UTC m=+1548.907508219"
Oct 31 19:45:19.161066 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:19.161034    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:19.273120 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:19.273071    4645 scope.go:122] "RemoveContainer" containerID="e263899d5c3d156e483a87f30c8d878ae4a89e6fdfb67c0ee69a861404237fd1"
Oct 31 19:45:19.273743 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:45:19.273631    4645 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-435728-29891265-tfbvw_odh-ai-gateway-infra(53b785a3-a308-43ec-92b9-ccd7d6a8bcad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-435728-29891265-tfbvw" podUID="53b785a3-a308-43ec-92b9-ccd7d6a8bcad"
Oct 31 19:45:19.275405 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:19.275379    4645 scope.go:122] "RemoveContainer" containerID="cd74e21b378df60e66d0039caadb0de062a9db8891dd59bed8e5863b5134ffe7"
Oct 31 19:45:19.275819 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:45:19.275791    4645 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-03dfee-29891265-njm8t_odh-ai-gateway-infra(3961f45f-4b0d-4880-b8d8-e55798d4abe8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-03dfee-29891265-njm8t" podUID="3961f45f-4b0d-4880-b8d8-e55798d4abe8"
Oct 31 19:45:19.277057 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:19.277034    4645 generic.go:356] "Generic (PLEG): container finished" podID="769ee829-1015-45b1-9dd7-d9cfe372fa9c" containerID="7b43a4d11228fc8314c3e91b60a4b756c64af2b7ac42fbcc21986e6d0cba2771" exitCode=60
Oct 31 19:45:19.277159 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:19.277126    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-fd1c91-29891265-rcrqk" event={"ID":"769ee829-1015-45b1-9dd7-d9cfe372fa9c","Type":"ContainerDied","Data":"7b43a4d11228fc8314c3e91b60a4b756c64af2b7ac42fbcc21986e6d0cba2771"}
Oct 31 19:45:19.277519 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:19.277167    4645 scope.go:122] "RemoveContainer" containerID="057510dd4613f5f5a98fd4ebbcd592be69d7302dcb127dc576167e795f1bb7c0"
Oct 31 19:45:19.277519 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:19.277487    4645 scope.go:122] "RemoveContainer" containerID="7b43a4d11228fc8314c3e91b60a4b756c64af2b7ac42fbcc21986e6d0cba2771"
Oct 31 19:45:19.277925 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:45:19.277904    4645 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-fd1c91-29891265-rcrqk_odh-ai-gateway-infra(769ee829-1015-45b1-9dd7-d9cfe372fa9c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-fd1c91-29891265-rcrqk" podUID="769ee829-1015-45b1-9dd7-d9cfe372fa9c"
Oct 31 19:45:19.279114 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:19.279092    4645 generic.go:356] "Generic (PLEG): container finished" podID="25370d12-c485-47b9-986a-a775c3fbd2b7" containerID="db6591caa16f6c6aad79a86e399510b6b943cb01a53175f6bbf8a971b988b3ee" exitCode=60
Oct 31 19:45:19.279213 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:19.279158    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-88eaf9-29891265-zttxn" event={"ID":"25370d12-c485-47b9-986a-a775c3fbd2b7","Type":"ContainerDied","Data":"db6591caa16f6c6aad79a86e399510b6b943cb01a53175f6bbf8a971b988b3ee"}
Oct 31 19:45:19.279520 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:19.279503    4645 scope.go:122] "RemoveContainer" containerID="db6591caa16f6c6aad79a86e399510b6b943cb01a53175f6bbf8a971b988b3ee"
Oct 31 19:45:19.281570 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:19.281550    4645 generic.go:356] "Generic (PLEG): container finished" podID="615544a2-1b13-4bce-aeb6-7b3b32e6a509" containerID="b7c6416e274fb6e0164de111d20cd44a8684834f78955789f14321b26b853dbe" exitCode=60
Oct 31 19:45:19.281678 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:19.281635    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-23843f-29891265-52rjc" event={"ID":"615544a2-1b13-4bce-aeb6-7b3b32e6a509","Type":"ContainerDied","Data":"b7c6416e274fb6e0164de111d20cd44a8684834f78955789f14321b26b853dbe"}
Oct 31 19:45:19.281906 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:19.281882    4645 scope.go:122] "RemoveContainer" containerID="b7c6416e274fb6e0164de111d20cd44a8684834f78955789f14321b26b853dbe"
Oct 31 19:45:19.282212 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:45:19.282189    4645 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-23843f-29891265-52rjc_odh-ai-gateway-infra(615544a2-1b13-4bce-aeb6-7b3b32e6a509)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-23843f-29891265-52rjc" podUID="615544a2-1b13-4bce-aeb6-7b3b32e6a509"
Oct 31 19:45:19.303514 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:19.303485    4645 scope.go:122] "RemoveContainer" containerID="1874b70e23d4508fcc45132c388072af9fed705d09a94acd6034702e51696a68"
Oct 31 19:45:19.303959 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:45:19.303926    4645 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-a891ce-29891265-md69g_odh-ai-gateway-infra(220956a9-b7c5-48b8-a598-61326c85691a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a891ce-29891265-md69g" podUID="220956a9-b7c5-48b8-a598-61326c85691a"
Oct 31 19:45:19.307857 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:19.307828    4645 generic.go:356] "Generic (PLEG): container finished" podID="e6f57036-6c45-4def-a9c1-0132ab845b3e" containerID="9e22ba3b319379c0fce668b1e733ca36e551a79dd8980e5b28c87a22f90b958a" exitCode=60
Oct 31 19:45:19.307976 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:19.307932    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891265-9vwg9" event={"ID":"e6f57036-6c45-4def-a9c1-0132ab845b3e","Type":"ContainerDied","Data":"9e22ba3b319379c0fce668b1e733ca36e551a79dd8980e5b28c87a22f90b958a"}
Oct 31 19:45:19.308204 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:19.308181    4645 scope.go:122] "RemoveContainer" containerID="9e22ba3b319379c0fce668b1e733ca36e551a79dd8980e5b28c87a22f90b958a"
Oct 31 19:45:19.309463 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:45:19.308552    4645 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891265-9vwg9_odh-ai-gateway-infra(e6f57036-6c45-4def-a9c1-0132ab845b3e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891265-9vwg9" podUID="e6f57036-6c45-4def-a9c1-0132ab845b3e"
Oct 31 19:45:19.316454 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:19.316433    4645 scope.go:122] "RemoveContainer" containerID="a1fb6962cafa58d9d92cb5bff0154fb44a21e803bd43c734aab516bb11e1ad14"
Oct 31 19:45:19.317298 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:19.317275    4645 generic.go:356] "Generic (PLEG): container finished" podID="e7198524-cd1a-4038-83c6-55f9a5391970" containerID="32dbade5037b8291ce61e8fcb623c66b9b4cf1fb48db6b83f724a52fb9bf2271" exitCode=60
Oct 31 19:45:19.317523 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:19.317473    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891265-r6vvt" event={"ID":"e7198524-cd1a-4038-83c6-55f9a5391970","Type":"ContainerDied","Data":"32dbade5037b8291ce61e8fcb623c66b9b4cf1fb48db6b83f724a52fb9bf2271"}
Oct 31 19:45:19.317891 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:19.317872    4645 scope.go:122] "RemoveContainer" containerID="32dbade5037b8291ce61e8fcb623c66b9b4cf1fb48db6b83f724a52fb9bf2271"
Oct 31 19:45:19.318411 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:45:19.318366    4645 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-8730d5-29891265-r6vvt_odh-ai-gateway-infra(e7198524-cd1a-4038-83c6-55f9a5391970)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891265-r6vvt" podUID="e7198524-cd1a-4038-83c6-55f9a5391970"
Oct 31 19:45:19.321592 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:19.321570    4645 generic.go:356] "Generic (PLEG): container finished" podID="4ae53805-07ba-4f39-bcbf-f19b67f0ad16" containerID="3d8e0da5a7cf672372700edd09b8cf8ab9a8ddc4a125e03fade92d9508b0dfb7" exitCode=60
Oct 31 19:45:19.321706 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:19.321622    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f5c1ab-29891265-96ksc" event={"ID":"4ae53805-07ba-4f39-bcbf-f19b67f0ad16","Type":"ContainerDied","Data":"3d8e0da5a7cf672372700edd09b8cf8ab9a8ddc4a125e03fade92d9508b0dfb7"}
Oct 31 19:45:19.322051 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:19.322023    4645 scope.go:122] "RemoveContainer" containerID="3d8e0da5a7cf672372700edd09b8cf8ab9a8ddc4a125e03fade92d9508b0dfb7"
Oct 31 19:45:19.322467 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:45:19.322438    4645 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-f5c1ab-29891265-96ksc_odh-ai-gateway-infra(4ae53805-07ba-4f39-bcbf-f19b67f0ad16)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f5c1ab-29891265-96ksc" podUID="4ae53805-07ba-4f39-bcbf-f19b67f0ad16"
Oct 31 19:45:19.324158 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:19.324136    4645 generic.go:356] "Generic (PLEG): container finished" podID="bbd4246c-22ba-4116-8e31-b53d3bdc2fbb" containerID="bedfa461ebbc34af1c8ee00169ea95fe31be59b33f410fe57163b88cc6b86e1a" exitCode=60
Oct 31 19:45:19.324405 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:19.324377    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8dcfa8-29891265-9lkmt" event={"ID":"bbd4246c-22ba-4116-8e31-b53d3bdc2fbb","Type":"ContainerDied","Data":"bedfa461ebbc34af1c8ee00169ea95fe31be59b33f410fe57163b88cc6b86e1a"}
Oct 31 19:45:19.324575 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:19.324559    4645 scope.go:122] "RemoveContainer" containerID="bedfa461ebbc34af1c8ee00169ea95fe31be59b33f410fe57163b88cc6b86e1a"
Oct 31 19:45:19.324919 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:45:19.324898    4645 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-8dcfa8-29891265-9lkmt_odh-ai-gateway-infra(bbd4246c-22ba-4116-8e31-b53d3bdc2fbb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8dcfa8-29891265-9lkmt" podUID="bbd4246c-22ba-4116-8e31-b53d3bdc2fbb"
Oct 31 19:45:19.327474 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:19.327439    4645 generic.go:356] "Generic (PLEG): container finished" podID="2e3605c3-ec3b-4fb7-be2a-40fe32c5adea" containerID="c16ea1602ad97fb5a566344e8f7cf5f40c8d9b639d55ae61eae4f88dfde4d22e" exitCode=60
Oct 31 19:45:19.328569 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:19.327856    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-785d26-29891265-8cn6g" event={"ID":"2e3605c3-ec3b-4fb7-be2a-40fe32c5adea","Type":"ContainerDied","Data":"c16ea1602ad97fb5a566344e8f7cf5f40c8d9b639d55ae61eae4f88dfde4d22e"}
Oct 31 19:45:19.328569 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:19.327909    4645 scope.go:122] "RemoveContainer" containerID="c16ea1602ad97fb5a566344e8f7cf5f40c8d9b639d55ae61eae4f88dfde4d22e"
Oct 31 19:45:19.328569 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:45:19.328263    4645 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-785d26-29891265-8cn6g_odh-ai-gateway-infra(2e3605c3-ec3b-4fb7-be2a-40fe32c5adea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-785d26-29891265-8cn6g" podUID="2e3605c3-ec3b-4fb7-be2a-40fe32c5adea"
Oct 31 19:45:19.330365 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:19.330344    4645 generic.go:356] "Generic (PLEG): container finished" podID="57977c65-f0c2-46d1-a05c-20bd8ed579c3" containerID="635a2d9c5a29f44dc065cb40324515a4364da5c1747b819ced38ef04f0d17697" exitCode=60
Oct 31 19:45:19.330505 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:19.330418    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1e8cf0-29891265-mbhwx" event={"ID":"57977c65-f0c2-46d1-a05c-20bd8ed579c3","Type":"ContainerDied","Data":"635a2d9c5a29f44dc065cb40324515a4364da5c1747b819ced38ef04f0d17697"}
Oct 31 19:45:19.330699 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:19.330675    4645 scope.go:122] "RemoveContainer" containerID="635a2d9c5a29f44dc065cb40324515a4364da5c1747b819ced38ef04f0d17697"
Oct 31 19:45:19.331024 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:45:19.331003    4645 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-1e8cf0-29891265-mbhwx_odh-ai-gateway-infra(57977c65-f0c2-46d1-a05c-20bd8ed579c3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1e8cf0-29891265-mbhwx" podUID="57977c65-f0c2-46d1-a05c-20bd8ed579c3"
Oct 31 19:45:19.338472 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:19.338453    4645 scope.go:122] "RemoveContainer" containerID="613208704e41a1fa0120083831fb7a8cb1c6d08bef79521d76b03c00865bb5c6"
Oct 31 19:45:19.338820 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:45:19.338796    4645 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-ae2e60-29891265-5s5f8_odh-ai-gateway-infra(3620ed03-6b03-4c6f-83be-484c41686458)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ae2e60-29891265-5s5f8" podUID="3620ed03-6b03-4c6f-83be-484c41686458"
Oct 31 19:45:19.341546 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:19.341494    4645 scope.go:122] "RemoveContainer" containerID="82fa4c9d5bede9a62e9c308b1c560f4d6a3d37aef819bb6ed06d4fe361fca00f"
Oct 31 19:45:19.359076 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:19.359059    4645 scope.go:122] "RemoveContainer" containerID="106c430a2500395ee4a8d208081034cea2a821cb49bd5a3bc7ec14844193f8df"
Oct 31 19:45:19.375338 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:19.375308    4645 scope.go:122] "RemoveContainer" containerID="f81944db505f31ccbbe57eb0b51acf00499f56387d5893e64f9ec4da02f30c4e"
Oct 31 19:45:19.389488 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:19.389469    4645 scope.go:122] "RemoveContainer" containerID="7eb6a73923070b715ba0af509fa6361f97b6650d7b82d8f436a6f4be3f3abe4a"
Oct 31 19:45:19.402827 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:19.402808    4645 scope.go:122] "RemoveContainer" containerID="f885fb780b2aca10240f93f5720cb25732566a400b1dec8b889809cafcae3717"
Oct 31 19:45:19.419861 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:19.419836    4645 scope.go:122] "RemoveContainer" containerID="009e16e2b4966db62f56b1ca9470e473315d3672073405432d27f31535f3b4a1"
Oct 31 19:45:19.790297 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:19.790252    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:19.918386 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:19.918302    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:20.104497 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:20.104412    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9"]
Oct 31 19:45:20.104764 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:20.104658    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9" podUID="7e5cadc6-cd7f-402d-9550-874d7b3d6376" containerName="istio-proxy" containerID="cri-o://8ded2c69a5be4f9a7ed997ea434b89bcf73ef6455045ec0c15c45cf4497ef976" gracePeriod=30
Oct 31 19:45:20.143700 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:20.143671    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:20.309140 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:20.309108    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:20.361103 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:20.361023    4645 scope.go:122] "RemoveContainer" containerID="9e22ba3b319379c0fce668b1e733ca36e551a79dd8980e5b28c87a22f90b958a"
Oct 31 19:45:20.361825 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:45:20.361792    4645 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891265-9vwg9_odh-ai-gateway-infra(e6f57036-6c45-4def-a9c1-0132ab845b3e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891265-9vwg9" podUID="e6f57036-6c45-4def-a9c1-0132ab845b3e"
Oct 31 19:45:20.370349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:20.370163    4645 scope.go:122] "RemoveContainer" containerID="3d8e0da5a7cf672372700edd09b8cf8ab9a8ddc4a125e03fade92d9508b0dfb7"
Oct 31 19:45:20.370976 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:45:20.370674    4645 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-f5c1ab-29891265-96ksc_odh-ai-gateway-infra(4ae53805-07ba-4f39-bcbf-f19b67f0ad16)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f5c1ab-29891265-96ksc" podUID="4ae53805-07ba-4f39-bcbf-f19b67f0ad16"
Oct 31 19:45:20.377942 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:20.377540    4645 scope.go:122] "RemoveContainer" containerID="bedfa461ebbc34af1c8ee00169ea95fe31be59b33f410fe57163b88cc6b86e1a"
Oct 31 19:45:20.377942 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:45:20.377889    4645 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-8dcfa8-29891265-9lkmt_odh-ai-gateway-infra(bbd4246c-22ba-4116-8e31-b53d3bdc2fbb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8dcfa8-29891265-9lkmt" podUID="bbd4246c-22ba-4116-8e31-b53d3bdc2fbb"
Oct 31 19:45:20.381111 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:20.381090    4645 scope.go:122] "RemoveContainer" containerID="c16ea1602ad97fb5a566344e8f7cf5f40c8d9b639d55ae61eae4f88dfde4d22e"
Oct 31 19:45:20.381636 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:45:20.381611    4645 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-785d26-29891265-8cn6g_odh-ai-gateway-infra(2e3605c3-ec3b-4fb7-be2a-40fe32c5adea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-785d26-29891265-8cn6g" podUID="2e3605c3-ec3b-4fb7-be2a-40fe32c5adea"
Oct 31 19:45:20.405268 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:20.400459    4645 scope.go:122] "RemoveContainer" containerID="635a2d9c5a29f44dc065cb40324515a4364da5c1747b819ced38ef04f0d17697"
Oct 31 19:45:20.405268 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:45:20.401385    4645 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-1e8cf0-29891265-mbhwx_odh-ai-gateway-infra(57977c65-f0c2-46d1-a05c-20bd8ed579c3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1e8cf0-29891265-mbhwx" podUID="57977c65-f0c2-46d1-a05c-20bd8ed579c3"
Oct 31 19:45:20.409374 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:20.408478    4645 scope.go:122] "RemoveContainer" containerID="b7c6416e274fb6e0164de111d20cd44a8684834f78955789f14321b26b853dbe"
Oct 31 19:45:20.409374 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:45:20.408912    4645 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-23843f-29891265-52rjc_odh-ai-gateway-infra(615544a2-1b13-4bce-aeb6-7b3b32e6a509)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-23843f-29891265-52rjc" podUID="615544a2-1b13-4bce-aeb6-7b3b32e6a509"
Oct 31 19:45:20.409823 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:20.406123    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-88eaf9-29891265-zttxn" event={"ID":"25370d12-c485-47b9-986a-a775c3fbd2b7","Type":"ContainerStarted","Data":"6e0fdf219940b19304fa4ae6614e0943e33083230d93614ed9bb893338915a19"}
Oct 31 19:45:20.409900 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:20.409857    4645 scope.go:122] "RemoveContainer" containerID="7b43a4d11228fc8314c3e91b60a4b756c64af2b7ac42fbcc21986e6d0cba2771"
Oct 31 19:45:20.410267 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:45:20.410240    4645 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-fd1c91-29891265-rcrqk_odh-ai-gateway-infra(769ee829-1015-45b1-9dd7-d9cfe372fa9c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-fd1c91-29891265-rcrqk" podUID="769ee829-1015-45b1-9dd7-d9cfe372fa9c"
Oct 31 19:45:20.477476 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:20.477411    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-88eaf9-29891265-zttxn" podStartSLOduration=4.724384108 podStartE2EDuration="20.477392729s" totalImagesPullingTime="15.753008621s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:45:20.46958298 +0000 UTC m=+1551.042817109" watchObservedRunningTime="2026-10-31 19:45:20.477392729 +0000 UTC m=+1551.050626859"
Oct 31 19:45:21.016426 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:21.016318    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:21.423238 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:21.423203    4645 generic.go:356] "Generic (PLEG): container finished" podID="25370d12-c485-47b9-986a-a775c3fbd2b7" containerID="6e0fdf219940b19304fa4ae6614e0943e33083230d93614ed9bb893338915a19" exitCode=60
Oct 31 19:45:21.423661 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:21.423270    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-88eaf9-29891265-zttxn" event={"ID":"25370d12-c485-47b9-986a-a775c3fbd2b7","Type":"ContainerDied","Data":"6e0fdf219940b19304fa4ae6614e0943e33083230d93614ed9bb893338915a19"}
Oct 31 19:45:21.423661 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:21.423394    4645 scope.go:122] "RemoveContainer" containerID="db6591caa16f6c6aad79a86e399510b6b943cb01a53175f6bbf8a971b988b3ee"
Oct 31 19:45:21.423873 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:21.423857    4645 scope.go:122] "RemoveContainer" containerID="6e0fdf219940b19304fa4ae6614e0943e33083230d93614ed9bb893338915a19"
Oct 31 19:45:21.424341 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:45:21.424298    4645 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-88eaf9-29891265-zttxn_odh-ai-gateway-infra(25370d12-c485-47b9-986a-a775c3fbd2b7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-88eaf9-29891265-zttxn" podUID="25370d12-c485-47b9-986a-a775c3fbd2b7"
Oct 31 19:45:21.458183 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:21.458158    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:21.603142 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:21.603105    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:21.741426 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:21.736717    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:22.498828 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:22.498794    4645 patch_prober.go:29] interesting pod/e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 19:45:22.499320 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:22.498848    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9" podUID="7e5cadc6-cd7f-402d-9550-874d7b3d6376" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 19:45:22.699175 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:22.698968    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:23.776646 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:23.776610    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:24.719185 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:24.719149    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:25.421254 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:25.421218    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9"
Oct 31 19:45:25.470561 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:25.470515    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d8c398-56d6bc9d68-8ddtd"]
Oct 31 19:45:25.472347 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:25.472302    4645 cpu_manager.go:414] "containerMap: removing container" podUID="7e5cadc6-cd7f-402d-9550-874d7b3d6376" containerName="istio-proxy"
Oct 31 19:45:25.472347 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:25.472347    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e5cadc6-cd7f-402d-9550-874d7b3d6376" containerName="istio-proxy"
Oct 31 19:45:25.472564 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:25.472373    4645 cpu_manager.go:414] "containerMap: removing container" podUID="7e5cadc6-cd7f-402d-9550-874d7b3d6376" containerName="istio-proxy"
Oct 31 19:45:25.472564 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:25.472381    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e5cadc6-cd7f-402d-9550-874d7b3d6376" containerName="istio-proxy"
Oct 31 19:45:25.472564 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:25.472498    4645 cpu_manager.go:414] "containerMap: removing container" podUID="97e726ce-2b00-4703-8c9e-277ea9bb4c4a" containerName="cleanup"
Oct 31 19:45:25.472564 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:25.472508    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97e726ce-2b00-4703-8c9e-277ea9bb4c4a" containerName="cleanup"
Oct 31 19:45:25.472764 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:25.472742    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="97e726ce-2b00-4703-8c9e-277ea9bb4c4a" containerName="cleanup"
Oct 31 19:45:25.472764 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:25.472763    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e5cadc6-cd7f-402d-9550-874d7b3d6376" containerName="istio-proxy"
Oct 31 19:45:25.472855 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:25.472778    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e5cadc6-cd7f-402d-9550-874d7b3d6376" containerName="istio-proxy"
Oct 31 19:45:25.473166 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:25.473137    4645 generic.go:356] "Generic (PLEG): container finished" podID="7e5cadc6-cd7f-402d-9550-874d7b3d6376" containerID="8ded2c69a5be4f9a7ed997ea434b89bcf73ef6455045ec0c15c45cf4497ef976" exitCode=0
Oct 31 19:45:25.473677 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:25.473233    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9"
Oct 31 19:45:25.483960 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:25.483931    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:25.485597 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:25.485570    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9" event={"ID":"7e5cadc6-cd7f-402d-9550-874d7b3d6376","Type":"ContainerDied","Data":"8ded2c69a5be4f9a7ed997ea434b89bcf73ef6455045ec0c15c45cf4497ef976"}
Oct 31 19:45:25.485707 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:25.485631    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9" event={"ID":"7e5cadc6-cd7f-402d-9550-874d7b3d6376","Type":"ContainerDied","Data":"917a571def81aba47cc4a8bee47c0b6ba1bc9897deba983b9648777d673eeb28"}
Oct 31 19:45:25.485707 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:25.485667    4645 scope.go:122] "RemoveContainer" containerID="8ded2c69a5be4f9a7ed997ea434b89bcf73ef6455045ec0c15c45cf4497ef976"
Oct 31 19:45:25.485867 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:25.485849    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d8c398-56d6bc9d68-8ddtd"
Oct 31 19:45:25.490315 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:25.490290    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-d8c398-serving-cert\""
Oct 31 19:45:25.492877 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:25.492746    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d8c398-56d6bc9d68-8ddtd"]
Oct 31 19:45:25.514809 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:25.514784    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7e5cadc6-cd7f-402d-9550-874d7b3d6376-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e5cadc6-cd7f-402d-9550-874d7b3d6376-istio-podinfo\") pod \"7e5cadc6-cd7f-402d-9550-874d7b3d6376\" (UID: \"7e5cadc6-cd7f-402d-9550-874d7b3d6376\") "
Oct 31 19:45:25.514924 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:25.514835    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e5cadc6-cd7f-402d-9550-874d7b3d6376-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e5cadc6-cd7f-402d-9550-874d7b3d6376-workload-socket\") pod \"7e5cadc6-cd7f-402d-9550-874d7b3d6376\" (UID: \"7e5cadc6-cd7f-402d-9550-874d7b3d6376\") "
Oct 31 19:45:25.514986 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:25.514938    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e5cadc6-cd7f-402d-9550-874d7b3d6376-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e5cadc6-cd7f-402d-9550-874d7b3d6376-workload-certs\") pod \"7e5cadc6-cd7f-402d-9550-874d7b3d6376\" (UID: \"7e5cadc6-cd7f-402d-9550-874d7b3d6376\") "
Oct 31 19:45:25.515044 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:25.514992    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e5cadc6-cd7f-402d-9550-874d7b3d6376-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e5cadc6-cd7f-402d-9550-874d7b3d6376-istio-envoy\") pod \"7e5cadc6-cd7f-402d-9550-874d7b3d6376\" (UID: \"7e5cadc6-cd7f-402d-9550-874d7b3d6376\") "
Oct 31 19:45:25.515125 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:25.515109    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e5cadc6-cd7f-402d-9550-874d7b3d6376-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e5cadc6-cd7f-402d-9550-874d7b3d6376-credential-socket\") pod \"7e5cadc6-cd7f-402d-9550-874d7b3d6376\" (UID: \"7e5cadc6-cd7f-402d-9550-874d7b3d6376\") "
Oct 31 19:45:25.515190 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:25.515175    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e5cadc6-cd7f-402d-9550-874d7b3d6376-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e5cadc6-cd7f-402d-9550-874d7b3d6376-istiod-ca-cert\") pod \"7e5cadc6-cd7f-402d-9550-874d7b3d6376\" (UID: \"7e5cadc6-cd7f-402d-9550-874d7b3d6376\") "
Oct 31 19:45:25.515412 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:25.515296    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e5cadc6-cd7f-402d-9550-874d7b3d6376-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e5cadc6-cd7f-402d-9550-874d7b3d6376-istio-data\") pod \"7e5cadc6-cd7f-402d-9550-874d7b3d6376\" (UID: \"7e5cadc6-cd7f-402d-9550-874d7b3d6376\") "
Oct 31 19:45:25.515412 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:25.515395    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e5cadc6-cd7f-402d-9550-874d7b3d6376-kube-api-access-khb2x\" (UniqueName: \"kubernetes.io/projected/7e5cadc6-cd7f-402d-9550-874d7b3d6376-kube-api-access-khb2x\") pod \"7e5cadc6-cd7f-402d-9550-874d7b3d6376\" (UID: \"7e5cadc6-cd7f-402d-9550-874d7b3d6376\") "
Oct 31 19:45:25.515592 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:25.515450    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e5cadc6-cd7f-402d-9550-874d7b3d6376-istio-token\" (UniqueName: \"kubernetes.io/projected/7e5cadc6-cd7f-402d-9550-874d7b3d6376-istio-token\") pod \"7e5cadc6-cd7f-402d-9550-874d7b3d6376\" (UID: \"7e5cadc6-cd7f-402d-9550-874d7b3d6376\") "
Oct 31 19:45:25.516055 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:25.515391    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e5cadc6-cd7f-402d-9550-874d7b3d6376-workload-certs" pod "7e5cadc6-cd7f-402d-9550-874d7b3d6376" (UID: "7e5cadc6-cd7f-402d-9550-874d7b3d6376"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:45:25.516554 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:25.516526    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e5cadc6-cd7f-402d-9550-874d7b3d6376-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:25.516554 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:25.515500    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e5cadc6-cd7f-402d-9550-874d7b3d6376-workload-socket" pod "7e5cadc6-cd7f-402d-9550-874d7b3d6376" (UID: "7e5cadc6-cd7f-402d-9550-874d7b3d6376"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:45:25.516696 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:25.515580    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e5cadc6-cd7f-402d-9550-874d7b3d6376-credential-socket" pod "7e5cadc6-cd7f-402d-9550-874d7b3d6376" (UID: "7e5cadc6-cd7f-402d-9550-874d7b3d6376"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:45:25.516696 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:25.515749    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e5cadc6-cd7f-402d-9550-874d7b3d6376-istio-data" pod "7e5cadc6-cd7f-402d-9550-874d7b3d6376" (UID: "7e5cadc6-cd7f-402d-9550-874d7b3d6376"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:45:25.516696 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:25.516020    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e5cadc6-cd7f-402d-9550-874d7b3d6376-istiod-ca-cert" pod "7e5cadc6-cd7f-402d-9550-874d7b3d6376" (UID: "7e5cadc6-cd7f-402d-9550-874d7b3d6376"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:45:25.522895 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:25.522761    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e5cadc6-cd7f-402d-9550-874d7b3d6376-istio-envoy" pod "7e5cadc6-cd7f-402d-9550-874d7b3d6376" (UID: "7e5cadc6-cd7f-402d-9550-874d7b3d6376"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:45:25.523103 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:25.523086    4645 scope.go:122] "RemoveContainer" containerID="2cf907396d4b6896492bbd4c364a1b67a16f35568fb0ef4e23fe0ed52b3a8514"
Oct 31 19:45:25.575347 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:25.569032    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e5cadc6-cd7f-402d-9550-874d7b3d6376-kube-api-access-khb2x" pod "7e5cadc6-cd7f-402d-9550-874d7b3d6376" (UID: "7e5cadc6-cd7f-402d-9550-874d7b3d6376"). InnerVolumeSpecName "kube-api-access-khb2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:45:25.575347 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:25.575199    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e5cadc6-cd7f-402d-9550-874d7b3d6376-istio-token" pod "7e5cadc6-cd7f-402d-9550-874d7b3d6376" (UID: "7e5cadc6-cd7f-402d-9550-874d7b3d6376"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:45:25.588468 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:25.588430    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e5cadc6-cd7f-402d-9550-874d7b3d6376-istio-podinfo" pod "7e5cadc6-cd7f-402d-9550-874d7b3d6376" (UID: "7e5cadc6-cd7f-402d-9550-874d7b3d6376"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:45:25.599638 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:25.599552    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:25.623595 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:25.623040    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25a436b2-50a2-404a-9e6d-3601d03edcc8-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d8c398-56d6bc9d68-8ddtd\" (UID: \"25a436b2-50a2-404a-9e6d-3601d03edcc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d8c398-56d6bc9d68-8ddtd"
Oct 31 19:45:25.623595 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:25.623319    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25a436b2-50a2-404a-9e6d-3601d03edcc8-metrics-certs\") pod \"maas-api-e2e-inf-b-d8c398-56d6bc9d68-8ddtd\" (UID: \"25a436b2-50a2-404a-9e6d-3601d03edcc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d8c398-56d6bc9d68-8ddtd"
Oct 31 19:45:25.623595 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:25.623451    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khd6h\" (UniqueName: \"kubernetes.io/projected/25a436b2-50a2-404a-9e6d-3601d03edcc8-kube-api-access-khd6h\") pod \"maas-api-e2e-inf-b-d8c398-56d6bc9d68-8ddtd\" (UID: \"25a436b2-50a2-404a-9e6d-3601d03edcc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d8c398-56d6bc9d68-8ddtd"
Oct 31 19:45:25.623595 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:25.623504    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25a436b2-50a2-404a-9e6d-3601d03edcc8-maas-api-tls\") pod \"maas-api-e2e-inf-b-d8c398-56d6bc9d68-8ddtd\" (UID: \"25a436b2-50a2-404a-9e6d-3601d03edcc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d8c398-56d6bc9d68-8ddtd"
Oct 31 19:45:25.623935 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:25.623675    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e5cadc6-cd7f-402d-9550-874d7b3d6376-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:25.623935 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:25.623693    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e5cadc6-cd7f-402d-9550-874d7b3d6376-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:25.623935 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:25.623706    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e5cadc6-cd7f-402d-9550-874d7b3d6376-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:25.623935 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:25.623719    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khb2x\" (UniqueName: \"kubernetes.io/projected/7e5cadc6-cd7f-402d-9550-874d7b3d6376-kube-api-access-khb2x\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:25.623935 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:25.623734    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e5cadc6-cd7f-402d-9550-874d7b3d6376-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:25.623935 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:25.623747    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e5cadc6-cd7f-402d-9550-874d7b3d6376-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:25.623935 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:25.623759    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e5cadc6-cd7f-402d-9550-874d7b3d6376-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:25.623935 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:25.623771    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e5cadc6-cd7f-402d-9550-874d7b3d6376-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:25.635391 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:25.628942    4645 scope.go:122] "RemoveContainer" containerID="8ded2c69a5be4f9a7ed997ea434b89bcf73ef6455045ec0c15c45cf4497ef976"
Oct 31 19:45:25.635391 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:45:25.629292    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ded2c69a5be4f9a7ed997ea434b89bcf73ef6455045ec0c15c45cf4497ef976\": container with ID starting with 8ded2c69a5be4f9a7ed997ea434b89bcf73ef6455045ec0c15c45cf4497ef976 not found: ID does not exist" containerID="8ded2c69a5be4f9a7ed997ea434b89bcf73ef6455045ec0c15c45cf4497ef976"
Oct 31 19:45:25.635391 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:25.629370    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ded2c69a5be4f9a7ed997ea434b89bcf73ef6455045ec0c15c45cf4497ef976"} err="failed to get container status \"8ded2c69a5be4f9a7ed997ea434b89bcf73ef6455045ec0c15c45cf4497ef976\": rpc error: code = NotFound desc = could not find container \"8ded2c69a5be4f9a7ed997ea434b89bcf73ef6455045ec0c15c45cf4497ef976\": container with ID starting with 8ded2c69a5be4f9a7ed997ea434b89bcf73ef6455045ec0c15c45cf4497ef976 not found: ID does not exist"
Oct 31 19:45:25.635391 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:25.629396    4645 scope.go:122] "RemoveContainer" containerID="2cf907396d4b6896492bbd4c364a1b67a16f35568fb0ef4e23fe0ed52b3a8514"
Oct 31 19:45:25.635391 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:45:25.629650    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cf907396d4b6896492bbd4c364a1b67a16f35568fb0ef4e23fe0ed52b3a8514\": container with ID starting with 2cf907396d4b6896492bbd4c364a1b67a16f35568fb0ef4e23fe0ed52b3a8514 not found: ID does not exist" containerID="2cf907396d4b6896492bbd4c364a1b67a16f35568fb0ef4e23fe0ed52b3a8514"
Oct 31 19:45:25.635391 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:25.629679    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cf907396d4b6896492bbd4c364a1b67a16f35568fb0ef4e23fe0ed52b3a8514"} err="failed to get container status \"2cf907396d4b6896492bbd4c364a1b67a16f35568fb0ef4e23fe0ed52b3a8514\": rpc error: code = NotFound desc = could not find container \"2cf907396d4b6896492bbd4c364a1b67a16f35568fb0ef4e23fe0ed52b3a8514\": container with ID starting with 2cf907396d4b6896492bbd4c364a1b67a16f35568fb0ef4e23fe0ed52b3a8514 not found: ID does not exist"
Oct 31 19:45:25.725042 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:25.725004    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25a436b2-50a2-404a-9e6d-3601d03edcc8-metrics-certs\") pod \"maas-api-e2e-inf-b-d8c398-56d6bc9d68-8ddtd\" (UID: \"25a436b2-50a2-404a-9e6d-3601d03edcc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d8c398-56d6bc9d68-8ddtd"
Oct 31 19:45:25.725232 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:25.725117    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khd6h\" (UniqueName: \"kubernetes.io/projected/25a436b2-50a2-404a-9e6d-3601d03edcc8-kube-api-access-khd6h\") pod \"maas-api-e2e-inf-b-d8c398-56d6bc9d68-8ddtd\" (UID: \"25a436b2-50a2-404a-9e6d-3601d03edcc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d8c398-56d6bc9d68-8ddtd"
Oct 31 19:45:25.725232 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:25.725176    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25a436b2-50a2-404a-9e6d-3601d03edcc8-maas-api-tls\") pod \"maas-api-e2e-inf-b-d8c398-56d6bc9d68-8ddtd\" (UID: \"25a436b2-50a2-404a-9e6d-3601d03edcc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d8c398-56d6bc9d68-8ddtd"
Oct 31 19:45:25.725358 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:25.725286    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25a436b2-50a2-404a-9e6d-3601d03edcc8-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d8c398-56d6bc9d68-8ddtd\" (UID: \"25a436b2-50a2-404a-9e6d-3601d03edcc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d8c398-56d6bc9d68-8ddtd"
Oct 31 19:45:25.725918 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:25.725889    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25a436b2-50a2-404a-9e6d-3601d03edcc8-openshift-service-ca\") pod \"maas-api-e2e-inf-b-d8c398-56d6bc9d68-8ddtd\" (UID: \"25a436b2-50a2-404a-9e6d-3601d03edcc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d8c398-56d6bc9d68-8ddtd"
Oct 31 19:45:25.727902 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:25.727873    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25a436b2-50a2-404a-9e6d-3601d03edcc8-metrics-certs\") pod \"maas-api-e2e-inf-b-d8c398-56d6bc9d68-8ddtd\" (UID: \"25a436b2-50a2-404a-9e6d-3601d03edcc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d8c398-56d6bc9d68-8ddtd"
Oct 31 19:45:25.727902 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:25.727890    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25a436b2-50a2-404a-9e6d-3601d03edcc8-maas-api-tls\") pod \"maas-api-e2e-inf-b-d8c398-56d6bc9d68-8ddtd\" (UID: \"25a436b2-50a2-404a-9e6d-3601d03edcc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d8c398-56d6bc9d68-8ddtd"
Oct 31 19:45:25.738726 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:25.738697    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khd6h\" (UniqueName: \"kubernetes.io/projected/25a436b2-50a2-404a-9e6d-3601d03edcc8-kube-api-access-khd6h\") pod \"maas-api-e2e-inf-b-d8c398-56d6bc9d68-8ddtd\" (UID: \"25a436b2-50a2-404a-9e6d-3601d03edcc8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d8c398-56d6bc9d68-8ddtd"
Oct 31 19:45:25.795832 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:25.795803    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9"]
Oct 31 19:45:25.798791 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:25.798763    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:25.801055 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:25.801034    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-a813e2-openshift-default-756f6db67-mzvc9"]
Oct 31 19:45:25.821690 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:25.821670    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d8c398-56d6bc9d68-8ddtd"
Oct 31 19:45:26.159734 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:45:26.159705    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25a436b2_50a2_404a_9e6d_3601d03edcc8.slice/crio-61fdf19f5dff3c63ca8224472dfb9abc4eb88e558dcb0cddf4ebf1a89ac9967b WatchSource:0}: Error finding container 61fdf19f5dff3c63ca8224472dfb9abc4eb88e558dcb0cddf4ebf1a89ac9967b: Status 404 returned error can't find the container with id 61fdf19f5dff3c63ca8224472dfb9abc4eb88e558dcb0cddf4ebf1a89ac9967b
Oct 31 19:45:26.161503 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:26.161474    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d8c398-56d6bc9d68-8ddtd"]
Oct 31 19:45:26.485305 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:26.485271    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d8c398-56d6bc9d68-8ddtd" event={"ID":"25a436b2-50a2-404a-9e6d-3601d03edcc8","Type":"ContainerStarted","Data":"61fdf19f5dff3c63ca8224472dfb9abc4eb88e558dcb0cddf4ebf1a89ac9967b"}
Oct 31 19:45:26.863798 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:26.863710    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:27.495857 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:27.495820    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d8c398-56d6bc9d68-8ddtd" event={"ID":"25a436b2-50a2-404a-9e6d-3601d03edcc8","Type":"ContainerStarted","Data":"4006a6ccf8c79645c2bdf6e4a95df64af55c2b9e7982cd24d760ae6d92697c81"}
Oct 31 19:45:27.496422 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:27.495915    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d8c398-56d6bc9d68-8ddtd"
Oct 31 19:45:27.517240 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:27.516888    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d8c398-56d6bc9d68-8ddtd" podStartSLOduration=2.069549308 podStartE2EDuration="2.516872733s" totalImagesPullingTime="447.323425ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:45:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:45:27.516016226 +0000 UTC m=+1558.089250355" watchObservedRunningTime="2026-10-31 19:45:27.516872733 +0000 UTC m=+1558.090106864"
Oct 31 19:45:27.621569 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:27.621540    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e5cadc6-cd7f-402d-9550-874d7b3d6376" path="/var/lib/kubelet/pods/7e5cadc6-cd7f-402d-9550-874d7b3d6376/volumes"
Oct 31 19:45:27.950929 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:27.950897    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:28.736476 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:28.736442    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:29.569691 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:29.569656    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:29.622008 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:29.621480    4645 scope.go:122] "RemoveContainer" containerID="613208704e41a1fa0120083831fb7a8cb1c6d08bef79521d76b03c00865bb5c6"
Oct 31 19:45:29.631311 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:29.622666    4645 scope.go:122] "RemoveContainer" containerID="cd74e21b378df60e66d0039caadb0de062a9db8891dd59bed8e5863b5134ffe7"
Oct 31 19:45:29.631311 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:29.623414    4645 scope.go:122] "RemoveContainer" containerID="3d8e0da5a7cf672372700edd09b8cf8ab9a8ddc4a125e03fade92d9508b0dfb7"
Oct 31 19:45:29.631311 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:29.624380    4645 scope.go:122] "RemoveContainer" containerID="1874b70e23d4508fcc45132c388072af9fed705d09a94acd6034702e51696a68"
Oct 31 19:45:29.631311 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:29.627075    4645 scope.go:122] "RemoveContainer" containerID="e263899d5c3d156e483a87f30c8d878ae4a89e6fdfb67c0ee69a861404237fd1"
Oct 31 19:45:29.641095 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:29.641071    4645 scope.go:122] "RemoveContainer" containerID="bedfa461ebbc34af1c8ee00169ea95fe31be59b33f410fe57163b88cc6b86e1a"
Oct 31 19:45:30.330778 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:30.330706    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:30.529637 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:30.529600    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f5c1ab-29891265-96ksc" event={"ID":"4ae53805-07ba-4f39-bcbf-f19b67f0ad16","Type":"ContainerStarted","Data":"32ee15b1d5ee43583d281cda09662661eaef57cec7173b94d344d0374af61a3f"}
Oct 31 19:45:30.531749 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:30.531717    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8dcfa8-29891265-9lkmt" event={"ID":"bbd4246c-22ba-4116-8e31-b53d3bdc2fbb","Type":"ContainerStarted","Data":"ef61a24e0614cc0f6b8c072b0a59cc05d3d213c03809cff94a2e6d1e7b9aa6e1"}
Oct 31 19:45:30.533847 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:30.533828    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ae2e60-29891265-5s5f8" event={"ID":"3620ed03-6b03-4c6f-83be-484c41686458","Type":"ContainerStarted","Data":"185a3f95ac3fde394f5f34cd3d28c254f1380d80bff0f833b5e702a775f58924"}
Oct 31 19:45:30.535737 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:30.535719    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-435728-29891265-tfbvw" event={"ID":"53b785a3-a308-43ec-92b9-ccd7d6a8bcad","Type":"ContainerStarted","Data":"210204d9d27cdeeb8b4b81bb984e28a075e830571b41387814575b0beeebd949"}
Oct 31 19:45:30.537627 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:30.537608    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-03dfee-29891265-njm8t" event={"ID":"3961f45f-4b0d-4880-b8d8-e55798d4abe8","Type":"ContainerStarted","Data":"9343c64c137e89bb1ff60deb44056999874904ae00b258f9636701cc3ab79cb5"}
Oct 31 19:45:30.539393 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:30.539367    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a891ce-29891265-md69g" event={"ID":"220956a9-b7c5-48b8-a598-61326c85691a","Type":"ContainerStarted","Data":"7f78c5a4204a76801ef461c028a1f5cecf1a85d0ffd2eaaf9bf39a55621ec06d"}
Oct 31 19:45:30.610933 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:30.610857    4645 scope.go:122] "RemoveContainer" containerID="b7c6416e274fb6e0164de111d20cd44a8684834f78955789f14321b26b853dbe"
Oct 31 19:45:30.611060 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:30.611044    4645 scope.go:122] "RemoveContainer" containerID="c16ea1602ad97fb5a566344e8f7cf5f40c8d9b639d55ae61eae4f88dfde4d22e"
Oct 31 19:45:30.611217 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:30.611199    4645 scope.go:122] "RemoveContainer" containerID="7b43a4d11228fc8314c3e91b60a4b756c64af2b7ac42fbcc21986e6d0cba2771"
Oct 31 19:45:30.611343 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:30.611304    4645 scope.go:122] "RemoveContainer" containerID="9e22ba3b319379c0fce668b1e733ca36e551a79dd8980e5b28c87a22f90b958a"
Oct 31 19:45:31.158310 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:31.158282    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:31.550007 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:31.549978    4645 generic.go:356] "Generic (PLEG): container finished" podID="bbd4246c-22ba-4116-8e31-b53d3bdc2fbb" containerID="ef61a24e0614cc0f6b8c072b0a59cc05d3d213c03809cff94a2e6d1e7b9aa6e1" exitCode=60
Oct 31 19:45:31.550447 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:31.550048    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8dcfa8-29891265-9lkmt" event={"ID":"bbd4246c-22ba-4116-8e31-b53d3bdc2fbb","Type":"ContainerDied","Data":"ef61a24e0614cc0f6b8c072b0a59cc05d3d213c03809cff94a2e6d1e7b9aa6e1"}
Oct 31 19:45:31.550447 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:31.550153    4645 scope.go:122] "RemoveContainer" containerID="bedfa461ebbc34af1c8ee00169ea95fe31be59b33f410fe57163b88cc6b86e1a"
Oct 31 19:45:31.550587 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:31.550567    4645 scope.go:122] "RemoveContainer" containerID="ef61a24e0614cc0f6b8c072b0a59cc05d3d213c03809cff94a2e6d1e7b9aa6e1"
Oct 31 19:45:31.551176 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:45:31.550910    4645 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-8dcfa8-29891265-9lkmt_odh-ai-gateway-infra(bbd4246c-22ba-4116-8e31-b53d3bdc2fbb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8dcfa8-29891265-9lkmt" podUID="bbd4246c-22ba-4116-8e31-b53d3bdc2fbb"
Oct 31 19:45:31.552523 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:31.552497    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-785d26-29891265-8cn6g" event={"ID":"2e3605c3-ec3b-4fb7-be2a-40fe32c5adea","Type":"ContainerStarted","Data":"90f2a2574676bc448598d57f47037ea167d5a844b53983b4f3e383d6bc046900"}
Oct 31 19:45:31.554830 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:31.554805    4645 generic.go:356] "Generic (PLEG): container finished" podID="3620ed03-6b03-4c6f-83be-484c41686458" containerID="185a3f95ac3fde394f5f34cd3d28c254f1380d80bff0f833b5e702a775f58924" exitCode=60
Oct 31 19:45:31.554958 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:31.554891    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ae2e60-29891265-5s5f8" event={"ID":"3620ed03-6b03-4c6f-83be-484c41686458","Type":"ContainerDied","Data":"185a3f95ac3fde394f5f34cd3d28c254f1380d80bff0f833b5e702a775f58924"}
Oct 31 19:45:31.557562 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:31.557537    4645 scope.go:122] "RemoveContainer" containerID="185a3f95ac3fde394f5f34cd3d28c254f1380d80bff0f833b5e702a775f58924"
Oct 31 19:45:31.557890 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:45:31.557865    4645 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-ae2e60-29891265-5s5f8_odh-ai-gateway-infra(3620ed03-6b03-4c6f-83be-484c41686458)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ae2e60-29891265-5s5f8" podUID="3620ed03-6b03-4c6f-83be-484c41686458"
Oct 31 19:45:31.568536 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:31.568505    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f5c1ab-29891265-96ksc"]
Oct 31 19:45:31.572563 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:31.572540    4645 generic.go:356] "Generic (PLEG): container finished" podID="53b785a3-a308-43ec-92b9-ccd7d6a8bcad" containerID="210204d9d27cdeeb8b4b81bb984e28a075e830571b41387814575b0beeebd949" exitCode=60
Oct 31 19:45:31.572715 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:31.572613    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-435728-29891265-tfbvw" event={"ID":"53b785a3-a308-43ec-92b9-ccd7d6a8bcad","Type":"ContainerDied","Data":"210204d9d27cdeeb8b4b81bb984e28a075e830571b41387814575b0beeebd949"}
Oct 31 19:45:31.573062 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:31.573043    4645 scope.go:122] "RemoveContainer" containerID="210204d9d27cdeeb8b4b81bb984e28a075e830571b41387814575b0beeebd949"
Oct 31 19:45:31.573920 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:45:31.573874    4645 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-435728-29891265-tfbvw_odh-ai-gateway-infra(53b785a3-a308-43ec-92b9-ccd7d6a8bcad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-435728-29891265-tfbvw" podUID="53b785a3-a308-43ec-92b9-ccd7d6a8bcad"
Oct 31 19:45:31.577866 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:31.577832    4645 generic.go:356] "Generic (PLEG): container finished" podID="3961f45f-4b0d-4880-b8d8-e55798d4abe8" containerID="9343c64c137e89bb1ff60deb44056999874904ae00b258f9636701cc3ab79cb5" exitCode=60
Oct 31 19:45:31.578113 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:31.578030    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-03dfee-29891265-njm8t" event={"ID":"3961f45f-4b0d-4880-b8d8-e55798d4abe8","Type":"ContainerDied","Data":"9343c64c137e89bb1ff60deb44056999874904ae00b258f9636701cc3ab79cb5"}
Oct 31 19:45:31.578555 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:31.578533    4645 scope.go:122] "RemoveContainer" containerID="9343c64c137e89bb1ff60deb44056999874904ae00b258f9636701cc3ab79cb5"
Oct 31 19:45:31.578886 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:45:31.578862    4645 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-03dfee-29891265-njm8t_odh-ai-gateway-infra(3961f45f-4b0d-4880-b8d8-e55798d4abe8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-03dfee-29891265-njm8t" podUID="3961f45f-4b0d-4880-b8d8-e55798d4abe8"
Oct 31 19:45:31.579631 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:31.579615    4645 scope.go:122] "RemoveContainer" containerID="613208704e41a1fa0120083831fb7a8cb1c6d08bef79521d76b03c00865bb5c6"
Oct 31 19:45:31.580105 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:31.580084    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-fd1c91-29891265-rcrqk" event={"ID":"769ee829-1015-45b1-9dd7-d9cfe372fa9c","Type":"ContainerStarted","Data":"ac8f707fe9dec466c5648a974c55766fd6f6a315ed9754cd31adbcf340360b1d"}
Oct 31 19:45:31.581212 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:31.581194    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ae2e60-29891265-5s5f8"]
Oct 31 19:45:31.582184 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:31.582160    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-23843f-29891265-52rjc" event={"ID":"615544a2-1b13-4bce-aeb6-7b3b32e6a509","Type":"ContainerStarted","Data":"9d70f7ad488c24d0cdd605f400ddf6241bfc449060eb6eb3c6867820fb2aaa45"}
Oct 31 19:45:31.584418 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:31.584400    4645 generic.go:356] "Generic (PLEG): container finished" podID="220956a9-b7c5-48b8-a598-61326c85691a" containerID="7f78c5a4204a76801ef461c028a1f5cecf1a85d0ffd2eaaf9bf39a55621ec06d" exitCode=60
Oct 31 19:45:31.584503 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:31.584481    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a891ce-29891265-md69g" event={"ID":"220956a9-b7c5-48b8-a598-61326c85691a","Type":"ContainerDied","Data":"7f78c5a4204a76801ef461c028a1f5cecf1a85d0ffd2eaaf9bf39a55621ec06d"}
Oct 31 19:45:31.584789 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:31.584767    4645 scope.go:122] "RemoveContainer" containerID="7f78c5a4204a76801ef461c028a1f5cecf1a85d0ffd2eaaf9bf39a55621ec06d"
Oct 31 19:45:31.585086 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:45:31.585062    4645 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-a891ce-29891265-md69g_odh-ai-gateway-infra(220956a9-b7c5-48b8-a598-61326c85691a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a891ce-29891265-md69g" podUID="220956a9-b7c5-48b8-a598-61326c85691a"
Oct 31 19:45:31.586573 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:31.586548    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891265-9vwg9" event={"ID":"e6f57036-6c45-4def-a9c1-0132ab845b3e","Type":"ContainerStarted","Data":"ab4229cdce00f7cdd2924c251bc483b331f43ee65baea7c23143c4b4d89cf45d"}
Oct 31 19:45:31.588485 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:31.588465    4645 generic.go:356] "Generic (PLEG): container finished" podID="4ae53805-07ba-4f39-bcbf-f19b67f0ad16" containerID="32ee15b1d5ee43583d281cda09662661eaef57cec7173b94d344d0374af61a3f" exitCode=60
Oct 31 19:45:31.588557 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:31.588490    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f5c1ab-29891265-96ksc" event={"ID":"4ae53805-07ba-4f39-bcbf-f19b67f0ad16","Type":"ContainerDied","Data":"32ee15b1d5ee43583d281cda09662661eaef57cec7173b94d344d0374af61a3f"}
Oct 31 19:45:31.597917 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:31.597824    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8dcfa8-29891265-9lkmt"]
Oct 31 19:45:31.597993 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:31.597943    4645 scope.go:122] "RemoveContainer" containerID="e263899d5c3d156e483a87f30c8d878ae4a89e6fdfb67c0ee69a861404237fd1"
Oct 31 19:45:31.611728 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:31.611410    4645 scope.go:122] "RemoveContainer" containerID="635a2d9c5a29f44dc065cb40324515a4364da5c1747b819ced38ef04f0d17697"
Oct 31 19:45:31.611728 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:31.611599    4645 scope.go:122] "RemoveContainer" containerID="32dbade5037b8291ce61e8fcb623c66b9b4cf1fb48db6b83f724a52fb9bf2271"
Oct 31 19:45:31.617032 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:31.616977    4645 scope.go:122] "RemoveContainer" containerID="cd74e21b378df60e66d0039caadb0de062a9db8891dd59bed8e5863b5134ffe7"
Oct 31 19:45:31.630091 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:31.630065    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-03dfee-29891265-njm8t"]
Oct 31 19:45:31.635623 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:31.635591    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a891ce-29891265-md69g"]
Oct 31 19:45:31.660052 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:31.660023    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-435728-29891265-tfbvw"]
Oct 31 19:45:31.695360 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:31.695313    4645 scope.go:122] "RemoveContainer" containerID="1874b70e23d4508fcc45132c388072af9fed705d09a94acd6034702e51696a68"
Oct 31 19:45:31.712717 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:31.712692    4645 scope.go:122] "RemoveContainer" containerID="3d8e0da5a7cf672372700edd09b8cf8ab9a8ddc4a125e03fade92d9508b0dfb7"
Oct 31 19:45:32.056189 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:32.056148    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:32.110359 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:32.090878    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f5c1ab-29891265-96ksc"
Oct 31 19:45:32.202813 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:32.202446    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ae53805-07ba-4f39-bcbf-f19b67f0ad16-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ae53805-07ba-4f39-bcbf-f19b67f0ad16-maas-api-service-ca\") pod \"4ae53805-07ba-4f39-bcbf-f19b67f0ad16\" (UID: \"4ae53805-07ba-4f39-bcbf-f19b67f0ad16\") "
Oct 31 19:45:32.202813 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:32.202589    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ae53805-07ba-4f39-bcbf-f19b67f0ad16-kube-api-access-ntwl4\" (UniqueName: \"kubernetes.io/projected/4ae53805-07ba-4f39-bcbf-f19b67f0ad16-kube-api-access-ntwl4\") pod \"4ae53805-07ba-4f39-bcbf-f19b67f0ad16\" (UID: \"4ae53805-07ba-4f39-bcbf-f19b67f0ad16\") "
Oct 31 19:45:32.205255 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:32.205223    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ae53805-07ba-4f39-bcbf-f19b67f0ad16-maas-api-service-ca" pod "4ae53805-07ba-4f39-bcbf-f19b67f0ad16" (UID: "4ae53805-07ba-4f39-bcbf-f19b67f0ad16"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:45:32.215001 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:32.214964    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ae53805-07ba-4f39-bcbf-f19b67f0ad16-kube-api-access-ntwl4" pod "4ae53805-07ba-4f39-bcbf-f19b67f0ad16" (UID: "4ae53805-07ba-4f39-bcbf-f19b67f0ad16"). InnerVolumeSpecName "kube-api-access-ntwl4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:45:32.307496 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:32.307465    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ae53805-07ba-4f39-bcbf-f19b67f0ad16-maas-api-service-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:32.307496 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:32.307492    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntwl4\" (UniqueName: \"kubernetes.io/projected/4ae53805-07ba-4f39-bcbf-f19b67f0ad16-kube-api-access-ntwl4\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:32.598596 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:32.598510    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f5c1ab-29891265-96ksc"
Oct 31 19:45:32.598596 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:32.598540    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f5c1ab-29891265-96ksc" event={"ID":"4ae53805-07ba-4f39-bcbf-f19b67f0ad16","Type":"ContainerDied","Data":"96510bec294712e6004b8f9047034d741f54d4dbac52192642b186eb42f282ee"}
Oct 31 19:45:32.599163 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:32.598602    4645 scope.go:122] "RemoveContainer" containerID="32ee15b1d5ee43583d281cda09662661eaef57cec7173b94d344d0374af61a3f"
Oct 31 19:45:32.603147 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:32.603121    4645 generic.go:356] "Generic (PLEG): container finished" podID="2e3605c3-ec3b-4fb7-be2a-40fe32c5adea" containerID="90f2a2574676bc448598d57f47037ea167d5a844b53983b4f3e383d6bc046900" exitCode=60
Oct 31 19:45:32.603276 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:32.603190    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-785d26-29891265-8cn6g" event={"ID":"2e3605c3-ec3b-4fb7-be2a-40fe32c5adea","Type":"ContainerDied","Data":"90f2a2574676bc448598d57f47037ea167d5a844b53983b4f3e383d6bc046900"}
Oct 31 19:45:32.603643 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:32.603620    4645 scope.go:122] "RemoveContainer" containerID="90f2a2574676bc448598d57f47037ea167d5a844b53983b4f3e383d6bc046900"
Oct 31 19:45:32.604064 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:45:32.604027    4645 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-785d26-29891265-8cn6g_odh-ai-gateway-infra(2e3605c3-ec3b-4fb7-be2a-40fe32c5adea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-785d26-29891265-8cn6g" podUID="2e3605c3-ec3b-4fb7-be2a-40fe32c5adea"
Oct 31 19:45:32.607020 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:32.606998    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1e8cf0-29891265-mbhwx" event={"ID":"57977c65-f0c2-46d1-a05c-20bd8ed579c3","Type":"ContainerStarted","Data":"4c862f620b30d7bb75ae8bed20dd6c62c02ed161fb5a42fe4bfedda138f097aa"}
Oct 31 19:45:32.613435 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:32.613413    4645 scope.go:122] "RemoveContainer" containerID="c16ea1602ad97fb5a566344e8f7cf5f40c8d9b639d55ae61eae4f88dfde4d22e"
Oct 31 19:45:32.616800 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:32.616646    4645 generic.go:356] "Generic (PLEG): container finished" podID="769ee829-1015-45b1-9dd7-d9cfe372fa9c" containerID="ac8f707fe9dec466c5648a974c55766fd6f6a315ed9754cd31adbcf340360b1d" exitCode=60
Oct 31 19:45:32.616800 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:32.616711    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-fd1c91-29891265-rcrqk" event={"ID":"769ee829-1015-45b1-9dd7-d9cfe372fa9c","Type":"ContainerDied","Data":"ac8f707fe9dec466c5648a974c55766fd6f6a315ed9754cd31adbcf340360b1d"}
Oct 31 19:45:32.617433 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:32.617413    4645 scope.go:122] "RemoveContainer" containerID="ac8f707fe9dec466c5648a974c55766fd6f6a315ed9754cd31adbcf340360b1d"
Oct 31 19:45:32.617831 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:45:32.617790    4645 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-fd1c91-29891265-rcrqk_odh-ai-gateway-infra(769ee829-1015-45b1-9dd7-d9cfe372fa9c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-fd1c91-29891265-rcrqk" podUID="769ee829-1015-45b1-9dd7-d9cfe372fa9c"
Oct 31 19:45:32.619966 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:32.619942    4645 generic.go:356] "Generic (PLEG): container finished" podID="615544a2-1b13-4bce-aeb6-7b3b32e6a509" containerID="9d70f7ad488c24d0cdd605f400ddf6241bfc449060eb6eb3c6867820fb2aaa45" exitCode=60
Oct 31 19:45:32.620109 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:32.620048    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-23843f-29891265-52rjc" event={"ID":"615544a2-1b13-4bce-aeb6-7b3b32e6a509","Type":"ContainerDied","Data":"9d70f7ad488c24d0cdd605f400ddf6241bfc449060eb6eb3c6867820fb2aaa45"}
Oct 31 19:45:32.621455 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:32.620435    4645 scope.go:122] "RemoveContainer" containerID="9d70f7ad488c24d0cdd605f400ddf6241bfc449060eb6eb3c6867820fb2aaa45"
Oct 31 19:45:32.621455 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:45:32.620742    4645 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-23843f-29891265-52rjc_odh-ai-gateway-infra(615544a2-1b13-4bce-aeb6-7b3b32e6a509)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-23843f-29891265-52rjc" podUID="615544a2-1b13-4bce-aeb6-7b3b32e6a509"
Oct 31 19:45:32.642740 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:32.637293    4645 scope.go:122] "RemoveContainer" containerID="7b43a4d11228fc8314c3e91b60a4b756c64af2b7ac42fbcc21986e6d0cba2771"
Oct 31 19:45:32.642740 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:32.637527    4645 generic.go:356] "Generic (PLEG): container finished" podID="e6f57036-6c45-4def-a9c1-0132ab845b3e" containerID="ab4229cdce00f7cdd2924c251bc483b331f43ee65baea7c23143c4b4d89cf45d" exitCode=60
Oct 31 19:45:32.642740 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:32.637617    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891265-9vwg9" event={"ID":"e6f57036-6c45-4def-a9c1-0132ab845b3e","Type":"ContainerDied","Data":"ab4229cdce00f7cdd2924c251bc483b331f43ee65baea7c23143c4b4d89cf45d"}
Oct 31 19:45:32.642740 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:32.638138    4645 scope.go:122] "RemoveContainer" containerID="ab4229cdce00f7cdd2924c251bc483b331f43ee65baea7c23143c4b4d89cf45d"
Oct 31 19:45:32.642740 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:45:32.639073    4645 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29891265-9vwg9_odh-ai-gateway-infra(e6f57036-6c45-4def-a9c1-0132ab845b3e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891265-9vwg9" podUID="e6f57036-6c45-4def-a9c1-0132ab845b3e"
Oct 31 19:45:32.651832 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:32.646785    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891265-r6vvt" event={"ID":"e7198524-cd1a-4038-83c6-55f9a5391970","Type":"ContainerStarted","Data":"0b7a923b65422c7da00f1fbd033b12779326e8d3f858ea417e007d2e26815aec"}
Oct 31 19:45:32.667738 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:32.667136    4645 scope.go:122] "RemoveContainer" containerID="b7c6416e274fb6e0164de111d20cd44a8684834f78955789f14321b26b853dbe"
Oct 31 19:45:32.684377 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:32.683486    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-785d26-29891265-8cn6g"]
Oct 31 19:45:32.720031 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:32.714092    4645 scope.go:122] "RemoveContainer" containerID="9e22ba3b319379c0fce668b1e733ca36e551a79dd8980e5b28c87a22f90b958a"
Oct 31 19:45:32.824238 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:32.824214    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-23843f-29891265-52rjc"]
Oct 31 19:45:32.832347 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:32.831982    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-fd1c91-29891265-rcrqk"]
Oct 31 19:45:32.849901 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:32.849837    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:32.850033 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:32.849922    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891265-9vwg9"]
Oct 31 19:45:32.973395 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:32.972155    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f5c1ab-29891265-96ksc"]
Oct 31 19:45:33.015398 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.013531    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8dcfa8-29891265-9lkmt"
Oct 31 19:45:33.033384 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.030012    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbd4246c-22ba-4116-8e31-b53d3bdc2fbb-kube-api-access-khhgd\" (UniqueName: \"kubernetes.io/projected/bbd4246c-22ba-4116-8e31-b53d3bdc2fbb-kube-api-access-khhgd\") pod \"bbd4246c-22ba-4116-8e31-b53d3bdc2fbb\" (UID: \"bbd4246c-22ba-4116-8e31-b53d3bdc2fbb\") "
Oct 31 19:45:33.033384 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.030514    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbd4246c-22ba-4116-8e31-b53d3bdc2fbb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbd4246c-22ba-4116-8e31-b53d3bdc2fbb-maas-api-service-ca\") pod \"bbd4246c-22ba-4116-8e31-b53d3bdc2fbb\" (UID: \"bbd4246c-22ba-4116-8e31-b53d3bdc2fbb\") "
Oct 31 19:45:33.033384 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.032138    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbd4246c-22ba-4116-8e31-b53d3bdc2fbb-maas-api-service-ca" pod "bbd4246c-22ba-4116-8e31-b53d3bdc2fbb" (UID: "bbd4246c-22ba-4116-8e31-b53d3bdc2fbb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:45:33.128113 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.127964    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-f5c1ab-29891265-96ksc"]
Oct 31 19:45:33.150849 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.134111    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbd4246c-22ba-4116-8e31-b53d3bdc2fbb-maas-api-service-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:33.242495 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.198611    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ae2e60-29891265-5s5f8"
Oct 31 19:45:33.242495 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.239057    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3620ed03-6b03-4c6f-83be-484c41686458-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3620ed03-6b03-4c6f-83be-484c41686458-maas-api-service-ca\") pod \"3620ed03-6b03-4c6f-83be-484c41686458\" (UID: \"3620ed03-6b03-4c6f-83be-484c41686458\") "
Oct 31 19:45:33.242495 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.239321    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3620ed03-6b03-4c6f-83be-484c41686458-kube-api-access-cqxnr\" (UniqueName: \"kubernetes.io/projected/3620ed03-6b03-4c6f-83be-484c41686458-kube-api-access-cqxnr\") pod \"3620ed03-6b03-4c6f-83be-484c41686458\" (UID: \"3620ed03-6b03-4c6f-83be-484c41686458\") "
Oct 31 19:45:33.242773 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.242692    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3620ed03-6b03-4c6f-83be-484c41686458-maas-api-service-ca" pod "3620ed03-6b03-4c6f-83be-484c41686458" (UID: "3620ed03-6b03-4c6f-83be-484c41686458"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:45:33.268383 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.261495    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbd4246c-22ba-4116-8e31-b53d3bdc2fbb-kube-api-access-khhgd" pod "bbd4246c-22ba-4116-8e31-b53d3bdc2fbb" (UID: "bbd4246c-22ba-4116-8e31-b53d3bdc2fbb"). InnerVolumeSpecName "kube-api-access-khhgd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:45:33.341734 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.335640    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3620ed03-6b03-4c6f-83be-484c41686458-kube-api-access-cqxnr" pod "3620ed03-6b03-4c6f-83be-484c41686458" (UID: "3620ed03-6b03-4c6f-83be-484c41686458"). InnerVolumeSpecName "kube-api-access-cqxnr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:45:33.342184 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.342122    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khhgd\" (UniqueName: \"kubernetes.io/projected/bbd4246c-22ba-4116-8e31-b53d3bdc2fbb-kube-api-access-khhgd\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:33.342313 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.342159    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3620ed03-6b03-4c6f-83be-484c41686458-maas-api-service-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:33.342313 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.342261    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqxnr\" (UniqueName: \"kubernetes.io/projected/3620ed03-6b03-4c6f-83be-484c41686458-kube-api-access-cqxnr\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:33.464361 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.440793    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9"]
Oct 31 19:45:33.464361 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.442192    4645 cpu_manager.go:414] "containerMap: removing container" podUID="3620ed03-6b03-4c6f-83be-484c41686458" containerName="cleanup"
Oct 31 19:45:33.464361 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.442215    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3620ed03-6b03-4c6f-83be-484c41686458" containerName="cleanup"
Oct 31 19:45:33.464361 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.442265    4645 cpu_manager.go:414] "containerMap: removing container" podUID="4ae53805-07ba-4f39-bcbf-f19b67f0ad16" containerName="cleanup"
Oct 31 19:45:33.464361 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.442273    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ae53805-07ba-4f39-bcbf-f19b67f0ad16" containerName="cleanup"
Oct 31 19:45:33.464361 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.442303    4645 cpu_manager.go:414] "containerMap: removing container" podUID="4ae53805-07ba-4f39-bcbf-f19b67f0ad16" containerName="cleanup"
Oct 31 19:45:33.464361 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.442311    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ae53805-07ba-4f39-bcbf-f19b67f0ad16" containerName="cleanup"
Oct 31 19:45:33.464361 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.442409    4645 cpu_manager.go:414] "containerMap: removing container" podUID="bbd4246c-22ba-4116-8e31-b53d3bdc2fbb" containerName="cleanup"
Oct 31 19:45:33.464361 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.442420    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbd4246c-22ba-4116-8e31-b53d3bdc2fbb" containerName="cleanup"
Oct 31 19:45:33.464361 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.442443    4645 cpu_manager.go:414] "containerMap: removing container" podUID="3620ed03-6b03-4c6f-83be-484c41686458" containerName="cleanup"
Oct 31 19:45:33.464361 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.442451    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3620ed03-6b03-4c6f-83be-484c41686458" containerName="cleanup"
Oct 31 19:45:33.464361 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.442475    4645 cpu_manager.go:414] "containerMap: removing container" podUID="bbd4246c-22ba-4116-8e31-b53d3bdc2fbb" containerName="cleanup"
Oct 31 19:45:33.464361 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.442490    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbd4246c-22ba-4116-8e31-b53d3bdc2fbb" containerName="cleanup"
Oct 31 19:45:33.464361 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.442500    4645 cpu_manager.go:414] "containerMap: removing container" podUID="4ae53805-07ba-4f39-bcbf-f19b67f0ad16" containerName="cleanup"
Oct 31 19:45:33.464361 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.442507    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ae53805-07ba-4f39-bcbf-f19b67f0ad16" containerName="cleanup"
Oct 31 19:45:33.464361 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.442525    4645 cpu_manager.go:414] "containerMap: removing container" podUID="bbd4246c-22ba-4116-8e31-b53d3bdc2fbb" containerName="cleanup"
Oct 31 19:45:33.464361 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.442532    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbd4246c-22ba-4116-8e31-b53d3bdc2fbb" containerName="cleanup"
Oct 31 19:45:33.464361 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.442809    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ae53805-07ba-4f39-bcbf-f19b67f0ad16" containerName="cleanup"
Oct 31 19:45:33.464361 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.442822    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbd4246c-22ba-4116-8e31-b53d3bdc2fbb" containerName="cleanup"
Oct 31 19:45:33.464361 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.442835    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="3620ed03-6b03-4c6f-83be-484c41686458" containerName="cleanup"
Oct 31 19:45:33.464361 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.442845    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbd4246c-22ba-4116-8e31-b53d3bdc2fbb" containerName="cleanup"
Oct 31 19:45:33.464361 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.442856    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ae53805-07ba-4f39-bcbf-f19b67f0ad16" containerName="cleanup"
Oct 31 19:45:33.464361 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.442869    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbd4246c-22ba-4116-8e31-b53d3bdc2fbb" containerName="cleanup"
Oct 31 19:45:33.464361 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.442888    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="3620ed03-6b03-4c6f-83be-484c41686458" containerName="cleanup"
Oct 31 19:45:33.464361 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.442903    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="3620ed03-6b03-4c6f-83be-484c41686458" containerName="cleanup"
Oct 31 19:45:33.478644 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.478611    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9"]
Oct 31 19:45:33.488898 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.488105    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9"
Oct 31 19:45:33.519722 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.510166    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-754e0a-openshift-default-dockercfg-t75h6\""
Oct 31 19:45:33.540666 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.539943    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d8c398-56d6bc9d68-8ddtd"
Oct 31 19:45:33.541896 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.541876    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-435728-29891265-tfbvw"
Oct 31 19:45:33.550558 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.550529    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-03dfee-29891265-njm8t"
Oct 31 19:45:33.558001 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.557969    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53b785a3-a308-43ec-92b9-ccd7d6a8bcad-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53b785a3-a308-43ec-92b9-ccd7d6a8bcad-maas-api-service-ca\") pod \"53b785a3-a308-43ec-92b9-ccd7d6a8bcad\" (UID: \"53b785a3-a308-43ec-92b9-ccd7d6a8bcad\") "
Oct 31 19:45:33.558266 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.558243    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53b785a3-a308-43ec-92b9-ccd7d6a8bcad-kube-api-access-28z8w\" (UniqueName: \"kubernetes.io/projected/53b785a3-a308-43ec-92b9-ccd7d6a8bcad-kube-api-access-28z8w\") pod \"53b785a3-a308-43ec-92b9-ccd7d6a8bcad\" (UID: \"53b785a3-a308-43ec-92b9-ccd7d6a8bcad\") "
Oct 31 19:45:33.562432 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.558588    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d734b787-3a19-4da6-8b9b-93aa92eaa6a5-istiod-ca-cert\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9\" (UID: \"d734b787-3a19-4da6-8b9b-93aa92eaa6a5\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9"
Oct 31 19:45:33.562432 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.558791    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d734b787-3a19-4da6-8b9b-93aa92eaa6a5-istio-envoy\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9\" (UID: \"d734b787-3a19-4da6-8b9b-93aa92eaa6a5\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9"
Oct 31 19:45:33.562432 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.558879    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d734b787-3a19-4da6-8b9b-93aa92eaa6a5-workload-socket\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9\" (UID: \"d734b787-3a19-4da6-8b9b-93aa92eaa6a5\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9"
Oct 31 19:45:33.562432 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.558930    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d734b787-3a19-4da6-8b9b-93aa92eaa6a5-istio-data\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9\" (UID: \"d734b787-3a19-4da6-8b9b-93aa92eaa6a5\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9"
Oct 31 19:45:33.562432 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.559109    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d734b787-3a19-4da6-8b9b-93aa92eaa6a5-credential-socket\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9\" (UID: \"d734b787-3a19-4da6-8b9b-93aa92eaa6a5\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9"
Oct 31 19:45:33.562432 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.559183    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5nth\" (UniqueName: \"kubernetes.io/projected/d734b787-3a19-4da6-8b9b-93aa92eaa6a5-kube-api-access-x5nth\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9\" (UID: \"d734b787-3a19-4da6-8b9b-93aa92eaa6a5\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9"
Oct 31 19:45:33.562432 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.559273    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d734b787-3a19-4da6-8b9b-93aa92eaa6a5-workload-certs\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9\" (UID: \"d734b787-3a19-4da6-8b9b-93aa92eaa6a5\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9"
Oct 31 19:45:33.562432 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.559379    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d734b787-3a19-4da6-8b9b-93aa92eaa6a5-istio-token\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9\" (UID: \"d734b787-3a19-4da6-8b9b-93aa92eaa6a5\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9"
Oct 31 19:45:33.562432 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.559500    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d734b787-3a19-4da6-8b9b-93aa92eaa6a5-istio-podinfo\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9\" (UID: \"d734b787-3a19-4da6-8b9b-93aa92eaa6a5\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9"
Oct 31 19:45:33.562432 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.559505    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a891ce-29891265-md69g"
Oct 31 19:45:33.562432 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.559939    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53b785a3-a308-43ec-92b9-ccd7d6a8bcad-maas-api-service-ca" pod "53b785a3-a308-43ec-92b9-ccd7d6a8bcad" (UID: "53b785a3-a308-43ec-92b9-ccd7d6a8bcad"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:45:33.563170 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.562510    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53b785a3-a308-43ec-92b9-ccd7d6a8bcad-kube-api-access-28z8w" pod "53b785a3-a308-43ec-92b9-ccd7d6a8bcad" (UID: "53b785a3-a308-43ec-92b9-ccd7d6a8bcad"). InnerVolumeSpecName "kube-api-access-28z8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:45:33.612877 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.612849    4645 scope.go:122] "RemoveContainer" containerID="6e0fdf219940b19304fa4ae6614e0943e33083230d93614ed9bb893338915a19"
Oct 31 19:45:33.643625 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.643590    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ae53805-07ba-4f39-bcbf-f19b67f0ad16" path="/var/lib/kubelet/pods/4ae53805-07ba-4f39-bcbf-f19b67f0ad16/volumes"
Oct 31 19:45:33.660810 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.660775    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3961f45f-4b0d-4880-b8d8-e55798d4abe8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3961f45f-4b0d-4880-b8d8-e55798d4abe8-maas-api-service-ca\") pod \"3961f45f-4b0d-4880-b8d8-e55798d4abe8\" (UID: \"3961f45f-4b0d-4880-b8d8-e55798d4abe8\") "
Oct 31 19:45:33.660990 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.660834    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/220956a9-b7c5-48b8-a598-61326c85691a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/220956a9-b7c5-48b8-a598-61326c85691a-maas-api-service-ca\") pod \"220956a9-b7c5-48b8-a598-61326c85691a\" (UID: \"220956a9-b7c5-48b8-a598-61326c85691a\") "
Oct 31 19:45:33.660990 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.660902    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3961f45f-4b0d-4880-b8d8-e55798d4abe8-kube-api-access-m9sfr\" (UniqueName: \"kubernetes.io/projected/3961f45f-4b0d-4880-b8d8-e55798d4abe8-kube-api-access-m9sfr\") pod \"3961f45f-4b0d-4880-b8d8-e55798d4abe8\" (UID: \"3961f45f-4b0d-4880-b8d8-e55798d4abe8\") "
Oct 31 19:45:33.661115 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.661094    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/220956a9-b7c5-48b8-a598-61326c85691a-kube-api-access-8jfnb\" (UniqueName: \"kubernetes.io/projected/220956a9-b7c5-48b8-a598-61326c85691a-kube-api-access-8jfnb\") pod \"220956a9-b7c5-48b8-a598-61326c85691a\" (UID: \"220956a9-b7c5-48b8-a598-61326c85691a\") "
Oct 31 19:45:33.661456 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.661432    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d734b787-3a19-4da6-8b9b-93aa92eaa6a5-credential-socket\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9\" (UID: \"d734b787-3a19-4da6-8b9b-93aa92eaa6a5\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9"
Oct 31 19:45:33.661567 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.661494    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5nth\" (UniqueName: \"kubernetes.io/projected/d734b787-3a19-4da6-8b9b-93aa92eaa6a5-kube-api-access-x5nth\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9\" (UID: \"d734b787-3a19-4da6-8b9b-93aa92eaa6a5\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9"
Oct 31 19:45:33.661635 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.661564    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d734b787-3a19-4da6-8b9b-93aa92eaa6a5-workload-certs\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9\" (UID: \"d734b787-3a19-4da6-8b9b-93aa92eaa6a5\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9"
Oct 31 19:45:33.661692 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.661650    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d734b787-3a19-4da6-8b9b-93aa92eaa6a5-istio-token\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9\" (UID: \"d734b787-3a19-4da6-8b9b-93aa92eaa6a5\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9"
Oct 31 19:45:33.661758 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.661745    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d734b787-3a19-4da6-8b9b-93aa92eaa6a5-istio-podinfo\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9\" (UID: \"d734b787-3a19-4da6-8b9b-93aa92eaa6a5\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9"
Oct 31 19:45:33.661816 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.661780    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d734b787-3a19-4da6-8b9b-93aa92eaa6a5-istiod-ca-cert\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9\" (UID: \"d734b787-3a19-4da6-8b9b-93aa92eaa6a5\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9"
Oct 31 19:45:33.661926 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.661910    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d734b787-3a19-4da6-8b9b-93aa92eaa6a5-istio-envoy\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9\" (UID: \"d734b787-3a19-4da6-8b9b-93aa92eaa6a5\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9"
Oct 31 19:45:33.662008 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.661993    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d734b787-3a19-4da6-8b9b-93aa92eaa6a5-workload-socket\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9\" (UID: \"d734b787-3a19-4da6-8b9b-93aa92eaa6a5\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9"
Oct 31 19:45:33.662067 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.662038    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d734b787-3a19-4da6-8b9b-93aa92eaa6a5-istio-data\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9\" (UID: \"d734b787-3a19-4da6-8b9b-93aa92eaa6a5\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9"
Oct 31 19:45:33.662226 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.662212    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28z8w\" (UniqueName: \"kubernetes.io/projected/53b785a3-a308-43ec-92b9-ccd7d6a8bcad-kube-api-access-28z8w\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:33.662293 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.662233    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53b785a3-a308-43ec-92b9-ccd7d6a8bcad-maas-api-service-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:33.662618 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.662596    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d734b787-3a19-4da6-8b9b-93aa92eaa6a5-istio-data\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9\" (UID: \"d734b787-3a19-4da6-8b9b-93aa92eaa6a5\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9"
Oct 31 19:45:33.662708 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.662622    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d734b787-3a19-4da6-8b9b-93aa92eaa6a5-workload-certs\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9\" (UID: \"d734b787-3a19-4da6-8b9b-93aa92eaa6a5\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9"
Oct 31 19:45:33.663280 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.663259    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d734b787-3a19-4da6-8b9b-93aa92eaa6a5-credential-socket\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9\" (UID: \"d734b787-3a19-4da6-8b9b-93aa92eaa6a5\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9"
Oct 31 19:45:33.664356 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.664319    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d734b787-3a19-4da6-8b9b-93aa92eaa6a5-istiod-ca-cert\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9\" (UID: \"d734b787-3a19-4da6-8b9b-93aa92eaa6a5\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9"
Oct 31 19:45:33.665149 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.664578    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3961f45f-4b0d-4880-b8d8-e55798d4abe8-maas-api-service-ca" pod "3961f45f-4b0d-4880-b8d8-e55798d4abe8" (UID: "3961f45f-4b0d-4880-b8d8-e55798d4abe8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:45:33.665541 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.665519    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d734b787-3a19-4da6-8b9b-93aa92eaa6a5-workload-socket\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9\" (UID: \"d734b787-3a19-4da6-8b9b-93aa92eaa6a5\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9"
Oct 31 19:45:33.670045 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.668071    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/220956a9-b7c5-48b8-a598-61326c85691a-maas-api-service-ca" pod "220956a9-b7c5-48b8-a598-61326c85691a" (UID: "220956a9-b7c5-48b8-a598-61326c85691a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:45:33.670994 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.670972    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d734b787-3a19-4da6-8b9b-93aa92eaa6a5-istio-envoy\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9\" (UID: \"d734b787-3a19-4da6-8b9b-93aa92eaa6a5\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9"
Oct 31 19:45:33.677903 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.677873    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-03dfee-29891265-njm8t" event={"ID":"3961f45f-4b0d-4880-b8d8-e55798d4abe8","Type":"ContainerDied","Data":"982f45c6f66cd2868373d25b98ff84111951da289dc0e241ed1172f6e1252b81"}
Oct 31 19:45:33.678069 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.678054    4645 scope.go:122] "RemoveContainer" containerID="9343c64c137e89bb1ff60deb44056999874904ae00b258f9636701cc3ab79cb5"
Oct 31 19:45:33.678313 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.678297    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-03dfee-29891265-njm8t"
Oct 31 19:45:33.702233 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.691693    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5nth\" (UniqueName: \"kubernetes.io/projected/d734b787-3a19-4da6-8b9b-93aa92eaa6a5-kube-api-access-x5nth\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9\" (UID: \"d734b787-3a19-4da6-8b9b-93aa92eaa6a5\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9"
Oct 31 19:45:33.727033 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.727008    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d734b787-3a19-4da6-8b9b-93aa92eaa6a5-istio-podinfo\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9\" (UID: \"d734b787-3a19-4da6-8b9b-93aa92eaa6a5\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9"
Oct 31 19:45:33.733237 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.729579    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/220956a9-b7c5-48b8-a598-61326c85691a-kube-api-access-8jfnb" pod "220956a9-b7c5-48b8-a598-61326c85691a" (UID: "220956a9-b7c5-48b8-a598-61326c85691a"). InnerVolumeSpecName "kube-api-access-8jfnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:45:33.733237 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.730294    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3961f45f-4b0d-4880-b8d8-e55798d4abe8-kube-api-access-m9sfr" pod "3961f45f-4b0d-4880-b8d8-e55798d4abe8" (UID: "3961f45f-4b0d-4880-b8d8-e55798d4abe8"). InnerVolumeSpecName "kube-api-access-m9sfr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:45:33.735421 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.735390    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a891ce-29891265-md69g" event={"ID":"220956a9-b7c5-48b8-a598-61326c85691a","Type":"ContainerDied","Data":"415b0234203ca2ef2c91e78b763cda02736ca4317b283d6bdfd3d848f074de18"}
Oct 31 19:45:33.735531 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.735459    4645 scope.go:122] "RemoveContainer" containerID="7f78c5a4204a76801ef461c028a1f5cecf1a85d0ffd2eaaf9bf39a55621ec06d"
Oct 31 19:45:33.735531 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.735458    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a891ce-29891265-md69g"
Oct 31 19:45:33.738078 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.738053    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d734b787-3a19-4da6-8b9b-93aa92eaa6a5-istio-token\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9\" (UID: \"d734b787-3a19-4da6-8b9b-93aa92eaa6a5\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9"
Oct 31 19:45:33.755694 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.752912    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1e8cf0-29891265-mbhwx"]
Oct 31 19:45:33.757301 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.757273    4645 generic.go:356] "Generic (PLEG): container finished" podID="e7198524-cd1a-4038-83c6-55f9a5391970" containerID="0b7a923b65422c7da00f1fbd033b12779326e8d3f858ea417e007d2e26815aec" exitCode=60
Oct 31 19:45:33.757599 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.757581    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891265-r6vvt" event={"ID":"e7198524-cd1a-4038-83c6-55f9a5391970","Type":"ContainerDied","Data":"0b7a923b65422c7da00f1fbd033b12779326e8d3f858ea417e007d2e26815aec"}
Oct 31 19:45:33.761391 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.760729    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8dcfa8-29891265-9lkmt" event={"ID":"bbd4246c-22ba-4116-8e31-b53d3bdc2fbb","Type":"ContainerDied","Data":"8475caf4e62a63a4ea6e546b905fe67af9c5a6da5cb440515abfe21cc4fcf9ea"}
Oct 31 19:45:33.761391 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.760816    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8dcfa8-29891265-9lkmt"
Oct 31 19:45:33.761642 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.761500    4645 scope.go:122] "RemoveContainer" containerID="32dbade5037b8291ce61e8fcb623c66b9b4cf1fb48db6b83f724a52fb9bf2271"
Oct 31 19:45:33.762148 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.761709    4645 scope.go:122] "RemoveContainer" containerID="0b7a923b65422c7da00f1fbd033b12779326e8d3f858ea417e007d2e26815aec"
Oct 31 19:45:33.762148 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:45:33.762085    4645 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-8730d5-29891265-r6vvt_odh-ai-gateway-infra(e7198524-cd1a-4038-83c6-55f9a5391970)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891265-r6vvt" podUID="e7198524-cd1a-4038-83c6-55f9a5391970"
Oct 31 19:45:33.765909 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.764480    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jfnb\" (UniqueName: \"kubernetes.io/projected/220956a9-b7c5-48b8-a598-61326c85691a-kube-api-access-8jfnb\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:33.765909 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.764507    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3961f45f-4b0d-4880-b8d8-e55798d4abe8-maas-api-service-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:33.765909 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.764523    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/220956a9-b7c5-48b8-a598-61326c85691a-maas-api-service-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:33.765909 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.764539    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9sfr\" (UniqueName: \"kubernetes.io/projected/3961f45f-4b0d-4880-b8d8-e55798d4abe8-kube-api-access-m9sfr\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:33.771371 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.771322    4645 generic.go:356] "Generic (PLEG): container finished" podID="57977c65-f0c2-46d1-a05c-20bd8ed579c3" containerID="4c862f620b30d7bb75ae8bed20dd6c62c02ed161fb5a42fe4bfedda138f097aa" exitCode=60
Oct 31 19:45:33.771495 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.771425    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1e8cf0-29891265-mbhwx" event={"ID":"57977c65-f0c2-46d1-a05c-20bd8ed579c3","Type":"ContainerDied","Data":"4c862f620b30d7bb75ae8bed20dd6c62c02ed161fb5a42fe4bfedda138f097aa"}
Oct 31 19:45:33.774371 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.774306    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ae2e60-29891265-5s5f8" event={"ID":"3620ed03-6b03-4c6f-83be-484c41686458","Type":"ContainerDied","Data":"e6678a4880ee9d086d1889bfb8b7446ef49b17f48b47be8ed21483b9b09b0dfd"}
Oct 31 19:45:33.774564 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.774545    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ae2e60-29891265-5s5f8"
Oct 31 19:45:33.780228 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.778787    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-435728-29891265-tfbvw" event={"ID":"53b785a3-a308-43ec-92b9-ccd7d6a8bcad","Type":"ContainerDied","Data":"b8a91d43e980aad7e6b95cea72cad226fef6924a3bcb6267bd378029c389c06d"}
Oct 31 19:45:33.780228 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.778906    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-435728-29891265-tfbvw"
Oct 31 19:45:33.800912 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.800882    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a891ce-29891265-md69g"]
Oct 31 19:45:33.801127 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.801099    4645 scope.go:122] "RemoveContainer" containerID="ef61a24e0614cc0f6b8c072b0a59cc05d3d213c03809cff94a2e6d1e7b9aa6e1"
Oct 31 19:45:33.827582 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.827555    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a891ce-29891265-md69g"]
Oct 31 19:45:33.831520 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.831495    4645 scope.go:122] "RemoveContainer" containerID="635a2d9c5a29f44dc065cb40324515a4364da5c1747b819ced38ef04f0d17697"
Oct 31 19:45:33.853897 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.853864    4645 scope.go:122] "RemoveContainer" containerID="185a3f95ac3fde394f5f34cd3d28c254f1380d80bff0f833b5e702a775f58924"
Oct 31 19:45:33.855167 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.855141    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9"
Oct 31 19:45:33.898139 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.898088    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891265-r6vvt"]
Oct 31 19:45:33.940297 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:33.940007    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:34.035408 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:34.020060    4645 scope.go:122] "RemoveContainer" containerID="210204d9d27cdeeb8b4b81bb984e28a075e830571b41387814575b0beeebd949"
Oct 31 19:45:34.035408 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:34.029543    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-435728-29891265-tfbvw"]
Oct 31 19:45:34.052708 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:34.052450    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-435728-29891265-tfbvw"]
Oct 31 19:45:34.064370 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:34.064293    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ae2e60-29891265-5s5f8"]
Oct 31 19:45:34.076967 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:34.076931    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-ae2e60-29891265-5s5f8"]
Oct 31 19:45:34.087030 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:34.086966    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8dcfa8-29891265-9lkmt"]
Oct 31 19:45:34.106517 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:34.105684    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-23843f-29891265-52rjc"
Oct 31 19:45:34.123586 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:34.123538    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-8dcfa8-29891265-9lkmt"]
Oct 31 19:45:34.133424 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:34.133368    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-03dfee-29891265-njm8t"]
Oct 31 19:45:34.139449 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:34.139409    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-03dfee-29891265-njm8t"]
Oct 31 19:45:34.234789 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:34.234758    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/615544a2-1b13-4bce-aeb6-7b3b32e6a509-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/615544a2-1b13-4bce-aeb6-7b3b32e6a509-maas-api-service-ca\") pod \"615544a2-1b13-4bce-aeb6-7b3b32e6a509\" (UID: \"615544a2-1b13-4bce-aeb6-7b3b32e6a509\") "
Oct 31 19:45:34.234972 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:34.234824    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/615544a2-1b13-4bce-aeb6-7b3b32e6a509-kube-api-access-qh449\" (UniqueName: \"kubernetes.io/projected/615544a2-1b13-4bce-aeb6-7b3b32e6a509-kube-api-access-qh449\") pod \"615544a2-1b13-4bce-aeb6-7b3b32e6a509\" (UID: \"615544a2-1b13-4bce-aeb6-7b3b32e6a509\") "
Oct 31 19:45:34.236275 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:34.236224    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/615544a2-1b13-4bce-aeb6-7b3b32e6a509-maas-api-service-ca" pod "615544a2-1b13-4bce-aeb6-7b3b32e6a509" (UID: "615544a2-1b13-4bce-aeb6-7b3b32e6a509"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:45:34.238772 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:34.238746    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/615544a2-1b13-4bce-aeb6-7b3b32e6a509-kube-api-access-qh449" pod "615544a2-1b13-4bce-aeb6-7b3b32e6a509" (UID: "615544a2-1b13-4bce-aeb6-7b3b32e6a509"). InnerVolumeSpecName "kube-api-access-qh449". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:45:34.337311 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:34.337272    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/615544a2-1b13-4bce-aeb6-7b3b32e6a509-maas-api-service-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:34.337311 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:34.337306    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qh449\" (UniqueName: \"kubernetes.io/projected/615544a2-1b13-4bce-aeb6-7b3b32e6a509-kube-api-access-qh449\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:34.409354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:34.407442    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-fd1c91-29891265-rcrqk"
Oct 31 19:45:34.587417 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:34.566679    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/769ee829-1015-45b1-9dd7-d9cfe372fa9c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/769ee829-1015-45b1-9dd7-d9cfe372fa9c-maas-api-service-ca\") pod \"769ee829-1015-45b1-9dd7-d9cfe372fa9c\" (UID: \"769ee829-1015-45b1-9dd7-d9cfe372fa9c\") "
Oct 31 19:45:34.587417 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:34.566783    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/769ee829-1015-45b1-9dd7-d9cfe372fa9c-kube-api-access-8jsfh\" (UniqueName: \"kubernetes.io/projected/769ee829-1015-45b1-9dd7-d9cfe372fa9c-kube-api-access-8jsfh\") pod \"769ee829-1015-45b1-9dd7-d9cfe372fa9c\" (UID: \"769ee829-1015-45b1-9dd7-d9cfe372fa9c\") "
Oct 31 19:45:34.656367 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:34.656222    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/769ee829-1015-45b1-9dd7-d9cfe372fa9c-maas-api-service-ca" pod "769ee829-1015-45b1-9dd7-d9cfe372fa9c" (UID: "769ee829-1015-45b1-9dd7-d9cfe372fa9c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:45:34.915177 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:34.914593    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891265-9vwg9"
Oct 31 19:45:34.986223 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:34.960898    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/769ee829-1015-45b1-9dd7-d9cfe372fa9c-maas-api-service-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:35.113635 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:35.113550    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6f57036-6c45-4def-a9c1-0132ab845b3e-kube-api-access-lxm98\" (UniqueName: \"kubernetes.io/projected/e6f57036-6c45-4def-a9c1-0132ab845b3e-kube-api-access-lxm98\") pod \"e6f57036-6c45-4def-a9c1-0132ab845b3e\" (UID: \"e6f57036-6c45-4def-a9c1-0132ab845b3e\") "
Oct 31 19:45:35.114147 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:35.114124    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6f57036-6c45-4def-a9c1-0132ab845b3e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6f57036-6c45-4def-a9c1-0132ab845b3e-maas-api-service-ca\") pod \"e6f57036-6c45-4def-a9c1-0132ab845b3e\" (UID: \"e6f57036-6c45-4def-a9c1-0132ab845b3e\") "
Oct 31 19:45:35.146352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:35.143164    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/769ee829-1015-45b1-9dd7-d9cfe372fa9c-kube-api-access-8jsfh" pod "769ee829-1015-45b1-9dd7-d9cfe372fa9c" (UID: "769ee829-1015-45b1-9dd7-d9cfe372fa9c"). InnerVolumeSpecName "kube-api-access-8jsfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:45:35.146752 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:35.146723    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6f57036-6c45-4def-a9c1-0132ab845b3e-maas-api-service-ca" pod "e6f57036-6c45-4def-a9c1-0132ab845b3e" (UID: "e6f57036-6c45-4def-a9c1-0132ab845b3e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:45:35.183205 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:35.183173    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-23843f-29891265-52rjc" event={"ID":"615544a2-1b13-4bce-aeb6-7b3b32e6a509","Type":"ContainerDied","Data":"e5e95bae79d9a6eac9ffb1eb124b7f223785facb8d11d443408b4abd750761e5"}
Oct 31 19:45:35.183459 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:35.183441    4645 scope.go:122] "RemoveContainer" containerID="9d70f7ad488c24d0cdd605f400ddf6241bfc449060eb6eb3c6867820fb2aaa45"
Oct 31 19:45:35.183661 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:35.183647    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-23843f-29891265-52rjc"
Oct 31 19:45:35.256116 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:35.256080    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6f57036-6c45-4def-a9c1-0132ab845b3e-maas-api-service-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:35.256266 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:35.256121    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jsfh\" (UniqueName: \"kubernetes.io/projected/769ee829-1015-45b1-9dd7-d9cfe372fa9c-kube-api-access-8jsfh\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:35.341351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:35.341070    4645 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l" podUID="12bea86c-61e5-465e-82fb-b4673a4d1be9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:45:35.367688 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:35.367585    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6f57036-6c45-4def-a9c1-0132ab845b3e-kube-api-access-lxm98" pod "e6f57036-6c45-4def-a9c1-0132ab845b3e" (UID: "e6f57036-6c45-4def-a9c1-0132ab845b3e"). InnerVolumeSpecName "kube-api-access-lxm98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:45:35.454913 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:35.454872    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-03dfee/test-model-03dfee-kserve-68c5b8bcbf-cwgcs" podUID="6e861821-e221-44d2-956a-66f0f656e1b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.135:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:45:35.457721 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:35.457688    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4" podUID="6032a997-b795-4561-805c-110cf67a9d5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:45:35.462954 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:35.458270    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891265-9vwg9" event={"ID":"e6f57036-6c45-4def-a9c1-0132ab845b3e","Type":"ContainerDied","Data":"57535395ac7e22369d5701eeddeb3f9608aa2db7ecfca66eee33fc79b36f7ee8"}
Oct 31 19:45:35.498753 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:35.463175    4645 scope.go:122] "RemoveContainer" containerID="ab4229cdce00f7cdd2924c251bc483b331f43ee65baea7c23143c4b4d89cf45d"
Oct 31 19:45:35.498753 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:35.458372    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891265-9vwg9"
Oct 31 19:45:35.572355 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:35.532795    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9"]
Oct 31 19:45:35.629538 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:35.599176    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw" podUID="d504b2d3-c26d-4dcf-a7eb-efe20732ae4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:45:35.643772 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:35.643735    4645 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-1e8cf0/test-model-1e8cf0-kserve-74747d94df-4gd76" podUID="b673f934-bc6d-4ae8-8adf-4acedf4a8df6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.158:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:45:35.644734 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:35.644704    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw" podUID="1287db54-a587-4c11-9eb6-1606de555894" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:45:35.658176 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:35.658131    4645 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-03dfee/test-model-03dfee-kserve-68c5b8bcbf-cwgcs" podUID="6e861821-e221-44d2-956a-66f0f656e1b5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.135:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:45:35.667762 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:35.658153    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-a4e312/test-model-a4e312-kserve-f4b656bdc-29rj9" podUID="7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.147:8000/ready\": context deadline exceeded"
Oct 31 19:45:35.667762 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:35.659742    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxm98\" (UniqueName: \"kubernetes.io/projected/e6f57036-6c45-4def-a9c1-0132ab845b3e-kube-api-access-lxm98\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:35.667762 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:35.665651    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1e8cf0-29891265-mbhwx"
Oct 31 19:45:35.683365 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:35.682871    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:45:35.683365 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:35.682960    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:45:35.683365 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:35.683002    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:45:35.727059 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:35.727027    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-785d26-29891265-8cn6g"
Oct 31 19:45:35.750152 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:35.750120    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="220956a9-b7c5-48b8-a598-61326c85691a" path="/var/lib/kubelet/pods/220956a9-b7c5-48b8-a598-61326c85691a/volumes"
Oct 31 19:45:35.751188 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:35.751119    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3620ed03-6b03-4c6f-83be-484c41686458" path="/var/lib/kubelet/pods/3620ed03-6b03-4c6f-83be-484c41686458/volumes"
Oct 31 19:45:35.751957 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:35.751938    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3961f45f-4b0d-4880-b8d8-e55798d4abe8" path="/var/lib/kubelet/pods/3961f45f-4b0d-4880-b8d8-e55798d4abe8/volumes"
Oct 31 19:45:35.757204 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:35.757182    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53b785a3-a308-43ec-92b9-ccd7d6a8bcad" path="/var/lib/kubelet/pods/53b785a3-a308-43ec-92b9-ccd7d6a8bcad/volumes"
Oct 31 19:45:35.757939 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:35.757922    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbd4246c-22ba-4116-8e31-b53d3bdc2fbb" path="/var/lib/kubelet/pods/bbd4246c-22ba-4116-8e31-b53d3bdc2fbb/volumes"
Oct 31 19:45:35.767670 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:35.767648    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57977c65-f0c2-46d1-a05c-20bd8ed579c3-kube-api-access-66szc\" (UniqueName: \"kubernetes.io/projected/57977c65-f0c2-46d1-a05c-20bd8ed579c3-kube-api-access-66szc\") pod \"57977c65-f0c2-46d1-a05c-20bd8ed579c3\" (UID: \"57977c65-f0c2-46d1-a05c-20bd8ed579c3\") "
Oct 31 19:45:35.768674 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:35.768652    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57977c65-f0c2-46d1-a05c-20bd8ed579c3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57977c65-f0c2-46d1-a05c-20bd8ed579c3-maas-api-service-ca\") pod \"57977c65-f0c2-46d1-a05c-20bd8ed579c3\" (UID: \"57977c65-f0c2-46d1-a05c-20bd8ed579c3\") "
Oct 31 19:45:35.774579 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:35.774556    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57977c65-f0c2-46d1-a05c-20bd8ed579c3-maas-api-service-ca" pod "57977c65-f0c2-46d1-a05c-20bd8ed579c3" (UID: "57977c65-f0c2-46d1-a05c-20bd8ed579c3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:45:35.850308 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:35.850204    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57977c65-f0c2-46d1-a05c-20bd8ed579c3-kube-api-access-66szc" pod "57977c65-f0c2-46d1-a05c-20bd8ed579c3" (UID: "57977c65-f0c2-46d1-a05c-20bd8ed579c3"). InnerVolumeSpecName "kube-api-access-66szc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:45:35.885295 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:35.885256    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e3605c3-ec3b-4fb7-be2a-40fe32c5adea-kube-api-access-b8nc4\" (UniqueName: \"kubernetes.io/projected/2e3605c3-ec3b-4fb7-be2a-40fe32c5adea-kube-api-access-b8nc4\") pod \"2e3605c3-ec3b-4fb7-be2a-40fe32c5adea\" (UID: \"2e3605c3-ec3b-4fb7-be2a-40fe32c5adea\") "
Oct 31 19:45:35.885507 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:35.885481    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e3605c3-ec3b-4fb7-be2a-40fe32c5adea-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e3605c3-ec3b-4fb7-be2a-40fe32c5adea-maas-api-service-ca\") pod \"2e3605c3-ec3b-4fb7-be2a-40fe32c5adea\" (UID: \"2e3605c3-ec3b-4fb7-be2a-40fe32c5adea\") "
Oct 31 19:45:35.889876 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:35.889846    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e3605c3-ec3b-4fb7-be2a-40fe32c5adea-kube-api-access-b8nc4" pod "2e3605c3-ec3b-4fb7-be2a-40fe32c5adea" (UID: "2e3605c3-ec3b-4fb7-be2a-40fe32c5adea"). InnerVolumeSpecName "kube-api-access-b8nc4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:45:35.890866 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:35.890840    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e3605c3-ec3b-4fb7-be2a-40fe32c5adea-maas-api-service-ca" pod "2e3605c3-ec3b-4fb7-be2a-40fe32c5adea" (UID: "2e3605c3-ec3b-4fb7-be2a-40fe32c5adea"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:45:35.901154 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:35.901134    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-23843f-29891265-52rjc"]
Oct 31 19:45:35.927510 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:35.927483    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66szc\" (UniqueName: \"kubernetes.io/projected/57977c65-f0c2-46d1-a05c-20bd8ed579c3-kube-api-access-66szc\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:35.927696 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:35.927681    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8nc4\" (UniqueName: \"kubernetes.io/projected/2e3605c3-ec3b-4fb7-be2a-40fe32c5adea-kube-api-access-b8nc4\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:35.927802 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:35.927790    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57977c65-f0c2-46d1-a05c-20bd8ed579c3-maas-api-service-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:35.927890 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:35.927879    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e3605c3-ec3b-4fb7-be2a-40fe32c5adea-maas-api-service-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:35.948754 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:35.943411    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-23843f-29891265-52rjc"]
Oct 31 19:45:36.002022 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:35.964148    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:36.002022 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:35.964824    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-fd1c91-29891265-rcrqk" event={"ID":"769ee829-1015-45b1-9dd7-d9cfe372fa9c","Type":"ContainerDied","Data":"f903486bf893265c23d3cd19a8290df2a8b3226ec37289fa53fb7397054b1c4a"}
Oct 31 19:45:36.002022 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:35.964944    4645 scope.go:122] "RemoveContainer" containerID="ac8f707fe9dec466c5648a974c55766fd6f6a315ed9754cd31adbcf340360b1d"
Oct 31 19:45:36.002022 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:35.965542    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-fd1c91-29891265-rcrqk"
Oct 31 19:45:36.002022 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:35.973695    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-88eaf9-29891265-zttxn" event={"ID":"25370d12-c485-47b9-986a-a775c3fbd2b7","Type":"ContainerStarted","Data":"7bad4130f50ad87d5e69f861980abe452a1840e961b6061d9829ea9c23286f39"}
Oct 31 19:45:36.002022 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:35.974730    4645 scope.go:122] "RemoveContainer" containerID="7bad4130f50ad87d5e69f861980abe452a1840e961b6061d9829ea9c23286f39"
Oct 31 19:45:36.002022 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:45:35.975517    4645 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-88eaf9-29891265-zttxn_odh-ai-gateway-infra(25370d12-c485-47b9-986a-a775c3fbd2b7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-88eaf9-29891265-zttxn" podUID="25370d12-c485-47b9-986a-a775c3fbd2b7"
Oct 31 19:45:36.064201 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.024392    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891265-9vwg9"]
Oct 31 19:45:36.064201 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.063751    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891265-9vwg9"]
Oct 31 19:45:36.087531 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.087505    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-fd1c91-29891265-rcrqk"]
Oct 31 19:45:36.095790 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.095759    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-fd1c91-29891265-rcrqk"]
Oct 31 19:45:36.155669 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.155645    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891265-r6vvt"
Oct 31 19:45:36.192288 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.192251    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v"]
Oct 31 19:45:36.193699 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.193674    4645 cpu_manager.go:414] "containerMap: removing container" podUID="3620ed03-6b03-4c6f-83be-484c41686458" containerName="cleanup"
Oct 31 19:45:36.193929 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.193914    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3620ed03-6b03-4c6f-83be-484c41686458" containerName="cleanup"
Oct 31 19:45:36.194037 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.194028    4645 cpu_manager.go:414] "containerMap: removing container" podUID="220956a9-b7c5-48b8-a598-61326c85691a" containerName="cleanup"
Oct 31 19:45:36.194143 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.194132    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="220956a9-b7c5-48b8-a598-61326c85691a" containerName="cleanup"
Oct 31 19:45:36.194221 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.194212    4645 cpu_manager.go:414] "containerMap: removing container" podUID="e6f57036-6c45-4def-a9c1-0132ab845b3e" containerName="cleanup"
Oct 31 19:45:36.194294 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.194285    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6f57036-6c45-4def-a9c1-0132ab845b3e" containerName="cleanup"
Oct 31 19:45:36.194410 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.194400    4645 cpu_manager.go:414] "containerMap: removing container" podUID="2e3605c3-ec3b-4fb7-be2a-40fe32c5adea" containerName="cleanup"
Oct 31 19:45:36.194480 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.194470    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e3605c3-ec3b-4fb7-be2a-40fe32c5adea" containerName="cleanup"
Oct 31 19:45:36.194561 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.194553    4645 cpu_manager.go:414] "containerMap: removing container" podUID="2e3605c3-ec3b-4fb7-be2a-40fe32c5adea" containerName="cleanup"
Oct 31 19:45:36.194639 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.194629    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e3605c3-ec3b-4fb7-be2a-40fe32c5adea" containerName="cleanup"
Oct 31 19:45:36.194712 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.194704    4645 cpu_manager.go:414] "containerMap: removing container" podUID="769ee829-1015-45b1-9dd7-d9cfe372fa9c" containerName="cleanup"
Oct 31 19:45:36.194780 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.194771    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="769ee829-1015-45b1-9dd7-d9cfe372fa9c" containerName="cleanup"
Oct 31 19:45:36.194892 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.194882    4645 cpu_manager.go:414] "containerMap: removing container" podUID="615544a2-1b13-4bce-aeb6-7b3b32e6a509" containerName="cleanup"
Oct 31 19:45:36.194963 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.194954    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="615544a2-1b13-4bce-aeb6-7b3b32e6a509" containerName="cleanup"
Oct 31 19:45:36.195040 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.195031    4645 cpu_manager.go:414] "containerMap: removing container" podUID="53b785a3-a308-43ec-92b9-ccd7d6a8bcad" containerName="cleanup"
Oct 31 19:45:36.195111 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.195102    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53b785a3-a308-43ec-92b9-ccd7d6a8bcad" containerName="cleanup"
Oct 31 19:45:36.195178 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.195169    4645 cpu_manager.go:414] "containerMap: removing container" podUID="769ee829-1015-45b1-9dd7-d9cfe372fa9c" containerName="cleanup"
Oct 31 19:45:36.195241 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.195232    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="769ee829-1015-45b1-9dd7-d9cfe372fa9c" containerName="cleanup"
Oct 31 19:45:36.195304 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.195296    4645 cpu_manager.go:414] "containerMap: removing container" podUID="53b785a3-a308-43ec-92b9-ccd7d6a8bcad" containerName="cleanup"
Oct 31 19:45:36.195389 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.195379    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53b785a3-a308-43ec-92b9-ccd7d6a8bcad" containerName="cleanup"
Oct 31 19:45:36.195468 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.195459    4645 cpu_manager.go:414] "containerMap: removing container" podUID="3961f45f-4b0d-4880-b8d8-e55798d4abe8" containerName="cleanup"
Oct 31 19:45:36.195544 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.195535    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3961f45f-4b0d-4880-b8d8-e55798d4abe8" containerName="cleanup"
Oct 31 19:45:36.195620 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.195612    4645 cpu_manager.go:414] "containerMap: removing container" podUID="57977c65-f0c2-46d1-a05c-20bd8ed579c3" containerName="cleanup"
Oct 31 19:45:36.195685 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.195676    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57977c65-f0c2-46d1-a05c-20bd8ed579c3" containerName="cleanup"
Oct 31 19:45:36.195762 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.195753    4645 cpu_manager.go:414] "containerMap: removing container" podUID="3961f45f-4b0d-4880-b8d8-e55798d4abe8" containerName="cleanup"
Oct 31 19:45:36.195838 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.195829    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3961f45f-4b0d-4880-b8d8-e55798d4abe8" containerName="cleanup"
Oct 31 19:45:36.195962 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.195951    4645 cpu_manager.go:414] "containerMap: removing container" podUID="e7198524-cd1a-4038-83c6-55f9a5391970" containerName="cleanup"
Oct 31 19:45:36.196030 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.196021    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7198524-cd1a-4038-83c6-55f9a5391970" containerName="cleanup"
Oct 31 19:45:36.196105 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.196097    4645 cpu_manager.go:414] "containerMap: removing container" podUID="2e3605c3-ec3b-4fb7-be2a-40fe32c5adea" containerName="cleanup"
Oct 31 19:45:36.196183 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.196172    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e3605c3-ec3b-4fb7-be2a-40fe32c5adea" containerName="cleanup"
Oct 31 19:45:36.196256 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.196247    4645 cpu_manager.go:414] "containerMap: removing container" podUID="e6f57036-6c45-4def-a9c1-0132ab845b3e" containerName="cleanup"
Oct 31 19:45:36.196344 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.196320    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6f57036-6c45-4def-a9c1-0132ab845b3e" containerName="cleanup"
Oct 31 19:45:36.196423 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.196414    4645 cpu_manager.go:414] "containerMap: removing container" podUID="57977c65-f0c2-46d1-a05c-20bd8ed579c3" containerName="cleanup"
Oct 31 19:45:36.196493 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.196484    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57977c65-f0c2-46d1-a05c-20bd8ed579c3" containerName="cleanup"
Oct 31 19:45:36.196566 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.196558    4645 cpu_manager.go:414] "containerMap: removing container" podUID="e7198524-cd1a-4038-83c6-55f9a5391970" containerName="cleanup"
Oct 31 19:45:36.196638 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.196629    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7198524-cd1a-4038-83c6-55f9a5391970" containerName="cleanup"
Oct 31 19:45:36.196707 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.196699    4645 cpu_manager.go:414] "containerMap: removing container" podUID="615544a2-1b13-4bce-aeb6-7b3b32e6a509" containerName="cleanup"
Oct 31 19:45:36.196776 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.196767    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="615544a2-1b13-4bce-aeb6-7b3b32e6a509" containerName="cleanup"
Oct 31 19:45:36.196854 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.196845    4645 cpu_manager.go:414] "containerMap: removing container" podUID="57977c65-f0c2-46d1-a05c-20bd8ed579c3" containerName="cleanup"
Oct 31 19:45:36.196926 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.196917    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57977c65-f0c2-46d1-a05c-20bd8ed579c3" containerName="cleanup"
Oct 31 19:45:36.197003 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.196994    4645 cpu_manager.go:414] "containerMap: removing container" podUID="e7198524-cd1a-4038-83c6-55f9a5391970" containerName="cleanup"
Oct 31 19:45:36.197067 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.197059    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7198524-cd1a-4038-83c6-55f9a5391970" containerName="cleanup"
Oct 31 19:45:36.197155 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.197146    4645 cpu_manager.go:414] "containerMap: removing container" podUID="220956a9-b7c5-48b8-a598-61326c85691a" containerName="cleanup"
Oct 31 19:45:36.197218 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.197209    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="220956a9-b7c5-48b8-a598-61326c85691a" containerName="cleanup"
Oct 31 19:45:36.197529 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.197515    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="220956a9-b7c5-48b8-a598-61326c85691a" containerName="cleanup"
Oct 31 19:45:36.197610 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.197602    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6f57036-6c45-4def-a9c1-0132ab845b3e" containerName="cleanup"
Oct 31 19:45:36.197690 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.197682    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e3605c3-ec3b-4fb7-be2a-40fe32c5adea" containerName="cleanup"
Oct 31 19:45:36.197756 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.197748    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e3605c3-ec3b-4fb7-be2a-40fe32c5adea" containerName="cleanup"
Oct 31 19:45:36.197827 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.197819    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="769ee829-1015-45b1-9dd7-d9cfe372fa9c" containerName="cleanup"
Oct 31 19:45:36.197898 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.197890    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="615544a2-1b13-4bce-aeb6-7b3b32e6a509" containerName="cleanup"
Oct 31 19:45:36.197963 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.197956    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="53b785a3-a308-43ec-92b9-ccd7d6a8bcad" containerName="cleanup"
Oct 31 19:45:36.198032 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.198024    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="769ee829-1015-45b1-9dd7-d9cfe372fa9c" containerName="cleanup"
Oct 31 19:45:36.198305 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.198293    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="53b785a3-a308-43ec-92b9-ccd7d6a8bcad" containerName="cleanup"
Oct 31 19:45:36.198414 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.198403    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="3961f45f-4b0d-4880-b8d8-e55798d4abe8" containerName="cleanup"
Oct 31 19:45:36.198485 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.198477    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6f57036-6c45-4def-a9c1-0132ab845b3e" containerName="cleanup"
Oct 31 19:45:36.198548 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.198540    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="57977c65-f0c2-46d1-a05c-20bd8ed579c3" containerName="cleanup"
Oct 31 19:45:36.198622 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.198614    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="615544a2-1b13-4bce-aeb6-7b3b32e6a509" containerName="cleanup"
Oct 31 19:45:36.198688 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.198680    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="3961f45f-4b0d-4880-b8d8-e55798d4abe8" containerName="cleanup"
Oct 31 19:45:36.198750 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.198742    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7198524-cd1a-4038-83c6-55f9a5391970" containerName="cleanup"
Oct 31 19:45:36.198820 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.198812    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6f57036-6c45-4def-a9c1-0132ab845b3e" containerName="cleanup"
Oct 31 19:45:36.198893 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.198885    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="57977c65-f0c2-46d1-a05c-20bd8ed579c3" containerName="cleanup"
Oct 31 19:45:36.198963 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.198954    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7198524-cd1a-4038-83c6-55f9a5391970" containerName="cleanup"
Oct 31 19:45:36.199033 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.199025    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="615544a2-1b13-4bce-aeb6-7b3b32e6a509" containerName="cleanup"
Oct 31 19:45:36.199104 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.199096    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ae53805-07ba-4f39-bcbf-f19b67f0ad16" containerName="cleanup"
Oct 31 19:45:36.199169 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.199161    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="57977c65-f0c2-46d1-a05c-20bd8ed579c3" containerName="cleanup"
Oct 31 19:45:36.199241 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.199232    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7198524-cd1a-4038-83c6-55f9a5391970" containerName="cleanup"
Oct 31 19:45:36.199319 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.199311    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="220956a9-b7c5-48b8-a598-61326c85691a" containerName="cleanup"
Oct 31 19:45:36.212193 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.212170    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v"
Oct 31 19:45:36.229189 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.226176    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v"]
Oct 31 19:45:36.279859 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:45:36.279828    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 19:45:36.326162 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.326124    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7198524-cd1a-4038-83c6-55f9a5391970-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7198524-cd1a-4038-83c6-55f9a5391970-maas-api-service-ca\") pod \"e7198524-cd1a-4038-83c6-55f9a5391970\" (UID: \"e7198524-cd1a-4038-83c6-55f9a5391970\") "
Oct 31 19:45:36.326385 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.326245    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7198524-cd1a-4038-83c6-55f9a5391970-kube-api-access-gknw9\" (UniqueName: \"kubernetes.io/projected/e7198524-cd1a-4038-83c6-55f9a5391970-kube-api-access-gknw9\") pod \"e7198524-cd1a-4038-83c6-55f9a5391970\" (UID: \"e7198524-cd1a-4038-83c6-55f9a5391970\") "
Oct 31 19:45:36.326785 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.326758    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t79d\" (UniqueName: \"kubernetes.io/projected/14ca45a7-b42e-45d3-91bf-3698df3303c7-kube-api-access-8t79d\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v\" (UID: \"14ca45a7-b42e-45d3-91bf-3698df3303c7\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v"
Oct 31 19:45:36.326860 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.326841    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14ca45a7-b42e-45d3-91bf-3698df3303c7-istiod-ca-cert\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v\" (UID: \"14ca45a7-b42e-45d3-91bf-3698df3303c7\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v"
Oct 31 19:45:36.326937 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.326908    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7198524-cd1a-4038-83c6-55f9a5391970-maas-api-service-ca" pod "e7198524-cd1a-4038-83c6-55f9a5391970" (UID: "e7198524-cd1a-4038-83c6-55f9a5391970"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:45:36.327087 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.327057    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14ca45a7-b42e-45d3-91bf-3698df3303c7-istio-podinfo\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v\" (UID: \"14ca45a7-b42e-45d3-91bf-3698df3303c7\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v"
Oct 31 19:45:36.327181 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.327161    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14ca45a7-b42e-45d3-91bf-3698df3303c7-istio-envoy\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v\" (UID: \"14ca45a7-b42e-45d3-91bf-3698df3303c7\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v"
Oct 31 19:45:36.327255 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.327237    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14ca45a7-b42e-45d3-91bf-3698df3303c7-istio-token\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v\" (UID: \"14ca45a7-b42e-45d3-91bf-3698df3303c7\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v"
Oct 31 19:45:36.327497 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.327474    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14ca45a7-b42e-45d3-91bf-3698df3303c7-workload-socket\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v\" (UID: \"14ca45a7-b42e-45d3-91bf-3698df3303c7\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v"
Oct 31 19:45:36.327590 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.327533    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14ca45a7-b42e-45d3-91bf-3698df3303c7-credential-socket\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v\" (UID: \"14ca45a7-b42e-45d3-91bf-3698df3303c7\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v"
Oct 31 19:45:36.327590 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.327568    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14ca45a7-b42e-45d3-91bf-3698df3303c7-workload-certs\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v\" (UID: \"14ca45a7-b42e-45d3-91bf-3698df3303c7\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v"
Oct 31 19:45:36.327701 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.327629    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14ca45a7-b42e-45d3-91bf-3698df3303c7-istio-data\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v\" (UID: \"14ca45a7-b42e-45d3-91bf-3698df3303c7\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v"
Oct 31 19:45:36.327781 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.327767    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7198524-cd1a-4038-83c6-55f9a5391970-maas-api-service-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:36.328693 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.328670    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7198524-cd1a-4038-83c6-55f9a5391970-kube-api-access-gknw9" pod "e7198524-cd1a-4038-83c6-55f9a5391970" (UID: "e7198524-cd1a-4038-83c6-55f9a5391970"). InnerVolumeSpecName "kube-api-access-gknw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:45:36.429409 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.429369    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14ca45a7-b42e-45d3-91bf-3698df3303c7-istio-token\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v\" (UID: \"14ca45a7-b42e-45d3-91bf-3698df3303c7\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v"
Oct 31 19:45:36.429807 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.429773    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14ca45a7-b42e-45d3-91bf-3698df3303c7-workload-socket\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v\" (UID: \"14ca45a7-b42e-45d3-91bf-3698df3303c7\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v"
Oct 31 19:45:36.429942 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.429923    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14ca45a7-b42e-45d3-91bf-3698df3303c7-credential-socket\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v\" (UID: \"14ca45a7-b42e-45d3-91bf-3698df3303c7\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v"
Oct 31 19:45:36.430015 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.429956    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14ca45a7-b42e-45d3-91bf-3698df3303c7-workload-certs\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v\" (UID: \"14ca45a7-b42e-45d3-91bf-3698df3303c7\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v"
Oct 31 19:45:36.430083 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.430013    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14ca45a7-b42e-45d3-91bf-3698df3303c7-istio-data\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v\" (UID: \"14ca45a7-b42e-45d3-91bf-3698df3303c7\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v"
Oct 31 19:45:36.430279 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.430231    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14ca45a7-b42e-45d3-91bf-3698df3303c7-workload-socket\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v\" (UID: \"14ca45a7-b42e-45d3-91bf-3698df3303c7\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v"
Oct 31 19:45:36.430437 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.430239    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t79d\" (UniqueName: \"kubernetes.io/projected/14ca45a7-b42e-45d3-91bf-3698df3303c7-kube-api-access-8t79d\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v\" (UID: \"14ca45a7-b42e-45d3-91bf-3698df3303c7\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v"
Oct 31 19:45:36.430704 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.430668    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14ca45a7-b42e-45d3-91bf-3698df3303c7-istiod-ca-cert\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v\" (UID: \"14ca45a7-b42e-45d3-91bf-3698df3303c7\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v"
Oct 31 19:45:36.430945 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.430913    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14ca45a7-b42e-45d3-91bf-3698df3303c7-istio-data\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v\" (UID: \"14ca45a7-b42e-45d3-91bf-3698df3303c7\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v"
Oct 31 19:45:36.431022 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.430747    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14ca45a7-b42e-45d3-91bf-3698df3303c7-workload-certs\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v\" (UID: \"14ca45a7-b42e-45d3-91bf-3698df3303c7\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v"
Oct 31 19:45:36.431121 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.430716    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14ca45a7-b42e-45d3-91bf-3698df3303c7-credential-socket\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v\" (UID: \"14ca45a7-b42e-45d3-91bf-3698df3303c7\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v"
Oct 31 19:45:36.431362 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.431292    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14ca45a7-b42e-45d3-91bf-3698df3303c7-istio-podinfo\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v\" (UID: \"14ca45a7-b42e-45d3-91bf-3698df3303c7\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v"
Oct 31 19:45:36.431695 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.431670    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14ca45a7-b42e-45d3-91bf-3698df3303c7-istiod-ca-cert\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v\" (UID: \"14ca45a7-b42e-45d3-91bf-3698df3303c7\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v"
Oct 31 19:45:36.431992 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.431676    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14ca45a7-b42e-45d3-91bf-3698df3303c7-istio-envoy\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v\" (UID: \"14ca45a7-b42e-45d3-91bf-3698df3303c7\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v"
Oct 31 19:45:36.432130 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.432113    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gknw9\" (UniqueName: \"kubernetes.io/projected/e7198524-cd1a-4038-83c6-55f9a5391970-kube-api-access-gknw9\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:36.440679 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.440651    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14ca45a7-b42e-45d3-91bf-3698df3303c7-istio-envoy\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v\" (UID: \"14ca45a7-b42e-45d3-91bf-3698df3303c7\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v"
Oct 31 19:45:36.441879 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.441853    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14ca45a7-b42e-45d3-91bf-3698df3303c7-istio-podinfo\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v\" (UID: \"14ca45a7-b42e-45d3-91bf-3698df3303c7\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v"
Oct 31 19:45:36.444714 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.444649    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t79d\" (UniqueName: \"kubernetes.io/projected/14ca45a7-b42e-45d3-91bf-3698df3303c7-kube-api-access-8t79d\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v\" (UID: \"14ca45a7-b42e-45d3-91bf-3698df3303c7\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v"
Oct 31 19:45:36.447626 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.447603    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14ca45a7-b42e-45d3-91bf-3698df3303c7-istio-token\") pod \"e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v\" (UID: \"14ca45a7-b42e-45d3-91bf-3698df3303c7\") " pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v"
Oct 31 19:45:36.549421 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.549361    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v"
Oct 31 19:45:36.868757 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:45:36.868719    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14ca45a7_b42e_45d3_91bf_3698df3303c7.slice/crio-1be67ee83d8301851b4f7d9b4fe126f44da3321c58625d56ecd12b01da4db5d1 WatchSource:0}: Error finding container 1be67ee83d8301851b4f7d9b4fe126f44da3321c58625d56ecd12b01da4db5d1: Status 404 returned error can't find the container with id 1be67ee83d8301851b4f7d9b4fe126f44da3321c58625d56ecd12b01da4db5d1
Oct 31 19:45:36.869785 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.869676    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v"]
Oct 31 19:45:36.874961 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.873205    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:45:36.874961 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.873306    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:45:36.874961 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.873371    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:45:36.901784 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:36.901748    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:37.006359 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:37.003198    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891265-r6vvt" event={"ID":"e7198524-cd1a-4038-83c6-55f9a5391970","Type":"ContainerDied","Data":"0fc69bfd6fba82b0e962a2ed3e1d978e2a12b839dd4bc1510ad0ca65e14d648a"}
Oct 31 19:45:37.006359 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:37.003412    4645 scope.go:122] "RemoveContainer" containerID="0b7a923b65422c7da00f1fbd033b12779326e8d3f858ea417e007d2e26815aec"
Oct 31 19:45:37.006359 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:37.003590    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891265-r6vvt"
Oct 31 19:45:37.027430 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:37.025653    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-88eaf9-29891265-zttxn"]
Oct 31 19:45:37.030010 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:37.029974    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-785d26-29891265-8cn6g" event={"ID":"2e3605c3-ec3b-4fb7-be2a-40fe32c5adea","Type":"ContainerDied","Data":"8ea0aa59bcc8af817bd845b6f8a2ca428d43cb3e13eb74e71d5e47a56035bb59"}
Oct 31 19:45:37.030221 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:37.030201    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-785d26-29891265-8cn6g"
Oct 31 19:45:37.037404 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:37.036236    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1e8cf0-29891265-mbhwx" event={"ID":"57977c65-f0c2-46d1-a05c-20bd8ed579c3","Type":"ContainerDied","Data":"e332333b2d18f171988ecc34257cfe03302668981cd9e783e345ab283e898628"}
Oct 31 19:45:37.037404 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:37.036473    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1e8cf0-29891265-mbhwx"
Oct 31 19:45:37.051677 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:37.051654    4645 scope.go:122] "RemoveContainer" containerID="90f2a2574676bc448598d57f47037ea167d5a844b53983b4f3e383d6bc046900"
Oct 31 19:45:37.053581 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:37.053544    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9" event={"ID":"d734b787-3a19-4da6-8b9b-93aa92eaa6a5","Type":"ContainerStarted","Data":"c710846a10f377410aaa0b72fde1570797cc359b3750714bf0cd93cbd7686fcf"}
Oct 31 19:45:37.053693 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:37.053610    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9" event={"ID":"d734b787-3a19-4da6-8b9b-93aa92eaa6a5","Type":"ContainerStarted","Data":"fd20c7d607dbd9b3ceaf78d3911197f6dcdf293e42ae083b14225a77ca84c891"}
Oct 31 19:45:37.056499 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:37.056154    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v" event={"ID":"14ca45a7-b42e-45d3-91bf-3698df3303c7","Type":"ContainerStarted","Data":"1be67ee83d8301851b4f7d9b4fe126f44da3321c58625d56ecd12b01da4db5d1"}
Oct 31 19:45:37.059790 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:37.059718    4645 generic.go:356] "Generic (PLEG): container finished" podID="25370d12-c485-47b9-986a-a775c3fbd2b7" containerID="7bad4130f50ad87d5e69f861980abe452a1840e961b6061d9829ea9c23286f39" exitCode=60
Oct 31 19:45:37.059974 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:37.059794    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-88eaf9-29891265-zttxn" event={"ID":"25370d12-c485-47b9-986a-a775c3fbd2b7","Type":"ContainerDied","Data":"7bad4130f50ad87d5e69f861980abe452a1840e961b6061d9829ea9c23286f39"}
Oct 31 19:45:37.065709 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:37.065682    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891265-r6vvt"]
Oct 31 19:45:37.072619 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:37.072591    4645 scope.go:122] "RemoveContainer" containerID="4c862f620b30d7bb75ae8bed20dd6c62c02ed161fb5a42fe4bfedda138f097aa"
Oct 31 19:45:37.078231 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:37.078203    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-8730d5-29891265-r6vvt"]
Oct 31 19:45:37.093634 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:37.093606    4645 scope.go:122] "RemoveContainer" containerID="6e0fdf219940b19304fa4ae6614e0943e33083230d93614ed9bb893338915a19"
Oct 31 19:45:37.102619 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:37.102596    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-785d26-29891265-8cn6g"]
Oct 31 19:45:37.130290 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:37.129219    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-785d26-29891265-8cn6g"]
Oct 31 19:45:37.179547 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:37.173228    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1e8cf0-29891265-mbhwx"]
Oct 31 19:45:37.199030 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:37.196615    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1e8cf0-29891265-mbhwx"]
Oct 31 19:45:37.217231 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:37.217144    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9" podStartSLOduration=16.21712711 podStartE2EDuration="16.21712711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:45:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:45:37.201858699 +0000 UTC m=+1567.775092830" watchObservedRunningTime="2026-10-31 19:45:37.21712711 +0000 UTC m=+1567.790361240"
Oct 31 19:45:37.385400 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:37.381600    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-88eaf9-29891265-zttxn"
Oct 31 19:45:37.577570 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:37.577535    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25370d12-c485-47b9-986a-a775c3fbd2b7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25370d12-c485-47b9-986a-a775c3fbd2b7-maas-api-service-ca\") pod \"25370d12-c485-47b9-986a-a775c3fbd2b7\" (UID: \"25370d12-c485-47b9-986a-a775c3fbd2b7\") "
Oct 31 19:45:37.577745 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:37.577666    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25370d12-c485-47b9-986a-a775c3fbd2b7-kube-api-access-g5j5s\" (UniqueName: \"kubernetes.io/projected/25370d12-c485-47b9-986a-a775c3fbd2b7-kube-api-access-g5j5s\") pod \"25370d12-c485-47b9-986a-a775c3fbd2b7\" (UID: \"25370d12-c485-47b9-986a-a775c3fbd2b7\") "
Oct 31 19:45:37.578200 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:37.578170    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25370d12-c485-47b9-986a-a775c3fbd2b7-maas-api-service-ca" pod "25370d12-c485-47b9-986a-a775c3fbd2b7" (UID: "25370d12-c485-47b9-986a-a775c3fbd2b7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:45:37.578793 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:37.578761    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25370d12-c485-47b9-986a-a775c3fbd2b7-maas-api-service-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:37.580293 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:37.580264    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25370d12-c485-47b9-986a-a775c3fbd2b7-kube-api-access-g5j5s" pod "25370d12-c485-47b9-986a-a775c3fbd2b7" (UID: "25370d12-c485-47b9-986a-a775c3fbd2b7"). InnerVolumeSpecName "kube-api-access-g5j5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:45:37.626154 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:37.626117    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e3605c3-ec3b-4fb7-be2a-40fe32c5adea" path="/var/lib/kubelet/pods/2e3605c3-ec3b-4fb7-be2a-40fe32c5adea/volumes"
Oct 31 19:45:37.626750 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:37.626728    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57977c65-f0c2-46d1-a05c-20bd8ed579c3" path="/var/lib/kubelet/pods/57977c65-f0c2-46d1-a05c-20bd8ed579c3/volumes"
Oct 31 19:45:37.627235 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:37.627212    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="615544a2-1b13-4bce-aeb6-7b3b32e6a509" path="/var/lib/kubelet/pods/615544a2-1b13-4bce-aeb6-7b3b32e6a509/volumes"
Oct 31 19:45:37.628358 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:37.628304    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="769ee829-1015-45b1-9dd7-d9cfe372fa9c" path="/var/lib/kubelet/pods/769ee829-1015-45b1-9dd7-d9cfe372fa9c/volumes"
Oct 31 19:45:37.628977 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:37.628953    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6f57036-6c45-4def-a9c1-0132ab845b3e" path="/var/lib/kubelet/pods/e6f57036-6c45-4def-a9c1-0132ab845b3e/volumes"
Oct 31 19:45:37.629619 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:37.629596    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7198524-cd1a-4038-83c6-55f9a5391970" path="/var/lib/kubelet/pods/e7198524-cd1a-4038-83c6-55f9a5391970/volumes"
Oct 31 19:45:37.681586 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:37.681541    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5j5s\" (UniqueName: \"kubernetes.io/projected/25370d12-c485-47b9-986a-a775c3fbd2b7-kube-api-access-g5j5s\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:45:37.876352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:37.859698    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9"
Oct 31 19:45:37.889562 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:37.887209    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9"
Oct 31 19:45:37.988304 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:37.988184    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:38.130491 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:38.130081    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-88eaf9-29891265-zttxn" event={"ID":"25370d12-c485-47b9-986a-a775c3fbd2b7","Type":"ContainerDied","Data":"1d03dfce658b08348f7cad84a0ad834f2cf7c7f1b11f6b5ec2a0e2d8b83ef824"}
Oct 31 19:45:38.130491 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:38.130159    4645 scope.go:122] "RemoveContainer" containerID="7bad4130f50ad87d5e69f861980abe452a1840e961b6061d9829ea9c23286f39"
Oct 31 19:45:38.137084 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:38.130834    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-88eaf9-29891265-zttxn"
Oct 31 19:45:38.144251 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:38.142583    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v" event={"ID":"14ca45a7-b42e-45d3-91bf-3698df3303c7","Type":"ContainerStarted","Data":"7e8510c621dbcba9d39a3f46ff71e3f92a47bffdd1af518ef26d926da7579781"}
Oct 31 19:45:38.144251 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:38.142869    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9"
Oct 31 19:45:38.144251 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:38.144113    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9"
Oct 31 19:45:38.189648 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:38.189614    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-88eaf9-29891265-zttxn"]
Oct 31 19:45:38.194411 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:38.194386    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-88eaf9-29891265-zttxn"]
Oct 31 19:45:38.243361 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:38.243265    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v" podStartSLOduration=36.243244772 podStartE2EDuration="36.243244772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:45:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:45:38.2289186 +0000 UTC m=+1568.802152730" watchObservedRunningTime="2026-10-31 19:45:38.243244772 +0000 UTC m=+1568.816478902"
Oct 31 19:45:38.550038 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:38.549999    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v"
Oct 31 19:45:38.551518 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:38.551488    4645 patch_prober.go:29] interesting pod/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Oct 31 19:45:38.551656 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:38.551540    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v" podUID="14ca45a7-b42e-45d3-91bf-3698df3303c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Oct 31 19:45:39.042119 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:39.042080    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:39.517786 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:39.517758    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:39.550177 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:39.550134    4645 patch_prober.go:29] interesting pod/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Oct 31 19:45:39.550374 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:39.550207    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v" podUID="14ca45a7-b42e-45d3-91bf-3698df3303c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Oct 31 19:45:39.623127 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:39.623092    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25370d12-c485-47b9-986a-a775c3fbd2b7" path="/var/lib/kubelet/pods/25370d12-c485-47b9-986a-a775c3fbd2b7/volumes"
Oct 31 19:45:39.643167 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:39.643130    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:40.145998 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:40.145964    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:40.549756 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:40.549721    4645 patch_prober.go:29] interesting pod/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Oct 31 19:45:40.549954 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:40.549786    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v" podUID="14ca45a7-b42e-45d3-91bf-3698df3303c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Oct 31 19:45:40.789354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:40.789303    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:40.911696 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:40.911623    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:41.194429 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:41.183669    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:41.553651 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:41.553555    4645 patch_prober.go:29] interesting pod/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Oct 31 19:45:41.553651 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:41.553622    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v" podUID="14ca45a7-b42e-45d3-91bf-3698df3303c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Oct 31 19:45:42.082353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:42.082202    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:42.550263 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:42.550228    4645 patch_prober.go:29] interesting pod/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Oct 31 19:45:42.550659 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:42.550296    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v" podUID="14ca45a7-b42e-45d3-91bf-3698df3303c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Oct 31 19:45:42.879413 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:42.878466    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:43.550386 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:43.550350    4645 patch_prober.go:29] interesting pod/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body=
Oct 31 19:45:43.551006 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:43.550411    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v" podUID="14ca45a7-b42e-45d3-91bf-3698df3303c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused"
Oct 31 19:45:43.791212 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:43.791185    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:44.558348 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:44.557819    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v"
Oct 31 19:45:44.562340 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:44.559457    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v"
Oct 31 19:45:44.562340 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:44.561148    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v"
Oct 31 19:45:44.570544 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:44.570493    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:45.356648 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:45.356610    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:46.072542 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:46.072506    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:46.802677 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:46.802634    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:47.602212 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:47.602179    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:48.342755 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:48.342723    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:49.142104 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:49.142067    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:50.008906 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:50.008871    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:50.957770 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:50.957732    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:51.702709 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:51.702675    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:52.858463 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:52.858427    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:53.744421 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:53.744383    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:54.635135 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:54.635102    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:54.746907 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:54.746870    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:55.555026 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:55.554987    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:56.138675 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:56.138638    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:56.821287 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:56.821240    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:57.667743 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:57.667706    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:58.446478 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:58.446442    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:45:59.352968 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:45:59.352937    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:00.239182 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:00.239140    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:01.170278 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:01.170244    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:02.159630 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:02.159597    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:03.231579 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:03.231542    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:04.332475 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:04.332434    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:04.536436 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:04.534258    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:05.510892 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:05.510858    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:06.248090 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:46:06.248051    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 19:46:06.302953 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:06.302922    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:07.125781 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:07.125749    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:07.990824 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:07.990789    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:08.730778 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:08.730750    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:09.478542 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:09.478509    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:10.401197 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:10.401159    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:11.117061 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:11.117016    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:11.896055 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:11.896019    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-d8c398/test-model-d8c398-kserve-57476967b9-v6ld4"]
Oct 31 19:46:11.897309 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:11.897287    4645 cpu_manager.go:414] "containerMap: removing container" podUID="25370d12-c485-47b9-986a-a775c3fbd2b7" containerName="cleanup"
Oct 31 19:46:11.897407 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:11.897314    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25370d12-c485-47b9-986a-a775c3fbd2b7" containerName="cleanup"
Oct 31 19:46:11.897407 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:11.897377    4645 cpu_manager.go:414] "containerMap: removing container" podUID="220956a9-b7c5-48b8-a598-61326c85691a" containerName="cleanup"
Oct 31 19:46:11.897407 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:11.897386    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="220956a9-b7c5-48b8-a598-61326c85691a" containerName="cleanup"
Oct 31 19:46:11.897407 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:11.897397    4645 cpu_manager.go:414] "containerMap: removing container" podUID="e6f57036-6c45-4def-a9c1-0132ab845b3e" containerName="cleanup"
Oct 31 19:46:11.897407 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:11.897405    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6f57036-6c45-4def-a9c1-0132ab845b3e" containerName="cleanup"
Oct 31 19:46:11.897575 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:11.897445    4645 cpu_manager.go:414] "containerMap: removing container" podUID="25370d12-c485-47b9-986a-a775c3fbd2b7" containerName="cleanup"
Oct 31 19:46:11.897575 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:11.897452    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25370d12-c485-47b9-986a-a775c3fbd2b7" containerName="cleanup"
Oct 31 19:46:11.897575 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:11.897470    4645 cpu_manager.go:414] "containerMap: removing container" podUID="615544a2-1b13-4bce-aeb6-7b3b32e6a509" containerName="cleanup"
Oct 31 19:46:11.897575 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:11.897478    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="615544a2-1b13-4bce-aeb6-7b3b32e6a509" containerName="cleanup"
Oct 31 19:46:11.897575 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:11.897497    4645 cpu_manager.go:414] "containerMap: removing container" podUID="769ee829-1015-45b1-9dd7-d9cfe372fa9c" containerName="cleanup"
Oct 31 19:46:11.897575 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:11.897504    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="769ee829-1015-45b1-9dd7-d9cfe372fa9c" containerName="cleanup"
Oct 31 19:46:11.897575 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:11.897514    4645 cpu_manager.go:414] "containerMap: removing container" podUID="53b785a3-a308-43ec-92b9-ccd7d6a8bcad" containerName="cleanup"
Oct 31 19:46:11.897575 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:11.897521    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53b785a3-a308-43ec-92b9-ccd7d6a8bcad" containerName="cleanup"
Oct 31 19:46:11.897575 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:11.897550    4645 cpu_manager.go:414] "containerMap: removing container" podUID="3961f45f-4b0d-4880-b8d8-e55798d4abe8" containerName="cleanup"
Oct 31 19:46:11.897575 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:11.897558    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3961f45f-4b0d-4880-b8d8-e55798d4abe8" containerName="cleanup"
Oct 31 19:46:11.897877 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:11.897748    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="25370d12-c485-47b9-986a-a775c3fbd2b7" containerName="cleanup"
Oct 31 19:46:11.897877 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:11.897765    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="220956a9-b7c5-48b8-a598-61326c85691a" containerName="cleanup"
Oct 31 19:46:11.897877 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:11.897776    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="25370d12-c485-47b9-986a-a775c3fbd2b7" containerName="cleanup"
Oct 31 19:46:11.897877 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:11.897791    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e3605c3-ec3b-4fb7-be2a-40fe32c5adea" containerName="cleanup"
Oct 31 19:46:11.897877 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:11.897805    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="769ee829-1015-45b1-9dd7-d9cfe372fa9c" containerName="cleanup"
Oct 31 19:46:11.897877 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:11.897817    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="53b785a3-a308-43ec-92b9-ccd7d6a8bcad" containerName="cleanup"
Oct 31 19:46:11.897877 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:11.897829    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="3961f45f-4b0d-4880-b8d8-e55798d4abe8" containerName="cleanup"
Oct 31 19:46:11.904111 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:11.904086    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d8c398/test-model-d8c398-kserve-57476967b9-v6ld4"
Oct 31 19:46:11.921364 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:11.921339    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-d8c398\"/\"openshift-service-ca.crt\""
Oct 31 19:46:11.921491 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:11.921443    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-d8c398\"/\"default-dockercfg-bs2b5\""
Oct 31 19:46:11.921590 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:11.921575    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-d8c398\"/\"test-model-d8c398-kserve-self-signed-certs\""
Oct 31 19:46:11.921952 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:11.921933    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-d8c398\"/\"kube-root-ca.crt\""
Oct 31 19:46:11.925953 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:11.925931    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-d8c398/test-model-d8c398-kserve-57476967b9-v6ld4"]
Oct 31 19:46:12.005953 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:12.005923    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15c43438-d0c9-41d4-92dd-788371b30f8f-tmp-dir\") pod \"test-model-d8c398-kserve-57476967b9-v6ld4\" (UID: \"15c43438-d0c9-41d4-92dd-788371b30f8f\") " pod="ai-tenant-e2e-inf-b-d8c398/test-model-d8c398-kserve-57476967b9-v6ld4"
Oct 31 19:46:12.006117 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:12.006014    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15c43438-d0c9-41d4-92dd-788371b30f8f-tls-certs\") pod \"test-model-d8c398-kserve-57476967b9-v6ld4\" (UID: \"15c43438-d0c9-41d4-92dd-788371b30f8f\") " pod="ai-tenant-e2e-inf-b-d8c398/test-model-d8c398-kserve-57476967b9-v6ld4"
Oct 31 19:46:12.006269 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:12.006253    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15c43438-d0c9-41d4-92dd-788371b30f8f-home\") pod \"test-model-d8c398-kserve-57476967b9-v6ld4\" (UID: \"15c43438-d0c9-41d4-92dd-788371b30f8f\") " pod="ai-tenant-e2e-inf-b-d8c398/test-model-d8c398-kserve-57476967b9-v6ld4"
Oct 31 19:46:12.006310 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:12.006284    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15c43438-d0c9-41d4-92dd-788371b30f8f-dshm\") pod \"test-model-d8c398-kserve-57476967b9-v6ld4\" (UID: \"15c43438-d0c9-41d4-92dd-788371b30f8f\") " pod="ai-tenant-e2e-inf-b-d8c398/test-model-d8c398-kserve-57476967b9-v6ld4"
Oct 31 19:46:12.006442 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:12.006426    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4zgl\" (UniqueName: \"kubernetes.io/projected/15c43438-d0c9-41d4-92dd-788371b30f8f-kube-api-access-q4zgl\") pod \"test-model-d8c398-kserve-57476967b9-v6ld4\" (UID: \"15c43438-d0c9-41d4-92dd-788371b30f8f\") " pod="ai-tenant-e2e-inf-b-d8c398/test-model-d8c398-kserve-57476967b9-v6ld4"
Oct 31 19:46:12.006482 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:12.006459    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15c43438-d0c9-41d4-92dd-788371b30f8f-model-cache\") pod \"test-model-d8c398-kserve-57476967b9-v6ld4\" (UID: \"15c43438-d0c9-41d4-92dd-788371b30f8f\") " pod="ai-tenant-e2e-inf-b-d8c398/test-model-d8c398-kserve-57476967b9-v6ld4"
Oct 31 19:46:12.036937 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:12.036906    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:12.107907 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:12.107872    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15c43438-d0c9-41d4-92dd-788371b30f8f-home\") pod \"test-model-d8c398-kserve-57476967b9-v6ld4\" (UID: \"15c43438-d0c9-41d4-92dd-788371b30f8f\") " pod="ai-tenant-e2e-inf-b-d8c398/test-model-d8c398-kserve-57476967b9-v6ld4"
Oct 31 19:46:12.108107 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:12.107918    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15c43438-d0c9-41d4-92dd-788371b30f8f-dshm\") pod \"test-model-d8c398-kserve-57476967b9-v6ld4\" (UID: \"15c43438-d0c9-41d4-92dd-788371b30f8f\") " pod="ai-tenant-e2e-inf-b-d8c398/test-model-d8c398-kserve-57476967b9-v6ld4"
Oct 31 19:46:12.108107 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:12.107972    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4zgl\" (UniqueName: \"kubernetes.io/projected/15c43438-d0c9-41d4-92dd-788371b30f8f-kube-api-access-q4zgl\") pod \"test-model-d8c398-kserve-57476967b9-v6ld4\" (UID: \"15c43438-d0c9-41d4-92dd-788371b30f8f\") " pod="ai-tenant-e2e-inf-b-d8c398/test-model-d8c398-kserve-57476967b9-v6ld4"
Oct 31 19:46:12.108107 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:12.107995    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15c43438-d0c9-41d4-92dd-788371b30f8f-model-cache\") pod \"test-model-d8c398-kserve-57476967b9-v6ld4\" (UID: \"15c43438-d0c9-41d4-92dd-788371b30f8f\") " pod="ai-tenant-e2e-inf-b-d8c398/test-model-d8c398-kserve-57476967b9-v6ld4"
Oct 31 19:46:12.108303 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:12.108106    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15c43438-d0c9-41d4-92dd-788371b30f8f-tmp-dir\") pod \"test-model-d8c398-kserve-57476967b9-v6ld4\" (UID: \"15c43438-d0c9-41d4-92dd-788371b30f8f\") " pod="ai-tenant-e2e-inf-b-d8c398/test-model-d8c398-kserve-57476967b9-v6ld4"
Oct 31 19:46:12.108303 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:12.108153    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15c43438-d0c9-41d4-92dd-788371b30f8f-tls-certs\") pod \"test-model-d8c398-kserve-57476967b9-v6ld4\" (UID: \"15c43438-d0c9-41d4-92dd-788371b30f8f\") " pod="ai-tenant-e2e-inf-b-d8c398/test-model-d8c398-kserve-57476967b9-v6ld4"
Oct 31 19:46:12.108435 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:12.108353    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15c43438-d0c9-41d4-92dd-788371b30f8f-home\") pod \"test-model-d8c398-kserve-57476967b9-v6ld4\" (UID: \"15c43438-d0c9-41d4-92dd-788371b30f8f\") " pod="ai-tenant-e2e-inf-b-d8c398/test-model-d8c398-kserve-57476967b9-v6ld4"
Oct 31 19:46:12.108485 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:12.108459    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15c43438-d0c9-41d4-92dd-788371b30f8f-model-cache\") pod \"test-model-d8c398-kserve-57476967b9-v6ld4\" (UID: \"15c43438-d0c9-41d4-92dd-788371b30f8f\") " pod="ai-tenant-e2e-inf-b-d8c398/test-model-d8c398-kserve-57476967b9-v6ld4"
Oct 31 19:46:12.108672 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:12.108654    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15c43438-d0c9-41d4-92dd-788371b30f8f-tmp-dir\") pod \"test-model-d8c398-kserve-57476967b9-v6ld4\" (UID: \"15c43438-d0c9-41d4-92dd-788371b30f8f\") " pod="ai-tenant-e2e-inf-b-d8c398/test-model-d8c398-kserve-57476967b9-v6ld4"
Oct 31 19:46:12.110344 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:12.110312    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15c43438-d0c9-41d4-92dd-788371b30f8f-dshm\") pod \"test-model-d8c398-kserve-57476967b9-v6ld4\" (UID: \"15c43438-d0c9-41d4-92dd-788371b30f8f\") " pod="ai-tenant-e2e-inf-b-d8c398/test-model-d8c398-kserve-57476967b9-v6ld4"
Oct 31 19:46:12.110691 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:12.110671    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15c43438-d0c9-41d4-92dd-788371b30f8f-tls-certs\") pod \"test-model-d8c398-kserve-57476967b9-v6ld4\" (UID: \"15c43438-d0c9-41d4-92dd-788371b30f8f\") " pod="ai-tenant-e2e-inf-b-d8c398/test-model-d8c398-kserve-57476967b9-v6ld4"
Oct 31 19:46:12.119657 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:12.119636    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4zgl\" (UniqueName: \"kubernetes.io/projected/15c43438-d0c9-41d4-92dd-788371b30f8f-kube-api-access-q4zgl\") pod \"test-model-d8c398-kserve-57476967b9-v6ld4\" (UID: \"15c43438-d0c9-41d4-92dd-788371b30f8f\") " pod="ai-tenant-e2e-inf-b-d8c398/test-model-d8c398-kserve-57476967b9-v6ld4"
Oct 31 19:46:12.227359 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:12.224804    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d8c398/test-model-d8c398-kserve-57476967b9-v6ld4"
Oct 31 19:46:12.637223 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:46:12.637192    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod15c43438_d0c9_41d4_92dd_788371b30f8f.slice/crio-5044e280505a6e32d2b513ac8b484fc1fbf636818d36861e8c1d69cd0b581e06 WatchSource:0}: Error finding container 5044e280505a6e32d2b513ac8b484fc1fbf636818d36861e8c1d69cd0b581e06: Status 404 returned error can't find the container with id 5044e280505a6e32d2b513ac8b484fc1fbf636818d36861e8c1d69cd0b581e06
Oct 31 19:46:12.637865 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:12.637837    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-d8c398/test-model-d8c398-kserve-57476967b9-v6ld4"]
Oct 31 19:46:12.902265 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:12.902163    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d8c398/test-model-d8c398-kserve-57476967b9-v6ld4" event={"ID":"15c43438-d0c9-41d4-92dd-788371b30f8f","Type":"ContainerStarted","Data":"99755302a08703e09405870dc95679697e366ff6bb8c71587119701327f23d70"}
Oct 31 19:46:12.902265 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:12.902216    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d8c398/test-model-d8c398-kserve-57476967b9-v6ld4" event={"ID":"15c43438-d0c9-41d4-92dd-788371b30f8f","Type":"ContainerStarted","Data":"5044e280505a6e32d2b513ac8b484fc1fbf636818d36861e8c1d69cd0b581e06"}
Oct 31 19:46:12.928044 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:12.927991    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-d8c398/test-model-d8c398-kserve-57476967b9-v6ld4" podStartSLOduration=1.927977314 podStartE2EDuration="1.927977314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:46:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:46:12.921777079 +0000 UTC m=+1603.495011208" watchObservedRunningTime="2026-10-31 19:46:12.927977314 +0000 UTC m=+1603.501211442"
Oct 31 19:46:12.952209 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:12.952174    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:13.233274 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:13.233238    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-d8c398/test-model-d8c398-kserve-57476967b9-v6ld4"
Oct 31 19:46:14.166407 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:14.166317    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:15.319351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:15.297242    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:16.356538 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:16.356501    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:16.692745 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:16.692696    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:17.360791 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:17.360737    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:17.508972 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:17.508940    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:18.260174 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:18.260117    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:19.089448 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:19.089409    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:19.894315 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:19.894276    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:20.854773 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:20.854738    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:21.748762 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:21.748720    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:22.233456 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:22.233419    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-d8c398/test-model-d8c398-kserve-57476967b9-v6ld4"
Oct 31 19:46:22.246910 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:22.246880    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-d8c398/test-model-d8c398-kserve-57476967b9-v6ld4"
Oct 31 19:46:22.663627 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:22.663590    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:23.038659 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:23.038588    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-d8c398/test-model-d8c398-kserve-57476967b9-v6ld4"
Oct 31 19:46:23.443471 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:23.443412    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:24.303967 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:24.303933    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:25.167980 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:25.167946    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:25.982769 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:25.982732    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:26.864407 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:26.864378    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:27.941442 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:27.941404    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:28.774957 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:28.774924    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:29.774208 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:29.774171    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:30.738546 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:30.738512    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:31.577719 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:31.577687    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:32.429880 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:32.429838    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:33.183508 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:33.183470    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:33.932391 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:33.932350    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:34.699876 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:34.699838    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:35.472807 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:35.472763    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:36.247279 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:46:36.247242    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 19:46:36.267495 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:36.267460    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:37.033098 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:37.033051    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:37.762348 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:37.762289    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:38.564050 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:38.564013    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:39.262378 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:39.262308    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:40.059096 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:40.059064    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:40.837777 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:40.837737    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:41.628259 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:41.628223    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:42.332479 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:42.332447    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:43.150149 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:43.150114    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:43.944104 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:43.944069    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:44.800911 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:44.800847    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:45.566252 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:45.566214    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:46.324750 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:46.324705    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:47.167042 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:47.167010    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:47.923118 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:47.923087    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:48.634909 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:48.634871    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:49.362907 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:49.362871    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:50.160255 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:50.160221    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:51.100817 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:51.100769    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:52.288784 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:52.288748    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:52.893474 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:52.893434    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:54.297347 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:54.297295    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:55.359354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:55.355584    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:56.189491 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:56.189452    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:56.310035 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:56.309966    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:56.476503 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:56.475634    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:57.487239 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:57.487204    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:58.371956 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:58.371925    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:46:59.178352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:46:59.170840    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:00.103241 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:00.103154    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:01.057366 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:01.057318    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:01.540498 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:01.540466    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:01.849801 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:01.849722    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:02.038088 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:02.038046    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:02.531819 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:02.531785    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:02.665461 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:02.665382    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:03.113843 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:03.113806    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:04.271522 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:04.271482    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:05.014508 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:05.014476    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:05.841413 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:05.841379    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:06.537213 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:06.537067    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:07.389042 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:47:07.389002    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 19:47:07.501638 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:07.501601    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:09.184479 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:09.184441    4645 patch_prober.go:29] interesting pod/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:47:09.199898 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:09.184499    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9" podUID="d734b787-3a19-4da6-8b9b-93aa92eaa6a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:47:09.318937 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:09.318871    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:09.629673 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:09.629647    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:09.748196 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:09.748168    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:09.815641 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:09.815549    4645 generic.go:356] "Generic (PLEG): container finished" podID="d734b787-3a19-4da6-8b9b-93aa92eaa6a5" containerID="c710846a10f377410aaa0b72fde1570797cc359b3750714bf0cd93cbd7686fcf" exitCode=0
Oct 31 19:47:09.815792 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:09.815768    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9" event={"ID":"d734b787-3a19-4da6-8b9b-93aa92eaa6a5","Type":"ContainerDied","Data":"c710846a10f377410aaa0b72fde1570797cc359b3750714bf0cd93cbd7686fcf"}
Oct 31 19:47:09.816337 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:09.816289    4645 scope.go:122] "RemoveContainer" containerID="c710846a10f377410aaa0b72fde1570797cc359b3750714bf0cd93cbd7686fcf"
Oct 31 19:47:09.819305 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:09.819265    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:47:09.819447 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:09.819378    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:47:09.819447 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:09.819425    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:47:10.547952 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:10.547905    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:10.826831 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:10.826743    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9" event={"ID":"d734b787-3a19-4da6-8b9b-93aa92eaa6a5","Type":"ContainerStarted","Data":"999b4f4360f74f671fabd6a034749d4ae588137a56eb551083bde28a51f099f2"}
Oct 31 19:47:10.886365 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:10.885791    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9"
Oct 31 19:47:10.886365 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:10.885950    4645 patch_prober.go:29] interesting pod/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body=
Oct 31 19:47:10.886365 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:10.886001    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9" podUID="d734b787-3a19-4da6-8b9b-93aa92eaa6a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused"
Oct 31 19:47:11.541277 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:11.541244    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:11.880409 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:11.872607    4645 patch_prober.go:29] interesting pod/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body=
Oct 31 19:47:11.880409 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:11.872671    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9" podUID="d734b787-3a19-4da6-8b9b-93aa92eaa6a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused"
Oct 31 19:47:12.233434 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:12.233396    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:12.859754 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:12.859713    4645 patch_prober.go:29] interesting pod/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body=
Oct 31 19:47:12.859994 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:12.859789    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9" podUID="d734b787-3a19-4da6-8b9b-93aa92eaa6a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused"
Oct 31 19:47:13.186589 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:13.186461    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:13.479863 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:13.479777    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:13.869409 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:13.862665    4645 patch_prober.go:29] interesting pod/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body=
Oct 31 19:47:13.869409 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:13.862786    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9" podUID="d734b787-3a19-4da6-8b9b-93aa92eaa6a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused"
Oct 31 19:47:14.195000 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:14.194952    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:14.867196 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:14.867157    4645 patch_prober.go:29] interesting pod/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body=
Oct 31 19:47:14.867430 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:14.867226    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9" podUID="d734b787-3a19-4da6-8b9b-93aa92eaa6a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused"
Oct 31 19:47:15.054543 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:15.054504    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:15.738234 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:15.738198    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c"]
Oct 31 19:47:15.738668 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:15.738434    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c" podUID="29a9754a-f686-404c-b428-a3b20ba0aa75" containerName="istio-proxy" containerID="cri-o://c9eb86dec8f35ac16479d9aa626e092211036e216812eb26aea6b69a9231c9d4" gracePeriod=30
Oct 31 19:47:15.747254 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:15.747227    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:15.886528 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:15.886477    4645 patch_prober.go:29] interesting pod/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body=
Oct 31 19:47:15.886852 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:15.886817    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9" podUID="d734b787-3a19-4da6-8b9b-93aa92eaa6a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused"
Oct 31 19:47:16.669315 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:16.669279    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:16.770513 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:16.770472    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:16.860728 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:16.860700    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9"
Oct 31 19:47:16.861198 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:16.861175    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9"
Oct 31 19:47:16.861989 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:16.861969    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9"
Oct 31 19:47:17.411574 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:17.411538    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:17.522654 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:17.522625    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:18.156148 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:18.156112    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:19.064686 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:19.064622    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:19.890724 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:19.890680    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:20.660917 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:20.660885    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:21.012041 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:21.012006    4645 generic.go:356] "Generic (PLEG): container finished" podID="29a9754a-f686-404c-b428-a3b20ba0aa75" containerID="c9eb86dec8f35ac16479d9aa626e092211036e216812eb26aea6b69a9231c9d4" exitCode=0
Oct 31 19:47:21.012486 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:21.012080    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c" event={"ID":"29a9754a-f686-404c-b428-a3b20ba0aa75","Type":"ContainerDied","Data":"c9eb86dec8f35ac16479d9aa626e092211036e216812eb26aea6b69a9231c9d4"}
Oct 31 19:47:21.012486 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:21.012156    4645 scope.go:122] "RemoveContainer" containerID="f64fb7efbcea841caf806db1831039c30ba96412296c4726fdb2790508abea98"
Oct 31 19:47:21.093899 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:21.093876    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c"
Oct 31 19:47:21.132225 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:21.126359    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29a9754a-f686-404c-b428-a3b20ba0aa75-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29a9754a-f686-404c-b428-a3b20ba0aa75-workload-certs\") pod \"29a9754a-f686-404c-b428-a3b20ba0aa75\" (UID: \"29a9754a-f686-404c-b428-a3b20ba0aa75\") "
Oct 31 19:47:21.132225 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:21.126470    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29a9754a-f686-404c-b428-a3b20ba0aa75-istio-token\" (UniqueName: \"kubernetes.io/projected/29a9754a-f686-404c-b428-a3b20ba0aa75-istio-token\") pod \"29a9754a-f686-404c-b428-a3b20ba0aa75\" (UID: \"29a9754a-f686-404c-b428-a3b20ba0aa75\") "
Oct 31 19:47:21.132225 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:21.126603    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29a9754a-f686-404c-b428-a3b20ba0aa75-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29a9754a-f686-404c-b428-a3b20ba0aa75-istiod-ca-cert\") pod \"29a9754a-f686-404c-b428-a3b20ba0aa75\" (UID: \"29a9754a-f686-404c-b428-a3b20ba0aa75\") "
Oct 31 19:47:21.132225 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:21.126820    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/29a9754a-f686-404c-b428-a3b20ba0aa75-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29a9754a-f686-404c-b428-a3b20ba0aa75-istio-podinfo\") pod \"29a9754a-f686-404c-b428-a3b20ba0aa75\" (UID: \"29a9754a-f686-404c-b428-a3b20ba0aa75\") "
Oct 31 19:47:21.132225 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:21.126903    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29a9754a-f686-404c-b428-a3b20ba0aa75-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29a9754a-f686-404c-b428-a3b20ba0aa75-credential-socket\") pod \"29a9754a-f686-404c-b428-a3b20ba0aa75\" (UID: \"29a9754a-f686-404c-b428-a3b20ba0aa75\") "
Oct 31 19:47:21.132225 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:21.127119    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29a9754a-f686-404c-b428-a3b20ba0aa75-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29a9754a-f686-404c-b428-a3b20ba0aa75-istio-envoy\") pod \"29a9754a-f686-404c-b428-a3b20ba0aa75\" (UID: \"29a9754a-f686-404c-b428-a3b20ba0aa75\") "
Oct 31 19:47:21.132225 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:21.127393    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29a9754a-f686-404c-b428-a3b20ba0aa75-kube-api-access-fs9sw\" (UniqueName: \"kubernetes.io/projected/29a9754a-f686-404c-b428-a3b20ba0aa75-kube-api-access-fs9sw\") pod \"29a9754a-f686-404c-b428-a3b20ba0aa75\" (UID: \"29a9754a-f686-404c-b428-a3b20ba0aa75\") "
Oct 31 19:47:21.132225 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:21.127432    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29a9754a-f686-404c-b428-a3b20ba0aa75-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29a9754a-f686-404c-b428-a3b20ba0aa75-workload-socket\") pod \"29a9754a-f686-404c-b428-a3b20ba0aa75\" (UID: \"29a9754a-f686-404c-b428-a3b20ba0aa75\") "
Oct 31 19:47:21.132225 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:21.127539    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29a9754a-f686-404c-b428-a3b20ba0aa75-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29a9754a-f686-404c-b428-a3b20ba0aa75-istio-data\") pod \"29a9754a-f686-404c-b428-a3b20ba0aa75\" (UID: \"29a9754a-f686-404c-b428-a3b20ba0aa75\") "
Oct 31 19:47:21.132225 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:21.127631    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29a9754a-f686-404c-b428-a3b20ba0aa75-istiod-ca-cert" pod "29a9754a-f686-404c-b428-a3b20ba0aa75" (UID: "29a9754a-f686-404c-b428-a3b20ba0aa75"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:47:21.132225 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:21.127967    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29a9754a-f686-404c-b428-a3b20ba0aa75-workload-certs" pod "29a9754a-f686-404c-b428-a3b20ba0aa75" (UID: "29a9754a-f686-404c-b428-a3b20ba0aa75"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:47:21.136228 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:21.136200    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29a9754a-f686-404c-b428-a3b20ba0aa75-workload-socket" pod "29a9754a-f686-404c-b428-a3b20ba0aa75" (UID: "29a9754a-f686-404c-b428-a3b20ba0aa75"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:47:21.136378 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:21.136296    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29a9754a-f686-404c-b428-a3b20ba0aa75-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:47:21.136378 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:21.136315    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29a9754a-f686-404c-b428-a3b20ba0aa75-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:47:21.136799 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:21.136435    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29a9754a-f686-404c-b428-a3b20ba0aa75-istio-data" pod "29a9754a-f686-404c-b428-a3b20ba0aa75" (UID: "29a9754a-f686-404c-b428-a3b20ba0aa75"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:47:21.136799 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:21.136549    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29a9754a-f686-404c-b428-a3b20ba0aa75-istio-token" pod "29a9754a-f686-404c-b428-a3b20ba0aa75" (UID: "29a9754a-f686-404c-b428-a3b20ba0aa75"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:47:21.137617 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:21.137565    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29a9754a-f686-404c-b428-a3b20ba0aa75-credential-socket" pod "29a9754a-f686-404c-b428-a3b20ba0aa75" (UID: "29a9754a-f686-404c-b428-a3b20ba0aa75"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:47:21.137865 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:21.137833    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29a9754a-f686-404c-b428-a3b20ba0aa75-kube-api-access-fs9sw" pod "29a9754a-f686-404c-b428-a3b20ba0aa75" (UID: "29a9754a-f686-404c-b428-a3b20ba0aa75"). InnerVolumeSpecName "kube-api-access-fs9sw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:47:21.139099 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:21.139061    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/29a9754a-f686-404c-b428-a3b20ba0aa75-istio-podinfo" pod "29a9754a-f686-404c-b428-a3b20ba0aa75" (UID: "29a9754a-f686-404c-b428-a3b20ba0aa75"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:47:21.142228 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:21.142205    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks"]
Oct 31 19:47:21.144139 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:21.143701    4645 cpu_manager.go:414] "containerMap: removing container" podUID="29a9754a-f686-404c-b428-a3b20ba0aa75" containerName="istio-proxy"
Oct 31 19:47:21.144139 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:21.143725    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29a9754a-f686-404c-b428-a3b20ba0aa75" containerName="istio-proxy"
Oct 31 19:47:21.145311 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:21.145291    4645 cpu_manager.go:414] "containerMap: removing container" podUID="25370d12-c485-47b9-986a-a775c3fbd2b7" containerName="cleanup"
Oct 31 19:47:21.145466 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:21.145311    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25370d12-c485-47b9-986a-a775c3fbd2b7" containerName="cleanup"
Oct 31 19:47:21.146393 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:21.145810    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="29a9754a-f686-404c-b428-a3b20ba0aa75" containerName="istio-proxy"
Oct 31 19:47:21.146393 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:21.145844    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="25370d12-c485-47b9-986a-a775c3fbd2b7" containerName="cleanup"
Oct 31 19:47:21.146393 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:21.145869    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="29a9754a-f686-404c-b428-a3b20ba0aa75" containerName="istio-proxy"
Oct 31 19:47:21.149050 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:21.149028    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29a9754a-f686-404c-b428-a3b20ba0aa75-istio-envoy" pod "29a9754a-f686-404c-b428-a3b20ba0aa75" (UID: "29a9754a-f686-404c-b428-a3b20ba0aa75"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:47:21.179987 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:21.179966    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks"]
Oct 31 19:47:21.180103 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:21.180073    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks"
Oct 31 19:47:21.239569 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:21.238741    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cf00cb6-1d65-42f3-9c91-f4622454a913-istio-envoy\") pod \"e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks\" (UID: \"1cf00cb6-1d65-42f3-9c91-f4622454a913\") " pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks"
Oct 31 19:47:21.239569 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:21.238860    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cf00cb6-1d65-42f3-9c91-f4622454a913-istio-token\") pod \"e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks\" (UID: \"1cf00cb6-1d65-42f3-9c91-f4622454a913\") " pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks"
Oct 31 19:47:21.239569 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:21.238971    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cf00cb6-1d65-42f3-9c91-f4622454a913-istio-data\") pod \"e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks\" (UID: \"1cf00cb6-1d65-42f3-9c91-f4622454a913\") " pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks"
Oct 31 19:47:21.239569 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:21.239029    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cf00cb6-1d65-42f3-9c91-f4622454a913-credential-socket\") pod \"e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks\" (UID: \"1cf00cb6-1d65-42f3-9c91-f4622454a913\") " pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks"
Oct 31 19:47:21.239981 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:21.239956    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cf00cb6-1d65-42f3-9c91-f4622454a913-istiod-ca-cert\") pod \"e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks\" (UID: \"1cf00cb6-1d65-42f3-9c91-f4622454a913\") " pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks"
Oct 31 19:47:21.240114 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:21.240098    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cf00cb6-1d65-42f3-9c91-f4622454a913-workload-socket\") pod \"e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks\" (UID: \"1cf00cb6-1d65-42f3-9c91-f4622454a913\") " pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks"
Oct 31 19:47:21.240316 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:21.240296    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s66vw\" (UniqueName: \"kubernetes.io/projected/1cf00cb6-1d65-42f3-9c91-f4622454a913-kube-api-access-s66vw\") pod \"e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks\" (UID: \"1cf00cb6-1d65-42f3-9c91-f4622454a913\") " pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks"
Oct 31 19:47:21.240667 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:21.240647    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cf00cb6-1d65-42f3-9c91-f4622454a913-istio-podinfo\") pod \"e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks\" (UID: \"1cf00cb6-1d65-42f3-9c91-f4622454a913\") " pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks"
Oct 31 19:47:21.240942 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:21.240909    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cf00cb6-1d65-42f3-9c91-f4622454a913-workload-certs\") pod \"e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks\" (UID: \"1cf00cb6-1d65-42f3-9c91-f4622454a913\") " pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks"
Oct 31 19:47:21.241250 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:21.241219    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29a9754a-f686-404c-b428-a3b20ba0aa75-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:47:21.241528 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:21.241510    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29a9754a-f686-404c-b428-a3b20ba0aa75-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:47:21.241682 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:21.241667    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29a9754a-f686-404c-b428-a3b20ba0aa75-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:47:21.241824 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:21.241809    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29a9754a-f686-404c-b428-a3b20ba0aa75-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:47:21.241963 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:21.241930    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fs9sw\" (UniqueName: \"kubernetes.io/projected/29a9754a-f686-404c-b428-a3b20ba0aa75-kube-api-access-fs9sw\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:47:21.242174 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:21.242070    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29a9754a-f686-404c-b428-a3b20ba0aa75-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:47:21.242320 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:21.242307    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29a9754a-f686-404c-b428-a3b20ba0aa75-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:47:21.344165 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:21.344132    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cf00cb6-1d65-42f3-9c91-f4622454a913-istiod-ca-cert\") pod \"e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks\" (UID: \"1cf00cb6-1d65-42f3-9c91-f4622454a913\") " pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks"
Oct 31 19:47:21.344289 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:21.344175    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cf00cb6-1d65-42f3-9c91-f4622454a913-workload-socket\") pod \"e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks\" (UID: \"1cf00cb6-1d65-42f3-9c91-f4622454a913\") " pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks"
Oct 31 19:47:21.344289 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:21.344209    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s66vw\" (UniqueName: \"kubernetes.io/projected/1cf00cb6-1d65-42f3-9c91-f4622454a913-kube-api-access-s66vw\") pod \"e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks\" (UID: \"1cf00cb6-1d65-42f3-9c91-f4622454a913\") " pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks"
Oct 31 19:47:21.344289 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:21.344236    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cf00cb6-1d65-42f3-9c91-f4622454a913-istio-podinfo\") pod \"e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks\" (UID: \"1cf00cb6-1d65-42f3-9c91-f4622454a913\") " pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks"
Oct 31 19:47:21.344289 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:21.344283    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cf00cb6-1d65-42f3-9c91-f4622454a913-workload-certs\") pod \"e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks\" (UID: \"1cf00cb6-1d65-42f3-9c91-f4622454a913\") " pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks"
Oct 31 19:47:21.344533 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:21.344398    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cf00cb6-1d65-42f3-9c91-f4622454a913-istio-envoy\") pod \"e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks\" (UID: \"1cf00cb6-1d65-42f3-9c91-f4622454a913\") " pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks"
Oct 31 19:47:21.344533 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:21.344484    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cf00cb6-1d65-42f3-9c91-f4622454a913-istio-token\") pod \"e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks\" (UID: \"1cf00cb6-1d65-42f3-9c91-f4622454a913\") " pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks"
Oct 31 19:47:21.344645 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:21.344577    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cf00cb6-1d65-42f3-9c91-f4622454a913-istio-data\") pod \"e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks\" (UID: \"1cf00cb6-1d65-42f3-9c91-f4622454a913\") " pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks"
Oct 31 19:47:21.344645 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:21.344618    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cf00cb6-1d65-42f3-9c91-f4622454a913-credential-socket\") pod \"e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks\" (UID: \"1cf00cb6-1d65-42f3-9c91-f4622454a913\") " pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks"
Oct 31 19:47:21.344758 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:21.344660    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cf00cb6-1d65-42f3-9c91-f4622454a913-workload-socket\") pod \"e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks\" (UID: \"1cf00cb6-1d65-42f3-9c91-f4622454a913\") " pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks"
Oct 31 19:47:21.344846 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:21.344812    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cf00cb6-1d65-42f3-9c91-f4622454a913-workload-certs\") pod \"e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks\" (UID: \"1cf00cb6-1d65-42f3-9c91-f4622454a913\") " pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks"
Oct 31 19:47:21.345010 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:21.344982    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cf00cb6-1d65-42f3-9c91-f4622454a913-istiod-ca-cert\") pod \"e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks\" (UID: \"1cf00cb6-1d65-42f3-9c91-f4622454a913\") " pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks"
Oct 31 19:47:21.345081 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:21.345054    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cf00cb6-1d65-42f3-9c91-f4622454a913-credential-socket\") pod \"e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks\" (UID: \"1cf00cb6-1d65-42f3-9c91-f4622454a913\") " pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks"
Oct 31 19:47:21.345251 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:21.345234    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cf00cb6-1d65-42f3-9c91-f4622454a913-istio-data\") pod \"e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks\" (UID: \"1cf00cb6-1d65-42f3-9c91-f4622454a913\") " pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks"
Oct 31 19:47:21.346807 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:21.346786    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cf00cb6-1d65-42f3-9c91-f4622454a913-istio-envoy\") pod \"e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks\" (UID: \"1cf00cb6-1d65-42f3-9c91-f4622454a913\") " pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks"
Oct 31 19:47:21.347035 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:21.347016    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cf00cb6-1d65-42f3-9c91-f4622454a913-istio-podinfo\") pod \"e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks\" (UID: \"1cf00cb6-1d65-42f3-9c91-f4622454a913\") " pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks"
Oct 31 19:47:21.356203 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:21.356181    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cf00cb6-1d65-42f3-9c91-f4622454a913-istio-token\") pod \"e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks\" (UID: \"1cf00cb6-1d65-42f3-9c91-f4622454a913\") " pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks"
Oct 31 19:47:21.356287 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:21.356230    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s66vw\" (UniqueName: \"kubernetes.io/projected/1cf00cb6-1d65-42f3-9c91-f4622454a913-kube-api-access-s66vw\") pod \"e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks\" (UID: \"1cf00cb6-1d65-42f3-9c91-f4622454a913\") " pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks"
Oct 31 19:47:21.442190 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:21.442167    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:21.503593 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:21.503499    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks"
Oct 31 19:47:21.849538 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:47:21.849505    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cf00cb6_1d65_42f3_9c91_f4622454a913.slice/crio-781193ce99a79a2b6eda0e2093dac7e0181c7a1261ba7e93eb7c7b288f4c0c11 WatchSource:0}: Error finding container 781193ce99a79a2b6eda0e2093dac7e0181c7a1261ba7e93eb7c7b288f4c0c11: Status 404 returned error can't find the container with id 781193ce99a79a2b6eda0e2093dac7e0181c7a1261ba7e93eb7c7b288f4c0c11
Oct 31 19:47:21.850053 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:21.850028    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks"]
Oct 31 19:47:21.852679 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:21.852644    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:47:21.852753 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:21.852716    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:47:21.852753 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:21.852748    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:47:22.023410 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:22.023366    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks" event={"ID":"1cf00cb6-1d65-42f3-9c91-f4622454a913","Type":"ContainerStarted","Data":"5a69f40deb4941134268160b1b5f8ce1a1c5a1ef5d30b28b7d699aeca47d4285"}
Oct 31 19:47:22.023872 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:22.023431    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks" event={"ID":"1cf00cb6-1d65-42f3-9c91-f4622454a913","Type":"ContainerStarted","Data":"781193ce99a79a2b6eda0e2093dac7e0181c7a1261ba7e93eb7c7b288f4c0c11"}
Oct 31 19:47:22.025004 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:22.024983    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c"
Oct 31 19:47:22.025124 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:22.024984    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c" event={"ID":"29a9754a-f686-404c-b428-a3b20ba0aa75","Type":"ContainerDied","Data":"270c644810a2b9e45180888280b0578ba49a024a0535f7f9ff152af78469e8f1"}
Oct 31 19:47:22.025124 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:22.025112    4645 scope.go:122] "RemoveContainer" containerID="c9eb86dec8f35ac16479d9aa626e092211036e216812eb26aea6b69a9231c9d4"
Oct 31 19:47:22.053758 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:22.053709    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks" podStartSLOduration=73.053690958 podStartE2EDuration="1m13.053690958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:46:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:47:22.046275345 +0000 UTC m=+1672.619509474" watchObservedRunningTime="2026-10-31 19:47:22.053690958 +0000 UTC m=+1672.626925090"
Oct 31 19:47:22.069926 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:22.069903    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c"]
Oct 31 19:47:22.077478 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:22.077454    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-9jz5c"]
Oct 31 19:47:22.258708 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:22.258681    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:22.504183 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:22.504143    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks"
Oct 31 19:47:22.506210 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:22.506184    4645 patch_prober.go:29] interesting pod/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body=
Oct 31 19:47:22.506388 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:22.506232    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks" podUID="1cf00cb6-1d65-42f3-9c91-f4622454a913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused"
Oct 31 19:47:23.248457 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:23.248412    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:23.504651 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:23.504552    4645 patch_prober.go:29] interesting pod/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body=
Oct 31 19:47:23.504651 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:23.504614    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks" podUID="1cf00cb6-1d65-42f3-9c91-f4622454a913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused"
Oct 31 19:47:23.621992 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:23.621961    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29a9754a-f686-404c-b428-a3b20ba0aa75" path="/var/lib/kubelet/pods/29a9754a-f686-404c-b428-a3b20ba0aa75/volumes"
Oct 31 19:47:23.954588 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:23.954558    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:24.505040 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:24.504470    4645 patch_prober.go:29] interesting pod/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body=
Oct 31 19:47:24.505040 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:24.504536    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks" podUID="1cf00cb6-1d65-42f3-9c91-f4622454a913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused"
Oct 31 19:47:24.700354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:24.700287    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:25.439916 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:25.439877    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:25.509484 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:25.508184    4645 patch_prober.go:29] interesting pod/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body=
Oct 31 19:47:25.509484 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:25.508254    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks" podUID="1cf00cb6-1d65-42f3-9c91-f4622454a913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused"
Oct 31 19:47:26.153479 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:26.153443    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:26.505059 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:26.505025    4645 patch_prober.go:29] interesting pod/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body=
Oct 31 19:47:26.505245 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:26.505072    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks" podUID="1cf00cb6-1d65-42f3-9c91-f4622454a913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused"
Oct 31 19:47:27.141365 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:27.138878    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:27.504941 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:27.504907    4645 patch_prober.go:29] interesting pod/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body=
Oct 31 19:47:27.505145 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:27.504961    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks" podUID="1cf00cb6-1d65-42f3-9c91-f4622454a913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused"
Oct 31 19:47:28.003074 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:28.003036    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:28.512444 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:28.512415    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks"
Oct 31 19:47:28.512894 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:28.512752    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks"
Oct 31 19:47:28.513338 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:28.513308    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks"
Oct 31 19:47:28.773938 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:28.773852    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:29.572368 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:29.572318    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:30.436501 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:30.436464    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:31.252378 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:31.252347    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:32.095375 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:32.095309    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:32.848541 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:32.848506    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:33.574876 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:33.574839    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:34.350280 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:34.350250    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:35.059677 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:35.059633    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:35.845043 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:35.845011    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:36.247557 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:47:36.247520    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 19:47:36.543983 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:36.543903    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:37.234960 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:37.234924    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:38.087099 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:38.087069    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:38.880206 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:38.880178    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:39.697815 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:39.697783    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:40.477199 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:40.477170    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:41.327050 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:41.327010    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:42.036124 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:42.036082    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:42.899076 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:42.899040    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:43.591602 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:43.591568    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:44.345230 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:44.345201    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:44.457592 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:44.457560    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:45.121635 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:45.121605    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:45.837277 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:45.837242    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:46.566426 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:46.566391    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:47.452994 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:47.452954    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:48.164587 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:48.164547    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:48.852368 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:48.852314    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:49.585531 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:49.585497    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:50.455701 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:50.455667    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:51.562810 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:51.562778    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:52.618483 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:52.618451    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:53.402632 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:53.402592    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:54.388110 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:54.388069    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:55.133983 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:55.133956    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:55.853593 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:55.853559    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:56.761885 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:56.761852    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:57.458448 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:57.458414    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:58.161150 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:58.161117    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:59.048763 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:59.048727    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:47:59.830318 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:47:59.830264    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:00.680167 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:00.680128    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:01.568200 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:01.568154    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:02.337921 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:02.337884    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:03.183051 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:03.183020    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:03.949633 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:03.949570    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:04.691386 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:04.691313    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:05.466137 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:05.466085    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:05.822686 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:05.822601    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w"]
Oct 31 19:48:05.822887 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:05.822851    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w" podUID="981d31f6-7349-4835-81d3-8bbf0408f0d0" containerName="istio-proxy" containerID="cri-o://d96ae52208eacbcc7609c5b75fee45de5cc01d64ba1eb915cec770ea57d8f827" gracePeriod=30
Oct 31 19:48:05.829741 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:05.829710    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4"]
Oct 31 19:48:05.829969 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:05.829943    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4" podUID="452d967b-8434-4766-9b19-690c49adc197" containerName="istio-proxy" containerID="cri-o://28b5fbd07c27a91c35c6eda7f308ee48d6e6d3c9ab9b1ad6f44b700acf30459b" gracePeriod=30
Oct 31 19:48:06.262180 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:48:06.262145    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 19:48:06.536782 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:06.536696    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:07.249496 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:07.249463    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:07.945096 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:07.945034    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:08.796407 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:08.796348    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:09.532772 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:09.532733    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:10.324789 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:10.324752    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:11.076177 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.076139    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:11.216574 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.216541    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w"
Oct 31 19:48:11.276982 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.276761    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g"]
Oct 31 19:48:11.278225 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.278185    4645 cpu_manager.go:414] "containerMap: removing container" podUID="29a9754a-f686-404c-b428-a3b20ba0aa75" containerName="istio-proxy"
Oct 31 19:48:11.278425 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.278406    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29a9754a-f686-404c-b428-a3b20ba0aa75" containerName="istio-proxy"
Oct 31 19:48:11.278608 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.278594    4645 cpu_manager.go:414] "containerMap: removing container" podUID="981d31f6-7349-4835-81d3-8bbf0408f0d0" containerName="istio-proxy"
Oct 31 19:48:11.278710 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.278697    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="981d31f6-7349-4835-81d3-8bbf0408f0d0" containerName="istio-proxy"
Oct 31 19:48:11.279039 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.279020    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="981d31f6-7349-4835-81d3-8bbf0408f0d0" containerName="istio-proxy"
Oct 31 19:48:11.284253 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.284125    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g"
Oct 31 19:48:11.296444 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.296416    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g"]
Oct 31 19:48:11.301285 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.300466    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4"
Oct 31 19:48:11.301285 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.301004    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/981d31f6-7349-4835-81d3-8bbf0408f0d0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/981d31f6-7349-4835-81d3-8bbf0408f0d0-istio-data\") pod \"981d31f6-7349-4835-81d3-8bbf0408f0d0\" (UID: \"981d31f6-7349-4835-81d3-8bbf0408f0d0\") "
Oct 31 19:48:11.301285 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.301076    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/981d31f6-7349-4835-81d3-8bbf0408f0d0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/981d31f6-7349-4835-81d3-8bbf0408f0d0-workload-socket\") pod \"981d31f6-7349-4835-81d3-8bbf0408f0d0\" (UID: \"981d31f6-7349-4835-81d3-8bbf0408f0d0\") "
Oct 31 19:48:11.301285 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.301116    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/981d31f6-7349-4835-81d3-8bbf0408f0d0-kube-api-access-jccxn\" (UniqueName: \"kubernetes.io/projected/981d31f6-7349-4835-81d3-8bbf0408f0d0-kube-api-access-jccxn\") pod \"981d31f6-7349-4835-81d3-8bbf0408f0d0\" (UID: \"981d31f6-7349-4835-81d3-8bbf0408f0d0\") "
Oct 31 19:48:11.301285 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.301171    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/981d31f6-7349-4835-81d3-8bbf0408f0d0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/981d31f6-7349-4835-81d3-8bbf0408f0d0-istio-podinfo\") pod \"981d31f6-7349-4835-81d3-8bbf0408f0d0\" (UID: \"981d31f6-7349-4835-81d3-8bbf0408f0d0\") "
Oct 31 19:48:11.301708 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.301300    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/981d31f6-7349-4835-81d3-8bbf0408f0d0-istio-data" pod "981d31f6-7349-4835-81d3-8bbf0408f0d0" (UID: "981d31f6-7349-4835-81d3-8bbf0408f0d0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:48:11.301708 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.301546    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/981d31f6-7349-4835-81d3-8bbf0408f0d0-workload-socket" pod "981d31f6-7349-4835-81d3-8bbf0408f0d0" (UID: "981d31f6-7349-4835-81d3-8bbf0408f0d0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:48:11.303559 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.301903    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/981d31f6-7349-4835-81d3-8bbf0408f0d0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/981d31f6-7349-4835-81d3-8bbf0408f0d0-credential-socket\") pod \"981d31f6-7349-4835-81d3-8bbf0408f0d0\" (UID: \"981d31f6-7349-4835-81d3-8bbf0408f0d0\") "
Oct 31 19:48:11.303559 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.302012    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/981d31f6-7349-4835-81d3-8bbf0408f0d0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/981d31f6-7349-4835-81d3-8bbf0408f0d0-workload-certs\") pod \"981d31f6-7349-4835-81d3-8bbf0408f0d0\" (UID: \"981d31f6-7349-4835-81d3-8bbf0408f0d0\") "
Oct 31 19:48:11.303559 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.302055    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/981d31f6-7349-4835-81d3-8bbf0408f0d0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/981d31f6-7349-4835-81d3-8bbf0408f0d0-istiod-ca-cert\") pod \"981d31f6-7349-4835-81d3-8bbf0408f0d0\" (UID: \"981d31f6-7349-4835-81d3-8bbf0408f0d0\") "
Oct 31 19:48:11.303559 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.302159    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/981d31f6-7349-4835-81d3-8bbf0408f0d0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/981d31f6-7349-4835-81d3-8bbf0408f0d0-istio-envoy\") pod \"981d31f6-7349-4835-81d3-8bbf0408f0d0\" (UID: \"981d31f6-7349-4835-81d3-8bbf0408f0d0\") "
Oct 31 19:48:11.303559 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.302378    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/981d31f6-7349-4835-81d3-8bbf0408f0d0-istio-token\" (UniqueName: \"kubernetes.io/projected/981d31f6-7349-4835-81d3-8bbf0408f0d0-istio-token\") pod \"981d31f6-7349-4835-81d3-8bbf0408f0d0\" (UID: \"981d31f6-7349-4835-81d3-8bbf0408f0d0\") "
Oct 31 19:48:11.303559 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.303257    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/981d31f6-7349-4835-81d3-8bbf0408f0d0-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:11.303559 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.303279    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/981d31f6-7349-4835-81d3-8bbf0408f0d0-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:11.304404 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.304302    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/981d31f6-7349-4835-81d3-8bbf0408f0d0-istio-podinfo" pod "981d31f6-7349-4835-81d3-8bbf0408f0d0" (UID: "981d31f6-7349-4835-81d3-8bbf0408f0d0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:48:11.305119 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.305101    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/981d31f6-7349-4835-81d3-8bbf0408f0d0-workload-certs" pod "981d31f6-7349-4835-81d3-8bbf0408f0d0" (UID: "981d31f6-7349-4835-81d3-8bbf0408f0d0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:48:11.305284 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.305267    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/981d31f6-7349-4835-81d3-8bbf0408f0d0-credential-socket" pod "981d31f6-7349-4835-81d3-8bbf0408f0d0" (UID: "981d31f6-7349-4835-81d3-8bbf0408f0d0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:48:11.305449 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.305432    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/981d31f6-7349-4835-81d3-8bbf0408f0d0-kube-api-access-jccxn" pod "981d31f6-7349-4835-81d3-8bbf0408f0d0" (UID: "981d31f6-7349-4835-81d3-8bbf0408f0d0"). InnerVolumeSpecName "kube-api-access-jccxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:48:11.305589 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.305571    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/981d31f6-7349-4835-81d3-8bbf0408f0d0-istio-token" pod "981d31f6-7349-4835-81d3-8bbf0408f0d0" (UID: "981d31f6-7349-4835-81d3-8bbf0408f0d0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:48:11.305698 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.305673    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/981d31f6-7349-4835-81d3-8bbf0408f0d0-istiod-ca-cert" pod "981d31f6-7349-4835-81d3-8bbf0408f0d0" (UID: "981d31f6-7349-4835-81d3-8bbf0408f0d0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:48:11.307510 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.307485    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/981d31f6-7349-4835-81d3-8bbf0408f0d0-istio-envoy" pod "981d31f6-7349-4835-81d3-8bbf0408f0d0" (UID: "981d31f6-7349-4835-81d3-8bbf0408f0d0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:48:11.360844 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.357025    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679"]
Oct 31 19:48:11.360844 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.358401    4645 cpu_manager.go:414] "containerMap: removing container" podUID="452d967b-8434-4766-9b19-690c49adc197" containerName="istio-proxy"
Oct 31 19:48:11.360844 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.358424    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="452d967b-8434-4766-9b19-690c49adc197" containerName="istio-proxy"
Oct 31 19:48:11.360844 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.358759    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="452d967b-8434-4766-9b19-690c49adc197" containerName="istio-proxy"
Oct 31 19:48:11.375114 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.375088    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679"]
Oct 31 19:48:11.375521 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.375495    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679"
Oct 31 19:48:11.403812 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.403777    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/452d967b-8434-4766-9b19-690c49adc197-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/452d967b-8434-4766-9b19-690c49adc197-istio-podinfo\") pod \"452d967b-8434-4766-9b19-690c49adc197\" (UID: \"452d967b-8434-4766-9b19-690c49adc197\") "
Oct 31 19:48:11.403950 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.403833    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/452d967b-8434-4766-9b19-690c49adc197-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/452d967b-8434-4766-9b19-690c49adc197-istio-data\") pod \"452d967b-8434-4766-9b19-690c49adc197\" (UID: \"452d967b-8434-4766-9b19-690c49adc197\") "
Oct 31 19:48:11.403950 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.403914    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/452d967b-8434-4766-9b19-690c49adc197-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/452d967b-8434-4766-9b19-690c49adc197-workload-certs\") pod \"452d967b-8434-4766-9b19-690c49adc197\" (UID: \"452d967b-8434-4766-9b19-690c49adc197\") "
Oct 31 19:48:11.404077 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.404030    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/452d967b-8434-4766-9b19-690c49adc197-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/452d967b-8434-4766-9b19-690c49adc197-istiod-ca-cert\") pod \"452d967b-8434-4766-9b19-690c49adc197\" (UID: \"452d967b-8434-4766-9b19-690c49adc197\") "
Oct 31 19:48:11.404077 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.404058    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/452d967b-8434-4766-9b19-690c49adc197-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/452d967b-8434-4766-9b19-690c49adc197-istio-envoy\") pod \"452d967b-8434-4766-9b19-690c49adc197\" (UID: \"452d967b-8434-4766-9b19-690c49adc197\") "
Oct 31 19:48:11.404192 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.404155    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/452d967b-8434-4766-9b19-690c49adc197-istio-token\" (UniqueName: \"kubernetes.io/projected/452d967b-8434-4766-9b19-690c49adc197-istio-token\") pod \"452d967b-8434-4766-9b19-690c49adc197\" (UID: \"452d967b-8434-4766-9b19-690c49adc197\") "
Oct 31 19:48:11.404348 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.404315    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/452d967b-8434-4766-9b19-690c49adc197-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/452d967b-8434-4766-9b19-690c49adc197-workload-socket\") pod \"452d967b-8434-4766-9b19-690c49adc197\" (UID: \"452d967b-8434-4766-9b19-690c49adc197\") "
Oct 31 19:48:11.404556 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.404530    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/452d967b-8434-4766-9b19-690c49adc197-kube-api-access-66r4p\" (UniqueName: \"kubernetes.io/projected/452d967b-8434-4766-9b19-690c49adc197-kube-api-access-66r4p\") pod \"452d967b-8434-4766-9b19-690c49adc197\" (UID: \"452d967b-8434-4766-9b19-690c49adc197\") "
Oct 31 19:48:11.404697 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.404410    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/452d967b-8434-4766-9b19-690c49adc197-workload-certs" pod "452d967b-8434-4766-9b19-690c49adc197" (UID: "452d967b-8434-4766-9b19-690c49adc197"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:48:11.404697 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.404639    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/452d967b-8434-4766-9b19-690c49adc197-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/452d967b-8434-4766-9b19-690c49adc197-credential-socket\") pod \"452d967b-8434-4766-9b19-690c49adc197\" (UID: \"452d967b-8434-4766-9b19-690c49adc197\") "
Oct 31 19:48:11.404697 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.404613    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/452d967b-8434-4766-9b19-690c49adc197-workload-socket" pod "452d967b-8434-4766-9b19-690c49adc197" (UID: "452d967b-8434-4766-9b19-690c49adc197"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:48:11.404946 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.404862    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/452d967b-8434-4766-9b19-690c49adc197-istiod-ca-cert" pod "452d967b-8434-4766-9b19-690c49adc197" (UID: "452d967b-8434-4766-9b19-690c49adc197"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:48:11.405049 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.405020    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8824b6ff-8d47-4df9-a907-8306aa909993-istio-envoy\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679\" (UID: \"8824b6ff-8d47-4df9-a907-8306aa909993\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679"
Oct 31 19:48:11.405132 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.405066    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8824b6ff-8d47-4df9-a907-8306aa909993-credential-socket\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679\" (UID: \"8824b6ff-8d47-4df9-a907-8306aa909993\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679"
Oct 31 19:48:11.405132 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.405099    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8824b6ff-8d47-4df9-a907-8306aa909993-istio-podinfo\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679\" (UID: \"8824b6ff-8d47-4df9-a907-8306aa909993\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679"
Oct 31 19:48:11.405299 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.405174    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8824b6ff-8d47-4df9-a907-8306aa909993-workload-certs\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679\" (UID: \"8824b6ff-8d47-4df9-a907-8306aa909993\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679"
Oct 31 19:48:11.405299 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.405187    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/452d967b-8434-4766-9b19-690c49adc197-credential-socket" pod "452d967b-8434-4766-9b19-690c49adc197" (UID: "452d967b-8434-4766-9b19-690c49adc197"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:48:11.405299 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.405281    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b-istio-token\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g\" (UID: \"f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g"
Oct 31 19:48:11.405618 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.405313    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b-workload-socket\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g\" (UID: \"f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g"
Oct 31 19:48:11.405618 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.405359    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b-istiod-ca-cert\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g\" (UID: \"f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g"
Oct 31 19:48:11.405618 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.405477    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/452d967b-8434-4766-9b19-690c49adc197-istio-data" pod "452d967b-8434-4766-9b19-690c49adc197" (UID: "452d967b-8434-4766-9b19-690c49adc197"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:48:11.405618 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.405545    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpw72\" (UniqueName: \"kubernetes.io/projected/8824b6ff-8d47-4df9-a907-8306aa909993-kube-api-access-fpw72\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679\" (UID: \"8824b6ff-8d47-4df9-a907-8306aa909993\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679"
Oct 31 19:48:11.405618 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.405601    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8824b6ff-8d47-4df9-a907-8306aa909993-istio-token\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679\" (UID: \"8824b6ff-8d47-4df9-a907-8306aa909993\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679"
Oct 31 19:48:11.405894 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.405694    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b-istio-podinfo\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g\" (UID: \"f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g"
Oct 31 19:48:11.405894 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.405732    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b-workload-certs\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g\" (UID: \"f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g"
Oct 31 19:48:11.405894 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.405764    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b-istio-data\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g\" (UID: \"f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g"
Oct 31 19:48:11.405894 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.405820    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8ltp\" (UniqueName: \"kubernetes.io/projected/f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b-kube-api-access-v8ltp\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g\" (UID: \"f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g"
Oct 31 19:48:11.405894 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.405867    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b-credential-socket\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g\" (UID: \"f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g"
Oct 31 19:48:11.405894 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.405881    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b-istio-envoy\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g\" (UID: \"f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g"
Oct 31 19:48:11.406203 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.405939    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8824b6ff-8d47-4df9-a907-8306aa909993-istiod-ca-cert\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679\" (UID: \"8824b6ff-8d47-4df9-a907-8306aa909993\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679"
Oct 31 19:48:11.406203 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.405973    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8824b6ff-8d47-4df9-a907-8306aa909993-workload-socket\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679\" (UID: \"8824b6ff-8d47-4df9-a907-8306aa909993\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679"
Oct 31 19:48:11.406203 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.406025    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8824b6ff-8d47-4df9-a907-8306aa909993-istio-data\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679\" (UID: \"8824b6ff-8d47-4df9-a907-8306aa909993\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679"
Oct 31 19:48:11.406203 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.406154    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/452d967b-8434-4766-9b19-690c49adc197-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:11.406203 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.406169    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/981d31f6-7349-4835-81d3-8bbf0408f0d0-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:11.406203 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.406177    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/452d967b-8434-4766-9b19-690c49adc197-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:11.406203 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.406187    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/452d967b-8434-4766-9b19-690c49adc197-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:11.406203 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.406196    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/981d31f6-7349-4835-81d3-8bbf0408f0d0-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:11.406720 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.406209    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/452d967b-8434-4766-9b19-690c49adc197-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:11.406720 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.406221    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jccxn\" (UniqueName: \"kubernetes.io/projected/981d31f6-7349-4835-81d3-8bbf0408f0d0-kube-api-access-jccxn\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:11.406720 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.406229    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/981d31f6-7349-4835-81d3-8bbf0408f0d0-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:11.406720 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.406237    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/452d967b-8434-4766-9b19-690c49adc197-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:11.406720 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.406245    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/981d31f6-7349-4835-81d3-8bbf0408f0d0-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:11.406720 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.406253    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/981d31f6-7349-4835-81d3-8bbf0408f0d0-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:11.406720 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.406261    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/981d31f6-7349-4835-81d3-8bbf0408f0d0-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:11.415827 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.415786    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/452d967b-8434-4766-9b19-690c49adc197-istio-podinfo" pod "452d967b-8434-4766-9b19-690c49adc197" (UID: "452d967b-8434-4766-9b19-690c49adc197"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:48:11.415929 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.415848    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/452d967b-8434-4766-9b19-690c49adc197-istio-envoy" pod "452d967b-8434-4766-9b19-690c49adc197" (UID: "452d967b-8434-4766-9b19-690c49adc197"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:48:11.416004 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.415942    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/452d967b-8434-4766-9b19-690c49adc197-istio-token" pod "452d967b-8434-4766-9b19-690c49adc197" (UID: "452d967b-8434-4766-9b19-690c49adc197"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:48:11.417735 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.417710    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/452d967b-8434-4766-9b19-690c49adc197-kube-api-access-66r4p" pod "452d967b-8434-4766-9b19-690c49adc197" (UID: "452d967b-8434-4766-9b19-690c49adc197"). InnerVolumeSpecName "kube-api-access-66r4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:48:11.507686 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.507653    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b-istio-podinfo\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g\" (UID: \"f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g"
Oct 31 19:48:11.507873 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.507698    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b-workload-certs\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g\" (UID: \"f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g"
Oct 31 19:48:11.507873 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.507719    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b-istio-data\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g\" (UID: \"f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g"
Oct 31 19:48:11.507873 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.507750    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8ltp\" (UniqueName: \"kubernetes.io/projected/f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b-kube-api-access-v8ltp\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g\" (UID: \"f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g"
Oct 31 19:48:11.507873 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.507795    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b-credential-socket\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g\" (UID: \"f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g"
Oct 31 19:48:11.507873 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.507817    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b-istio-envoy\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g\" (UID: \"f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g"
Oct 31 19:48:11.508128 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.508003    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8824b6ff-8d47-4df9-a907-8306aa909993-istiod-ca-cert\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679\" (UID: \"8824b6ff-8d47-4df9-a907-8306aa909993\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679"
Oct 31 19:48:11.508128 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.508080    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b-workload-certs\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g\" (UID: \"f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g"
Oct 31 19:48:11.508128 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.508088    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8824b6ff-8d47-4df9-a907-8306aa909993-workload-socket\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679\" (UID: \"8824b6ff-8d47-4df9-a907-8306aa909993\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679"
Oct 31 19:48:11.508809 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.508121    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b-istio-data\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g\" (UID: \"f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g"
Oct 31 19:48:11.508809 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.508171    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b-credential-socket\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g\" (UID: \"f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g"
Oct 31 19:48:11.508809 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.508226    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8824b6ff-8d47-4df9-a907-8306aa909993-istio-data\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679\" (UID: \"8824b6ff-8d47-4df9-a907-8306aa909993\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679"
Oct 31 19:48:11.508809 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.508388    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8824b6ff-8d47-4df9-a907-8306aa909993-istio-envoy\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679\" (UID: \"8824b6ff-8d47-4df9-a907-8306aa909993\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679"
Oct 31 19:48:11.508809 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.508431    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8824b6ff-8d47-4df9-a907-8306aa909993-credential-socket\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679\" (UID: \"8824b6ff-8d47-4df9-a907-8306aa909993\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679"
Oct 31 19:48:11.508809 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.508465    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8824b6ff-8d47-4df9-a907-8306aa909993-istio-podinfo\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679\" (UID: \"8824b6ff-8d47-4df9-a907-8306aa909993\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679"
Oct 31 19:48:11.508809 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.508549    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8824b6ff-8d47-4df9-a907-8306aa909993-workload-socket\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679\" (UID: \"8824b6ff-8d47-4df9-a907-8306aa909993\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679"
Oct 31 19:48:11.508809 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.508567    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8824b6ff-8d47-4df9-a907-8306aa909993-workload-certs\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679\" (UID: \"8824b6ff-8d47-4df9-a907-8306aa909993\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679"
Oct 31 19:48:11.508809 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.508606    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b-istio-token\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g\" (UID: \"f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g"
Oct 31 19:48:11.508809 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.508664    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b-workload-socket\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g\" (UID: \"f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g"
Oct 31 19:48:11.508809 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.508700    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b-istiod-ca-cert\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g\" (UID: \"f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g"
Oct 31 19:48:11.509343 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.508820    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8824b6ff-8d47-4df9-a907-8306aa909993-credential-socket\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679\" (UID: \"8824b6ff-8d47-4df9-a907-8306aa909993\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679"
Oct 31 19:48:11.509343 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.508851    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpw72\" (UniqueName: \"kubernetes.io/projected/8824b6ff-8d47-4df9-a907-8306aa909993-kube-api-access-fpw72\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679\" (UID: \"8824b6ff-8d47-4df9-a907-8306aa909993\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679"
Oct 31 19:48:11.509343 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.508892    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8824b6ff-8d47-4df9-a907-8306aa909993-istio-token\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679\" (UID: \"8824b6ff-8d47-4df9-a907-8306aa909993\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679"
Oct 31 19:48:11.509343 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.508920    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8824b6ff-8d47-4df9-a907-8306aa909993-workload-certs\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679\" (UID: \"8824b6ff-8d47-4df9-a907-8306aa909993\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679"
Oct 31 19:48:11.509343 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.508664    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8824b6ff-8d47-4df9-a907-8306aa909993-istio-data\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679\" (UID: \"8824b6ff-8d47-4df9-a907-8306aa909993\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679"
Oct 31 19:48:11.509343 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.509218    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8824b6ff-8d47-4df9-a907-8306aa909993-istiod-ca-cert\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679\" (UID: \"8824b6ff-8d47-4df9-a907-8306aa909993\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679"
Oct 31 19:48:11.509343 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.509272    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b-workload-socket\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g\" (UID: \"f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g"
Oct 31 19:48:11.509343 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.509316    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/452d967b-8434-4766-9b19-690c49adc197-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:11.509703 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.509362    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66r4p\" (UniqueName: \"kubernetes.io/projected/452d967b-8434-4766-9b19-690c49adc197-kube-api-access-66r4p\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:11.509703 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.509393    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/452d967b-8434-4766-9b19-690c49adc197-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:11.509703 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.509408    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/452d967b-8434-4766-9b19-690c49adc197-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:11.510068 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.510042    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b-istiod-ca-cert\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g\" (UID: \"f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g"
Oct 31 19:48:11.510922 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.510896    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b-istio-envoy\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g\" (UID: \"f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g"
Oct 31 19:48:11.511033 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.510990    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b-istio-podinfo\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g\" (UID: \"f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g"
Oct 31 19:48:11.511668 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.511646    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8824b6ff-8d47-4df9-a907-8306aa909993-istio-podinfo\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679\" (UID: \"8824b6ff-8d47-4df9-a907-8306aa909993\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679"
Oct 31 19:48:11.511757 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.511702    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8824b6ff-8d47-4df9-a907-8306aa909993-istio-envoy\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679\" (UID: \"8824b6ff-8d47-4df9-a907-8306aa909993\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679"
Oct 31 19:48:11.521599 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.521571    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8ltp\" (UniqueName: \"kubernetes.io/projected/f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b-kube-api-access-v8ltp\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g\" (UID: \"f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g"
Oct 31 19:48:11.522702 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.522681    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b-istio-token\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g\" (UID: \"f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g"
Oct 31 19:48:11.522783 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.522714    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8824b6ff-8d47-4df9-a907-8306aa909993-istio-token\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679\" (UID: \"8824b6ff-8d47-4df9-a907-8306aa909993\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679"
Oct 31 19:48:11.522838 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.522822    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpw72\" (UniqueName: \"kubernetes.io/projected/8824b6ff-8d47-4df9-a907-8306aa909993-kube-api-access-fpw72\") pod \"e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679\" (UID: \"8824b6ff-8d47-4df9-a907-8306aa909993\") " pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679"
Oct 31 19:48:11.614047 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.614019    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g"
Oct 31 19:48:11.694647 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.693910    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679"
Oct 31 19:48:11.736624 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.736562    4645 generic.go:356] "Generic (PLEG): container finished" podID="981d31f6-7349-4835-81d3-8bbf0408f0d0" containerID="d96ae52208eacbcc7609c5b75fee45de5cc01d64ba1eb915cec770ea57d8f827" exitCode=0
Oct 31 19:48:11.736797 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.736676    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w" event={"ID":"981d31f6-7349-4835-81d3-8bbf0408f0d0","Type":"ContainerDied","Data":"d96ae52208eacbcc7609c5b75fee45de5cc01d64ba1eb915cec770ea57d8f827"}
Oct 31 19:48:11.736797 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.736732    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w" event={"ID":"981d31f6-7349-4835-81d3-8bbf0408f0d0","Type":"ContainerDied","Data":"7f352ef3150f230937aa1424290ae09e7b862bd63ef9b705b9f4734a8f4da679"}
Oct 31 19:48:11.736797 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.736764    4645 scope.go:122] "RemoveContainer" containerID="d96ae52208eacbcc7609c5b75fee45de5cc01d64ba1eb915cec770ea57d8f827"
Oct 31 19:48:11.736990 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.736945    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w"
Oct 31 19:48:11.749070 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.748760    4645 generic.go:356] "Generic (PLEG): container finished" podID="452d967b-8434-4766-9b19-690c49adc197" containerID="28b5fbd07c27a91c35c6eda7f308ee48d6e6d3c9ab9b1ad6f44b700acf30459b" exitCode=0
Oct 31 19:48:11.749070 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.748997    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4" event={"ID":"452d967b-8434-4766-9b19-690c49adc197","Type":"ContainerDied","Data":"28b5fbd07c27a91c35c6eda7f308ee48d6e6d3c9ab9b1ad6f44b700acf30459b"}
Oct 31 19:48:11.749070 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.749040    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4" event={"ID":"452d967b-8434-4766-9b19-690c49adc197","Type":"ContainerDied","Data":"0464259c471388e0e11dcf72a2832bfd707edf68a59f8e617c5c3f59f791d646"}
Oct 31 19:48:11.749298 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.749116    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4"
Oct 31 19:48:11.772664 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.772606    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w"]
Oct 31 19:48:11.781430 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.781277    4645 scope.go:122] "RemoveContainer" containerID="d96ae52208eacbcc7609c5b75fee45de5cc01d64ba1eb915cec770ea57d8f827"
Oct 31 19:48:11.782591 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:48:11.782442    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d96ae52208eacbcc7609c5b75fee45de5cc01d64ba1eb915cec770ea57d8f827\": container with ID starting with d96ae52208eacbcc7609c5b75fee45de5cc01d64ba1eb915cec770ea57d8f827 not found: ID does not exist" containerID="d96ae52208eacbcc7609c5b75fee45de5cc01d64ba1eb915cec770ea57d8f827"
Oct 31 19:48:11.782591 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.782479    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d96ae52208eacbcc7609c5b75fee45de5cc01d64ba1eb915cec770ea57d8f827"} err="failed to get container status \"d96ae52208eacbcc7609c5b75fee45de5cc01d64ba1eb915cec770ea57d8f827\": rpc error: code = NotFound desc = could not find container \"d96ae52208eacbcc7609c5b75fee45de5cc01d64ba1eb915cec770ea57d8f827\": container with ID starting with d96ae52208eacbcc7609c5b75fee45de5cc01d64ba1eb915cec770ea57d8f827 not found: ID does not exist"
Oct 31 19:48:11.782591 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.782503    4645 scope.go:122] "RemoveContainer" containerID="28b5fbd07c27a91c35c6eda7f308ee48d6e6d3c9ab9b1ad6f44b700acf30459b"
Oct 31 19:48:11.785168 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.785150    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-6528w"]
Oct 31 19:48:11.791400 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.791374    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4"]
Oct 31 19:48:11.796281 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.796193    4645 scope.go:122] "RemoveContainer" containerID="28b5fbd07c27a91c35c6eda7f308ee48d6e6d3c9ab9b1ad6f44b700acf30459b"
Oct 31 19:48:11.796598 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:48:11.796540    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28b5fbd07c27a91c35c6eda7f308ee48d6e6d3c9ab9b1ad6f44b700acf30459b\": container with ID starting with 28b5fbd07c27a91c35c6eda7f308ee48d6e6d3c9ab9b1ad6f44b700acf30459b not found: ID does not exist" containerID="28b5fbd07c27a91c35c6eda7f308ee48d6e6d3c9ab9b1ad6f44b700acf30459b"
Oct 31 19:48:11.796598 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.796565    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28b5fbd07c27a91c35c6eda7f308ee48d6e6d3c9ab9b1ad6f44b700acf30459b"} err="failed to get container status \"28b5fbd07c27a91c35c6eda7f308ee48d6e6d3c9ab9b1ad6f44b700acf30459b\": rpc error: code = NotFound desc = could not find container \"28b5fbd07c27a91c35c6eda7f308ee48d6e6d3c9ab9b1ad6f44b700acf30459b\": container with ID starting with 28b5fbd07c27a91c35c6eda7f308ee48d6e6d3c9ab9b1ad6f44b700acf30459b not found: ID does not exist"
Oct 31 19:48:11.797176 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.796854    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-kztj4"]
Oct 31 19:48:11.911262 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.911118    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g"]
Oct 31 19:48:11.913925 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:48:11.913896    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf540b229_b1ea_4fe2_b7ac_a3e3bc140a1b.slice/crio-873fce5daf2b2d684e79a31ff1a3b3430d94f11ccf65ab31a5dcfec22fdc93d0 WatchSource:0}: Error finding container 873fce5daf2b2d684e79a31ff1a3b3430d94f11ccf65ab31a5dcfec22fdc93d0: Status 404 returned error can't find the container with id 873fce5daf2b2d684e79a31ff1a3b3430d94f11ccf65ab31a5dcfec22fdc93d0
Oct 31 19:48:11.916458 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.916422    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:48:11.916561 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.916488    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:48:11.916561 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.916520    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:48:11.965920 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:11.965893    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:12.072571 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:12.071428    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679"]
Oct 31 19:48:12.081363 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:48:12.078802    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8824b6ff_8d47_4df9_a907_8306aa909993.slice/crio-857fdd596bb5e97c022bd228c4aca9f7ab2f37d5fe195c59955034a44ef4dfb5 WatchSource:0}: Error finding container 857fdd596bb5e97c022bd228c4aca9f7ab2f37d5fe195c59955034a44ef4dfb5: Status 404 returned error can't find the container with id 857fdd596bb5e97c022bd228c4aca9f7ab2f37d5fe195c59955034a44ef4dfb5
Oct 31 19:48:12.087443 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:12.084662    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:48:12.087443 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:12.084732    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:48:12.087443 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:12.084766    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:48:12.758707 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:12.758674    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g" event={"ID":"f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b","Type":"ContainerStarted","Data":"8c66143ccc602e9d179ba7b3f611d5f0ef71cc1f87c4bf05a0ff1829a9637efa"}
Oct 31 19:48:12.758894 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:12.758726    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g" event={"ID":"f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b","Type":"ContainerStarted","Data":"873fce5daf2b2d684e79a31ff1a3b3430d94f11ccf65ab31a5dcfec22fdc93d0"}
Oct 31 19:48:12.762462 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:12.762436    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679" event={"ID":"8824b6ff-8d47-4df9-a907-8306aa909993","Type":"ContainerStarted","Data":"4b7f911128454c22b30bd5bee1def6d15118dd4942154af22635dd3ad1d9caa5"}
Oct 31 19:48:12.762575 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:12.762475    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679" event={"ID":"8824b6ff-8d47-4df9-a907-8306aa909993","Type":"ContainerStarted","Data":"857fdd596bb5e97c022bd228c4aca9f7ab2f37d5fe195c59955034a44ef4dfb5"}
Oct 31 19:48:12.784306 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:12.784254    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g" podStartSLOduration=66.784241713 podStartE2EDuration="1m6.784241713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:47:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:48:12.781751137 +0000 UTC m=+1723.354985264" watchObservedRunningTime="2026-10-31 19:48:12.784241713 +0000 UTC m=+1723.357475843"
Oct 31 19:48:12.814149 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:12.814090    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679" podStartSLOduration=66.814075209 podStartE2EDuration="1m6.814075209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:47:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:48:12.813635424 +0000 UTC m=+1723.386869552" watchObservedRunningTime="2026-10-31 19:48:12.814075209 +0000 UTC m=+1723.387309336"
Oct 31 19:48:12.843292 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:12.843261    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:13.616455 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:13.616422    4645 patch_prober.go:29] interesting pod/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Oct 31 19:48:13.616930 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:13.616489    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g" podUID="f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Oct 31 19:48:13.622893 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:13.622865    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="452d967b-8434-4766-9b19-690c49adc197" path="/var/lib/kubelet/pods/452d967b-8434-4766-9b19-690c49adc197/volumes"
Oct 31 19:48:13.623435 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:13.623419    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="981d31f6-7349-4835-81d3-8bbf0408f0d0" path="/var/lib/kubelet/pods/981d31f6-7349-4835-81d3-8bbf0408f0d0/volumes"
Oct 31 19:48:13.623779 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:13.623766    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g"
Oct 31 19:48:13.710377 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:13.703380    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679"
Oct 31 19:48:13.711911 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:13.711873    4645 patch_prober.go:29] interesting pod/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Oct 31 19:48:13.712066 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:13.711935    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679" podUID="8824b6ff-8d47-4df9-a907-8306aa909993" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Oct 31 19:48:13.813394 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:13.811656    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:14.615700 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:14.614681    4645 patch_prober.go:29] interesting pod/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Oct 31 19:48:14.615700 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:14.614727    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g" podUID="f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Oct 31 19:48:14.708793 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:14.708287    4645 patch_prober.go:29] interesting pod/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Oct 31 19:48:14.708793 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:14.708375    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679" podUID="8824b6ff-8d47-4df9-a907-8306aa909993" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Oct 31 19:48:15.092353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:15.091835    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:15.325451 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:15.320266    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:15.486353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:15.477502    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:15.616352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:15.614864    4645 patch_prober.go:29] interesting pod/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Oct 31 19:48:15.616352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:15.614925    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g" podUID="f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Oct 31 19:48:15.712350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:15.698491    4645 patch_prober.go:29] interesting pod/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Oct 31 19:48:15.712350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:15.698549    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679" podUID="8824b6ff-8d47-4df9-a907-8306aa909993" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Oct 31 19:48:16.302689 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:16.302614    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:16.615559 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:16.615467    4645 patch_prober.go:29] interesting pod/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Oct 31 19:48:16.615559 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:16.615527    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g" podUID="f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Oct 31 19:48:16.710355 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:16.709358    4645 patch_prober.go:29] interesting pod/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Oct 31 19:48:16.710355 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:16.709425    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679" podUID="8824b6ff-8d47-4df9-a907-8306aa909993" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Oct 31 19:48:16.827543 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:16.827308    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:16.948037 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:16.948011    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:17.625795 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:17.625755    4645 patch_prober.go:29] interesting pod/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Oct 31 19:48:17.626098 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:17.626063    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g" podUID="f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Oct 31 19:48:17.731345 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:17.723952    4645 patch_prober.go:29] interesting pod/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Oct 31 19:48:17.731345 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:17.724021    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679" podUID="8824b6ff-8d47-4df9-a907-8306aa909993" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Oct 31 19:48:17.739168 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:17.738643    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:17.979353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:17.976374    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:18.614827 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:18.614795    4645 patch_prober.go:29] interesting pod/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Oct 31 19:48:18.615010 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:18.614856    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g" podUID="f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Oct 31 19:48:18.700096 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:18.698542    4645 patch_prober.go:29] interesting pod/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body=
Oct 31 19:48:18.700096 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:18.698626    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679" podUID="8824b6ff-8d47-4df9-a907-8306aa909993" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused"
Oct 31 19:48:18.988949 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:18.988761    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:19.622274 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:19.622243    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g"
Oct 31 19:48:19.622742 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:19.622722    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g"
Oct 31 19:48:19.623483 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:19.623464    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g"
Oct 31 19:48:19.728666 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:19.728632    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679"
Oct 31 19:48:19.729576 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:19.729534    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679"
Oct 31 19:48:19.730300 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:19.730276    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679"
Oct 31 19:48:19.842766 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:19.842733    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:20.713110 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:20.713079    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:21.606033 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:21.606001    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:21.800098 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:21.800061    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-l6wqz"]
Oct 31 19:48:21.800617 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:21.800395    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-l6wqz" podUID="5cc617a4-ee55-4218-9907-1f43b5d0a18a" containerName="istio-proxy" containerID="cri-o://1ee327e16fe4e3f2196d2d2d2d355098434730dae143457b2c46e503404fa44a" gracePeriod=30
Oct 31 19:48:21.804983 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:21.804958    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-msrfk"]
Oct 31 19:48:21.805143 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:21.805120    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-msrfk" podUID="48b0949c-0c89-4c80-a1dc-64d14839f620" containerName="istio-proxy" containerID="cri-o://ae2f8d98f60d251c600ecf1c895d943af254d216228ec8e3d6119638df3befcb" gracePeriod=30
Oct 31 19:48:22.385669 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:22.385639    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:23.262158 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:23.262124    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:24.039881 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:24.039851    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:24.939600 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:24.939569    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:25.858622 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:25.858587    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:26.591500 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:26.591462    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:26.988544 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:26.988358    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-l6wqz" event={"ID":"5cc617a4-ee55-4218-9907-1f43b5d0a18a","Type":"ContainerDied","Data":"1ee327e16fe4e3f2196d2d2d2d355098434730dae143457b2c46e503404fa44a"}
Oct 31 19:48:26.988544 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:26.988310    4645 generic.go:356] "Generic (PLEG): container finished" podID="5cc617a4-ee55-4218-9907-1f43b5d0a18a" containerID="1ee327e16fe4e3f2196d2d2d2d355098434730dae143457b2c46e503404fa44a" exitCode=0
Oct 31 19:48:26.990830 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:26.990803    4645 generic.go:356] "Generic (PLEG): container finished" podID="48b0949c-0c89-4c80-a1dc-64d14839f620" containerID="ae2f8d98f60d251c600ecf1c895d943af254d216228ec8e3d6119638df3befcb" exitCode=0
Oct 31 19:48:26.990966 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:26.990874    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-msrfk" event={"ID":"48b0949c-0c89-4c80-a1dc-64d14839f620","Type":"ContainerDied","Data":"ae2f8d98f60d251c600ecf1c895d943af254d216228ec8e3d6119638df3befcb"}
Oct 31 19:48:27.128813 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:27.128786    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-l6wqz"
Oct 31 19:48:27.161377 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:27.161338    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-msrfk"
Oct 31 19:48:27.170226 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:27.170197    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7c892a-98d9d78c9-qrnvx"]
Oct 31 19:48:27.171061 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:27.171039    4645 cpu_manager.go:414] "containerMap: removing container" podUID="5cc617a4-ee55-4218-9907-1f43b5d0a18a" containerName="istio-proxy"
Oct 31 19:48:27.171061 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:27.171061    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cc617a4-ee55-4218-9907-1f43b5d0a18a" containerName="istio-proxy"
Oct 31 19:48:27.171212 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:27.171128    4645 cpu_manager.go:414] "containerMap: removing container" podUID="48b0949c-0c89-4c80-a1dc-64d14839f620" containerName="istio-proxy"
Oct 31 19:48:27.171212 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:27.171135    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48b0949c-0c89-4c80-a1dc-64d14839f620" containerName="istio-proxy"
Oct 31 19:48:27.171371 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:27.171350    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cc617a4-ee55-4218-9907-1f43b5d0a18a" containerName="istio-proxy"
Oct 31 19:48:27.171450 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:27.171383    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="48b0949c-0c89-4c80-a1dc-64d14839f620" containerName="istio-proxy"
Oct 31 19:48:27.175655 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:27.175633    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7c892a-98d9d78c9-qrnvx"
Oct 31 19:48:27.182833 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:27.180872    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-7c892a\""
Oct 31 19:48:27.182833 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:27.181000    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-7c892a-dockercfg-j2rrp\""
Oct 31 19:48:27.183015 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:27.182981    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7c892a-98d9d78c9-qrnvx"]
Oct 31 19:48:27.283556 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:27.283527    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48b0949c-0c89-4c80-a1dc-64d14839f620-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48b0949c-0c89-4c80-a1dc-64d14839f620-credential-socket\") pod \"48b0949c-0c89-4c80-a1dc-64d14839f620\" (UID: \"48b0949c-0c89-4c80-a1dc-64d14839f620\") "
Oct 31 19:48:27.283708 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:27.283622    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48b0949c-0c89-4c80-a1dc-64d14839f620-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48b0949c-0c89-4c80-a1dc-64d14839f620-workload-socket\") pod \"48b0949c-0c89-4c80-a1dc-64d14839f620\" (UID: \"48b0949c-0c89-4c80-a1dc-64d14839f620\") "
Oct 31 19:48:27.283708 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:27.283650    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48b0949c-0c89-4c80-a1dc-64d14839f620-istio-token\" (UniqueName: \"kubernetes.io/projected/48b0949c-0c89-4c80-a1dc-64d14839f620-istio-token\") pod \"48b0949c-0c89-4c80-a1dc-64d14839f620\" (UID: \"48b0949c-0c89-4c80-a1dc-64d14839f620\") "
Oct 31 19:48:27.283708 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:27.283693    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5cc617a4-ee55-4218-9907-1f43b5d0a18a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cc617a4-ee55-4218-9907-1f43b5d0a18a-istio-podinfo\") pod \"5cc617a4-ee55-4218-9907-1f43b5d0a18a\" (UID: \"5cc617a4-ee55-4218-9907-1f43b5d0a18a\") "
Oct 31 19:48:27.283859 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:27.283727    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/48b0949c-0c89-4c80-a1dc-64d14839f620-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48b0949c-0c89-4c80-a1dc-64d14839f620-istio-podinfo\") pod \"48b0949c-0c89-4c80-a1dc-64d14839f620\" (UID: \"48b0949c-0c89-4c80-a1dc-64d14839f620\") "
Oct 31 19:48:27.283859 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:27.283776    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cc617a4-ee55-4218-9907-1f43b5d0a18a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cc617a4-ee55-4218-9907-1f43b5d0a18a-istio-envoy\") pod \"5cc617a4-ee55-4218-9907-1f43b5d0a18a\" (UID: \"5cc617a4-ee55-4218-9907-1f43b5d0a18a\") "
Oct 31 19:48:27.283859 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:27.283779    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48b0949c-0c89-4c80-a1dc-64d14839f620-credential-socket" pod "48b0949c-0c89-4c80-a1dc-64d14839f620" (UID: "48b0949c-0c89-4c80-a1dc-64d14839f620"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:48:27.284023 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:27.283851    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48b0949c-0c89-4c80-a1dc-64d14839f620-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48b0949c-0c89-4c80-a1dc-64d14839f620-workload-certs\") pod \"48b0949c-0c89-4c80-a1dc-64d14839f620\" (UID: \"48b0949c-0c89-4c80-a1dc-64d14839f620\") "
Oct 31 19:48:27.284023 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:27.283898    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48b0949c-0c89-4c80-a1dc-64d14839f620-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48b0949c-0c89-4c80-a1dc-64d14839f620-istiod-ca-cert\") pod \"48b0949c-0c89-4c80-a1dc-64d14839f620\" (UID: \"48b0949c-0c89-4c80-a1dc-64d14839f620\") "
Oct 31 19:48:27.284023 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:27.283930    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cc617a4-ee55-4218-9907-1f43b5d0a18a-kube-api-access-kch7w\" (UniqueName: \"kubernetes.io/projected/5cc617a4-ee55-4218-9907-1f43b5d0a18a-kube-api-access-kch7w\") pod \"5cc617a4-ee55-4218-9907-1f43b5d0a18a\" (UID: \"5cc617a4-ee55-4218-9907-1f43b5d0a18a\") "
Oct 31 19:48:27.284023 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:27.283943    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48b0949c-0c89-4c80-a1dc-64d14839f620-workload-socket" pod "48b0949c-0c89-4c80-a1dc-64d14839f620" (UID: "48b0949c-0c89-4c80-a1dc-64d14839f620"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:48:27.284023 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:27.284013    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cc617a4-ee55-4218-9907-1f43b5d0a18a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cc617a4-ee55-4218-9907-1f43b5d0a18a-istiod-ca-cert\") pod \"5cc617a4-ee55-4218-9907-1f43b5d0a18a\" (UID: \"5cc617a4-ee55-4218-9907-1f43b5d0a18a\") "
Oct 31 19:48:27.284385 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:27.284069    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48b0949c-0c89-4c80-a1dc-64d14839f620-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48b0949c-0c89-4c80-a1dc-64d14839f620-istio-envoy\") pod \"48b0949c-0c89-4c80-a1dc-64d14839f620\" (UID: \"48b0949c-0c89-4c80-a1dc-64d14839f620\") "
Oct 31 19:48:27.284385 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:27.284100    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48b0949c-0c89-4c80-a1dc-64d14839f620-workload-certs" pod "48b0949c-0c89-4c80-a1dc-64d14839f620" (UID: "48b0949c-0c89-4c80-a1dc-64d14839f620"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:48:27.284385 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:27.284283    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48b0949c-0c89-4c80-a1dc-64d14839f620-istiod-ca-cert" pod "48b0949c-0c89-4c80-a1dc-64d14839f620" (UID: "48b0949c-0c89-4c80-a1dc-64d14839f620"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:48:27.284385 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:27.284287    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cc617a4-ee55-4218-9907-1f43b5d0a18a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cc617a4-ee55-4218-9907-1f43b5d0a18a-credential-socket\") pod \"5cc617a4-ee55-4218-9907-1f43b5d0a18a\" (UID: \"5cc617a4-ee55-4218-9907-1f43b5d0a18a\") "
Oct 31 19:48:27.284621 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:27.284404    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48b0949c-0c89-4c80-a1dc-64d14839f620-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48b0949c-0c89-4c80-a1dc-64d14839f620-istio-data\") pod \"48b0949c-0c89-4c80-a1dc-64d14839f620\" (UID: \"48b0949c-0c89-4c80-a1dc-64d14839f620\") "
Oct 31 19:48:27.284621 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:27.284437    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cc617a4-ee55-4218-9907-1f43b5d0a18a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cc617a4-ee55-4218-9907-1f43b5d0a18a-workload-socket\") pod \"5cc617a4-ee55-4218-9907-1f43b5d0a18a\" (UID: \"5cc617a4-ee55-4218-9907-1f43b5d0a18a\") "
Oct 31 19:48:27.284621 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:27.284527    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cc617a4-ee55-4218-9907-1f43b5d0a18a-istiod-ca-cert" pod "5cc617a4-ee55-4218-9907-1f43b5d0a18a" (UID: "5cc617a4-ee55-4218-9907-1f43b5d0a18a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:48:27.284798 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:27.284626    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cc617a4-ee55-4218-9907-1f43b5d0a18a-credential-socket" pod "5cc617a4-ee55-4218-9907-1f43b5d0a18a" (UID: "5cc617a4-ee55-4218-9907-1f43b5d0a18a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:48:27.284798 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:27.284653    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48b0949c-0c89-4c80-a1dc-64d14839f620-istio-data" pod "48b0949c-0c89-4c80-a1dc-64d14839f620" (UID: "48b0949c-0c89-4c80-a1dc-64d14839f620"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:48:27.284798 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:27.284712    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cc617a4-ee55-4218-9907-1f43b5d0a18a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cc617a4-ee55-4218-9907-1f43b5d0a18a-workload-certs\") pod \"5cc617a4-ee55-4218-9907-1f43b5d0a18a\" (UID: \"5cc617a4-ee55-4218-9907-1f43b5d0a18a\") "
Oct 31 19:48:27.284969 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:27.284811    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cc617a4-ee55-4218-9907-1f43b5d0a18a-workload-socket" pod "5cc617a4-ee55-4218-9907-1f43b5d0a18a" (UID: "5cc617a4-ee55-4218-9907-1f43b5d0a18a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:48:27.284969 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:27.284920    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48b0949c-0c89-4c80-a1dc-64d14839f620-kube-api-access-kldvq\" (UniqueName: \"kubernetes.io/projected/48b0949c-0c89-4c80-a1dc-64d14839f620-kube-api-access-kldvq\") pod \"48b0949c-0c89-4c80-a1dc-64d14839f620\" (UID: \"48b0949c-0c89-4c80-a1dc-64d14839f620\") "
Oct 31 19:48:27.285137 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:27.284970    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cc617a4-ee55-4218-9907-1f43b5d0a18a-workload-certs" pod "5cc617a4-ee55-4218-9907-1f43b5d0a18a" (UID: "5cc617a4-ee55-4218-9907-1f43b5d0a18a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:48:27.285414 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:27.285262    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cc617a4-ee55-4218-9907-1f43b5d0a18a-istio-token\" (UniqueName: \"kubernetes.io/projected/5cc617a4-ee55-4218-9907-1f43b5d0a18a-istio-token\") pod \"5cc617a4-ee55-4218-9907-1f43b5d0a18a\" (UID: \"5cc617a4-ee55-4218-9907-1f43b5d0a18a\") "
Oct 31 19:48:27.285414 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:27.285338    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cc617a4-ee55-4218-9907-1f43b5d0a18a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cc617a4-ee55-4218-9907-1f43b5d0a18a-istio-data\") pod \"5cc617a4-ee55-4218-9907-1f43b5d0a18a\" (UID: \"5cc617a4-ee55-4218-9907-1f43b5d0a18a\") "
Oct 31 19:48:27.285672 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:27.285654    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cc617a4-ee55-4218-9907-1f43b5d0a18a-istio-data" pod "5cc617a4-ee55-4218-9907-1f43b5d0a18a" (UID: "5cc617a4-ee55-4218-9907-1f43b5d0a18a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:48:27.286474 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:27.286444    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dad64038-a18c-4944-80c8-b3f052a98dbb-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-7c892a-98d9d78c9-qrnvx\" (UID: \"dad64038-a18c-4944-80c8-b3f052a98dbb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7c892a-98d9d78c9-qrnvx"
Oct 31 19:48:27.286474 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:27.286450    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48b0949c-0c89-4c80-a1dc-64d14839f620-istio-podinfo" pod "48b0949c-0c89-4c80-a1dc-64d14839f620" (UID: "48b0949c-0c89-4c80-a1dc-64d14839f620"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:48:27.286727 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:27.286545    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cc617a4-ee55-4218-9907-1f43b5d0a18a-istio-envoy" pod "5cc617a4-ee55-4218-9907-1f43b5d0a18a" (UID: "5cc617a4-ee55-4218-9907-1f43b5d0a18a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:48:27.286727 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:27.286696    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f946j\" (UniqueName: \"kubernetes.io/projected/dad64038-a18c-4944-80c8-b3f052a98dbb-kube-api-access-f946j\") pod \"payload-pre-processing-e2e-inf-b-7c892a-98d9d78c9-qrnvx\" (UID: \"dad64038-a18c-4944-80c8-b3f052a98dbb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7c892a-98d9d78c9-qrnvx"
Oct 31 19:48:27.286991 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:27.286971    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48b0949c-0c89-4c80-a1dc-64d14839f620-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:27.287074 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:27.286998    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48b0949c-0c89-4c80-a1dc-64d14839f620-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:27.287074 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:27.287012    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48b0949c-0c89-4c80-a1dc-64d14839f620-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:27.287074 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:27.287026    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cc617a4-ee55-4218-9907-1f43b5d0a18a-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:27.287074 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:27.287041    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48b0949c-0c89-4c80-a1dc-64d14839f620-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:27.287074 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:27.287053    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48b0949c-0c89-4c80-a1dc-64d14839f620-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:27.287074 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:27.287065    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cc617a4-ee55-4218-9907-1f43b5d0a18a-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:27.287074 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:27.287077    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cc617a4-ee55-4218-9907-1f43b5d0a18a-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:27.287496 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:27.287089    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48b0949c-0c89-4c80-a1dc-64d14839f620-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:27.287496 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:27.287104    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cc617a4-ee55-4218-9907-1f43b5d0a18a-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:27.287496 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:27.287116    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cc617a4-ee55-4218-9907-1f43b5d0a18a-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:27.287496 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:27.287127    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cc617a4-ee55-4218-9907-1f43b5d0a18a-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:27.287496 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:27.287214    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48b0949c-0c89-4c80-a1dc-64d14839f620-istio-envoy" pod "48b0949c-0c89-4c80-a1dc-64d14839f620" (UID: "48b0949c-0c89-4c80-a1dc-64d14839f620"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:48:27.287496 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:27.287275    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48b0949c-0c89-4c80-a1dc-64d14839f620-istio-token" pod "48b0949c-0c89-4c80-a1dc-64d14839f620" (UID: "48b0949c-0c89-4c80-a1dc-64d14839f620"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:48:27.287496 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:27.287348    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5cc617a4-ee55-4218-9907-1f43b5d0a18a-istio-podinfo" pod "5cc617a4-ee55-4218-9907-1f43b5d0a18a" (UID: "5cc617a4-ee55-4218-9907-1f43b5d0a18a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:48:27.287496 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:27.287382    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48b0949c-0c89-4c80-a1dc-64d14839f620-kube-api-access-kldvq" pod "48b0949c-0c89-4c80-a1dc-64d14839f620" (UID: "48b0949c-0c89-4c80-a1dc-64d14839f620"). InnerVolumeSpecName "kube-api-access-kldvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:48:27.287496 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:27.287446    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cc617a4-ee55-4218-9907-1f43b5d0a18a-kube-api-access-kch7w" pod "5cc617a4-ee55-4218-9907-1f43b5d0a18a" (UID: "5cc617a4-ee55-4218-9907-1f43b5d0a18a"). InnerVolumeSpecName "kube-api-access-kch7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:48:27.288461 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:27.288443    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cc617a4-ee55-4218-9907-1f43b5d0a18a-istio-token" pod "5cc617a4-ee55-4218-9907-1f43b5d0a18a" (UID: "5cc617a4-ee55-4218-9907-1f43b5d0a18a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:48:27.397791 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:27.397686    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f946j\" (UniqueName: \"kubernetes.io/projected/dad64038-a18c-4944-80c8-b3f052a98dbb-kube-api-access-f946j\") pod \"payload-pre-processing-e2e-inf-b-7c892a-98d9d78c9-qrnvx\" (UID: \"dad64038-a18c-4944-80c8-b3f052a98dbb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7c892a-98d9d78c9-qrnvx"
Oct 31 19:48:27.398431 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:27.398392    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dad64038-a18c-4944-80c8-b3f052a98dbb-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-7c892a-98d9d78c9-qrnvx\" (UID: \"dad64038-a18c-4944-80c8-b3f052a98dbb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7c892a-98d9d78c9-qrnvx"
Oct 31 19:48:27.398906 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:27.398885    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48b0949c-0c89-4c80-a1dc-64d14839f620-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:27.399068 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:27.399042    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kldvq\" (UniqueName: \"kubernetes.io/projected/48b0949c-0c89-4c80-a1dc-64d14839f620-kube-api-access-kldvq\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:27.399175 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:27.399162    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cc617a4-ee55-4218-9907-1f43b5d0a18a-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:27.399264 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:27.399253    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48b0949c-0c89-4c80-a1dc-64d14839f620-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:27.399355 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:27.399344    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cc617a4-ee55-4218-9907-1f43b5d0a18a-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:27.399436 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:27.399425    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kch7w\" (UniqueName: \"kubernetes.io/projected/5cc617a4-ee55-4218-9907-1f43b5d0a18a-kube-api-access-kch7w\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:48:27.400475 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:27.400436    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dad64038-a18c-4944-80c8-b3f052a98dbb-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-7c892a-98d9d78c9-qrnvx\" (UID: \"dad64038-a18c-4944-80c8-b3f052a98dbb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7c892a-98d9d78c9-qrnvx"
Oct 31 19:48:27.401761 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:27.401603    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:27.412552 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:27.412524    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f946j\" (UniqueName: \"kubernetes.io/projected/dad64038-a18c-4944-80c8-b3f052a98dbb-kube-api-access-f946j\") pod \"payload-pre-processing-e2e-inf-b-7c892a-98d9d78c9-qrnvx\" (UID: \"dad64038-a18c-4944-80c8-b3f052a98dbb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7c892a-98d9d78c9-qrnvx"
Oct 31 19:48:27.533190 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:27.533157    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7c892a-98d9d78c9-qrnvx"
Oct 31 19:48:27.822486 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:48:27.822456    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddad64038_a18c_4944_80c8_b3f052a98dbb.slice/crio-f8a56bee5ec8381858d7d72c43bd85e82a70ac7a4f61859beacec99835cf1fd1 WatchSource:0}: Error finding container f8a56bee5ec8381858d7d72c43bd85e82a70ac7a4f61859beacec99835cf1fd1: Status 404 returned error can't find the container with id f8a56bee5ec8381858d7d72c43bd85e82a70ac7a4f61859beacec99835cf1fd1
Oct 31 19:48:27.822813 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:27.822664    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7c892a-98d9d78c9-qrnvx"]
Oct 31 19:48:28.000931 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:28.000898    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7c892a-98d9d78c9-qrnvx" event={"ID":"dad64038-a18c-4944-80c8-b3f052a98dbb","Type":"ContainerStarted","Data":"a3ed5340dd2ca8710d32df3a1e01e31447636ccd988cad30a18044d290cbe69f"}
Oct 31 19:48:28.001072 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:28.000970    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7c892a-98d9d78c9-qrnvx"
Oct 31 19:48:28.001072 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:28.000992    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7c892a-98d9d78c9-qrnvx" event={"ID":"dad64038-a18c-4944-80c8-b3f052a98dbb","Type":"ContainerStarted","Data":"f8a56bee5ec8381858d7d72c43bd85e82a70ac7a4f61859beacec99835cf1fd1"}
Oct 31 19:48:28.002393 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:28.002371    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-l6wqz" event={"ID":"5cc617a4-ee55-4218-9907-1f43b5d0a18a","Type":"ContainerDied","Data":"1b01726061b9fa14bd5dc8abd9590b2b8c6f83d357c826bb033ce161a9a189b7"}
Oct 31 19:48:28.002497 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:28.002433    4645 scope.go:122] "RemoveContainer" containerID="1ee327e16fe4e3f2196d2d2d2d355098434730dae143457b2c46e503404fa44a"
Oct 31 19:48:28.002497 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:28.002379    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-l6wqz"
Oct 31 19:48:28.004635 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:28.003999    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-msrfk"
Oct 31 19:48:28.004635 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:28.004012    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-msrfk" event={"ID":"48b0949c-0c89-4c80-a1dc-64d14839f620","Type":"ContainerDied","Data":"4776e22c2b14ec115176f74ad372e5f5b3c1a8e12e852e1ab9e1cc58298eb863"}
Oct 31 19:48:28.013413 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:28.013386    4645 scope.go:122] "RemoveContainer" containerID="ae2f8d98f60d251c600ecf1c895d943af254d216228ec8e3d6119638df3befcb"
Oct 31 19:48:28.022518 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:28.022462    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7c892a-98d9d78c9-qrnvx" podStartSLOduration=12.022446889 podStartE2EDuration="12.022446889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:48:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:48:28.019192001 +0000 UTC m=+1738.592426129" watchObservedRunningTime="2026-10-31 19:48:28.022446889 +0000 UTC m=+1738.595681017"
Oct 31 19:48:28.038430 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:28.038406    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-msrfk"]
Oct 31 19:48:28.045082 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:28.045061    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-msrfk"]
Oct 31 19:48:28.057978 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:28.057956    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-l6wqz"]
Oct 31 19:48:28.064583 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:28.064561    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-l6wqz"]
Oct 31 19:48:28.067244 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:28.067225    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:28.840220 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:28.840165    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:29.922034 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:29.908587    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:29.996718 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:29.994963    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48b0949c-0c89-4c80-a1dc-64d14839f620" path="/var/lib/kubelet/pods/48b0949c-0c89-4c80-a1dc-64d14839f620/volumes"
Oct 31 19:48:29.996718 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:29.996004    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cc617a4-ee55-4218-9907-1f43b5d0a18a" path="/var/lib/kubelet/pods/5cc617a4-ee55-4218-9907-1f43b5d0a18a/volumes"
Oct 31 19:48:30.847879 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:30.847838    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:31.548945 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:31.548905    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:32.512214 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:32.512155    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:33.324701 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:33.324666    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:34.142821 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:34.142784    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:34.941478 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:34.941439    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:35.742111 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:35.742073    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:36.252493 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:48:36.252458    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 19:48:36.557644 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:36.557554    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:37.339994 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:37.339961    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:38.206525 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:38.206489    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:38.956796 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:38.956764    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:39.020276 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:39.020242    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7c892a-98d9d78c9-qrnvx"
Oct 31 19:48:39.738939 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:39.738903    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:40.376930 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:40.376895    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7c892a-6f48cdf487-v46nr"]
Oct 31 19:48:40.407353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:40.404853    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7c892a-6f48cdf487-v46nr"
Oct 31 19:48:40.412188 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:40.412156    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-7c892a-serving-cert\""
Oct 31 19:48:40.426039 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:40.426008    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7c892a-6f48cdf487-v46nr"]
Oct 31 19:48:40.431441 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:40.431401    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:40.437620 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:40.437596    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88gsc\" (UniqueName: \"kubernetes.io/projected/4d444a98-d697-4fe3-ba5c-ed7158998ab7-kube-api-access-88gsc\") pod \"maas-api-e2e-inf-b-7c892a-6f48cdf487-v46nr\" (UID: \"4d444a98-d697-4fe3-ba5c-ed7158998ab7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7c892a-6f48cdf487-v46nr"
Oct 31 19:48:40.437872 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:40.437850    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d444a98-d697-4fe3-ba5c-ed7158998ab7-openshift-service-ca\") pod \"maas-api-e2e-inf-b-7c892a-6f48cdf487-v46nr\" (UID: \"4d444a98-d697-4fe3-ba5c-ed7158998ab7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7c892a-6f48cdf487-v46nr"
Oct 31 19:48:40.437954 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:40.437903    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d444a98-d697-4fe3-ba5c-ed7158998ab7-maas-api-tls\") pod \"maas-api-e2e-inf-b-7c892a-6f48cdf487-v46nr\" (UID: \"4d444a98-d697-4fe3-ba5c-ed7158998ab7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7c892a-6f48cdf487-v46nr"
Oct 31 19:48:40.438070 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:40.438050    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d444a98-d697-4fe3-ba5c-ed7158998ab7-metrics-certs\") pod \"maas-api-e2e-inf-b-7c892a-6f48cdf487-v46nr\" (UID: \"4d444a98-d697-4fe3-ba5c-ed7158998ab7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7c892a-6f48cdf487-v46nr"
Oct 31 19:48:40.538416 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:40.538381    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:40.541623 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:40.540089    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88gsc\" (UniqueName: \"kubernetes.io/projected/4d444a98-d697-4fe3-ba5c-ed7158998ab7-kube-api-access-88gsc\") pod \"maas-api-e2e-inf-b-7c892a-6f48cdf487-v46nr\" (UID: \"4d444a98-d697-4fe3-ba5c-ed7158998ab7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7c892a-6f48cdf487-v46nr"
Oct 31 19:48:40.542166 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:40.542122    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d444a98-d697-4fe3-ba5c-ed7158998ab7-openshift-service-ca\") pod \"maas-api-e2e-inf-b-7c892a-6f48cdf487-v46nr\" (UID: \"4d444a98-d697-4fe3-ba5c-ed7158998ab7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7c892a-6f48cdf487-v46nr"
Oct 31 19:48:40.547537 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:40.547507    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d444a98-d697-4fe3-ba5c-ed7158998ab7-maas-api-tls\") pod \"maas-api-e2e-inf-b-7c892a-6f48cdf487-v46nr\" (UID: \"4d444a98-d697-4fe3-ba5c-ed7158998ab7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7c892a-6f48cdf487-v46nr"
Oct 31 19:48:40.547866 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:40.547844    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d444a98-d697-4fe3-ba5c-ed7158998ab7-metrics-certs\") pod \"maas-api-e2e-inf-b-7c892a-6f48cdf487-v46nr\" (UID: \"4d444a98-d697-4fe3-ba5c-ed7158998ab7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7c892a-6f48cdf487-v46nr"
Oct 31 19:48:40.548061 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:48:40.548037    4645 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-7c892a-serving-cert: secret "maas-api-e2e-inf-b-7c892a-serving-cert" not found
Oct 31 19:48:40.548366 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:40.547900    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d444a98-d697-4fe3-ba5c-ed7158998ab7-openshift-service-ca\") pod \"maas-api-e2e-inf-b-7c892a-6f48cdf487-v46nr\" (UID: \"4d444a98-d697-4fe3-ba5c-ed7158998ab7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7c892a-6f48cdf487-v46nr"
Oct 31 19:48:40.562363 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:48:40.559517    4645 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d444a98-d697-4fe3-ba5c-ed7158998ab7-maas-api-tls podName:4d444a98-d697-4fe3-ba5c-ed7158998ab7 nodeName:}" failed. No retries permitted until 2026-10-31 19:48:41.050214773 +0000 UTC m=+1751.623448886 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4d444a98-d697-4fe3-ba5c-ed7158998ab7-maas-api-tls") pod "maas-api-e2e-inf-b-7c892a-6f48cdf487-v46nr" (UID: "4d444a98-d697-4fe3-ba5c-ed7158998ab7") : secret "maas-api-e2e-inf-b-7c892a-serving-cert" not found
Oct 31 19:48:40.581143 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:40.581116    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d444a98-d697-4fe3-ba5c-ed7158998ab7-metrics-certs\") pod \"maas-api-e2e-inf-b-7c892a-6f48cdf487-v46nr\" (UID: \"4d444a98-d697-4fe3-ba5c-ed7158998ab7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7c892a-6f48cdf487-v46nr"
Oct 31 19:48:40.581598 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:40.581578    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88gsc\" (UniqueName: \"kubernetes.io/projected/4d444a98-d697-4fe3-ba5c-ed7158998ab7-kube-api-access-88gsc\") pod \"maas-api-e2e-inf-b-7c892a-6f48cdf487-v46nr\" (UID: \"4d444a98-d697-4fe3-ba5c-ed7158998ab7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7c892a-6f48cdf487-v46nr"
Oct 31 19:48:40.902831 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:40.902793    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:41.077351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:41.067138    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d444a98-d697-4fe3-ba5c-ed7158998ab7-maas-api-tls\") pod \"maas-api-e2e-inf-b-7c892a-6f48cdf487-v46nr\" (UID: \"4d444a98-d697-4fe3-ba5c-ed7158998ab7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7c892a-6f48cdf487-v46nr"
Oct 31 19:48:41.077351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:41.072088    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d444a98-d697-4fe3-ba5c-ed7158998ab7-maas-api-tls\") pod \"maas-api-e2e-inf-b-7c892a-6f48cdf487-v46nr\" (UID: \"4d444a98-d697-4fe3-ba5c-ed7158998ab7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7c892a-6f48cdf487-v46nr"
Oct 31 19:48:41.337456 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:41.337421    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7c892a-6f48cdf487-v46nr"
Oct 31 19:48:41.412720 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:41.412585    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:41.627544 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:41.627515    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7c892a-6f48cdf487-v46nr"]
Oct 31 19:48:41.627663 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:48:41.627645    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d444a98_d697_4fe3_ba5c_ed7158998ab7.slice/crio-7dc8d8e22c0a10802100d121b9ea2df0d68aa227ee16c007f5623126eafceb11 WatchSource:0}: Error finding container 7dc8d8e22c0a10802100d121b9ea2df0d68aa227ee16c007f5623126eafceb11: Status 404 returned error can't find the container with id 7dc8d8e22c0a10802100d121b9ea2df0d68aa227ee16c007f5623126eafceb11
Oct 31 19:48:42.086978 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:42.086948    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:42.139889 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:42.139859    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7c892a-6f48cdf487-v46nr" event={"ID":"4d444a98-d697-4fe3-ba5c-ed7158998ab7","Type":"ContainerStarted","Data":"ab24419c9537a6219a4369c7bf63ee457dc3085cab2df6b5c8fa7e7b95e0c2c9"}
Oct 31 19:48:42.140027 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:42.139921    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7c892a-6f48cdf487-v46nr" event={"ID":"4d444a98-d697-4fe3-ba5c-ed7158998ab7","Type":"ContainerStarted","Data":"7dc8d8e22c0a10802100d121b9ea2df0d68aa227ee16c007f5623126eafceb11"}
Oct 31 19:48:42.140027 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:42.139964    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7c892a-6f48cdf487-v46nr"
Oct 31 19:48:42.167944 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:42.167898    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7c892a-6f48cdf487-v46nr" podStartSLOduration=1.801019037 podStartE2EDuration="2.16788534s" totalImagesPullingTime="366.866303ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:48:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:48:42.162176242 +0000 UTC m=+1752.735410370" watchObservedRunningTime="2026-10-31 19:48:42.16788534 +0000 UTC m=+1752.741119469"
Oct 31 19:48:42.206443 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:42.206414    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:42.312119 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:42.312043    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:43.137245 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:43.137213    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:44.162960 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:44.162923    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:44.968030 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:44.967995    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:45.700517 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:45.700485    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:46.458863 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:46.458829    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:47.176947 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:47.176911    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:48.060778 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:48.060745    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:48.156965 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:48.156934    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7c892a-6f48cdf487-v46nr"
Oct 31 19:48:48.945685 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:48.945531    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:49.628274 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:49.628239    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:50.347149 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:50.347114    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:51.138504 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:51.138465    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:51.246064 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:51.246026    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:51.990167 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:51.990137    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:52.751260 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:52.751225    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:53.460473 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:53.460421    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:54.268078 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:54.268033    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:54.966895 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:54.966858    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:55.666472 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:55.666437    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:56.454861 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:56.454824    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:57.245738 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:57.245687    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:58.104300 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:58.104244    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:58.852807 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:58.852767    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:48:59.541288 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:48:59.541248    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:00.296135 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:00.296100    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:01.051463 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:01.051428    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:01.838611 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:01.837237    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:02.552117 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:02.552084    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:03.335205 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:03.335065    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:04.036284 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:04.036255    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:05.191715 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:05.191682    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:05.789976 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:05.789942    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf"]
Oct 31 19:49:05.790175 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:05.790155    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf" podUID="506d452a-2e17-44c0-9b4e-c7ea88e7e5bf" containerName="istio-proxy" containerID="cri-o://3b2ed82f1f816debce75a6206b7bc06a278d5d8a913b9a23776168803d3123a9" gracePeriod=30
Oct 31 19:49:06.121807 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:06.121720    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:06.255614 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:49:06.255498    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 19:49:06.258054 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:06.257989    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:06.938712 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:06.938677    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:07.456596 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:07.456563    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-754e0a/test-model-754e0a-kserve-767c88f75f-8xwns"]
Oct 31 19:49:07.463235 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:07.463206    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-754e0a/test-model-754e0a-kserve-767c88f75f-8xwns"
Oct 31 19:49:07.471385 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:07.471357    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-754e0a\"/\"kube-root-ca.crt\""
Oct 31 19:49:07.471648 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:07.471623    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-754e0a\"/\"openshift-service-ca.crt\""
Oct 31 19:49:07.478321 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:07.478289    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-754e0a/test-model-754e0a-kserve-767c88f75f-8xwns"]
Oct 31 19:49:07.483614 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:07.483589    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-754e0a\"/\"test-model-754e0a-kserve-self-signed-certs\""
Oct 31 19:49:07.483735 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:07.483661    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-754e0a\"/\"default-dockercfg-kg49l\""
Oct 31 19:49:07.562176 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:07.562135    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55751f79-d0e3-474e-9623-5e37f8d1f396-tmp-dir\") pod \"test-model-754e0a-kserve-767c88f75f-8xwns\" (UID: \"55751f79-d0e3-474e-9623-5e37f8d1f396\") " pod="ai-tenant-e2e-inf-a-754e0a/test-model-754e0a-kserve-767c88f75f-8xwns"
Oct 31 19:49:07.562387 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:07.562280    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55751f79-d0e3-474e-9623-5e37f8d1f396-model-cache\") pod \"test-model-754e0a-kserve-767c88f75f-8xwns\" (UID: \"55751f79-d0e3-474e-9623-5e37f8d1f396\") " pod="ai-tenant-e2e-inf-a-754e0a/test-model-754e0a-kserve-767c88f75f-8xwns"
Oct 31 19:49:07.562387 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:07.562306    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55751f79-d0e3-474e-9623-5e37f8d1f396-tls-certs\") pod \"test-model-754e0a-kserve-767c88f75f-8xwns\" (UID: \"55751f79-d0e3-474e-9623-5e37f8d1f396\") " pod="ai-tenant-e2e-inf-a-754e0a/test-model-754e0a-kserve-767c88f75f-8xwns"
Oct 31 19:49:07.562387 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:07.562371    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55751f79-d0e3-474e-9623-5e37f8d1f396-dshm\") pod \"test-model-754e0a-kserve-767c88f75f-8xwns\" (UID: \"55751f79-d0e3-474e-9623-5e37f8d1f396\") " pod="ai-tenant-e2e-inf-a-754e0a/test-model-754e0a-kserve-767c88f75f-8xwns"
Oct 31 19:49:07.562600 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:07.562583    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkzhb\" (UniqueName: \"kubernetes.io/projected/55751f79-d0e3-474e-9623-5e37f8d1f396-kube-api-access-hkzhb\") pod \"test-model-754e0a-kserve-767c88f75f-8xwns\" (UID: \"55751f79-d0e3-474e-9623-5e37f8d1f396\") " pod="ai-tenant-e2e-inf-a-754e0a/test-model-754e0a-kserve-767c88f75f-8xwns"
Oct 31 19:49:07.562645 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:07.562619    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55751f79-d0e3-474e-9623-5e37f8d1f396-home\") pod \"test-model-754e0a-kserve-767c88f75f-8xwns\" (UID: \"55751f79-d0e3-474e-9623-5e37f8d1f396\") " pod="ai-tenant-e2e-inf-a-754e0a/test-model-754e0a-kserve-767c88f75f-8xwns"
Oct 31 19:49:07.686354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:07.666215    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55751f79-d0e3-474e-9623-5e37f8d1f396-tmp-dir\") pod \"test-model-754e0a-kserve-767c88f75f-8xwns\" (UID: \"55751f79-d0e3-474e-9623-5e37f8d1f396\") " pod="ai-tenant-e2e-inf-a-754e0a/test-model-754e0a-kserve-767c88f75f-8xwns"
Oct 31 19:49:07.754219 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:07.754124    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55751f79-d0e3-474e-9623-5e37f8d1f396-model-cache\") pod \"test-model-754e0a-kserve-767c88f75f-8xwns\" (UID: \"55751f79-d0e3-474e-9623-5e37f8d1f396\") " pod="ai-tenant-e2e-inf-a-754e0a/test-model-754e0a-kserve-767c88f75f-8xwns"
Oct 31 19:49:07.754484 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:07.754459    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55751f79-d0e3-474e-9623-5e37f8d1f396-tls-certs\") pod \"test-model-754e0a-kserve-767c88f75f-8xwns\" (UID: \"55751f79-d0e3-474e-9623-5e37f8d1f396\") " pod="ai-tenant-e2e-inf-a-754e0a/test-model-754e0a-kserve-767c88f75f-8xwns"
Oct 31 19:49:07.754629 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:07.754615    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55751f79-d0e3-474e-9623-5e37f8d1f396-dshm\") pod \"test-model-754e0a-kserve-767c88f75f-8xwns\" (UID: \"55751f79-d0e3-474e-9623-5e37f8d1f396\") " pod="ai-tenant-e2e-inf-a-754e0a/test-model-754e0a-kserve-767c88f75f-8xwns"
Oct 31 19:49:07.755072 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:07.755051    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkzhb\" (UniqueName: \"kubernetes.io/projected/55751f79-d0e3-474e-9623-5e37f8d1f396-kube-api-access-hkzhb\") pod \"test-model-754e0a-kserve-767c88f75f-8xwns\" (UID: \"55751f79-d0e3-474e-9623-5e37f8d1f396\") " pod="ai-tenant-e2e-inf-a-754e0a/test-model-754e0a-kserve-767c88f75f-8xwns"
Oct 31 19:49:07.755226 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:07.755198    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55751f79-d0e3-474e-9623-5e37f8d1f396-home\") pod \"test-model-754e0a-kserve-767c88f75f-8xwns\" (UID: \"55751f79-d0e3-474e-9623-5e37f8d1f396\") " pod="ai-tenant-e2e-inf-a-754e0a/test-model-754e0a-kserve-767c88f75f-8xwns"
Oct 31 19:49:07.756084 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:07.756062    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55751f79-d0e3-474e-9623-5e37f8d1f396-home\") pod \"test-model-754e0a-kserve-767c88f75f-8xwns\" (UID: \"55751f79-d0e3-474e-9623-5e37f8d1f396\") " pod="ai-tenant-e2e-inf-a-754e0a/test-model-754e0a-kserve-767c88f75f-8xwns"
Oct 31 19:49:07.764346 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:07.757916    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55751f79-d0e3-474e-9623-5e37f8d1f396-model-cache\") pod \"test-model-754e0a-kserve-767c88f75f-8xwns\" (UID: \"55751f79-d0e3-474e-9623-5e37f8d1f396\") " pod="ai-tenant-e2e-inf-a-754e0a/test-model-754e0a-kserve-767c88f75f-8xwns"
Oct 31 19:49:07.766485 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:07.766446    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:07.796063 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:07.795787    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55751f79-d0e3-474e-9623-5e37f8d1f396-tmp-dir\") pod \"test-model-754e0a-kserve-767c88f75f-8xwns\" (UID: \"55751f79-d0e3-474e-9623-5e37f8d1f396\") " pod="ai-tenant-e2e-inf-a-754e0a/test-model-754e0a-kserve-767c88f75f-8xwns"
Oct 31 19:49:07.841426 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:07.840121    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkzhb\" (UniqueName: \"kubernetes.io/projected/55751f79-d0e3-474e-9623-5e37f8d1f396-kube-api-access-hkzhb\") pod \"test-model-754e0a-kserve-767c88f75f-8xwns\" (UID: \"55751f79-d0e3-474e-9623-5e37f8d1f396\") " pod="ai-tenant-e2e-inf-a-754e0a/test-model-754e0a-kserve-767c88f75f-8xwns"
Oct 31 19:49:07.859190 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:07.850744    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55751f79-d0e3-474e-9623-5e37f8d1f396-dshm\") pod \"test-model-754e0a-kserve-767c88f75f-8xwns\" (UID: \"55751f79-d0e3-474e-9623-5e37f8d1f396\") " pod="ai-tenant-e2e-inf-a-754e0a/test-model-754e0a-kserve-767c88f75f-8xwns"
Oct 31 19:49:07.860668 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:07.860457    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55751f79-d0e3-474e-9623-5e37f8d1f396-tls-certs\") pod \"test-model-754e0a-kserve-767c88f75f-8xwns\" (UID: \"55751f79-d0e3-474e-9623-5e37f8d1f396\") " pod="ai-tenant-e2e-inf-a-754e0a/test-model-754e0a-kserve-767c88f75f-8xwns"
Oct 31 19:49:08.037805 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:08.037704    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:08.085361 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:08.085116    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-754e0a/test-model-754e0a-kserve-767c88f75f-8xwns"
Oct 31 19:49:08.413681 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:08.413652    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-754e0a/test-model-754e0a-kserve-767c88f75f-8xwns"]
Oct 31 19:49:08.414071 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:49:08.414049    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod55751f79_d0e3_474e_9623_5e37f8d1f396.slice/crio-0fea81aa49f4f5b108823c3136a47aa2843f9106a008a275935c37743bcc8469 WatchSource:0}: Error finding container 0fea81aa49f4f5b108823c3136a47aa2843f9106a008a275935c37743bcc8469: Status 404 returned error can't find the container with id 0fea81aa49f4f5b108823c3136a47aa2843f9106a008a275935c37743bcc8469
Oct 31 19:49:08.467525 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:08.467498    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-754e0a/test-model-754e0a-kserve-767c88f75f-8xwns" event={"ID":"55751f79-d0e3-474e-9623-5e37f8d1f396","Type":"ContainerStarted","Data":"0fea81aa49f4f5b108823c3136a47aa2843f9106a008a275935c37743bcc8469"}
Oct 31 19:49:08.978054 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:08.978021    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:09.477491 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:09.477457    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-754e0a/test-model-754e0a-kserve-767c88f75f-8xwns" event={"ID":"55751f79-d0e3-474e-9623-5e37f8d1f396","Type":"ContainerStarted","Data":"82940fe1be15510f34d139309ad98a64aeb93293ab80eaf0b63709b8f1c7b6be"}
Oct 31 19:49:09.500886 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:09.500839    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-754e0a/test-model-754e0a-kserve-767c88f75f-8xwns" podStartSLOduration=2.500825249 podStartE2EDuration="2.500825249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:49:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:49:09.495311157 +0000 UTC m=+1780.068545287" watchObservedRunningTime="2026-10-31 19:49:09.500825249 +0000 UTC m=+1780.074059377"
Oct 31 19:49:09.701312 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:09.701268    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:10.085882 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:10.085847    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-754e0a/test-model-754e0a-kserve-767c88f75f-8xwns"
Oct 31 19:49:10.782146 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:10.782115    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:11.120146 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:11.120118    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf"
Oct 31 19:49:11.237348 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:11.232348    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/506d452a-2e17-44c0-9b4e-c7ea88e7e5bf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/506d452a-2e17-44c0-9b4e-c7ea88e7e5bf-istiod-ca-cert\") pod \"506d452a-2e17-44c0-9b4e-c7ea88e7e5bf\" (UID: \"506d452a-2e17-44c0-9b4e-c7ea88e7e5bf\") "
Oct 31 19:49:11.237348 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:11.232474    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/506d452a-2e17-44c0-9b4e-c7ea88e7e5bf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/506d452a-2e17-44c0-9b4e-c7ea88e7e5bf-istio-podinfo\") pod \"506d452a-2e17-44c0-9b4e-c7ea88e7e5bf\" (UID: \"506d452a-2e17-44c0-9b4e-c7ea88e7e5bf\") "
Oct 31 19:49:11.237348 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:11.232707    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/506d452a-2e17-44c0-9b4e-c7ea88e7e5bf-istio-token\" (UniqueName: \"kubernetes.io/projected/506d452a-2e17-44c0-9b4e-c7ea88e7e5bf-istio-token\") pod \"506d452a-2e17-44c0-9b4e-c7ea88e7e5bf\" (UID: \"506d452a-2e17-44c0-9b4e-c7ea88e7e5bf\") "
Oct 31 19:49:11.237348 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:11.232774    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/506d452a-2e17-44c0-9b4e-c7ea88e7e5bf-kube-api-access-jghk8\" (UniqueName: \"kubernetes.io/projected/506d452a-2e17-44c0-9b4e-c7ea88e7e5bf-kube-api-access-jghk8\") pod \"506d452a-2e17-44c0-9b4e-c7ea88e7e5bf\" (UID: \"506d452a-2e17-44c0-9b4e-c7ea88e7e5bf\") "
Oct 31 19:49:11.237348 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:11.232828    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/506d452a-2e17-44c0-9b4e-c7ea88e7e5bf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/506d452a-2e17-44c0-9b4e-c7ea88e7e5bf-workload-certs\") pod \"506d452a-2e17-44c0-9b4e-c7ea88e7e5bf\" (UID: \"506d452a-2e17-44c0-9b4e-c7ea88e7e5bf\") "
Oct 31 19:49:11.237348 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:11.232909    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/506d452a-2e17-44c0-9b4e-c7ea88e7e5bf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/506d452a-2e17-44c0-9b4e-c7ea88e7e5bf-workload-socket\") pod \"506d452a-2e17-44c0-9b4e-c7ea88e7e5bf\" (UID: \"506d452a-2e17-44c0-9b4e-c7ea88e7e5bf\") "
Oct 31 19:49:11.237348 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:11.232941    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/506d452a-2e17-44c0-9b4e-c7ea88e7e5bf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/506d452a-2e17-44c0-9b4e-c7ea88e7e5bf-istio-data\") pod \"506d452a-2e17-44c0-9b4e-c7ea88e7e5bf\" (UID: \"506d452a-2e17-44c0-9b4e-c7ea88e7e5bf\") "
Oct 31 19:49:11.237348 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:11.233127    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/506d452a-2e17-44c0-9b4e-c7ea88e7e5bf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/506d452a-2e17-44c0-9b4e-c7ea88e7e5bf-istio-envoy\") pod \"506d452a-2e17-44c0-9b4e-c7ea88e7e5bf\" (UID: \"506d452a-2e17-44c0-9b4e-c7ea88e7e5bf\") "
Oct 31 19:49:11.237348 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:11.233165    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/506d452a-2e17-44c0-9b4e-c7ea88e7e5bf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/506d452a-2e17-44c0-9b4e-c7ea88e7e5bf-credential-socket\") pod \"506d452a-2e17-44c0-9b4e-c7ea88e7e5bf\" (UID: \"506d452a-2e17-44c0-9b4e-c7ea88e7e5bf\") "
Oct 31 19:49:11.237348 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:11.234669    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/506d452a-2e17-44c0-9b4e-c7ea88e7e5bf-credential-socket" pod "506d452a-2e17-44c0-9b4e-c7ea88e7e5bf" (UID: "506d452a-2e17-44c0-9b4e-c7ea88e7e5bf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:49:11.237348 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:11.235070    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/506d452a-2e17-44c0-9b4e-c7ea88e7e5bf-istiod-ca-cert" pod "506d452a-2e17-44c0-9b4e-c7ea88e7e5bf" (UID: "506d452a-2e17-44c0-9b4e-c7ea88e7e5bf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:49:11.237845 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:11.237443    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/506d452a-2e17-44c0-9b4e-c7ea88e7e5bf-istio-podinfo" pod "506d452a-2e17-44c0-9b4e-c7ea88e7e5bf" (UID: "506d452a-2e17-44c0-9b4e-c7ea88e7e5bf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:49:11.237845 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:11.237608    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/506d452a-2e17-44c0-9b4e-c7ea88e7e5bf-workload-socket" pod "506d452a-2e17-44c0-9b4e-c7ea88e7e5bf" (UID: "506d452a-2e17-44c0-9b4e-c7ea88e7e5bf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:49:11.239438 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:11.239397    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/506d452a-2e17-44c0-9b4e-c7ea88e7e5bf-workload-certs" pod "506d452a-2e17-44c0-9b4e-c7ea88e7e5bf" (UID: "506d452a-2e17-44c0-9b4e-c7ea88e7e5bf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:49:11.239802 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:11.239765    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/506d452a-2e17-44c0-9b4e-c7ea88e7e5bf-kube-api-access-jghk8" pod "506d452a-2e17-44c0-9b4e-c7ea88e7e5bf" (UID: "506d452a-2e17-44c0-9b4e-c7ea88e7e5bf"). InnerVolumeSpecName "kube-api-access-jghk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:49:11.241578 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:11.241552    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/506d452a-2e17-44c0-9b4e-c7ea88e7e5bf-istio-envoy" pod "506d452a-2e17-44c0-9b4e-c7ea88e7e5bf" (UID: "506d452a-2e17-44c0-9b4e-c7ea88e7e5bf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:49:11.241715 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:11.241690    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/506d452a-2e17-44c0-9b4e-c7ea88e7e5bf-istio-token" pod "506d452a-2e17-44c0-9b4e-c7ea88e7e5bf" (UID: "506d452a-2e17-44c0-9b4e-c7ea88e7e5bf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:49:11.242134 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:11.242109    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/506d452a-2e17-44c0-9b4e-c7ea88e7e5bf-istio-data" pod "506d452a-2e17-44c0-9b4e-c7ea88e7e5bf" (UID: "506d452a-2e17-44c0-9b4e-c7ea88e7e5bf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:49:11.336439 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:11.336348    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/506d452a-2e17-44c0-9b4e-c7ea88e7e5bf-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:49:11.336439 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:11.336375    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/506d452a-2e17-44c0-9b4e-c7ea88e7e5bf-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:49:11.336439 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:11.336389    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/506d452a-2e17-44c0-9b4e-c7ea88e7e5bf-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:49:11.336439 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:11.336402    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jghk8\" (UniqueName: \"kubernetes.io/projected/506d452a-2e17-44c0-9b4e-c7ea88e7e5bf-kube-api-access-jghk8\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:49:11.336439 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:11.336414    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/506d452a-2e17-44c0-9b4e-c7ea88e7e5bf-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:49:11.336439 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:11.336423    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/506d452a-2e17-44c0-9b4e-c7ea88e7e5bf-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:49:11.336439 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:11.336431    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/506d452a-2e17-44c0-9b4e-c7ea88e7e5bf-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:49:11.336439 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:11.336442    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/506d452a-2e17-44c0-9b4e-c7ea88e7e5bf-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:49:11.336873 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:11.336451    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/506d452a-2e17-44c0-9b4e-c7ea88e7e5bf-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:49:11.495514 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:11.495478    4645 generic.go:356] "Generic (PLEG): container finished" podID="506d452a-2e17-44c0-9b4e-c7ea88e7e5bf" containerID="3b2ed82f1f816debce75a6206b7bc06a278d5d8a913b9a23776168803d3123a9" exitCode=0
Oct 31 19:49:11.495711 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:11.495514    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf" event={"ID":"506d452a-2e17-44c0-9b4e-c7ea88e7e5bf","Type":"ContainerDied","Data":"3b2ed82f1f816debce75a6206b7bc06a278d5d8a913b9a23776168803d3123a9"}
Oct 31 19:49:11.495711 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:11.495548    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf"
Oct 31 19:49:11.495711 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:11.495589    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf" event={"ID":"506d452a-2e17-44c0-9b4e-c7ea88e7e5bf","Type":"ContainerDied","Data":"0618340424f1ba1e6ca6b2934f037b519dd5166898361ba92285750f916ef180"}
Oct 31 19:49:11.495711 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:11.495619    4645 scope.go:122] "RemoveContainer" containerID="3b2ed82f1f816debce75a6206b7bc06a278d5d8a913b9a23776168803d3123a9"
Oct 31 19:49:11.520420 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:11.519353    4645 scope.go:122] "RemoveContainer" containerID="3b2ed82f1f816debce75a6206b7bc06a278d5d8a913b9a23776168803d3123a9"
Oct 31 19:49:11.520420 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:49:11.519785    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b2ed82f1f816debce75a6206b7bc06a278d5d8a913b9a23776168803d3123a9\": container with ID starting with 3b2ed82f1f816debce75a6206b7bc06a278d5d8a913b9a23776168803d3123a9 not found: ID does not exist" containerID="3b2ed82f1f816debce75a6206b7bc06a278d5d8a913b9a23776168803d3123a9"
Oct 31 19:49:11.520420 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:11.519832    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b2ed82f1f816debce75a6206b7bc06a278d5d8a913b9a23776168803d3123a9"} err="failed to get container status \"3b2ed82f1f816debce75a6206b7bc06a278d5d8a913b9a23776168803d3123a9\": rpc error: code = NotFound desc = could not find container \"3b2ed82f1f816debce75a6206b7bc06a278d5d8a913b9a23776168803d3123a9\": container with ID starting with 3b2ed82f1f816debce75a6206b7bc06a278d5d8a913b9a23776168803d3123a9 not found: ID does not exist"
Oct 31 19:49:11.529683 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:11.529657    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf"]
Oct 31 19:49:11.534396 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:11.534371    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-pf6kf"]
Oct 31 19:49:11.535410 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:11.535384    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:11.628895 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:11.628815    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="506d452a-2e17-44c0-9b4e-c7ea88e7e5bf" path="/var/lib/kubelet/pods/506d452a-2e17-44c0-9b4e-c7ea88e7e5bf/volumes"
Oct 31 19:49:11.755907 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:11.755880    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ceb3f51e-hwggz"]
Oct 31 19:49:11.756760 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:11.756743    4645 cpu_manager.go:414] "containerMap: removing container" podUID="506d452a-2e17-44c0-9b4e-c7ea88e7e5bf" containerName="istio-proxy"
Oct 31 19:49:11.756813 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:11.756762    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="506d452a-2e17-44c0-9b4e-c7ea88e7e5bf" containerName="istio-proxy"
Oct 31 19:49:11.756932 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:11.756923    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="506d452a-2e17-44c0-9b4e-c7ea88e7e5bf" containerName="istio-proxy"
Oct 31 19:49:11.762354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:11.762322    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceb3f51e-hwggz"
Oct 31 19:49:11.765781 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:11.765757    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:49:11.765990 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:11.765965    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 19:49:11.770714 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:11.770689    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ceb3f51e-hwggz"]
Oct 31 19:49:11.844690 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:11.844665    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzjp7\" (UniqueName: \"kubernetes.io/projected/d6eea5ad-a82b-47ba-a674-a3adecfacb68-kube-api-access-gzjp7\") pod \"maas-api-revoke-sub-ceb3f51e-hwggz\" (UID: \"d6eea5ad-a82b-47ba-a674-a3adecfacb68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceb3f51e-hwggz"
Oct 31 19:49:11.845049 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:11.844888    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6eea5ad-a82b-47ba-a674-a3adecfacb68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ceb3f51e-hwggz\" (UID: \"d6eea5ad-a82b-47ba-a674-a3adecfacb68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceb3f51e-hwggz"
Oct 31 19:49:11.845049 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:11.844992    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6eea5ad-a82b-47ba-a674-a3adecfacb68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ceb3f51e-hwggz\" (UID: \"d6eea5ad-a82b-47ba-a674-a3adecfacb68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceb3f51e-hwggz"
Oct 31 19:49:11.860090 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:11.860069    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:11.946586 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:11.946559    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6eea5ad-a82b-47ba-a674-a3adecfacb68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ceb3f51e-hwggz\" (UID: \"d6eea5ad-a82b-47ba-a674-a3adecfacb68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceb3f51e-hwggz"
Oct 31 19:49:11.946712 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:11.946699    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzjp7\" (UniqueName: \"kubernetes.io/projected/d6eea5ad-a82b-47ba-a674-a3adecfacb68-kube-api-access-gzjp7\") pod \"maas-api-revoke-sub-ceb3f51e-hwggz\" (UID: \"d6eea5ad-a82b-47ba-a674-a3adecfacb68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceb3f51e-hwggz"
Oct 31 19:49:11.946867 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:11.946854    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6eea5ad-a82b-47ba-a674-a3adecfacb68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ceb3f51e-hwggz\" (UID: \"d6eea5ad-a82b-47ba-a674-a3adecfacb68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceb3f51e-hwggz"
Oct 31 19:49:11.947296 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:11.947270    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6eea5ad-a82b-47ba-a674-a3adecfacb68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ceb3f51e-hwggz\" (UID: \"d6eea5ad-a82b-47ba-a674-a3adecfacb68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceb3f51e-hwggz"
Oct 31 19:49:11.949394 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:11.949373    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6eea5ad-a82b-47ba-a674-a3adecfacb68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ceb3f51e-hwggz\" (UID: \"d6eea5ad-a82b-47ba-a674-a3adecfacb68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceb3f51e-hwggz"
Oct 31 19:49:11.959154 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:11.959133    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzjp7\" (UniqueName: \"kubernetes.io/projected/d6eea5ad-a82b-47ba-a674-a3adecfacb68-kube-api-access-gzjp7\") pod \"maas-api-revoke-sub-ceb3f51e-hwggz\" (UID: \"d6eea5ad-a82b-47ba-a674-a3adecfacb68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceb3f51e-hwggz"
Oct 31 19:49:11.969782 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:11.969759    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:12.084030 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:12.084012    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceb3f51e-hwggz"
Oct 31 19:49:12.374190 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:49:12.374157    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6eea5ad_a82b_47ba_a674_a3adecfacb68.slice/crio-71fc319d0b96e6381099a3dde9f444e479c244a6edb7d3b59c56b909dac8974b WatchSource:0}: Error finding container 71fc319d0b96e6381099a3dde9f444e479c244a6edb7d3b59c56b909dac8974b: Status 404 returned error can't find the container with id 71fc319d0b96e6381099a3dde9f444e479c244a6edb7d3b59c56b909dac8974b
Oct 31 19:49:12.375100 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:12.375072    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ceb3f51e-hwggz"]
Oct 31 19:49:12.496097 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:12.496049    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:12.508141 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:12.508113    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceb3f51e-hwggz" event={"ID":"d6eea5ad-a82b-47ba-a674-a3adecfacb68","Type":"ContainerStarted","Data":"71fc319d0b96e6381099a3dde9f444e479c244a6edb7d3b59c56b909dac8974b"}
Oct 31 19:49:13.389113 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:13.389070    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:13.523048 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:13.523019    4645 generic.go:356] "Generic (PLEG): container finished" podID="d6eea5ad-a82b-47ba-a674-a3adecfacb68" containerID="7b99f12652fa8c4c40f711674d62c4f47524f1511395b30f98bd2c72bc2c60ca" exitCode=0
Oct 31 19:49:13.523200 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:13.523102    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceb3f51e-hwggz" event={"ID":"d6eea5ad-a82b-47ba-a674-a3adecfacb68","Type":"ContainerDied","Data":"7b99f12652fa8c4c40f711674d62c4f47524f1511395b30f98bd2c72bc2c60ca"}
Oct 31 19:49:14.132673 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:14.132628    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:14.723645 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:14.723617    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceb3f51e-hwggz"
Oct 31 19:49:14.817444 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:14.817367    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6eea5ad-a82b-47ba-a674-a3adecfacb68-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6eea5ad-a82b-47ba-a674-a3adecfacb68-maas-api-openshift-ca\") pod \"d6eea5ad-a82b-47ba-a674-a3adecfacb68\" (UID: \"d6eea5ad-a82b-47ba-a674-a3adecfacb68\") "
Oct 31 19:49:14.817612 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:14.817553    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6eea5ad-a82b-47ba-a674-a3adecfacb68-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6eea5ad-a82b-47ba-a674-a3adecfacb68-maas-api-xks-ca\") pod \"d6eea5ad-a82b-47ba-a674-a3adecfacb68\" (UID: \"d6eea5ad-a82b-47ba-a674-a3adecfacb68\") "
Oct 31 19:49:14.817692 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:14.817667    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6eea5ad-a82b-47ba-a674-a3adecfacb68-kube-api-access-gzjp7\" (UniqueName: \"kubernetes.io/projected/d6eea5ad-a82b-47ba-a674-a3adecfacb68-kube-api-access-gzjp7\") pod \"d6eea5ad-a82b-47ba-a674-a3adecfacb68\" (UID: \"d6eea5ad-a82b-47ba-a674-a3adecfacb68\") "
Oct 31 19:49:14.817788 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:14.817740    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6eea5ad-a82b-47ba-a674-a3adecfacb68-maas-api-openshift-ca" pod "d6eea5ad-a82b-47ba-a674-a3adecfacb68" (UID: "d6eea5ad-a82b-47ba-a674-a3adecfacb68"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:49:14.818418 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:14.818395    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6eea5ad-a82b-47ba-a674-a3adecfacb68-maas-api-openshift-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:49:14.819831 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:14.819803    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6eea5ad-a82b-47ba-a674-a3adecfacb68-maas-api-xks-ca" pod "d6eea5ad-a82b-47ba-a674-a3adecfacb68" (UID: "d6eea5ad-a82b-47ba-a674-a3adecfacb68"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:49:14.819831 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:14.819823    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6eea5ad-a82b-47ba-a674-a3adecfacb68-kube-api-access-gzjp7" pod "d6eea5ad-a82b-47ba-a674-a3adecfacb68" (UID: "d6eea5ad-a82b-47ba-a674-a3adecfacb68"). InnerVolumeSpecName "kube-api-access-gzjp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:49:14.920270 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:14.920237    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzjp7\" (UniqueName: \"kubernetes.io/projected/d6eea5ad-a82b-47ba-a674-a3adecfacb68-kube-api-access-gzjp7\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:49:14.920270 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:14.920264    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6eea5ad-a82b-47ba-a674-a3adecfacb68-maas-api-xks-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:49:14.927256 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:14.927222    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:15.550396 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:15.550368    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceb3f51e-hwggz"
Oct 31 19:49:15.550602 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:15.550365    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceb3f51e-hwggz" event={"ID":"d6eea5ad-a82b-47ba-a674-a3adecfacb68","Type":"ContainerDied","Data":"71fc319d0b96e6381099a3dde9f444e479c244a6edb7d3b59c56b909dac8974b"}
Oct 31 19:49:15.550602 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:15.550525    4645 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71fc319d0b96e6381099a3dde9f444e479c244a6edb7d3b59c56b909dac8974b"
Oct 31 19:49:15.737921 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:15.737889    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:16.556653 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:16.556616    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:16.846653 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:16.846571    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:17.506472 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:17.504654    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:17.792788 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:17.792705    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:18.086479 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:18.086398    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-754e0a/test-model-754e0a-kserve-767c88f75f-8xwns"
Oct 31 19:49:18.099380 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:18.099354    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-754e0a/test-model-754e0a-kserve-767c88f75f-8xwns"
Oct 31 19:49:18.438945 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:18.438914    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:18.597865 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:18.597829    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-754e0a/test-model-754e0a-kserve-767c88f75f-8xwns"
Oct 31 19:49:19.478685 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:19.478649    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:20.348223 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:20.348182    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:20.772704 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:20.772673    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5"]
Oct 31 19:49:20.773131 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:20.772858    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5" podUID="2bf496d2-b4cd-417b-b5dc-3afbac91f26b" containerName="istio-proxy" containerID="cri-o://a5b0e166ca64ced17af9ede634b7c94b9c98be1c40972abd9ac1e175bcf6acf7" gracePeriod=30
Oct 31 19:49:21.145198 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:21.145112    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:22.046145 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:22.046109    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:22.854823 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:22.854783    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:23.339177 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:23.339143    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-7c892a/test-model-7c892a-kserve-57875c9b5f-r4j8f"]
Oct 31 19:49:23.340053 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:23.340033    4645 cpu_manager.go:414] "containerMap: removing container" podUID="d6eea5ad-a82b-47ba-a674-a3adecfacb68" containerName="revoke-keys"
Oct 31 19:49:23.340117 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:23.340054    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6eea5ad-a82b-47ba-a674-a3adecfacb68" containerName="revoke-keys"
Oct 31 19:49:23.340240 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:23.340229    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6eea5ad-a82b-47ba-a674-a3adecfacb68" containerName="revoke-keys"
Oct 31 19:49:23.344265 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:23.344243    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7c892a/test-model-7c892a-kserve-57875c9b5f-r4j8f"
Oct 31 19:49:23.350166 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:23.350145    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-7c892a\"/\"openshift-service-ca.crt\""
Oct 31 19:49:23.350293 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:23.350273    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-7c892a\"/\"kube-root-ca.crt\""
Oct 31 19:49:23.352417 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:23.352392    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-7c892a\"/\"default-dockercfg-lqsnk\""
Oct 31 19:49:23.352515 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:23.352494    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-7c892a\"/\"test-model-7c892a-kserve-self-signed-certs\""
Oct 31 19:49:23.359226 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:23.359201    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-7c892a/test-model-7c892a-kserve-57875c9b5f-r4j8f"]
Oct 31 19:49:23.413132 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:23.413097    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:23.415703 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:23.415675    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7be66b1c-b038-45f8-8e28-cb3b3cd3c526-home\") pod \"test-model-7c892a-kserve-57875c9b5f-r4j8f\" (UID: \"7be66b1c-b038-45f8-8e28-cb3b3cd3c526\") " pod="ai-tenant-e2e-inf-b-7c892a/test-model-7c892a-kserve-57875c9b5f-r4j8f"
Oct 31 19:49:23.415846 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:23.415791    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7be66b1c-b038-45f8-8e28-cb3b3cd3c526-tmp-dir\") pod \"test-model-7c892a-kserve-57875c9b5f-r4j8f\" (UID: \"7be66b1c-b038-45f8-8e28-cb3b3cd3c526\") " pod="ai-tenant-e2e-inf-b-7c892a/test-model-7c892a-kserve-57875c9b5f-r4j8f"
Oct 31 19:49:23.415924 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:23.415904    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7be66b1c-b038-45f8-8e28-cb3b3cd3c526-model-cache\") pod \"test-model-7c892a-kserve-57875c9b5f-r4j8f\" (UID: \"7be66b1c-b038-45f8-8e28-cb3b3cd3c526\") " pod="ai-tenant-e2e-inf-b-7c892a/test-model-7c892a-kserve-57875c9b5f-r4j8f"
Oct 31 19:49:23.416122 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:23.416094    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnwdz\" (UniqueName: \"kubernetes.io/projected/7be66b1c-b038-45f8-8e28-cb3b3cd3c526-kube-api-access-cnwdz\") pod \"test-model-7c892a-kserve-57875c9b5f-r4j8f\" (UID: \"7be66b1c-b038-45f8-8e28-cb3b3cd3c526\") " pod="ai-tenant-e2e-inf-b-7c892a/test-model-7c892a-kserve-57875c9b5f-r4j8f"
Oct 31 19:49:23.416245 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:23.416188    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7be66b1c-b038-45f8-8e28-cb3b3cd3c526-dshm\") pod \"test-model-7c892a-kserve-57875c9b5f-r4j8f\" (UID: \"7be66b1c-b038-45f8-8e28-cb3b3cd3c526\") " pod="ai-tenant-e2e-inf-b-7c892a/test-model-7c892a-kserve-57875c9b5f-r4j8f"
Oct 31 19:49:23.416320 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:23.416263    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7be66b1c-b038-45f8-8e28-cb3b3cd3c526-tls-certs\") pod \"test-model-7c892a-kserve-57875c9b5f-r4j8f\" (UID: \"7be66b1c-b038-45f8-8e28-cb3b3cd3c526\") " pod="ai-tenant-e2e-inf-b-7c892a/test-model-7c892a-kserve-57875c9b5f-r4j8f"
Oct 31 19:49:23.520885 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:23.520836    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7be66b1c-b038-45f8-8e28-cb3b3cd3c526-tmp-dir\") pod \"test-model-7c892a-kserve-57875c9b5f-r4j8f\" (UID: \"7be66b1c-b038-45f8-8e28-cb3b3cd3c526\") " pod="ai-tenant-e2e-inf-b-7c892a/test-model-7c892a-kserve-57875c9b5f-r4j8f"
Oct 31 19:49:23.522040 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:23.522013    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7be66b1c-b038-45f8-8e28-cb3b3cd3c526-model-cache\") pod \"test-model-7c892a-kserve-57875c9b5f-r4j8f\" (UID: \"7be66b1c-b038-45f8-8e28-cb3b3cd3c526\") " pod="ai-tenant-e2e-inf-b-7c892a/test-model-7c892a-kserve-57875c9b5f-r4j8f"
Oct 31 19:49:23.522891 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:23.522868    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnwdz\" (UniqueName: \"kubernetes.io/projected/7be66b1c-b038-45f8-8e28-cb3b3cd3c526-kube-api-access-cnwdz\") pod \"test-model-7c892a-kserve-57875c9b5f-r4j8f\" (UID: \"7be66b1c-b038-45f8-8e28-cb3b3cd3c526\") " pod="ai-tenant-e2e-inf-b-7c892a/test-model-7c892a-kserve-57875c9b5f-r4j8f"
Oct 31 19:49:23.571918 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:23.571883    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7be66b1c-b038-45f8-8e28-cb3b3cd3c526-dshm\") pod \"test-model-7c892a-kserve-57875c9b5f-r4j8f\" (UID: \"7be66b1c-b038-45f8-8e28-cb3b3cd3c526\") " pod="ai-tenant-e2e-inf-b-7c892a/test-model-7c892a-kserve-57875c9b5f-r4j8f"
Oct 31 19:49:23.572217 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:23.572192    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7be66b1c-b038-45f8-8e28-cb3b3cd3c526-tls-certs\") pod \"test-model-7c892a-kserve-57875c9b5f-r4j8f\" (UID: \"7be66b1c-b038-45f8-8e28-cb3b3cd3c526\") " pod="ai-tenant-e2e-inf-b-7c892a/test-model-7c892a-kserve-57875c9b5f-r4j8f"
Oct 31 19:49:23.572598 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:23.572576    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7be66b1c-b038-45f8-8e28-cb3b3cd3c526-home\") pod \"test-model-7c892a-kserve-57875c9b5f-r4j8f\" (UID: \"7be66b1c-b038-45f8-8e28-cb3b3cd3c526\") " pod="ai-tenant-e2e-inf-b-7c892a/test-model-7c892a-kserve-57875c9b5f-r4j8f"
Oct 31 19:49:23.573225 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:23.573200    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7be66b1c-b038-45f8-8e28-cb3b3cd3c526-home\") pod \"test-model-7c892a-kserve-57875c9b5f-r4j8f\" (UID: \"7be66b1c-b038-45f8-8e28-cb3b3cd3c526\") " pod="ai-tenant-e2e-inf-b-7c892a/test-model-7c892a-kserve-57875c9b5f-r4j8f"
Oct 31 19:49:23.581438 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:23.521912    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7be66b1c-b038-45f8-8e28-cb3b3cd3c526-tmp-dir\") pod \"test-model-7c892a-kserve-57875c9b5f-r4j8f\" (UID: \"7be66b1c-b038-45f8-8e28-cb3b3cd3c526\") " pod="ai-tenant-e2e-inf-b-7c892a/test-model-7c892a-kserve-57875c9b5f-r4j8f"
Oct 31 19:49:23.581601 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:23.522648    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7be66b1c-b038-45f8-8e28-cb3b3cd3c526-model-cache\") pod \"test-model-7c892a-kserve-57875c9b5f-r4j8f\" (UID: \"7be66b1c-b038-45f8-8e28-cb3b3cd3c526\") " pod="ai-tenant-e2e-inf-b-7c892a/test-model-7c892a-kserve-57875c9b5f-r4j8f"
Oct 31 19:49:23.603401 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:23.595519    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7be66b1c-b038-45f8-8e28-cb3b3cd3c526-dshm\") pod \"test-model-7c892a-kserve-57875c9b5f-r4j8f\" (UID: \"7be66b1c-b038-45f8-8e28-cb3b3cd3c526\") " pod="ai-tenant-e2e-inf-b-7c892a/test-model-7c892a-kserve-57875c9b5f-r4j8f"
Oct 31 19:49:23.620954 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:23.620910    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnwdz\" (UniqueName: \"kubernetes.io/projected/7be66b1c-b038-45f8-8e28-cb3b3cd3c526-kube-api-access-cnwdz\") pod \"test-model-7c892a-kserve-57875c9b5f-r4j8f\" (UID: \"7be66b1c-b038-45f8-8e28-cb3b3cd3c526\") " pod="ai-tenant-e2e-inf-b-7c892a/test-model-7c892a-kserve-57875c9b5f-r4j8f"
Oct 31 19:49:23.634169 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:23.634126    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7be66b1c-b038-45f8-8e28-cb3b3cd3c526-tls-certs\") pod \"test-model-7c892a-kserve-57875c9b5f-r4j8f\" (UID: \"7be66b1c-b038-45f8-8e28-cb3b3cd3c526\") " pod="ai-tenant-e2e-inf-b-7c892a/test-model-7c892a-kserve-57875c9b5f-r4j8f"
Oct 31 19:49:23.665362 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:23.661104    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7c892a/test-model-7c892a-kserve-57875c9b5f-r4j8f"
Oct 31 19:49:23.972228 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:23.972201    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:24.066501 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:49:24.066472    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7be66b1c_b038_45f8_8e28_cb3b3cd3c526.slice/crio-bdf2b43aca451df0cdde4f192b9beda33588babd00c5037fca57fc602fbb777e WatchSource:0}: Error finding container bdf2b43aca451df0cdde4f192b9beda33588babd00c5037fca57fc602fbb777e: Status 404 returned error can't find the container with id bdf2b43aca451df0cdde4f192b9beda33588babd00c5037fca57fc602fbb777e
Oct 31 19:49:24.066620 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:24.066505    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-7c892a/test-model-7c892a-kserve-57875c9b5f-r4j8f"]
Oct 31 19:49:24.352566 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:24.352314    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:24.696010 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:24.695981    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7c892a/test-model-7c892a-kserve-57875c9b5f-r4j8f" event={"ID":"7be66b1c-b038-45f8-8e28-cb3b3cd3c526","Type":"ContainerStarted","Data":"9295a364f0bea976491737426de95a27251994ea30dbe97442874ec97cb985e7"}
Oct 31 19:49:24.696194 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:24.696024    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7c892a/test-model-7c892a-kserve-57875c9b5f-r4j8f" event={"ID":"7be66b1c-b038-45f8-8e28-cb3b3cd3c526","Type":"ContainerStarted","Data":"bdf2b43aca451df0cdde4f192b9beda33588babd00c5037fca57fc602fbb777e"}
Oct 31 19:49:24.719315 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:24.719271    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-7c892a/test-model-7c892a-kserve-57875c9b5f-r4j8f" podStartSLOduration=1.719259733 podStartE2EDuration="1.719259733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:49:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:49:24.714672393 +0000 UTC m=+1795.287906524" watchObservedRunningTime="2026-10-31 19:49:24.719259733 +0000 UTC m=+1795.292493882"
Oct 31 19:49:24.855873 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:24.855847    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:25.646377 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:25.646310    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:25.661687 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:25.661666    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-7c892a/test-model-7c892a-kserve-57875c9b5f-r4j8f"
Oct 31 19:49:26.087883 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:26.087858    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5"
Oct 31 19:49:26.140540 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:26.140504    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2bf496d2-b4cd-417b-b5dc-3afbac91f26b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bf496d2-b4cd-417b-b5dc-3afbac91f26b-istio-podinfo\") pod \"2bf496d2-b4cd-417b-b5dc-3afbac91f26b\" (UID: \"2bf496d2-b4cd-417b-b5dc-3afbac91f26b\") "
Oct 31 19:49:26.140717 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:26.140558    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bf496d2-b4cd-417b-b5dc-3afbac91f26b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bf496d2-b4cd-417b-b5dc-3afbac91f26b-workload-socket\") pod \"2bf496d2-b4cd-417b-b5dc-3afbac91f26b\" (UID: \"2bf496d2-b4cd-417b-b5dc-3afbac91f26b\") "
Oct 31 19:49:26.140717 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:26.140625    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bf496d2-b4cd-417b-b5dc-3afbac91f26b-kube-api-access-m88qn\" (UniqueName: \"kubernetes.io/projected/2bf496d2-b4cd-417b-b5dc-3afbac91f26b-kube-api-access-m88qn\") pod \"2bf496d2-b4cd-417b-b5dc-3afbac91f26b\" (UID: \"2bf496d2-b4cd-417b-b5dc-3afbac91f26b\") "
Oct 31 19:49:26.140854 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:26.140784    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bf496d2-b4cd-417b-b5dc-3afbac91f26b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bf496d2-b4cd-417b-b5dc-3afbac91f26b-istio-data\") pod \"2bf496d2-b4cd-417b-b5dc-3afbac91f26b\" (UID: \"2bf496d2-b4cd-417b-b5dc-3afbac91f26b\") "
Oct 31 19:49:26.140933 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:26.140904    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bf496d2-b4cd-417b-b5dc-3afbac91f26b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bf496d2-b4cd-417b-b5dc-3afbac91f26b-istiod-ca-cert\") pod \"2bf496d2-b4cd-417b-b5dc-3afbac91f26b\" (UID: \"2bf496d2-b4cd-417b-b5dc-3afbac91f26b\") "
Oct 31 19:49:26.141220 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:26.141160    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bf496d2-b4cd-417b-b5dc-3afbac91f26b-workload-socket" pod "2bf496d2-b4cd-417b-b5dc-3afbac91f26b" (UID: "2bf496d2-b4cd-417b-b5dc-3afbac91f26b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:49:26.141686 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:26.141618    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bf496d2-b4cd-417b-b5dc-3afbac91f26b-istio-data" pod "2bf496d2-b4cd-417b-b5dc-3afbac91f26b" (UID: "2bf496d2-b4cd-417b-b5dc-3afbac91f26b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:49:26.141686 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:26.141645    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bf496d2-b4cd-417b-b5dc-3afbac91f26b-istiod-ca-cert" pod "2bf496d2-b4cd-417b-b5dc-3afbac91f26b" (UID: "2bf496d2-b4cd-417b-b5dc-3afbac91f26b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:49:26.142115 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:26.142090    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bf496d2-b4cd-417b-b5dc-3afbac91f26b-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:49:26.142236 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:26.142153    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bf496d2-b4cd-417b-b5dc-3afbac91f26b-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:49:26.142236 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:26.142172    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bf496d2-b4cd-417b-b5dc-3afbac91f26b-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:49:26.143813 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:26.143386    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2bf496d2-b4cd-417b-b5dc-3afbac91f26b-istio-podinfo" pod "2bf496d2-b4cd-417b-b5dc-3afbac91f26b" (UID: "2bf496d2-b4cd-417b-b5dc-3afbac91f26b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:49:26.144814 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:26.144787    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bf496d2-b4cd-417b-b5dc-3afbac91f26b-kube-api-access-m88qn" pod "2bf496d2-b4cd-417b-b5dc-3afbac91f26b" (UID: "2bf496d2-b4cd-417b-b5dc-3afbac91f26b"). InnerVolumeSpecName "kube-api-access-m88qn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:49:26.243200 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:26.243121    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bf496d2-b4cd-417b-b5dc-3afbac91f26b-istio-token\" (UniqueName: \"kubernetes.io/projected/2bf496d2-b4cd-417b-b5dc-3afbac91f26b-istio-token\") pod \"2bf496d2-b4cd-417b-b5dc-3afbac91f26b\" (UID: \"2bf496d2-b4cd-417b-b5dc-3afbac91f26b\") "
Oct 31 19:49:26.243370 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:26.243285    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bf496d2-b4cd-417b-b5dc-3afbac91f26b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bf496d2-b4cd-417b-b5dc-3afbac91f26b-istio-envoy\") pod \"2bf496d2-b4cd-417b-b5dc-3afbac91f26b\" (UID: \"2bf496d2-b4cd-417b-b5dc-3afbac91f26b\") "
Oct 31 19:49:26.243370 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:26.243347    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bf496d2-b4cd-417b-b5dc-3afbac91f26b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bf496d2-b4cd-417b-b5dc-3afbac91f26b-credential-socket\") pod \"2bf496d2-b4cd-417b-b5dc-3afbac91f26b\" (UID: \"2bf496d2-b4cd-417b-b5dc-3afbac91f26b\") "
Oct 31 19:49:26.243468 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:26.243384    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bf496d2-b4cd-417b-b5dc-3afbac91f26b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bf496d2-b4cd-417b-b5dc-3afbac91f26b-workload-certs\") pod \"2bf496d2-b4cd-417b-b5dc-3afbac91f26b\" (UID: \"2bf496d2-b4cd-417b-b5dc-3afbac91f26b\") "
Oct 31 19:49:26.243636 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:26.243613    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bf496d2-b4cd-417b-b5dc-3afbac91f26b-credential-socket" pod "2bf496d2-b4cd-417b-b5dc-3afbac91f26b" (UID: "2bf496d2-b4cd-417b-b5dc-3afbac91f26b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:49:26.243728 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:26.243701    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bf496d2-b4cd-417b-b5dc-3afbac91f26b-workload-certs" pod "2bf496d2-b4cd-417b-b5dc-3afbac91f26b" (UID: "2bf496d2-b4cd-417b-b5dc-3afbac91f26b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:49:26.246352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:26.244686    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bf496d2-b4cd-417b-b5dc-3afbac91f26b-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:49:26.246352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:26.244725    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bf496d2-b4cd-417b-b5dc-3afbac91f26b-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:49:26.246352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:26.244753    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bf496d2-b4cd-417b-b5dc-3afbac91f26b-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:49:26.246352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:26.244768    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m88qn\" (UniqueName: \"kubernetes.io/projected/2bf496d2-b4cd-417b-b5dc-3afbac91f26b-kube-api-access-m88qn\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:49:26.265956 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:26.265913    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bf496d2-b4cd-417b-b5dc-3afbac91f26b-istio-envoy" pod "2bf496d2-b4cd-417b-b5dc-3afbac91f26b" (UID: "2bf496d2-b4cd-417b-b5dc-3afbac91f26b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:49:26.266129 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:26.265989    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bf496d2-b4cd-417b-b5dc-3afbac91f26b-istio-token" pod "2bf496d2-b4cd-417b-b5dc-3afbac91f26b" (UID: "2bf496d2-b4cd-417b-b5dc-3afbac91f26b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:49:26.346794 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:26.346755    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bf496d2-b4cd-417b-b5dc-3afbac91f26b-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:49:26.346794 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:26.346785    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bf496d2-b4cd-417b-b5dc-3afbac91f26b-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:49:26.473987 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:26.473947    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:26.755389 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:26.755354    4645 generic.go:356] "Generic (PLEG): container finished" podID="2bf496d2-b4cd-417b-b5dc-3afbac91f26b" containerID="a5b0e166ca64ced17af9ede634b7c94b9c98be1c40972abd9ac1e175bcf6acf7" exitCode=0
Oct 31 19:49:26.755825 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:26.755395    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5" event={"ID":"2bf496d2-b4cd-417b-b5dc-3afbac91f26b","Type":"ContainerDied","Data":"a5b0e166ca64ced17af9ede634b7c94b9c98be1c40972abd9ac1e175bcf6acf7"}
Oct 31 19:49:26.755825 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:26.755429    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5"
Oct 31 19:49:26.755825 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:26.755448    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5" event={"ID":"2bf496d2-b4cd-417b-b5dc-3afbac91f26b","Type":"ContainerDied","Data":"a68085ff91a5b2f4be6d84f9f5c159e47d0ab3e0913f471fcb71e133320783cf"}
Oct 31 19:49:26.755825 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:26.755480    4645 scope.go:122] "RemoveContainer" containerID="a5b0e166ca64ced17af9ede634b7c94b9c98be1c40972abd9ac1e175bcf6acf7"
Oct 31 19:49:26.773655 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:26.773630    4645 scope.go:122] "RemoveContainer" containerID="a5b0e166ca64ced17af9ede634b7c94b9c98be1c40972abd9ac1e175bcf6acf7"
Oct 31 19:49:26.773958 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:49:26.773937    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5b0e166ca64ced17af9ede634b7c94b9c98be1c40972abd9ac1e175bcf6acf7\": container with ID starting with a5b0e166ca64ced17af9ede634b7c94b9c98be1c40972abd9ac1e175bcf6acf7 not found: ID does not exist" containerID="a5b0e166ca64ced17af9ede634b7c94b9c98be1c40972abd9ac1e175bcf6acf7"
Oct 31 19:49:26.774026 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:26.773970    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5b0e166ca64ced17af9ede634b7c94b9c98be1c40972abd9ac1e175bcf6acf7"} err="failed to get container status \"a5b0e166ca64ced17af9ede634b7c94b9c98be1c40972abd9ac1e175bcf6acf7\": rpc error: code = NotFound desc = could not find container \"a5b0e166ca64ced17af9ede634b7c94b9c98be1c40972abd9ac1e175bcf6acf7\": container with ID starting with a5b0e166ca64ced17af9ede634b7c94b9c98be1c40972abd9ac1e175bcf6acf7 not found: ID does not exist"
Oct 31 19:49:26.787700 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:26.787673    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5"]
Oct 31 19:49:26.796572 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:26.796538    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-jstv5"]
Oct 31 19:49:27.406728 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:27.406694    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:27.661434 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:27.661400    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bf496d2-b4cd-417b-b5dc-3afbac91f26b" path="/var/lib/kubelet/pods/2bf496d2-b4cd-417b-b5dc-3afbac91f26b/volumes"
Oct 31 19:49:28.479370 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:28.479302    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:28.562827 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:28.562782    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-03dfee/test-model-03dfee-kserve-68c5b8bcbf-cwgcs"]
Oct 31 19:49:28.632379 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:28.632319    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:29.118270 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:29.118212    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5e1b35a-cnbfq"]
Oct 31 19:49:29.119982 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:29.119950    4645 cpu_manager.go:414] "containerMap: removing container" podUID="2bf496d2-b4cd-417b-b5dc-3afbac91f26b" containerName="istio-proxy"
Oct 31 19:49:29.120228 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:29.120208    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bf496d2-b4cd-417b-b5dc-3afbac91f26b" containerName="istio-proxy"
Oct 31 19:49:29.120802 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:29.120774    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bf496d2-b4cd-417b-b5dc-3afbac91f26b" containerName="istio-proxy"
Oct 31 19:49:29.140742 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:29.140709    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5e1b35a-cnbfq"
Oct 31 19:49:29.146785 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:29.146748    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5e1b35a-cnbfq"]
Oct 31 19:49:29.150243 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:29.150033    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:49:29.150243 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:29.150166    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 19:49:29.237836 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:29.237240    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nx5v5\" (UniqueName: \"kubernetes.io/projected/cd48fc95-b72d-427d-ac18-b86487770086-kube-api-access-nx5v5\") pod \"maas-api-revoke-sub-f5e1b35a-cnbfq\" (UID: \"cd48fc95-b72d-427d-ac18-b86487770086\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5e1b35a-cnbfq"
Oct 31 19:49:29.237836 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:29.237536    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd48fc95-b72d-427d-ac18-b86487770086-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5e1b35a-cnbfq\" (UID: \"cd48fc95-b72d-427d-ac18-b86487770086\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5e1b35a-cnbfq"
Oct 31 19:49:29.237836 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:29.237591    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd48fc95-b72d-427d-ac18-b86487770086-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5e1b35a-cnbfq\" (UID: \"cd48fc95-b72d-427d-ac18-b86487770086\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5e1b35a-cnbfq"
Oct 31 19:49:29.343950 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:29.343909    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd48fc95-b72d-427d-ac18-b86487770086-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5e1b35a-cnbfq\" (UID: \"cd48fc95-b72d-427d-ac18-b86487770086\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5e1b35a-cnbfq"
Oct 31 19:49:29.344276 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:29.344254    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd48fc95-b72d-427d-ac18-b86487770086-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5e1b35a-cnbfq\" (UID: \"cd48fc95-b72d-427d-ac18-b86487770086\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5e1b35a-cnbfq"
Oct 31 19:49:29.344593 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:29.344573    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx5v5\" (UniqueName: \"kubernetes.io/projected/cd48fc95-b72d-427d-ac18-b86487770086-kube-api-access-nx5v5\") pod \"maas-api-revoke-sub-f5e1b35a-cnbfq\" (UID: \"cd48fc95-b72d-427d-ac18-b86487770086\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5e1b35a-cnbfq"
Oct 31 19:49:29.353186 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:29.353156    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd48fc95-b72d-427d-ac18-b86487770086-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5e1b35a-cnbfq\" (UID: \"cd48fc95-b72d-427d-ac18-b86487770086\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5e1b35a-cnbfq"
Oct 31 19:49:29.353417 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:29.353396    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd48fc95-b72d-427d-ac18-b86487770086-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5e1b35a-cnbfq\" (UID: \"cd48fc95-b72d-427d-ac18-b86487770086\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5e1b35a-cnbfq"
Oct 31 19:49:29.357810 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:29.357760    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx5v5\" (UniqueName: \"kubernetes.io/projected/cd48fc95-b72d-427d-ac18-b86487770086-kube-api-access-nx5v5\") pod \"maas-api-revoke-sub-f5e1b35a-cnbfq\" (UID: \"cd48fc95-b72d-427d-ac18-b86487770086\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5e1b35a-cnbfq"
Oct 31 19:49:29.574350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:29.572864    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 19:49:29.581115 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:29.580217    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5e1b35a-cnbfq"
Oct 31 19:49:29.800241 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:29.800204    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:30.006155 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:30.006066    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5e1b35a-cnbfq"]
Oct 31 19:49:30.025282 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:49:30.020265    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd48fc95_b72d_427d_ac18_b86487770086.slice/crio-cab225fa2ba3711f914d06a852d277dfc4c3df14d7a68b1517843159381f79bb WatchSource:0}: Error finding container cab225fa2ba3711f914d06a852d277dfc4c3df14d7a68b1517843159381f79bb: Status 404 returned error can't find the container with id cab225fa2ba3711f914d06a852d277dfc4c3df14d7a68b1517843159381f79bb
Oct 31 19:49:30.775621 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:30.775588    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:30.832401 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:30.832363    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5e1b35a-cnbfq" event={"ID":"cd48fc95-b72d-427d-ac18-b86487770086","Type":"ContainerStarted","Data":"6733c2051b46505dc74e54ebd20a03f1993f062851339d2c76632b64dbc42f1d"}
Oct 31 19:49:30.832608 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:30.832422    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5e1b35a-cnbfq" event={"ID":"cd48fc95-b72d-427d-ac18-b86487770086","Type":"ContainerStarted","Data":"cab225fa2ba3711f914d06a852d277dfc4c3df14d7a68b1517843159381f79bb"}
Oct 31 19:49:30.856848 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:30.856751    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5e1b35a-cnbfq" podStartSLOduration=1.85673624 podStartE2EDuration="1.85673624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:49:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:49:30.85003536 +0000 UTC m=+1801.423269490" watchObservedRunningTime="2026-10-31 19:49:30.85673624 +0000 UTC m=+1801.429970399"
Oct 31 19:49:31.828204 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:31.827545    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:31.845696 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:31.845653    4645 generic.go:356] "Generic (PLEG): container finished" podID="cd48fc95-b72d-427d-ac18-b86487770086" containerID="6733c2051b46505dc74e54ebd20a03f1993f062851339d2c76632b64dbc42f1d" exitCode=0
Oct 31 19:49:31.845878 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:31.845745    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5e1b35a-cnbfq" event={"ID":"cd48fc95-b72d-427d-ac18-b86487770086","Type":"ContainerDied","Data":"6733c2051b46505dc74e54ebd20a03f1993f062851339d2c76632b64dbc42f1d"}
Oct 31 19:49:32.644554 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:32.644518    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:33.040403 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:33.040380    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5e1b35a-cnbfq"
Oct 31 19:49:33.181461 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:33.181377    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd48fc95-b72d-427d-ac18-b86487770086-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd48fc95-b72d-427d-ac18-b86487770086-maas-api-openshift-ca\") pod \"cd48fc95-b72d-427d-ac18-b86487770086\" (UID: \"cd48fc95-b72d-427d-ac18-b86487770086\") "
Oct 31 19:49:33.181634 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:33.181485    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd48fc95-b72d-427d-ac18-b86487770086-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd48fc95-b72d-427d-ac18-b86487770086-maas-api-xks-ca\") pod \"cd48fc95-b72d-427d-ac18-b86487770086\" (UID: \"cd48fc95-b72d-427d-ac18-b86487770086\") "
Oct 31 19:49:33.181634 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:33.181532    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd48fc95-b72d-427d-ac18-b86487770086-kube-api-access-nx5v5\" (UniqueName: \"kubernetes.io/projected/cd48fc95-b72d-427d-ac18-b86487770086-kube-api-access-nx5v5\") pod \"cd48fc95-b72d-427d-ac18-b86487770086\" (UID: \"cd48fc95-b72d-427d-ac18-b86487770086\") "
Oct 31 19:49:33.181793 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:33.181766    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd48fc95-b72d-427d-ac18-b86487770086-maas-api-openshift-ca" pod "cd48fc95-b72d-427d-ac18-b86487770086" (UID: "cd48fc95-b72d-427d-ac18-b86487770086"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:49:33.182240 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:33.182210    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd48fc95-b72d-427d-ac18-b86487770086-maas-api-openshift-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:49:33.183779 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:33.183750    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd48fc95-b72d-427d-ac18-b86487770086-maas-api-xks-ca" pod "cd48fc95-b72d-427d-ac18-b86487770086" (UID: "cd48fc95-b72d-427d-ac18-b86487770086"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:49:33.183779 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:33.183766    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd48fc95-b72d-427d-ac18-b86487770086-kube-api-access-nx5v5" pod "cd48fc95-b72d-427d-ac18-b86487770086" (UID: "cd48fc95-b72d-427d-ac18-b86487770086"). InnerVolumeSpecName "kube-api-access-nx5v5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:49:33.283179 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:33.283151    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd48fc95-b72d-427d-ac18-b86487770086-maas-api-xks-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:49:33.283274 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:33.283180    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nx5v5\" (UniqueName: \"kubernetes.io/projected/cd48fc95-b72d-427d-ac18-b86487770086-kube-api-access-nx5v5\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:49:33.422528 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:33.422494    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:33.662139 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:33.662111    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-7c892a/test-model-7c892a-kserve-57875c9b5f-r4j8f"
Oct 31 19:49:33.674743 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:33.674720    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-7c892a/test-model-7c892a-kserve-57875c9b5f-r4j8f"
Oct 31 19:49:33.864053 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:33.864025    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5e1b35a-cnbfq"
Oct 31 19:49:33.864206 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:33.864062    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5e1b35a-cnbfq" event={"ID":"cd48fc95-b72d-427d-ac18-b86487770086","Type":"ContainerDied","Data":"cab225fa2ba3711f914d06a852d277dfc4c3df14d7a68b1517843159381f79bb"}
Oct 31 19:49:33.864206 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:33.864112    4645 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cab225fa2ba3711f914d06a852d277dfc4c3df14d7a68b1517843159381f79bb"
Oct 31 19:49:33.875866 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:33.875837    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-7c892a/test-model-7c892a-kserve-57875c9b5f-r4j8f"
Oct 31 19:49:34.394293 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:34.394260    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:34.626037 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:34.626000    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:34.776850 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:34.776814    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:35.350456 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:35.350426    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:36.077308 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:36.077265    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:36.251849 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:49:36.251814    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 19:49:36.956066 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:36.956008    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:37.717953 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:37.717920    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:38.558475 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:38.558436    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:39.439501 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:39.439467    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:40.265683 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:40.265638    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:41.061567 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:41.061526    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:41.942744 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:41.942710    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:42.727064 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:42.727018    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:43.604621 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:43.604576    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-03dfee/test-model-03dfee-kserve-68c5b8bcbf-cwgcs" podUID="6e861821-e221-44d2-956a-66f0f656e1b5" containerName="main" containerID="cri-o://415f7634423a5489e29c5860279bb8e5c3020257b8ed7b665b0fa3936f3e749a" gracePeriod=45
Oct 31 19:49:43.660036 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:43.660002    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:43.789495 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:43.789460    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-88eaf9/test-model-88eaf9-kserve-7f86f56bdf-28gmk"]
Oct 31 19:49:43.821682 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:43.821649    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:43.974014 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:43.973970    4645 generic.go:356] "Generic (PLEG): container finished" podID="6e861821-e221-44d2-956a-66f0f656e1b5" containerID="415f7634423a5489e29c5860279bb8e5c3020257b8ed7b665b0fa3936f3e749a" exitCode=0
Oct 31 19:49:43.974362 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:43.974292    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-03dfee/test-model-03dfee-kserve-68c5b8bcbf-cwgcs" event={"ID":"6e861821-e221-44d2-956a-66f0f656e1b5","Type":"ContainerDied","Data":"415f7634423a5489e29c5860279bb8e5c3020257b8ed7b665b0fa3936f3e749a"}
Oct 31 19:49:44.102927 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:44.102893    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-03dfee-7r697"]
Oct 31 19:49:44.104379 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:44.104354    4645 cpu_manager.go:414] "containerMap: removing container" podUID="cd48fc95-b72d-427d-ac18-b86487770086" containerName="revoke-keys"
Oct 31 19:49:44.104379 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:44.104380    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd48fc95-b72d-427d-ac18-b86487770086" containerName="revoke-keys"
Oct 31 19:49:44.104698 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:44.104682    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd48fc95-b72d-427d-ac18-b86487770086" containerName="revoke-keys"
Oct 31 19:49:44.121387 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:44.110416    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-03dfee-7r697"
Oct 31 19:49:44.128370 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:44.126933    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:49:44.128370 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:44.127054    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 19:49:44.134782 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:44.134756    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-03dfee-7r697"]
Oct 31 19:49:44.150512 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:44.150487    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b907681a-4af7-4147-b812-3f0de1f710c8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-03dfee-7r697\" (UID: \"b907681a-4af7-4147-b812-3f0de1f710c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-03dfee-7r697"
Oct 31 19:49:44.150651 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:44.150625    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b907681a-4af7-4147-b812-3f0de1f710c8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-03dfee-7r697\" (UID: \"b907681a-4af7-4147-b812-3f0de1f710c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-03dfee-7r697"
Oct 31 19:49:44.150757 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:44.150665    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xlzj\" (UniqueName: \"kubernetes.io/projected/b907681a-4af7-4147-b812-3f0de1f710c8-kube-api-access-7xlzj\") pod \"maas-api-revoke-keys-e2e-inf-a-03dfee-7r697\" (UID: \"b907681a-4af7-4147-b812-3f0de1f710c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-03dfee-7r697"
Oct 31 19:49:44.190273 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:44.190248    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-03dfee/test-model-03dfee-kserve-68c5b8bcbf-cwgcs"
Oct 31 19:49:44.252103 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:44.252033    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e861821-e221-44d2-956a-66f0f656e1b5-home\" (UniqueName: \"kubernetes.io/empty-dir/6e861821-e221-44d2-956a-66f0f656e1b5-home\") pod \"6e861821-e221-44d2-956a-66f0f656e1b5\" (UID: \"6e861821-e221-44d2-956a-66f0f656e1b5\") "
Oct 31 19:49:44.252255 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:44.252134    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e861821-e221-44d2-956a-66f0f656e1b5-tls-certs\" (UniqueName: \"kubernetes.io/secret/6e861821-e221-44d2-956a-66f0f656e1b5-tls-certs\") pod \"6e861821-e221-44d2-956a-66f0f656e1b5\" (UID: \"6e861821-e221-44d2-956a-66f0f656e1b5\") "
Oct 31 19:49:44.252255 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:44.252198    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e861821-e221-44d2-956a-66f0f656e1b5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e861821-e221-44d2-956a-66f0f656e1b5-tmp-dir\") pod \"6e861821-e221-44d2-956a-66f0f656e1b5\" (UID: \"6e861821-e221-44d2-956a-66f0f656e1b5\") "
Oct 31 19:49:44.252255 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:44.252226    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e861821-e221-44d2-956a-66f0f656e1b5-kube-api-access-5gp6z\" (UniqueName: \"kubernetes.io/projected/6e861821-e221-44d2-956a-66f0f656e1b5-kube-api-access-5gp6z\") pod \"6e861821-e221-44d2-956a-66f0f656e1b5\" (UID: \"6e861821-e221-44d2-956a-66f0f656e1b5\") "
Oct 31 19:49:44.252438 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:44.252354    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e861821-e221-44d2-956a-66f0f656e1b5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e861821-e221-44d2-956a-66f0f656e1b5-dshm\") pod \"6e861821-e221-44d2-956a-66f0f656e1b5\" (UID: \"6e861821-e221-44d2-956a-66f0f656e1b5\") "
Oct 31 19:49:44.252438 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:44.252314    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e861821-e221-44d2-956a-66f0f656e1b5-home" pod "6e861821-e221-44d2-956a-66f0f656e1b5" (UID: "6e861821-e221-44d2-956a-66f0f656e1b5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:49:44.252548 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:44.252451    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e861821-e221-44d2-956a-66f0f656e1b5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e861821-e221-44d2-956a-66f0f656e1b5-model-cache\") pod \"6e861821-e221-44d2-956a-66f0f656e1b5\" (UID: \"6e861821-e221-44d2-956a-66f0f656e1b5\") "
Oct 31 19:49:44.252913 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:44.252832    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b907681a-4af7-4147-b812-3f0de1f710c8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-03dfee-7r697\" (UID: \"b907681a-4af7-4147-b812-3f0de1f710c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-03dfee-7r697"
Oct 31 19:49:44.253517 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:44.253037    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e861821-e221-44d2-956a-66f0f656e1b5-tmp-dir" pod "6e861821-e221-44d2-956a-66f0f656e1b5" (UID: "6e861821-e221-44d2-956a-66f0f656e1b5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:49:44.253613 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:44.253100    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e861821-e221-44d2-956a-66f0f656e1b5-model-cache" pod "6e861821-e221-44d2-956a-66f0f656e1b5" (UID: "6e861821-e221-44d2-956a-66f0f656e1b5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:49:44.253613 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:44.253473    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b907681a-4af7-4147-b812-3f0de1f710c8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-03dfee-7r697\" (UID: \"b907681a-4af7-4147-b812-3f0de1f710c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-03dfee-7r697"
Oct 31 19:49:44.254350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:44.254290    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b907681a-4af7-4147-b812-3f0de1f710c8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-03dfee-7r697\" (UID: \"b907681a-4af7-4147-b812-3f0de1f710c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-03dfee-7r697"
Oct 31 19:49:44.254583 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:44.254563    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xlzj\" (UniqueName: \"kubernetes.io/projected/b907681a-4af7-4147-b812-3f0de1f710c8-kube-api-access-7xlzj\") pod \"maas-api-revoke-keys-e2e-inf-a-03dfee-7r697\" (UID: \"b907681a-4af7-4147-b812-3f0de1f710c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-03dfee-7r697"
Oct 31 19:49:44.254983 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:44.254958    4645 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e861821-e221-44d2-956a-66f0f656e1b5-home\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:49:44.254983 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:44.254978    4645 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e861821-e221-44d2-956a-66f0f656e1b5-tmp-dir\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:49:44.255151 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:44.254987    4645 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e861821-e221-44d2-956a-66f0f656e1b5-model-cache\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:49:44.255406 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:44.255385    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e861821-e221-44d2-956a-66f0f656e1b5-dshm" pod "6e861821-e221-44d2-956a-66f0f656e1b5" (UID: "6e861821-e221-44d2-956a-66f0f656e1b5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:49:44.255986 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:44.255959    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b907681a-4af7-4147-b812-3f0de1f710c8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-03dfee-7r697\" (UID: \"b907681a-4af7-4147-b812-3f0de1f710c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-03dfee-7r697"
Oct 31 19:49:44.256695 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:44.256671    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e861821-e221-44d2-956a-66f0f656e1b5-tls-certs" pod "6e861821-e221-44d2-956a-66f0f656e1b5" (UID: "6e861821-e221-44d2-956a-66f0f656e1b5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:49:44.257878 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:44.257854    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e861821-e221-44d2-956a-66f0f656e1b5-kube-api-access-5gp6z" pod "6e861821-e221-44d2-956a-66f0f656e1b5" (UID: "6e861821-e221-44d2-956a-66f0f656e1b5"). InnerVolumeSpecName "kube-api-access-5gp6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:49:44.266719 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:44.266696    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xlzj\" (UniqueName: \"kubernetes.io/projected/b907681a-4af7-4147-b812-3f0de1f710c8-kube-api-access-7xlzj\") pod \"maas-api-revoke-keys-e2e-inf-a-03dfee-7r697\" (UID: \"b907681a-4af7-4147-b812-3f0de1f710c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-03dfee-7r697"
Oct 31 19:49:44.356865 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:44.356838    4645 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e861821-e221-44d2-956a-66f0f656e1b5-tls-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:49:44.356865 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:44.356863    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gp6z\" (UniqueName: \"kubernetes.io/projected/6e861821-e221-44d2-956a-66f0f656e1b5-kube-api-access-5gp6z\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:49:44.357027 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:44.356874    4645 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e861821-e221-44d2-956a-66f0f656e1b5-dshm\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:49:44.487206 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:44.487183    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-03dfee-7r697"
Oct 31 19:49:44.781611 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:49:44.781582    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb907681a_4af7_4147_b812_3f0de1f710c8.slice/crio-0559dbd702df62a982401712ca4a1bbc4c09bfe73fcf138cd0f87daed08b0c3b WatchSource:0}: Error finding container 0559dbd702df62a982401712ca4a1bbc4c09bfe73fcf138cd0f87daed08b0c3b: Status 404 returned error can't find the container with id 0559dbd702df62a982401712ca4a1bbc4c09bfe73fcf138cd0f87daed08b0c3b
Oct 31 19:49:44.781766 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:44.781744    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-03dfee-7r697"]
Oct 31 19:49:44.849288 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:44.849264    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:44.985229 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:44.985195    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-03dfee/test-model-03dfee-kserve-68c5b8bcbf-cwgcs"
Oct 31 19:49:44.985424 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:44.985231    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-03dfee/test-model-03dfee-kserve-68c5b8bcbf-cwgcs" event={"ID":"6e861821-e221-44d2-956a-66f0f656e1b5","Type":"ContainerDied","Data":"c523a66a71fe9f46498375c8ad4e86cb4818bcc5bbe8922838aa494ed536b625"}
Oct 31 19:49:44.985424 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:44.985282    4645 scope.go:122] "RemoveContainer" containerID="415f7634423a5489e29c5860279bb8e5c3020257b8ed7b665b0fa3936f3e749a"
Oct 31 19:49:44.986934 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:44.986911    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-03dfee-7r697" event={"ID":"b907681a-4af7-4147-b812-3f0de1f710c8","Type":"ContainerStarted","Data":"0559dbd702df62a982401712ca4a1bbc4c09bfe73fcf138cd0f87daed08b0c3b"}
Oct 31 19:49:45.017101 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:45.017070    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-03dfee/test-model-03dfee-kserve-68c5b8bcbf-cwgcs"]
Oct 31 19:49:45.022321 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:45.022298    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-03dfee/test-model-03dfee-kserve-68c5b8bcbf-cwgcs"]
Oct 31 19:49:45.621629 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:45.621593    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e861821-e221-44d2-956a-66f0f656e1b5" path="/var/lib/kubelet/pods/6e861821-e221-44d2-956a-66f0f656e1b5/volumes"
Oct 31 19:49:45.731264 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:45.731240    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:45.997212 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:45.997180    4645 generic.go:356] "Generic (PLEG): container finished" podID="b907681a-4af7-4147-b812-3f0de1f710c8" containerID="5ace4d970d9936d0fba97d5352a10d2453d8caa8a9205e732b43f50b79744b3e" exitCode=0
Oct 31 19:49:45.997681 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:45.997259    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-03dfee-7r697" event={"ID":"b907681a-4af7-4147-b812-3f0de1f710c8","Type":"ContainerDied","Data":"5ace4d970d9936d0fba97d5352a10d2453d8caa8a9205e732b43f50b79744b3e"}
Oct 31 19:49:46.433312 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:46.433282    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:47.189526 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:47.189499    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-03dfee-7r697"
Oct 31 19:49:47.219370 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:47.215818    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b907681a-4af7-4147-b812-3f0de1f710c8-kube-api-access-7xlzj\" (UniqueName: \"kubernetes.io/projected/b907681a-4af7-4147-b812-3f0de1f710c8-kube-api-access-7xlzj\") pod \"b907681a-4af7-4147-b812-3f0de1f710c8\" (UID: \"b907681a-4af7-4147-b812-3f0de1f710c8\") "
Oct 31 19:49:47.219370 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:47.215866    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b907681a-4af7-4147-b812-3f0de1f710c8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b907681a-4af7-4147-b812-3f0de1f710c8-maas-api-xks-ca\") pod \"b907681a-4af7-4147-b812-3f0de1f710c8\" (UID: \"b907681a-4af7-4147-b812-3f0de1f710c8\") "
Oct 31 19:49:47.219370 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:47.215935    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b907681a-4af7-4147-b812-3f0de1f710c8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b907681a-4af7-4147-b812-3f0de1f710c8-maas-api-openshift-ca\") pod \"b907681a-4af7-4147-b812-3f0de1f710c8\" (UID: \"b907681a-4af7-4147-b812-3f0de1f710c8\") "
Oct 31 19:49:47.219370 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:47.217249    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b907681a-4af7-4147-b812-3f0de1f710c8-maas-api-openshift-ca" pod "b907681a-4af7-4147-b812-3f0de1f710c8" (UID: "b907681a-4af7-4147-b812-3f0de1f710c8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:49:47.219736 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:47.219709    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b907681a-4af7-4147-b812-3f0de1f710c8-kube-api-access-7xlzj" pod "b907681a-4af7-4147-b812-3f0de1f710c8" (UID: "b907681a-4af7-4147-b812-3f0de1f710c8"). InnerVolumeSpecName "kube-api-access-7xlzj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:49:47.224463 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:47.224435    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b907681a-4af7-4147-b812-3f0de1f710c8-maas-api-xks-ca" pod "b907681a-4af7-4147-b812-3f0de1f710c8" (UID: "b907681a-4af7-4147-b812-3f0de1f710c8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:49:47.258679 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:47.258574    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:47.323285 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:47.323244    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xlzj\" (UniqueName: \"kubernetes.io/projected/b907681a-4af7-4147-b812-3f0de1f710c8-kube-api-access-7xlzj\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:49:47.323285 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:47.323287    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b907681a-4af7-4147-b812-3f0de1f710c8-maas-api-xks-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:49:47.323567 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:47.323302    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b907681a-4af7-4147-b812-3f0de1f710c8-maas-api-openshift-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:49:48.017807 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:48.017780    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-03dfee-7r697"
Oct 31 19:49:48.018019 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:48.017808    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-03dfee-7r697" event={"ID":"b907681a-4af7-4147-b812-3f0de1f710c8","Type":"ContainerDied","Data":"0559dbd702df62a982401712ca4a1bbc4c09bfe73fcf138cd0f87daed08b0c3b"}
Oct 31 19:49:48.018019 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:48.017856    4645 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0559dbd702df62a982401712ca4a1bbc4c09bfe73fcf138cd0f87daed08b0c3b"
Oct 31 19:49:48.330217 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:48.330126    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:49.000916 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:49.000873    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:49.240839 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:49.240799    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:49.827183 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:49.827129    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:50.637625 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:50.637592    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:51.437100 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:51.437059    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:52.302781 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:52.302683    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:53.067666 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:53.067617    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:53.890424 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:53.890318    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:54.360411 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:54.360372    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2683f624-g7tlm"]
Oct 31 19:49:54.361298 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:54.361277    4645 cpu_manager.go:414] "containerMap: removing container" podUID="b907681a-4af7-4147-b812-3f0de1f710c8" containerName="revoke-keys"
Oct 31 19:49:54.361298 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:54.361298    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b907681a-4af7-4147-b812-3f0de1f710c8" containerName="revoke-keys"
Oct 31 19:49:54.361488 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:54.361349    4645 cpu_manager.go:414] "containerMap: removing container" podUID="6e861821-e221-44d2-956a-66f0f656e1b5" containerName="main"
Oct 31 19:49:54.361488 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:54.361355    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e861821-e221-44d2-956a-66f0f656e1b5" containerName="main"
Oct 31 19:49:54.361562 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:54.361525    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="b907681a-4af7-4147-b812-3f0de1f710c8" containerName="revoke-keys"
Oct 31 19:49:54.361562 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:54.361540    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e861821-e221-44d2-956a-66f0f656e1b5" containerName="main"
Oct 31 19:49:54.365700 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:54.365671    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2683f624-g7tlm"
Oct 31 19:49:54.370586 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:54.370475    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:49:54.370876 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:54.370849    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 19:49:54.376889 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:54.376856    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2683f624-g7tlm"]
Oct 31 19:49:54.433543 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:54.433508    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:54.472347 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:54.471954    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krlrp\" (UniqueName: \"kubernetes.io/projected/aa03581c-60ab-4f65-a8ea-e5ba6b47a16d-kube-api-access-krlrp\") pod \"maas-api-revoke-sub-2683f624-g7tlm\" (UID: \"aa03581c-60ab-4f65-a8ea-e5ba6b47a16d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2683f624-g7tlm"
Oct 31 19:49:54.472347 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:54.472202    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa03581c-60ab-4f65-a8ea-e5ba6b47a16d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2683f624-g7tlm\" (UID: \"aa03581c-60ab-4f65-a8ea-e5ba6b47a16d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2683f624-g7tlm"
Oct 31 19:49:54.472347 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:54.472314    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa03581c-60ab-4f65-a8ea-e5ba6b47a16d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2683f624-g7tlm\" (UID: \"aa03581c-60ab-4f65-a8ea-e5ba6b47a16d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2683f624-g7tlm"
Oct 31 19:49:54.539209 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:54.539171    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:54.574377 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:54.574293    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa03581c-60ab-4f65-a8ea-e5ba6b47a16d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2683f624-g7tlm\" (UID: \"aa03581c-60ab-4f65-a8ea-e5ba6b47a16d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2683f624-g7tlm"
Oct 31 19:49:54.574596 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:54.574580    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krlrp\" (UniqueName: \"kubernetes.io/projected/aa03581c-60ab-4f65-a8ea-e5ba6b47a16d-kube-api-access-krlrp\") pod \"maas-api-revoke-sub-2683f624-g7tlm\" (UID: \"aa03581c-60ab-4f65-a8ea-e5ba6b47a16d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2683f624-g7tlm"
Oct 31 19:49:54.574745 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:54.574733    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa03581c-60ab-4f65-a8ea-e5ba6b47a16d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2683f624-g7tlm\" (UID: \"aa03581c-60ab-4f65-a8ea-e5ba6b47a16d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2683f624-g7tlm"
Oct 31 19:49:54.575404 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:54.575377    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa03581c-60ab-4f65-a8ea-e5ba6b47a16d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2683f624-g7tlm\" (UID: \"aa03581c-60ab-4f65-a8ea-e5ba6b47a16d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2683f624-g7tlm"
Oct 31 19:49:54.576862 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:54.576838    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa03581c-60ab-4f65-a8ea-e5ba6b47a16d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2683f624-g7tlm\" (UID: \"aa03581c-60ab-4f65-a8ea-e5ba6b47a16d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2683f624-g7tlm"
Oct 31 19:49:54.589358 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:54.589312    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krlrp\" (UniqueName: \"kubernetes.io/projected/aa03581c-60ab-4f65-a8ea-e5ba6b47a16d-kube-api-access-krlrp\") pod \"maas-api-revoke-sub-2683f624-g7tlm\" (UID: \"aa03581c-60ab-4f65-a8ea-e5ba6b47a16d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2683f624-g7tlm"
Oct 31 19:49:54.702357 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:54.696782    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d448ce9-mm5sz"]
Oct 31 19:49:54.705702 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:54.705666    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2683f624-g7tlm"
Oct 31 19:49:54.795115 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:54.795063    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d448ce9-mm5sz"]
Oct 31 19:49:54.795815 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:54.795776    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d448ce9-mm5sz"
Oct 31 19:49:54.902660 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:54.902613    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2f889ae-016e-472f-ab38-346ceb2cd980-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d448ce9-mm5sz\" (UID: \"d2f889ae-016e-472f-ab38-346ceb2cd980\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d448ce9-mm5sz"
Oct 31 19:49:54.905017 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:54.904985    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89bcw\" (UniqueName: \"kubernetes.io/projected/d2f889ae-016e-472f-ab38-346ceb2cd980-kube-api-access-89bcw\") pod \"maas-api-revoke-sub-7d448ce9-mm5sz\" (UID: \"d2f889ae-016e-472f-ab38-346ceb2cd980\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d448ce9-mm5sz"
Oct 31 19:49:54.905513 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:54.905488    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2f889ae-016e-472f-ab38-346ceb2cd980-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d448ce9-mm5sz\" (UID: \"d2f889ae-016e-472f-ab38-346ceb2cd980\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d448ce9-mm5sz"
Oct 31 19:49:55.018141 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:55.018104    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2f889ae-016e-472f-ab38-346ceb2cd980-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d448ce9-mm5sz\" (UID: \"d2f889ae-016e-472f-ab38-346ceb2cd980\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d448ce9-mm5sz"
Oct 31 19:49:55.019147 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:55.019105    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2f889ae-016e-472f-ab38-346ceb2cd980-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d448ce9-mm5sz\" (UID: \"d2f889ae-016e-472f-ab38-346ceb2cd980\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d448ce9-mm5sz"
Oct 31 19:49:55.019622 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:55.019601    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89bcw\" (UniqueName: \"kubernetes.io/projected/d2f889ae-016e-472f-ab38-346ceb2cd980-kube-api-access-89bcw\") pod \"maas-api-revoke-sub-7d448ce9-mm5sz\" (UID: \"d2f889ae-016e-472f-ab38-346ceb2cd980\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d448ce9-mm5sz"
Oct 31 19:49:55.020441 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:55.020423    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2f889ae-016e-472f-ab38-346ceb2cd980-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d448ce9-mm5sz\" (UID: \"d2f889ae-016e-472f-ab38-346ceb2cd980\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d448ce9-mm5sz"
Oct 31 19:49:55.026791 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:55.026767    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2f889ae-016e-472f-ab38-346ceb2cd980-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d448ce9-mm5sz\" (UID: \"d2f889ae-016e-472f-ab38-346ceb2cd980\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d448ce9-mm5sz"
Oct 31 19:49:55.043073 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:55.043034    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89bcw\" (UniqueName: \"kubernetes.io/projected/d2f889ae-016e-472f-ab38-346ceb2cd980-kube-api-access-89bcw\") pod \"maas-api-revoke-sub-7d448ce9-mm5sz\" (UID: \"d2f889ae-016e-472f-ab38-346ceb2cd980\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d448ce9-mm5sz"
Oct 31 19:49:55.108290 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:55.108248    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:55.206893 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:55.206858    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d448ce9-mm5sz"
Oct 31 19:49:55.223370 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:55.223320    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7b1e78a-qg2kl"]
Oct 31 19:49:55.229480 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:55.229401    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7b1e78a-qg2kl"
Oct 31 19:49:55.236995 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:55.236954    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7b1e78a-qg2kl"]
Oct 31 19:49:55.313542 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:55.312965    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2683f624-g7tlm"]
Oct 31 19:49:55.324417 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:55.324356    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znxpl\" (UniqueName: \"kubernetes.io/projected/3003da29-b66f-4a97-a303-decfacf3250a-kube-api-access-znxpl\") pod \"maas-api-revoke-sub-f7b1e78a-qg2kl\" (UID: \"3003da29-b66f-4a97-a303-decfacf3250a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7b1e78a-qg2kl"
Oct 31 19:49:55.324597 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:55.324435    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3003da29-b66f-4a97-a303-decfacf3250a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7b1e78a-qg2kl\" (UID: \"3003da29-b66f-4a97-a303-decfacf3250a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7b1e78a-qg2kl"
Oct 31 19:49:55.324597 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:55.324581    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3003da29-b66f-4a97-a303-decfacf3250a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7b1e78a-qg2kl\" (UID: \"3003da29-b66f-4a97-a303-decfacf3250a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7b1e78a-qg2kl"
Oct 31 19:49:55.325937 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:49:55.325895    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa03581c_60ab_4f65_a8ea_e5ba6b47a16d.slice/crio-94e86384754442cb8ca919058302f2e64a76768763a9d40683f61667737732c4 WatchSource:0}: Error finding container 94e86384754442cb8ca919058302f2e64a76768763a9d40683f61667737732c4: Status 404 returned error can't find the container with id 94e86384754442cb8ca919058302f2e64a76768763a9d40683f61667737732c4
Oct 31 19:49:55.426395 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:55.426284    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znxpl\" (UniqueName: \"kubernetes.io/projected/3003da29-b66f-4a97-a303-decfacf3250a-kube-api-access-znxpl\") pod \"maas-api-revoke-sub-f7b1e78a-qg2kl\" (UID: \"3003da29-b66f-4a97-a303-decfacf3250a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7b1e78a-qg2kl"
Oct 31 19:49:55.426395 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:55.426382    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3003da29-b66f-4a97-a303-decfacf3250a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7b1e78a-qg2kl\" (UID: \"3003da29-b66f-4a97-a303-decfacf3250a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7b1e78a-qg2kl"
Oct 31 19:49:55.426995 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:55.426441    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3003da29-b66f-4a97-a303-decfacf3250a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7b1e78a-qg2kl\" (UID: \"3003da29-b66f-4a97-a303-decfacf3250a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7b1e78a-qg2kl"
Oct 31 19:49:55.427231 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:55.427209    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3003da29-b66f-4a97-a303-decfacf3250a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7b1e78a-qg2kl\" (UID: \"3003da29-b66f-4a97-a303-decfacf3250a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7b1e78a-qg2kl"
Oct 31 19:49:55.429348 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:55.429310    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3003da29-b66f-4a97-a303-decfacf3250a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7b1e78a-qg2kl\" (UID: \"3003da29-b66f-4a97-a303-decfacf3250a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7b1e78a-qg2kl"
Oct 31 19:49:55.438590 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:55.438557    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znxpl\" (UniqueName: \"kubernetes.io/projected/3003da29-b66f-4a97-a303-decfacf3250a-kube-api-access-znxpl\") pod \"maas-api-revoke-sub-f7b1e78a-qg2kl\" (UID: \"3003da29-b66f-4a97-a303-decfacf3250a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7b1e78a-qg2kl"
Oct 31 19:49:55.555625 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:55.555592    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7b1e78a-qg2kl"
Oct 31 19:49:55.562826 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:55.560247    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d448ce9-mm5sz"]
Oct 31 19:49:55.562826 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:49:55.560375    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2f889ae_016e_472f_ab38_346ceb2cd980.slice/crio-0278db3182a4b3bfbd35057a098076d5b1594af3066c7a746f4f3c595c2e9c2a WatchSource:0}: Error finding container 0278db3182a4b3bfbd35057a098076d5b1594af3066c7a746f4f3c595c2e9c2a: Status 404 returned error can't find the container with id 0278db3182a4b3bfbd35057a098076d5b1594af3066c7a746f4f3c595c2e9c2a
Oct 31 19:49:55.864198 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:55.864162    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7b1e78a-qg2kl"]
Oct 31 19:49:55.864391 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:49:55.864149    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3003da29_b66f_4a97_a303_decfacf3250a.slice/crio-a6ca5dc194a412da624e88244091337f421a7e3a9848d716528f8d4d995fdec4 WatchSource:0}: Error finding container a6ca5dc194a412da624e88244091337f421a7e3a9848d716528f8d4d995fdec4: Status 404 returned error can't find the container with id a6ca5dc194a412da624e88244091337f421a7e3a9848d716528f8d4d995fdec4
Oct 31 19:49:55.953435 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:55.953395    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:56.096875 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:56.096834    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d448ce9-mm5sz" event={"ID":"d2f889ae-016e-472f-ab38-346ceb2cd980","Type":"ContainerStarted","Data":"896d2fcb2bbf70573710480e8f8493ab08d97a4e109aa4fad5bf4f55af4dbd43"}
Oct 31 19:49:56.097085 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:56.096945    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d448ce9-mm5sz" event={"ID":"d2f889ae-016e-472f-ab38-346ceb2cd980","Type":"ContainerStarted","Data":"0278db3182a4b3bfbd35057a098076d5b1594af3066c7a746f4f3c595c2e9c2a"}
Oct 31 19:49:56.098731 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:56.098701    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2683f624-g7tlm" event={"ID":"aa03581c-60ab-4f65-a8ea-e5ba6b47a16d","Type":"ContainerStarted","Data":"37e487f42e91062eeeb1a3a9b1fad11b450b86067e96d8912ddc748ceb865121"}
Oct 31 19:49:56.098875 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:56.098758    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2683f624-g7tlm" event={"ID":"aa03581c-60ab-4f65-a8ea-e5ba6b47a16d","Type":"ContainerStarted","Data":"94e86384754442cb8ca919058302f2e64a76768763a9d40683f61667737732c4"}
Oct 31 19:49:56.100227 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:56.100204    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7b1e78a-qg2kl" event={"ID":"3003da29-b66f-4a97-a303-decfacf3250a","Type":"ContainerStarted","Data":"a6ca5dc194a412da624e88244091337f421a7e3a9848d716528f8d4d995fdec4"}
Oct 31 19:49:56.125023 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:56.124913    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d448ce9-mm5sz" podStartSLOduration=2.124897013 podStartE2EDuration="2.124897013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:49:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:49:56.118738852 +0000 UTC m=+1826.691972981" watchObservedRunningTime="2026-10-31 19:49:56.124897013 +0000 UTC m=+1826.698131143"
Oct 31 19:49:56.140603 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:56.140542    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2683f624-g7tlm" podStartSLOduration=2.14052584 podStartE2EDuration="2.14052584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:49:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:49:56.135663932 +0000 UTC m=+1826.708898043" watchObservedRunningTime="2026-10-31 19:49:56.14052584 +0000 UTC m=+1826.713759969"
Oct 31 19:49:56.784631 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:56.784535    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:57.111005 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:57.110973    4645 generic.go:356] "Generic (PLEG): container finished" podID="d2f889ae-016e-472f-ab38-346ceb2cd980" containerID="896d2fcb2bbf70573710480e8f8493ab08d97a4e109aa4fad5bf4f55af4dbd43" exitCode=0
Oct 31 19:49:57.111208 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:57.111010    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d448ce9-mm5sz" event={"ID":"d2f889ae-016e-472f-ab38-346ceb2cd980","Type":"ContainerDied","Data":"896d2fcb2bbf70573710480e8f8493ab08d97a4e109aa4fad5bf4f55af4dbd43"}
Oct 31 19:49:57.112941 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:57.112914    4645 generic.go:356] "Generic (PLEG): container finished" podID="aa03581c-60ab-4f65-a8ea-e5ba6b47a16d" containerID="37e487f42e91062eeeb1a3a9b1fad11b450b86067e96d8912ddc748ceb865121" exitCode=0
Oct 31 19:49:57.113068 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:57.112995    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2683f624-g7tlm" event={"ID":"aa03581c-60ab-4f65-a8ea-e5ba6b47a16d","Type":"ContainerDied","Data":"37e487f42e91062eeeb1a3a9b1fad11b450b86067e96d8912ddc748ceb865121"}
Oct 31 19:49:57.114835 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:57.114810    4645 generic.go:356] "Generic (PLEG): container finished" podID="3003da29-b66f-4a97-a303-decfacf3250a" containerID="7f49c83612a8ab8d67a97adac1df12964bb1b4949eacc3260b89a87fcbffce6c" exitCode=0
Oct 31 19:49:57.114936 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:57.114888    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7b1e78a-qg2kl" event={"ID":"3003da29-b66f-4a97-a303-decfacf3250a","Type":"ContainerDied","Data":"7f49c83612a8ab8d67a97adac1df12964bb1b4949eacc3260b89a87fcbffce6c"}
Oct 31 19:49:57.573276 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:57.573243    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:58.153984 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:58.152908    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-03dfee-7c45f6ff5c-swqcn"]
Oct 31 19:49:58.153984 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:58.153203    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-03dfee-7c45f6ff5c-swqcn" podUID="bf7da3cd-1911-44f0-9e13-b102f000b7d5" containerName="maas-api" containerID="cri-o://99c23c35bc4e7974b79555b922bdfac8457a10f289ce5beb8827a730140350d5" gracePeriod=30
Oct 31 19:49:58.238179 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:58.237987    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:58.538052 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:58.537936    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:58.641379 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:58.638644    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt"]
Oct 31 19:49:58.648890 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:58.648566    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2683f624-g7tlm"
Oct 31 19:49:58.689086 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:58.666054    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:58.689086 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:58.668118    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt"
Oct 31 19:49:58.756269 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:58.754793    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl9l8\" (UniqueName: \"kubernetes.io/projected/63076e22-91cb-460d-8d0f-579d4da51d8c-kube-api-access-bl9l8\") pod \"e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt\" (UID: \"63076e22-91cb-460d-8d0f-579d4da51d8c\") " pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt"
Oct 31 19:49:58.756269 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:58.754925    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63076e22-91cb-460d-8d0f-579d4da51d8c-istio-data\") pod \"e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt\" (UID: \"63076e22-91cb-460d-8d0f-579d4da51d8c\") " pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt"
Oct 31 19:49:58.756269 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:58.754966    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63076e22-91cb-460d-8d0f-579d4da51d8c-workload-certs\") pod \"e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt\" (UID: \"63076e22-91cb-460d-8d0f-579d4da51d8c\") " pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt"
Oct 31 19:49:58.756269 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:58.755035    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63076e22-91cb-460d-8d0f-579d4da51d8c-istio-envoy\") pod \"e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt\" (UID: \"63076e22-91cb-460d-8d0f-579d4da51d8c\") " pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt"
Oct 31 19:49:58.756269 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:58.755156    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63076e22-91cb-460d-8d0f-579d4da51d8c-workload-socket\") pod \"e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt\" (UID: \"63076e22-91cb-460d-8d0f-579d4da51d8c\") " pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt"
Oct 31 19:49:58.756269 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:58.755309    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63076e22-91cb-460d-8d0f-579d4da51d8c-istio-podinfo\") pod \"e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt\" (UID: \"63076e22-91cb-460d-8d0f-579d4da51d8c\") " pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt"
Oct 31 19:49:58.756269 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:58.755426    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63076e22-91cb-460d-8d0f-579d4da51d8c-credential-socket\") pod \"e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt\" (UID: \"63076e22-91cb-460d-8d0f-579d4da51d8c\") " pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt"
Oct 31 19:49:58.756269 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:58.755458    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63076e22-91cb-460d-8d0f-579d4da51d8c-istiod-ca-cert\") pod \"e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt\" (UID: \"63076e22-91cb-460d-8d0f-579d4da51d8c\") " pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt"
Oct 31 19:49:58.756269 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:58.755552    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63076e22-91cb-460d-8d0f-579d4da51d8c-istio-token\") pod \"e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt\" (UID: \"63076e22-91cb-460d-8d0f-579d4da51d8c\") " pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt"
Oct 31 19:49:58.795787 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:58.795751    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt"]
Oct 31 19:49:58.833096 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:58.833066    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-03dfee-7r697"]
Oct 31 19:49:58.845291 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:58.845248    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-03dfee-7r697"]
Oct 31 19:49:58.858067 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:58.856803    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d448ce9-mm5sz"
Oct 31 19:49:58.859300 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:58.859251    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7b1e78a-qg2kl"
Oct 31 19:49:58.859979 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:58.859957    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa03581c-60ab-4f65-a8ea-e5ba6b47a16d-kube-api-access-krlrp\" (UniqueName: \"kubernetes.io/projected/aa03581c-60ab-4f65-a8ea-e5ba6b47a16d-kube-api-access-krlrp\") pod \"aa03581c-60ab-4f65-a8ea-e5ba6b47a16d\" (UID: \"aa03581c-60ab-4f65-a8ea-e5ba6b47a16d\") "
Oct 31 19:49:58.860093 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:58.860002    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa03581c-60ab-4f65-a8ea-e5ba6b47a16d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa03581c-60ab-4f65-a8ea-e5ba6b47a16d-maas-api-openshift-ca\") pod \"aa03581c-60ab-4f65-a8ea-e5ba6b47a16d\" (UID: \"aa03581c-60ab-4f65-a8ea-e5ba6b47a16d\") "
Oct 31 19:49:58.860209 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:58.860193    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa03581c-60ab-4f65-a8ea-e5ba6b47a16d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa03581c-60ab-4f65-a8ea-e5ba6b47a16d-maas-api-xks-ca\") pod \"aa03581c-60ab-4f65-a8ea-e5ba6b47a16d\" (UID: \"aa03581c-60ab-4f65-a8ea-e5ba6b47a16d\") "
Oct 31 19:49:58.860807 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:58.860781    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63076e22-91cb-460d-8d0f-579d4da51d8c-istio-podinfo\") pod \"e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt\" (UID: \"63076e22-91cb-460d-8d0f-579d4da51d8c\") " pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt"
Oct 31 19:49:58.860992 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:58.860973    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63076e22-91cb-460d-8d0f-579d4da51d8c-credential-socket\") pod \"e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt\" (UID: \"63076e22-91cb-460d-8d0f-579d4da51d8c\") " pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt"
Oct 31 19:49:58.861112 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:58.861096    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63076e22-91cb-460d-8d0f-579d4da51d8c-istiod-ca-cert\") pod \"e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt\" (UID: \"63076e22-91cb-460d-8d0f-579d4da51d8c\") " pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt"
Oct 31 19:49:58.861363 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:58.861324    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63076e22-91cb-460d-8d0f-579d4da51d8c-istio-token\") pod \"e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt\" (UID: \"63076e22-91cb-460d-8d0f-579d4da51d8c\") " pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt"
Oct 31 19:49:58.861732 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:58.861707    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl9l8\" (UniqueName: \"kubernetes.io/projected/63076e22-91cb-460d-8d0f-579d4da51d8c-kube-api-access-bl9l8\") pod \"e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt\" (UID: \"63076e22-91cb-460d-8d0f-579d4da51d8c\") " pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt"
Oct 31 19:49:58.861828 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:58.861813    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63076e22-91cb-460d-8d0f-579d4da51d8c-istio-data\") pod \"e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt\" (UID: \"63076e22-91cb-460d-8d0f-579d4da51d8c\") " pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt"
Oct 31 19:49:58.861887 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:58.861859    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63076e22-91cb-460d-8d0f-579d4da51d8c-workload-certs\") pod \"e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt\" (UID: \"63076e22-91cb-460d-8d0f-579d4da51d8c\") " pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt"
Oct 31 19:49:58.861939 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:58.861925    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63076e22-91cb-460d-8d0f-579d4da51d8c-istio-envoy\") pod \"e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt\" (UID: \"63076e22-91cb-460d-8d0f-579d4da51d8c\") " pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt"
Oct 31 19:49:58.862037 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:58.862022    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63076e22-91cb-460d-8d0f-579d4da51d8c-workload-socket\") pod \"e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt\" (UID: \"63076e22-91cb-460d-8d0f-579d4da51d8c\") " pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt"
Oct 31 19:49:58.863152 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:58.862319    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-88eaf9/test-model-88eaf9-kserve-7f86f56bdf-28gmk" podUID="244c8e8b-236e-47cf-99c6-69a0a1fb6af6" containerName="main" containerID="cri-o://8bdbf5432e2a53e1ed07dbb3e8b4ee292e895247fa70c95c29bf09d782a7899c" gracePeriod=45
Oct 31 19:49:58.863152 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:58.862447    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63076e22-91cb-460d-8d0f-579d4da51d8c-credential-socket\") pod \"e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt\" (UID: \"63076e22-91cb-460d-8d0f-579d4da51d8c\") " pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt"
Oct 31 19:49:58.863152 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:58.862525    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63076e22-91cb-460d-8d0f-579d4da51d8c-workload-socket\") pod \"e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt\" (UID: \"63076e22-91cb-460d-8d0f-579d4da51d8c\") " pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt"
Oct 31 19:49:58.863534 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:58.863211    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63076e22-91cb-460d-8d0f-579d4da51d8c-istiod-ca-cert\") pod \"e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt\" (UID: \"63076e22-91cb-460d-8d0f-579d4da51d8c\") " pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt"
Oct 31 19:49:58.863943 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:58.863921    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63076e22-91cb-460d-8d0f-579d4da51d8c-istio-data\") pod \"e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt\" (UID: \"63076e22-91cb-460d-8d0f-579d4da51d8c\") " pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt"
Oct 31 19:49:58.864151 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:58.864130    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63076e22-91cb-460d-8d0f-579d4da51d8c-workload-certs\") pod \"e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt\" (UID: \"63076e22-91cb-460d-8d0f-579d4da51d8c\") " pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt"
Oct 31 19:49:58.868176 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:58.868144    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63076e22-91cb-460d-8d0f-579d4da51d8c-istio-envoy\") pod \"e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt\" (UID: \"63076e22-91cb-460d-8d0f-579d4da51d8c\") " pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt"
Oct 31 19:49:58.881801 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:58.869753    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa03581c-60ab-4f65-a8ea-e5ba6b47a16d-maas-api-openshift-ca" pod "aa03581c-60ab-4f65-a8ea-e5ba6b47a16d" (UID: "aa03581c-60ab-4f65-a8ea-e5ba6b47a16d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:49:58.881801 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:58.872351    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa03581c-60ab-4f65-a8ea-e5ba6b47a16d-maas-api-xks-ca" pod "aa03581c-60ab-4f65-a8ea-e5ba6b47a16d" (UID: "aa03581c-60ab-4f65-a8ea-e5ba6b47a16d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:49:58.881801 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:58.875092    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63076e22-91cb-460d-8d0f-579d4da51d8c-istio-podinfo\") pod \"e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt\" (UID: \"63076e22-91cb-460d-8d0f-579d4da51d8c\") " pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt"
Oct 31 19:49:58.883215 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:58.882902    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa03581c-60ab-4f65-a8ea-e5ba6b47a16d-kube-api-access-krlrp" pod "aa03581c-60ab-4f65-a8ea-e5ba6b47a16d" (UID: "aa03581c-60ab-4f65-a8ea-e5ba6b47a16d"). InnerVolumeSpecName "kube-api-access-krlrp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:49:58.890909 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:58.887793    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63076e22-91cb-460d-8d0f-579d4da51d8c-istio-token\") pod \"e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt\" (UID: \"63076e22-91cb-460d-8d0f-579d4da51d8c\") " pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt"
Oct 31 19:49:58.894017 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:58.893995    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl9l8\" (UniqueName: \"kubernetes.io/projected/63076e22-91cb-460d-8d0f-579d4da51d8c-kube-api-access-bl9l8\") pod \"e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt\" (UID: \"63076e22-91cb-460d-8d0f-579d4da51d8c\") " pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt"
Oct 31 19:49:58.940361 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:58.940246    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:58.966819 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:58.966784    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2f889ae-016e-472f-ab38-346ceb2cd980-kube-api-access-89bcw\" (UniqueName: \"kubernetes.io/projected/d2f889ae-016e-472f-ab38-346ceb2cd980-kube-api-access-89bcw\") pod \"d2f889ae-016e-472f-ab38-346ceb2cd980\" (UID: \"d2f889ae-016e-472f-ab38-346ceb2cd980\") "
Oct 31 19:49:58.967215 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:58.967174    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3003da29-b66f-4a97-a303-decfacf3250a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3003da29-b66f-4a97-a303-decfacf3250a-maas-api-openshift-ca\") pod \"3003da29-b66f-4a97-a303-decfacf3250a\" (UID: \"3003da29-b66f-4a97-a303-decfacf3250a\") "
Oct 31 19:49:58.967453 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:58.967417    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2f889ae-016e-472f-ab38-346ceb2cd980-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2f889ae-016e-472f-ab38-346ceb2cd980-maas-api-xks-ca\") pod \"d2f889ae-016e-472f-ab38-346ceb2cd980\" (UID: \"d2f889ae-016e-472f-ab38-346ceb2cd980\") "
Oct 31 19:49:58.971504 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:58.967743    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2f889ae-016e-472f-ab38-346ceb2cd980-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2f889ae-016e-472f-ab38-346ceb2cd980-maas-api-openshift-ca\") pod \"d2f889ae-016e-472f-ab38-346ceb2cd980\" (UID: \"d2f889ae-016e-472f-ab38-346ceb2cd980\") "
Oct 31 19:49:58.971807 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:58.971772    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3003da29-b66f-4a97-a303-decfacf3250a-kube-api-access-znxpl\" (UniqueName: \"kubernetes.io/projected/3003da29-b66f-4a97-a303-decfacf3250a-kube-api-access-znxpl\") pod \"3003da29-b66f-4a97-a303-decfacf3250a\" (UID: \"3003da29-b66f-4a97-a303-decfacf3250a\") "
Oct 31 19:49:58.980658 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:58.979711    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3003da29-b66f-4a97-a303-decfacf3250a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3003da29-b66f-4a97-a303-decfacf3250a-maas-api-xks-ca\") pod \"3003da29-b66f-4a97-a303-decfacf3250a\" (UID: \"3003da29-b66f-4a97-a303-decfacf3250a\") "
Oct 31 19:49:58.996990 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:58.981006    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa03581c-60ab-4f65-a8ea-e5ba6b47a16d-maas-api-openshift-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:49:58.996990 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:58.981037    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa03581c-60ab-4f65-a8ea-e5ba6b47a16d-maas-api-xks-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:49:58.996990 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:58.981052    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krlrp\" (UniqueName: \"kubernetes.io/projected/aa03581c-60ab-4f65-a8ea-e5ba6b47a16d-kube-api-access-krlrp\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:49:58.996990 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:58.969677    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3003da29-b66f-4a97-a303-decfacf3250a-maas-api-openshift-ca" pod "3003da29-b66f-4a97-a303-decfacf3250a" (UID: "3003da29-b66f-4a97-a303-decfacf3250a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:49:58.996990 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:58.970285    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2f889ae-016e-472f-ab38-346ceb2cd980-maas-api-openshift-ca" pod "d2f889ae-016e-472f-ab38-346ceb2cd980" (UID: "d2f889ae-016e-472f-ab38-346ceb2cd980"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:49:58.998691 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:58.998323    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3003da29-b66f-4a97-a303-decfacf3250a-maas-api-xks-ca" pod "3003da29-b66f-4a97-a303-decfacf3250a" (UID: "3003da29-b66f-4a97-a303-decfacf3250a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:49:59.003032 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:59.003004    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-03dfee-7c45f6ff5c-swqcn"
Oct 31 19:49:59.012258 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:59.012086    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3003da29-b66f-4a97-a303-decfacf3250a-kube-api-access-znxpl" pod "3003da29-b66f-4a97-a303-decfacf3250a" (UID: "3003da29-b66f-4a97-a303-decfacf3250a"). InnerVolumeSpecName "kube-api-access-znxpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:49:59.012258 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:59.012222    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2f889ae-016e-472f-ab38-346ceb2cd980-kube-api-access-89bcw" pod "d2f889ae-016e-472f-ab38-346ceb2cd980" (UID: "d2f889ae-016e-472f-ab38-346ceb2cd980"). InnerVolumeSpecName "kube-api-access-89bcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:49:59.013574 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:59.013548    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2f889ae-016e-472f-ab38-346ceb2cd980-maas-api-xks-ca" pod "d2f889ae-016e-472f-ab38-346ceb2cd980" (UID: "d2f889ae-016e-472f-ab38-346ceb2cd980"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:49:59.082429 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:59.082393    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf7da3cd-1911-44f0-9e13-b102f000b7d5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf7da3cd-1911-44f0-9e13-b102f000b7d5-openshift-service-ca\") pod \"bf7da3cd-1911-44f0-9e13-b102f000b7d5\" (UID: \"bf7da3cd-1911-44f0-9e13-b102f000b7d5\") "
Oct 31 19:49:59.082625 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:59.082576    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf7da3cd-1911-44f0-9e13-b102f000b7d5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf7da3cd-1911-44f0-9e13-b102f000b7d5-maas-api-tls\") pod \"bf7da3cd-1911-44f0-9e13-b102f000b7d5\" (UID: \"bf7da3cd-1911-44f0-9e13-b102f000b7d5\") "
Oct 31 19:49:59.082707 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:59.082635    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf7da3cd-1911-44f0-9e13-b102f000b7d5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf7da3cd-1911-44f0-9e13-b102f000b7d5-metrics-certs\") pod \"bf7da3cd-1911-44f0-9e13-b102f000b7d5\" (UID: \"bf7da3cd-1911-44f0-9e13-b102f000b7d5\") "
Oct 31 19:49:59.082849 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:59.082824    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf7da3cd-1911-44f0-9e13-b102f000b7d5-openshift-service-ca" pod "bf7da3cd-1911-44f0-9e13-b102f000b7d5" (UID: "bf7da3cd-1911-44f0-9e13-b102f000b7d5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:49:59.082914 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:59.082878    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf7da3cd-1911-44f0-9e13-b102f000b7d5-kube-api-access-4nv87\" (UniqueName: \"kubernetes.io/projected/bf7da3cd-1911-44f0-9e13-b102f000b7d5-kube-api-access-4nv87\") pod \"bf7da3cd-1911-44f0-9e13-b102f000b7d5\" (UID: \"bf7da3cd-1911-44f0-9e13-b102f000b7d5\") "
Oct 31 19:49:59.083862 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:59.083837    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2f889ae-016e-472f-ab38-346ceb2cd980-maas-api-openshift-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:49:59.084018 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:59.084000    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znxpl\" (UniqueName: \"kubernetes.io/projected/3003da29-b66f-4a97-a303-decfacf3250a-kube-api-access-znxpl\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:49:59.084098 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:59.084022    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3003da29-b66f-4a97-a303-decfacf3250a-maas-api-xks-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:49:59.084098 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:59.084037    4645 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf7da3cd-1911-44f0-9e13-b102f000b7d5-openshift-service-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:49:59.084098 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:59.084051    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89bcw\" (UniqueName: \"kubernetes.io/projected/d2f889ae-016e-472f-ab38-346ceb2cd980-kube-api-access-89bcw\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:49:59.084098 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:59.084066    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3003da29-b66f-4a97-a303-decfacf3250a-maas-api-openshift-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:49:59.084098 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:59.084081    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2f889ae-016e-472f-ab38-346ceb2cd980-maas-api-xks-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:49:59.085233 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:59.085201    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf7da3cd-1911-44f0-9e13-b102f000b7d5-metrics-certs" pod "bf7da3cd-1911-44f0-9e13-b102f000b7d5" (UID: "bf7da3cd-1911-44f0-9e13-b102f000b7d5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:49:59.085415 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:59.085277    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf7da3cd-1911-44f0-9e13-b102f000b7d5-maas-api-tls" pod "bf7da3cd-1911-44f0-9e13-b102f000b7d5" (UID: "bf7da3cd-1911-44f0-9e13-b102f000b7d5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:49:59.085598 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:59.085581    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf7da3cd-1911-44f0-9e13-b102f000b7d5-kube-api-access-4nv87" pod "bf7da3cd-1911-44f0-9e13-b102f000b7d5" (UID: "bf7da3cd-1911-44f0-9e13-b102f000b7d5"). InnerVolumeSpecName "kube-api-access-4nv87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:49:59.133318 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:59.133291    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt"
Oct 31 19:49:59.140854 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:59.139340    4645 generic.go:356] "Generic (PLEG): container finished" podID="bf7da3cd-1911-44f0-9e13-b102f000b7d5" containerID="99c23c35bc4e7974b79555b922bdfac8457a10f289ce5beb8827a730140350d5" exitCode=0
Oct 31 19:49:59.140854 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:59.139418    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-03dfee-7c45f6ff5c-swqcn" event={"ID":"bf7da3cd-1911-44f0-9e13-b102f000b7d5","Type":"ContainerDied","Data":"99c23c35bc4e7974b79555b922bdfac8457a10f289ce5beb8827a730140350d5"}
Oct 31 19:49:59.140854 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:59.139529    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-03dfee-7c45f6ff5c-swqcn"
Oct 31 19:49:59.140854 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:59.139580    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-03dfee-7c45f6ff5c-swqcn" event={"ID":"bf7da3cd-1911-44f0-9e13-b102f000b7d5","Type":"ContainerDied","Data":"062a512e51c355c70b02bd7f000338b254582d93864f3dfd6cc564c5f1a2dfa6"}
Oct 31 19:49:59.140854 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:59.139639    4645 scope.go:122] "RemoveContainer" containerID="99c23c35bc4e7974b79555b922bdfac8457a10f289ce5beb8827a730140350d5"
Oct 31 19:49:59.161524 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:59.151777    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d448ce9-mm5sz" event={"ID":"d2f889ae-016e-472f-ab38-346ceb2cd980","Type":"ContainerDied","Data":"0278db3182a4b3bfbd35057a098076d5b1594af3066c7a746f4f3c595c2e9c2a"}
Oct 31 19:49:59.161524 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:59.151833    4645 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0278db3182a4b3bfbd35057a098076d5b1594af3066c7a746f4f3c595c2e9c2a"
Oct 31 19:49:59.161524 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:59.151916    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d448ce9-mm5sz"
Oct 31 19:49:59.164238 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:59.164200    4645 generic.go:356] "Generic (PLEG): container finished" podID="244c8e8b-236e-47cf-99c6-69a0a1fb6af6" containerID="8bdbf5432e2a53e1ed07dbb3e8b4ee292e895247fa70c95c29bf09d782a7899c" exitCode=0
Oct 31 19:49:59.164384 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:59.164313    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-88eaf9/test-model-88eaf9-kserve-7f86f56bdf-28gmk" event={"ID":"244c8e8b-236e-47cf-99c6-69a0a1fb6af6","Type":"ContainerDied","Data":"8bdbf5432e2a53e1ed07dbb3e8b4ee292e895247fa70c95c29bf09d782a7899c"}
Oct 31 19:49:59.167550 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:59.167527    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2683f624-g7tlm"
Oct 31 19:49:59.167645 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:59.167541    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2683f624-g7tlm" event={"ID":"aa03581c-60ab-4f65-a8ea-e5ba6b47a16d","Type":"ContainerDied","Data":"94e86384754442cb8ca919058302f2e64a76768763a9d40683f61667737732c4"}
Oct 31 19:49:59.167645 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:59.167606    4645 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94e86384754442cb8ca919058302f2e64a76768763a9d40683f61667737732c4"
Oct 31 19:49:59.169878 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:59.169849    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7b1e78a-qg2kl" event={"ID":"3003da29-b66f-4a97-a303-decfacf3250a","Type":"ContainerDied","Data":"a6ca5dc194a412da624e88244091337f421a7e3a9848d716528f8d4d995fdec4"}
Oct 31 19:49:59.169988 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:59.169899    4645 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6ca5dc194a412da624e88244091337f421a7e3a9848d716528f8d4d995fdec4"
Oct 31 19:49:59.169988 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:59.169961    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7b1e78a-qg2kl"
Oct 31 19:49:59.183323 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:59.183307    4645 scope.go:122] "RemoveContainer" containerID="99c23c35bc4e7974b79555b922bdfac8457a10f289ce5beb8827a730140350d5"
Oct 31 19:49:59.183323 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:49:59.183584    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99c23c35bc4e7974b79555b922bdfac8457a10f289ce5beb8827a730140350d5\": container with ID starting with 99c23c35bc4e7974b79555b922bdfac8457a10f289ce5beb8827a730140350d5 not found: ID does not exist" containerID="99c23c35bc4e7974b79555b922bdfac8457a10f289ce5beb8827a730140350d5"
Oct 31 19:49:59.183323 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:59.183620    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99c23c35bc4e7974b79555b922bdfac8457a10f289ce5beb8827a730140350d5"} err="failed to get container status \"99c23c35bc4e7974b79555b922bdfac8457a10f289ce5beb8827a730140350d5\": rpc error: code = NotFound desc = could not find container \"99c23c35bc4e7974b79555b922bdfac8457a10f289ce5beb8827a730140350d5\": container with ID starting with 99c23c35bc4e7974b79555b922bdfac8457a10f289ce5beb8827a730140350d5 not found: ID does not exist"
Oct 31 19:49:59.185774 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:59.185744    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nv87\" (UniqueName: \"kubernetes.io/projected/bf7da3cd-1911-44f0-9e13-b102f000b7d5-kube-api-access-4nv87\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:49:59.185858 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:59.185783    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bf7da3cd-1911-44f0-9e13-b102f000b7d5-maas-api-tls\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:49:59.185858 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:59.185802    4645 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bf7da3cd-1911-44f0-9e13-b102f000b7d5-metrics-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:49:59.228798 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:59.227584    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-03dfee-7c45f6ff5c-swqcn"]
Oct 31 19:49:59.246230 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:49:59.246193    4645 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2f889ae_016e_472f_ab38_346ceb2cd980.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf7da3cd_1911_44f0_9e13_b102f000b7d5.slice/crio-062a512e51c355c70b02bd7f000338b254582d93864f3dfd6cc564c5f1a2dfa6\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf7da3cd_1911_44f0_9e13_b102f000b7d5.slice\": RecentStats: unable to find data in memory cache]"
Oct 31 19:49:59.251502 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:59.251407    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-03dfee-7c45f6ff5c-swqcn"]
Oct 31 19:49:59.280182 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:59.280152    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-88eaf9/test-model-88eaf9-kserve-7f86f56bdf-28gmk"
Oct 31 19:49:59.310379 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:59.310298    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:49:59.388293 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:59.388266    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/244c8e8b-236e-47cf-99c6-69a0a1fb6af6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/244c8e8b-236e-47cf-99c6-69a0a1fb6af6-tmp-dir\") pod \"244c8e8b-236e-47cf-99c6-69a0a1fb6af6\" (UID: \"244c8e8b-236e-47cf-99c6-69a0a1fb6af6\") "
Oct 31 19:49:59.388467 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:59.388453    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/244c8e8b-236e-47cf-99c6-69a0a1fb6af6-kube-api-access-cfmmj\" (UniqueName: \"kubernetes.io/projected/244c8e8b-236e-47cf-99c6-69a0a1fb6af6-kube-api-access-cfmmj\") pod \"244c8e8b-236e-47cf-99c6-69a0a1fb6af6\" (UID: \"244c8e8b-236e-47cf-99c6-69a0a1fb6af6\") "
Oct 31 19:49:59.388527 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:59.388501    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/244c8e8b-236e-47cf-99c6-69a0a1fb6af6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/244c8e8b-236e-47cf-99c6-69a0a1fb6af6-model-cache\") pod \"244c8e8b-236e-47cf-99c6-69a0a1fb6af6\" (UID: \"244c8e8b-236e-47cf-99c6-69a0a1fb6af6\") "
Oct 31 19:49:59.388571 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:59.388553    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/244c8e8b-236e-47cf-99c6-69a0a1fb6af6-tmp-dir" pod "244c8e8b-236e-47cf-99c6-69a0a1fb6af6" (UID: "244c8e8b-236e-47cf-99c6-69a0a1fb6af6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:49:59.388631 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:59.388616    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/244c8e8b-236e-47cf-99c6-69a0a1fb6af6-tls-certs\" (UniqueName: \"kubernetes.io/secret/244c8e8b-236e-47cf-99c6-69a0a1fb6af6-tls-certs\") pod \"244c8e8b-236e-47cf-99c6-69a0a1fb6af6\" (UID: \"244c8e8b-236e-47cf-99c6-69a0a1fb6af6\") "
Oct 31 19:49:59.388797 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:59.388776    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/244c8e8b-236e-47cf-99c6-69a0a1fb6af6-model-cache" pod "244c8e8b-236e-47cf-99c6-69a0a1fb6af6" (UID: "244c8e8b-236e-47cf-99c6-69a0a1fb6af6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:49:59.388860 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:59.388848    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/244c8e8b-236e-47cf-99c6-69a0a1fb6af6-home\" (UniqueName: \"kubernetes.io/empty-dir/244c8e8b-236e-47cf-99c6-69a0a1fb6af6-home\") pod \"244c8e8b-236e-47cf-99c6-69a0a1fb6af6\" (UID: \"244c8e8b-236e-47cf-99c6-69a0a1fb6af6\") "
Oct 31 19:49:59.388907 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:59.388882    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/244c8e8b-236e-47cf-99c6-69a0a1fb6af6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/244c8e8b-236e-47cf-99c6-69a0a1fb6af6-dshm\") pod \"244c8e8b-236e-47cf-99c6-69a0a1fb6af6\" (UID: \"244c8e8b-236e-47cf-99c6-69a0a1fb6af6\") "
Oct 31 19:49:59.389098 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:59.389077    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/244c8e8b-236e-47cf-99c6-69a0a1fb6af6-home" pod "244c8e8b-236e-47cf-99c6-69a0a1fb6af6" (UID: "244c8e8b-236e-47cf-99c6-69a0a1fb6af6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:49:59.389620 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:59.389601    4645 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/244c8e8b-236e-47cf-99c6-69a0a1fb6af6-model-cache\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:49:59.389707 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:59.389624    4645 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/244c8e8b-236e-47cf-99c6-69a0a1fb6af6-home\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:49:59.389707 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:59.389638    4645 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/244c8e8b-236e-47cf-99c6-69a0a1fb6af6-tmp-dir\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:49:59.390724 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:59.390707    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/244c8e8b-236e-47cf-99c6-69a0a1fb6af6-tls-certs" pod "244c8e8b-236e-47cf-99c6-69a0a1fb6af6" (UID: "244c8e8b-236e-47cf-99c6-69a0a1fb6af6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:49:59.390830 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:59.390811    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/244c8e8b-236e-47cf-99c6-69a0a1fb6af6-kube-api-access-cfmmj" pod "244c8e8b-236e-47cf-99c6-69a0a1fb6af6" (UID: "244c8e8b-236e-47cf-99c6-69a0a1fb6af6"). InnerVolumeSpecName "kube-api-access-cfmmj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:49:59.390869 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:59.390826    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/244c8e8b-236e-47cf-99c6-69a0a1fb6af6-dshm" pod "244c8e8b-236e-47cf-99c6-69a0a1fb6af6" (UID: "244c8e8b-236e-47cf-99c6-69a0a1fb6af6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:49:59.483892 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:59.483865    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt"]
Oct 31 19:49:59.484551 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:49:59.484529    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63076e22_91cb_460d_8d0f_579d4da51d8c.slice/crio-18331802d8bb37b545bf019e23e0d113d53e127c1b14f7f5c5bab3ab4bbd011d WatchSource:0}: Error finding container 18331802d8bb37b545bf019e23e0d113d53e127c1b14f7f5c5bab3ab4bbd011d: Status 404 returned error can't find the container with id 18331802d8bb37b545bf019e23e0d113d53e127c1b14f7f5c5bab3ab4bbd011d
Oct 31 19:49:59.487710 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:59.487678    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:49:59.487789 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:59.487742    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:49:59.487789 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:59.487772    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:49:59.491591 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:59.491403    4645 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/244c8e8b-236e-47cf-99c6-69a0a1fb6af6-tls-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:49:59.491591 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:59.491426    4645 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/244c8e8b-236e-47cf-99c6-69a0a1fb6af6-dshm\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:49:59.491591 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:59.491440    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfmmj\" (UniqueName: \"kubernetes.io/projected/244c8e8b-236e-47cf-99c6-69a0a1fb6af6-kube-api-access-cfmmj\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:49:59.625323 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:59.625288    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b907681a-4af7-4147-b812-3f0de1f710c8" path="/var/lib/kubelet/pods/b907681a-4af7-4147-b812-3f0de1f710c8/volumes"
Oct 31 19:49:59.625847 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:59.625829    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf7da3cd-1911-44f0-9e13-b102f000b7d5" path="/var/lib/kubelet/pods/bf7da3cd-1911-44f0-9e13-b102f000b7d5/volumes"
Oct 31 19:49:59.829204 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:49:59.829114    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:00.181055 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:00.181013    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt" event={"ID":"63076e22-91cb-460d-8d0f-579d4da51d8c","Type":"ContainerStarted","Data":"2ba6f95de5abd698e58dbde9e45f200890221940dc84b9deda9d3b2e70c54279"}
Oct 31 19:50:00.181532 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:00.181084    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt" event={"ID":"63076e22-91cb-460d-8d0f-579d4da51d8c","Type":"ContainerStarted","Data":"18331802d8bb37b545bf019e23e0d113d53e127c1b14f7f5c5bab3ab4bbd011d"}
Oct 31 19:50:00.182630 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:00.182609    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-88eaf9/test-model-88eaf9-kserve-7f86f56bdf-28gmk"
Oct 31 19:50:00.182732 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:00.182649    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-88eaf9/test-model-88eaf9-kserve-7f86f56bdf-28gmk" event={"ID":"244c8e8b-236e-47cf-99c6-69a0a1fb6af6","Type":"ContainerDied","Data":"f109811389ae13edbd72ba99b678c6100f9280133bc50c1c014d814ee7653ab1"}
Oct 31 19:50:00.182732 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:00.182697    4645 scope.go:122] "RemoveContainer" containerID="8bdbf5432e2a53e1ed07dbb3e8b4ee292e895247fa70c95c29bf09d782a7899c"
Oct 31 19:50:00.213688 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:00.213636    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt" podStartSLOduration=2.213623047 podStartE2EDuration="2.213623047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:49:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:50:00.209565434 +0000 UTC m=+1830.782799563" watchObservedRunningTime="2026-10-31 19:50:00.213623047 +0000 UTC m=+1830.786857172"
Oct 31 19:50:00.230755 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:00.230731    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-88eaf9/test-model-88eaf9-kserve-7f86f56bdf-28gmk"]
Oct 31 19:50:00.235650 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:00.235628    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-88eaf9/test-model-88eaf9-kserve-7f86f56bdf-28gmk"]
Oct 31 19:50:00.637871 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:00.637777    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:01.134411 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:01.134378    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt"
Oct 31 19:50:01.136248 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:01.136214    4645 patch_prober.go:29] interesting pod/e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Oct 31 19:50:01.136403 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:01.136278    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt" podUID="63076e22-91cb-460d-8d0f-579d4da51d8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Oct 31 19:50:01.477926 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:01.477890    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:01.623127 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:01.623092    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="244c8e8b-236e-47cf-99c6-69a0a1fb6af6" path="/var/lib/kubelet/pods/244c8e8b-236e-47cf-99c6-69a0a1fb6af6/volumes"
Oct 31 19:50:02.134542 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:02.134508    4645 patch_prober.go:29] interesting pod/e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Oct 31 19:50:02.134777 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:02.134554    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt" podUID="63076e22-91cb-460d-8d0f-579d4da51d8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Oct 31 19:50:02.383664 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:02.383630    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:03.134123 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:03.134087    4645 patch_prober.go:29] interesting pod/e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Oct 31 19:50:03.134549 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:03.134142    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt" podUID="63076e22-91cb-460d-8d0f-579d4da51d8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Oct 31 19:50:03.233241 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:03.233152    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:03.981874 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:03.981839    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:04.133849 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:04.133817    4645 patch_prober.go:29] interesting pod/e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Oct 31 19:50:04.134024 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:04.133876    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt" podUID="63076e22-91cb-460d-8d0f-579d4da51d8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Oct 31 19:50:04.854869 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:04.854823    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:05.137445 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:05.137358    4645 patch_prober.go:29] interesting pod/e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Oct 31 19:50:05.137445 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:05.137416    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt" podUID="63076e22-91cb-460d-8d0f-579d4da51d8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Oct 31 19:50:05.142258 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:05.142228    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-b1fae7/test-model-b1fae7-kserve-7c67554bff-4dwfc"]
Oct 31 19:50:05.195990 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:05.195960    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:05.684381 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:05.683272    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-278c04/test-model-278c04-kserve-6b6f77c967-2f8mt"]
Oct 31 19:50:05.789461 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:05.789424    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:05.837753 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:05.836094    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e160cf2-wqzvm"]
Oct 31 19:50:05.837753 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:05.837349    4645 cpu_manager.go:414] "containerMap: removing container" podUID="244c8e8b-236e-47cf-99c6-69a0a1fb6af6" containerName="main"
Oct 31 19:50:05.838269 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:05.838197    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="244c8e8b-236e-47cf-99c6-69a0a1fb6af6" containerName="main"
Oct 31 19:50:05.838441 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:05.838406    4645 cpu_manager.go:414] "containerMap: removing container" podUID="3003da29-b66f-4a97-a303-decfacf3250a" containerName="revoke-keys"
Oct 31 19:50:05.838596 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:05.838531    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3003da29-b66f-4a97-a303-decfacf3250a" containerName="revoke-keys"
Oct 31 19:50:05.838783 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:05.838711    4645 cpu_manager.go:414] "containerMap: removing container" podUID="aa03581c-60ab-4f65-a8ea-e5ba6b47a16d" containerName="revoke-keys"
Oct 31 19:50:05.838783 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:05.838728    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa03581c-60ab-4f65-a8ea-e5ba6b47a16d" containerName="revoke-keys"
Oct 31 19:50:05.839143 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:05.838927    4645 cpu_manager.go:414] "containerMap: removing container" podUID="d2f889ae-016e-472f-ab38-346ceb2cd980" containerName="revoke-keys"
Oct 31 19:50:05.839143 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:05.838940    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2f889ae-016e-472f-ab38-346ceb2cd980" containerName="revoke-keys"
Oct 31 19:50:05.839143 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:05.839011    4645 cpu_manager.go:414] "containerMap: removing container" podUID="bf7da3cd-1911-44f0-9e13-b102f000b7d5" containerName="maas-api"
Oct 31 19:50:05.839143 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:05.839019    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf7da3cd-1911-44f0-9e13-b102f000b7d5" containerName="maas-api"
Oct 31 19:50:05.839848 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:05.839654    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="3003da29-b66f-4a97-a303-decfacf3250a" containerName="revoke-keys"
Oct 31 19:50:05.839848 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:05.839681    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa03581c-60ab-4f65-a8ea-e5ba6b47a16d" containerName="revoke-keys"
Oct 31 19:50:05.839848 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:05.839692    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2f889ae-016e-472f-ab38-346ceb2cd980" containerName="revoke-keys"
Oct 31 19:50:05.839848 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:05.839707    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf7da3cd-1911-44f0-9e13-b102f000b7d5" containerName="maas-api"
Oct 31 19:50:05.839848 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:05.839723    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="244c8e8b-236e-47cf-99c6-69a0a1fb6af6" containerName="main"
Oct 31 19:50:05.847932 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:05.846775    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e160cf2-wqzvm"
Oct 31 19:50:05.863389 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:05.859556    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:50:05.863389 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:05.859674    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 19:50:05.915929 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:05.915896    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e160cf2-wqzvm"]
Oct 31 19:50:05.941811 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:05.940906    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c13db3a-c224-4a77-b079-5d491d2592a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e160cf2-wqzvm\" (UID: \"5c13db3a-c224-4a77-b079-5d491d2592a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e160cf2-wqzvm"
Oct 31 19:50:05.941811 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:05.941114    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c13db3a-c224-4a77-b079-5d491d2592a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e160cf2-wqzvm\" (UID: \"5c13db3a-c224-4a77-b079-5d491d2592a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e160cf2-wqzvm"
Oct 31 19:50:05.941811 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:05.941167    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7krvg\" (UniqueName: \"kubernetes.io/projected/5c13db3a-c224-4a77-b079-5d491d2592a0-kube-api-access-7krvg\") pod \"maas-api-revoke-sub-0e160cf2-wqzvm\" (UID: \"5c13db3a-c224-4a77-b079-5d491d2592a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e160cf2-wqzvm"
Oct 31 19:50:05.959403 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:05.956842    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk"]
Oct 31 19:50:05.959403 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:05.957082    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk" podUID="1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99" containerName="istio-proxy" containerID="cri-o://9764f894f229b36d3d6d8d7485ec8d3f328d69d6524026ca88a4fd6d1b51a530" gracePeriod=30
Oct 31 19:50:06.095348 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:06.095284    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c13db3a-c224-4a77-b079-5d491d2592a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e160cf2-wqzvm\" (UID: \"5c13db3a-c224-4a77-b079-5d491d2592a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e160cf2-wqzvm"
Oct 31 19:50:06.097656 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:06.095585    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c13db3a-c224-4a77-b079-5d491d2592a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e160cf2-wqzvm\" (UID: \"5c13db3a-c224-4a77-b079-5d491d2592a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e160cf2-wqzvm"
Oct 31 19:50:06.097656 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:06.095705    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7krvg\" (UniqueName: \"kubernetes.io/projected/5c13db3a-c224-4a77-b079-5d491d2592a0-kube-api-access-7krvg\") pod \"maas-api-revoke-sub-0e160cf2-wqzvm\" (UID: \"5c13db3a-c224-4a77-b079-5d491d2592a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e160cf2-wqzvm"
Oct 31 19:50:06.098140 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:06.098109    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c13db3a-c224-4a77-b079-5d491d2592a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e160cf2-wqzvm\" (UID: \"5c13db3a-c224-4a77-b079-5d491d2592a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e160cf2-wqzvm"
Oct 31 19:50:06.101555 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:06.101359    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c13db3a-c224-4a77-b079-5d491d2592a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e160cf2-wqzvm\" (UID: \"5c13db3a-c224-4a77-b079-5d491d2592a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e160cf2-wqzvm"
Oct 31 19:50:06.132005 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:06.131973    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7krvg\" (UniqueName: \"kubernetes.io/projected/5c13db3a-c224-4a77-b079-5d491d2592a0-kube-api-access-7krvg\") pod \"maas-api-revoke-sub-0e160cf2-wqzvm\" (UID: \"5c13db3a-c224-4a77-b079-5d491d2592a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e160cf2-wqzvm"
Oct 31 19:50:06.144380 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:06.142656    4645 patch_prober.go:29] interesting pod/e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Oct 31 19:50:06.144380 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:06.142712    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt" podUID="63076e22-91cb-460d-8d0f-579d4da51d8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Oct 31 19:50:06.192868 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:06.192785    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e160cf2-wqzvm"
Oct 31 19:50:06.258825 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:50:06.258788    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 19:50:06.406162 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:06.406128    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d914121-zfkpr"]
Oct 31 19:50:06.412268 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:06.412246    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d914121-zfkpr"
Oct 31 19:50:06.426471 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:06.426443    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d914121-zfkpr"]
Oct 31 19:50:06.459037 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:06.458931    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:06.534102 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:06.534001    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bda9b5d-489e-4834-9676-1ed463ed81e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d914121-zfkpr\" (UID: \"4bda9b5d-489e-4834-9676-1ed463ed81e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d914121-zfkpr"
Oct 31 19:50:06.534574 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:06.534546    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cdf2\" (UniqueName: \"kubernetes.io/projected/4bda9b5d-489e-4834-9676-1ed463ed81e0-kube-api-access-4cdf2\") pod \"maas-api-revoke-sub-8d914121-zfkpr\" (UID: \"4bda9b5d-489e-4834-9676-1ed463ed81e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d914121-zfkpr"
Oct 31 19:50:06.534993 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:06.534971    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bda9b5d-489e-4834-9676-1ed463ed81e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d914121-zfkpr\" (UID: \"4bda9b5d-489e-4834-9676-1ed463ed81e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d914121-zfkpr"
Oct 31 19:50:06.543718 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:06.543689    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e160cf2-wqzvm"]
Oct 31 19:50:06.637079 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:06.637055    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bda9b5d-489e-4834-9676-1ed463ed81e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d914121-zfkpr\" (UID: \"4bda9b5d-489e-4834-9676-1ed463ed81e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d914121-zfkpr"
Oct 31 19:50:06.637210 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:06.637157    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bda9b5d-489e-4834-9676-1ed463ed81e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d914121-zfkpr\" (UID: \"4bda9b5d-489e-4834-9676-1ed463ed81e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d914121-zfkpr"
Oct 31 19:50:06.637307 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:06.637293    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cdf2\" (UniqueName: \"kubernetes.io/projected/4bda9b5d-489e-4834-9676-1ed463ed81e0-kube-api-access-4cdf2\") pod \"maas-api-revoke-sub-8d914121-zfkpr\" (UID: \"4bda9b5d-489e-4834-9676-1ed463ed81e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d914121-zfkpr"
Oct 31 19:50:06.637837 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:06.637813    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bda9b5d-489e-4834-9676-1ed463ed81e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d914121-zfkpr\" (UID: \"4bda9b5d-489e-4834-9676-1ed463ed81e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d914121-zfkpr"
Oct 31 19:50:06.639501 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:06.639482    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bda9b5d-489e-4834-9676-1ed463ed81e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d914121-zfkpr\" (UID: \"4bda9b5d-489e-4834-9676-1ed463ed81e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d914121-zfkpr"
Oct 31 19:50:06.649780 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:06.649755    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cdf2\" (UniqueName: \"kubernetes.io/projected/4bda9b5d-489e-4834-9676-1ed463ed81e0-kube-api-access-4cdf2\") pod \"maas-api-revoke-sub-8d914121-zfkpr\" (UID: \"4bda9b5d-489e-4834-9676-1ed463ed81e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d914121-zfkpr"
Oct 31 19:50:06.733418 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:06.733359    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d914121-zfkpr"
Oct 31 19:50:07.023294 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:50:07.023266    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bda9b5d_489e_4834_9676_1ed463ed81e0.slice/crio-66e35a96c2c3363451106581ce0c23d0eec997e56cdf0a6542801d0023ae66d2 WatchSource:0}: Error finding container 66e35a96c2c3363451106581ce0c23d0eec997e56cdf0a6542801d0023ae66d2: Status 404 returned error can't find the container with id 66e35a96c2c3363451106581ce0c23d0eec997e56cdf0a6542801d0023ae66d2
Oct 31 19:50:07.024411 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:07.024388    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d914121-zfkpr"]
Oct 31 19:50:07.139789 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:07.139763    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt"
Oct 31 19:50:07.140358 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:07.140305    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt"
Oct 31 19:50:07.141120 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:07.141095    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt"
Oct 31 19:50:07.252560 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:07.252525    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e160cf2-wqzvm" event={"ID":"5c13db3a-c224-4a77-b079-5d491d2592a0","Type":"ContainerStarted","Data":"0bccce786d3a1f1d55d96a02b4ad8e53aea0f0ecc0c826b785512076d4551e1e"}
Oct 31 19:50:07.252771 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:07.252589    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e160cf2-wqzvm" event={"ID":"5c13db3a-c224-4a77-b079-5d491d2592a0","Type":"ContainerStarted","Data":"9fffb0b349cc4bdc2f723c94337b85ec3d0a9bed476640aa4cf3993353007912"}
Oct 31 19:50:07.254321 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:07.254294    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d914121-zfkpr" event={"ID":"4bda9b5d-489e-4834-9676-1ed463ed81e0","Type":"ContainerStarted","Data":"66e35a96c2c3363451106581ce0c23d0eec997e56cdf0a6542801d0023ae66d2"}
Oct 31 19:50:07.273907 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:07.273827    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e160cf2-wqzvm" podStartSLOduration=2.273814412 podStartE2EDuration="2.273814412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:50:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:50:07.271131509 +0000 UTC m=+1837.844365637" watchObservedRunningTime="2026-10-31 19:50:07.273814412 +0000 UTC m=+1837.847048540"
Oct 31 19:50:07.432280 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:07.432230    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:08.266043 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:08.266012    4645 generic.go:356] "Generic (PLEG): container finished" podID="4bda9b5d-489e-4834-9676-1ed463ed81e0" containerID="1524c1647e87702dc653abc736a896cf5f0e26cefcf35c787e4ae5828515fe4a" exitCode=0
Oct 31 19:50:08.266499 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:08.266100    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d914121-zfkpr" event={"ID":"4bda9b5d-489e-4834-9676-1ed463ed81e0","Type":"ContainerDied","Data":"1524c1647e87702dc653abc736a896cf5f0e26cefcf35c787e4ae5828515fe4a"}
Oct 31 19:50:08.267873 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:08.267843    4645 generic.go:356] "Generic (PLEG): container finished" podID="5c13db3a-c224-4a77-b079-5d491d2592a0" containerID="0bccce786d3a1f1d55d96a02b4ad8e53aea0f0ecc0c826b785512076d4551e1e" exitCode=0
Oct 31 19:50:08.267992 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:08.267923    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e160cf2-wqzvm" event={"ID":"5c13db3a-c224-4a77-b079-5d491d2592a0","Type":"ContainerDied","Data":"0bccce786d3a1f1d55d96a02b4ad8e53aea0f0ecc0c826b785512076d4551e1e"}
Oct 31 19:50:08.289226 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:08.289195    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:08.403654 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:08.400593    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:08.776262 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:08.776227    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:09.228085 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:09.228053    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-88eaf9-r9646"]
Oct 31 19:50:09.232945 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:09.232926    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-88eaf9-r9646"
Oct 31 19:50:09.244467 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:09.244443    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-88eaf9-r9646"]
Oct 31 19:50:09.310434 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:09.310401    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/231fb65d-49fe-497c-871c-0e15eafce2f9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-88eaf9-r9646\" (UID: \"231fb65d-49fe-497c-871c-0e15eafce2f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-88eaf9-r9646"
Oct 31 19:50:09.310914 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:09.310596    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lszvv\" (UniqueName: \"kubernetes.io/projected/231fb65d-49fe-497c-871c-0e15eafce2f9-kube-api-access-lszvv\") pod \"maas-api-revoke-keys-e2e-inf-b-88eaf9-r9646\" (UID: \"231fb65d-49fe-497c-871c-0e15eafce2f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-88eaf9-r9646"
Oct 31 19:50:09.310914 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:09.310798    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/231fb65d-49fe-497c-871c-0e15eafce2f9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-88eaf9-r9646\" (UID: \"231fb65d-49fe-497c-871c-0e15eafce2f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-88eaf9-r9646"
Oct 31 19:50:09.413040 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:09.413004    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/231fb65d-49fe-497c-871c-0e15eafce2f9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-88eaf9-r9646\" (UID: \"231fb65d-49fe-497c-871c-0e15eafce2f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-88eaf9-r9646"
Oct 31 19:50:09.413276 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:09.413257    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/231fb65d-49fe-497c-871c-0e15eafce2f9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-88eaf9-r9646\" (UID: \"231fb65d-49fe-497c-871c-0e15eafce2f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-88eaf9-r9646"
Oct 31 19:50:09.413406 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:09.413390    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lszvv\" (UniqueName: \"kubernetes.io/projected/231fb65d-49fe-497c-871c-0e15eafce2f9-kube-api-access-lszvv\") pod \"maas-api-revoke-keys-e2e-inf-b-88eaf9-r9646\" (UID: \"231fb65d-49fe-497c-871c-0e15eafce2f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-88eaf9-r9646"
Oct 31 19:50:09.415029 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:09.414997    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/231fb65d-49fe-497c-871c-0e15eafce2f9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-88eaf9-r9646\" (UID: \"231fb65d-49fe-497c-871c-0e15eafce2f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-88eaf9-r9646"
Oct 31 19:50:09.417039 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:09.417007    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/231fb65d-49fe-497c-871c-0e15eafce2f9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-88eaf9-r9646\" (UID: \"231fb65d-49fe-497c-871c-0e15eafce2f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-88eaf9-r9646"
Oct 31 19:50:09.426865 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:09.426837    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lszvv\" (UniqueName: \"kubernetes.io/projected/231fb65d-49fe-497c-871c-0e15eafce2f9-kube-api-access-lszvv\") pod \"maas-api-revoke-keys-e2e-inf-b-88eaf9-r9646\" (UID: \"231fb65d-49fe-497c-871c-0e15eafce2f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-88eaf9-r9646"
Oct 31 19:50:09.481739 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:09.481675    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e160cf2-wqzvm"
Oct 31 19:50:09.517322 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:09.517294    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c13db3a-c224-4a77-b079-5d491d2592a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c13db3a-c224-4a77-b079-5d491d2592a0-maas-api-xks-ca\") pod \"5c13db3a-c224-4a77-b079-5d491d2592a0\" (UID: \"5c13db3a-c224-4a77-b079-5d491d2592a0\") "
Oct 31 19:50:09.517541 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:09.517449    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c13db3a-c224-4a77-b079-5d491d2592a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c13db3a-c224-4a77-b079-5d491d2592a0-maas-api-openshift-ca\") pod \"5c13db3a-c224-4a77-b079-5d491d2592a0\" (UID: \"5c13db3a-c224-4a77-b079-5d491d2592a0\") "
Oct 31 19:50:09.517609 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:09.517582    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c13db3a-c224-4a77-b079-5d491d2592a0-kube-api-access-7krvg\" (UniqueName: \"kubernetes.io/projected/5c13db3a-c224-4a77-b079-5d491d2592a0-kube-api-access-7krvg\") pod \"5c13db3a-c224-4a77-b079-5d491d2592a0\" (UID: \"5c13db3a-c224-4a77-b079-5d491d2592a0\") "
Oct 31 19:50:09.519534 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:09.519501    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c13db3a-c224-4a77-b079-5d491d2592a0-maas-api-openshift-ca" pod "5c13db3a-c224-4a77-b079-5d491d2592a0" (UID: "5c13db3a-c224-4a77-b079-5d491d2592a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:50:09.526733 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:09.526703    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c13db3a-c224-4a77-b079-5d491d2592a0-maas-api-xks-ca" pod "5c13db3a-c224-4a77-b079-5d491d2592a0" (UID: "5c13db3a-c224-4a77-b079-5d491d2592a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:50:09.532800 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:09.532673    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c13db3a-c224-4a77-b079-5d491d2592a0-kube-api-access-7krvg" pod "5c13db3a-c224-4a77-b079-5d491d2592a0" (UID: "5c13db3a-c224-4a77-b079-5d491d2592a0"). InnerVolumeSpecName "kube-api-access-7krvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:50:09.541194 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:09.541156    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:09.550337 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:09.550303    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-88eaf9-r9646"
Oct 31 19:50:09.568877 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:09.568813    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d914121-zfkpr"
Oct 31 19:50:09.627715 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:09.627299    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4bda9b5d-489e-4834-9676-1ed463ed81e0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bda9b5d-489e-4834-9676-1ed463ed81e0-maas-api-xks-ca\") pod \"4bda9b5d-489e-4834-9676-1ed463ed81e0\" (UID: \"4bda9b5d-489e-4834-9676-1ed463ed81e0\") "
Oct 31 19:50:09.627715 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:09.627503    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bda9b5d-489e-4834-9676-1ed463ed81e0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bda9b5d-489e-4834-9676-1ed463ed81e0-maas-api-openshift-ca\") pod \"4bda9b5d-489e-4834-9676-1ed463ed81e0\" (UID: \"4bda9b5d-489e-4834-9676-1ed463ed81e0\") "
Oct 31 19:50:09.627715 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:09.627560    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bda9b5d-489e-4834-9676-1ed463ed81e0-kube-api-access-4cdf2\" (UniqueName: \"kubernetes.io/projected/4bda9b5d-489e-4834-9676-1ed463ed81e0-kube-api-access-4cdf2\") pod \"4bda9b5d-489e-4834-9676-1ed463ed81e0\" (UID: \"4bda9b5d-489e-4834-9676-1ed463ed81e0\") "
Oct 31 19:50:09.628526 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:09.628447    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c13db3a-c224-4a77-b079-5d491d2592a0-maas-api-xks-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:09.628526 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:09.628478    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c13db3a-c224-4a77-b079-5d491d2592a0-maas-api-openshift-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:09.628526 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:09.628497    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7krvg\" (UniqueName: \"kubernetes.io/projected/5c13db3a-c224-4a77-b079-5d491d2592a0-kube-api-access-7krvg\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:09.638504 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:09.638455    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bda9b5d-489e-4834-9676-1ed463ed81e0-maas-api-openshift-ca" pod "4bda9b5d-489e-4834-9676-1ed463ed81e0" (UID: "4bda9b5d-489e-4834-9676-1ed463ed81e0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:50:09.638829 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:09.638806    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bda9b5d-489e-4834-9676-1ed463ed81e0-kube-api-access-4cdf2" pod "4bda9b5d-489e-4834-9676-1ed463ed81e0" (UID: "4bda9b5d-489e-4834-9676-1ed463ed81e0"). InnerVolumeSpecName "kube-api-access-4cdf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:50:09.641654 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:09.641612    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bda9b5d-489e-4834-9676-1ed463ed81e0-maas-api-xks-ca" pod "4bda9b5d-489e-4834-9676-1ed463ed81e0" (UID: "4bda9b5d-489e-4834-9676-1ed463ed81e0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:50:09.736148 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:09.736121    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bda9b5d-489e-4834-9676-1ed463ed81e0-maas-api-openshift-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:09.736148 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:09.736146    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cdf2\" (UniqueName: \"kubernetes.io/projected/4bda9b5d-489e-4834-9676-1ed463ed81e0-kube-api-access-4cdf2\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:09.736903 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:09.736173    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bda9b5d-489e-4834-9676-1ed463ed81e0-maas-api-xks-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:09.972972 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:50:09.972938    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod231fb65d_49fe_497c_871c_0e15eafce2f9.slice/crio-e6057760258541654de6ca8ba3250ce1673a66a6aa1d763c1ba2a995a43ea0f3 WatchSource:0}: Error finding container e6057760258541654de6ca8ba3250ce1673a66a6aa1d763c1ba2a995a43ea0f3: Status 404 returned error can't find the container with id e6057760258541654de6ca8ba3250ce1673a66a6aa1d763c1ba2a995a43ea0f3
Oct 31 19:50:09.974299 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:09.974274    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-88eaf9-r9646"]
Oct 31 19:50:10.289833 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:10.289754    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d914121-zfkpr"
Oct 31 19:50:10.289833 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:10.289775    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d914121-zfkpr" event={"ID":"4bda9b5d-489e-4834-9676-1ed463ed81e0","Type":"ContainerDied","Data":"66e35a96c2c3363451106581ce0c23d0eec997e56cdf0a6542801d0023ae66d2"}
Oct 31 19:50:10.289833 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:10.289828    4645 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66e35a96c2c3363451106581ce0c23d0eec997e56cdf0a6542801d0023ae66d2"
Oct 31 19:50:10.291291 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:10.291269    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e160cf2-wqzvm"
Oct 31 19:50:10.291501 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:10.291308    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e160cf2-wqzvm" event={"ID":"5c13db3a-c224-4a77-b079-5d491d2592a0","Type":"ContainerDied","Data":"9fffb0b349cc4bdc2f723c94337b85ec3d0a9bed476640aa4cf3993353007912"}
Oct 31 19:50:10.291501 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:10.291367    4645 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9fffb0b349cc4bdc2f723c94337b85ec3d0a9bed476640aa4cf3993353007912"
Oct 31 19:50:10.294341 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:10.294304    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-88eaf9-r9646" event={"ID":"231fb65d-49fe-497c-871c-0e15eafce2f9","Type":"ContainerStarted","Data":"e6057760258541654de6ca8ba3250ce1673a66a6aa1d763c1ba2a995a43ea0f3"}
Oct 31 19:50:10.627669 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:10.627585    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:11.372243 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:11.371528    4645 generic.go:356] "Generic (PLEG): container finished" podID="231fb65d-49fe-497c-871c-0e15eafce2f9" containerID="9fa564799d0d32b7e29caae6df8363518d394ba3ccb8069669d4fcf60e2b6a06" exitCode=0
Oct 31 19:50:11.372243 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:11.371709    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-88eaf9-r9646" event={"ID":"231fb65d-49fe-497c-871c-0e15eafce2f9","Type":"ContainerDied","Data":"9fa564799d0d32b7e29caae6df8363518d394ba3ccb8069669d4fcf60e2b6a06"}
Oct 31 19:50:11.378825 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:11.378496    4645 generic.go:356] "Generic (PLEG): container finished" podID="1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99" containerID="9764f894f229b36d3d6d8d7485ec8d3f328d69d6524026ca88a4fd6d1b51a530" exitCode=0
Oct 31 19:50:11.378825 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:11.378738    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk" event={"ID":"1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99","Type":"ContainerDied","Data":"9764f894f229b36d3d6d8d7485ec8d3f328d69d6524026ca88a4fd6d1b51a530"}
Oct 31 19:50:12.084365 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:12.079675    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:12.293782 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:12.293744    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk"
Oct 31 19:50:12.458353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:12.449843    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99-istio-envoy\") pod \"1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99\" (UID: \"1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99\") "
Oct 31 19:50:12.458353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:12.449943    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99-workload-socket\") pod \"1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99\" (UID: \"1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99\") "
Oct 31 19:50:12.458353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:12.450050    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99-istio-data\") pod \"1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99\" (UID: \"1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99\") "
Oct 31 19:50:12.458353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:12.450091    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99-workload-certs\") pod \"1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99\" (UID: \"1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99\") "
Oct 31 19:50:12.458353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:12.450142    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99-istio-token\" (UniqueName: \"kubernetes.io/projected/1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99-istio-token\") pod \"1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99\" (UID: \"1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99\") "
Oct 31 19:50:12.458353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:12.450236    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99-istiod-ca-cert\") pod \"1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99\" (UID: \"1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99\") "
Oct 31 19:50:12.458353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:12.450477    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99-credential-socket\") pod \"1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99\" (UID: \"1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99\") "
Oct 31 19:50:12.458353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:12.450659    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99-istio-podinfo\") pod \"1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99\" (UID: \"1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99\") "
Oct 31 19:50:12.458353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:12.450691    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99-kube-api-access-n4b8r\" (UniqueName: \"kubernetes.io/projected/1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99-kube-api-access-n4b8r\") pod \"1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99\" (UID: \"1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99\") "
Oct 31 19:50:12.458353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:12.456148    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99-workload-socket" pod "1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99" (UID: "1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:12.458353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:12.456396    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99-istio-data" pod "1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99" (UID: "1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:12.458353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:12.456613    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99-credential-socket" pod "1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99" (UID: "1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:12.458353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:12.457665    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99-workload-certs" pod "1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99" (UID: "1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:12.458353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:12.457825    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99-istiod-ca-cert" pod "1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99" (UID: "1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:50:12.469802 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:12.462190    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk"
Oct 31 19:50:12.469802 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:12.462635    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk" event={"ID":"1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99","Type":"ContainerDied","Data":"2db29144451663fb446b4a368a6cca61c6e28e44851271a81815a8216ce52a27"}
Oct 31 19:50:12.469802 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:12.462692    4645 scope.go:122] "RemoveContainer" containerID="9764f894f229b36d3d6d8d7485ec8d3f328d69d6524026ca88a4fd6d1b51a530"
Oct 31 19:50:12.491352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:12.470269    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99-istio-envoy" pod "1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99" (UID: "1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:12.544416 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:12.544273    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99-kube-api-access-n4b8r" pod "1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99" (UID: "1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99"). InnerVolumeSpecName "kube-api-access-n4b8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:50:12.555359 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:12.548416    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-b1fae7/test-model-b1fae7-kserve-7c67554bff-4dwfc" podUID="e692773b-183e-4265-a650-edab04468a46" containerName="main" probeResult="failure" output="Get \"https://10.133.0.156:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:50:12.555359 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:12.550547    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99-istio-token" pod "1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99" (UID: "1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:50:12.584352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:12.583688    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:12.584352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:12.583722    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:12.584352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:12.583737    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:12.584352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:12.583750    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:12.584352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:12.583763    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:12.584352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:12.583777    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4b8r\" (UniqueName: \"kubernetes.io/projected/1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99-kube-api-access-n4b8r\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:12.584352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:12.583910    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:12.584352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:12.583926    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:12.692352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:12.688494    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669" podUID="9285bee0-bf7c-4fd4-9da7-a1db021cae05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:50:12.732350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:12.728465    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw" podUID="1287db54-a587-4c11-9eb6-1606de555894" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": context deadline exceeded"
Oct 31 19:50:12.732350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:12.728906    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-754e0a/test-model-754e0a-kserve-767c88f75f-8xwns" podUID="55751f79-d0e3-474e-9623-5e37f8d1f396" containerName="main" probeResult="failure" output="Get \"https://10.133.0.184:8000/ready\": context deadline exceeded"
Oct 31 19:50:12.732350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:12.729221    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x" podUID="a774e286-dcb8-4a2f-8541-26bf231eb08d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 19:50:12.783360 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:12.783197    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-1e8cf0/test-model-1e8cf0-kserve-74747d94df-4gd76" podUID="b673f934-bc6d-4ae8-8adf-4acedf4a8df6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.158:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:50:12.880856 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:12.880811    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-7c892a/test-model-7c892a-kserve-57875c9b5f-r4j8f" podUID="7be66b1c-b038-45f8-8e28-cb3b3cd3c526" containerName="main" probeResult="failure" output="Get \"https://10.133.0.186:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:50:12.979664 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:12.562635    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n" podUID="dbb97168-db93-4757-9a1b-f9c5eb704fc5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 19:50:12.979664 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:12.973644    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99-istio-podinfo" pod "1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99" (UID: "1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:50:13.037714 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:13.028569    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l" podUID="12bea86c-61e5-465e-82fb-b4673a4d1be9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:50:13.058375 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:13.057899    4645 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-b-f8c3f7/test-model-f8c3f7-kserve-6fccbd6557-ts9sg" podUID="2e6b6da3-9983-463c-a814-e07d9a22ed1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.157:8000/health\": context deadline exceeded"
Oct 31 19:50:13.078517 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:13.078476    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:13.082359 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:13.081415    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-d8c398/test-model-d8c398-kserve-57476967b9-v6ld4" podUID="15c43438-d0c9-41d4-92dd-788371b30f8f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.178:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:50:13.197703 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:13.197664    4645 patch_prober.go:29] interesting pod/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:50:13.198245 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:13.198210    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk" podUID="1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:50:13.201861 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:13.201704    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-d45f6c/test-model-d45f6c-kserve-784b7b884c-7mgtp" podUID="16133210-4877-4f71-9d0d-fe90c29cca43" containerName="main" probeResult="failure" output="Get \"https://10.133.0.150:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:50:13.313728 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:13.313661    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk"]
Oct 31 19:50:13.316359 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:13.316252    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:13.325597 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:13.323299    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-t95xk"]
Oct 31 19:50:13.416872 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:13.416842    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-88eaf9-r9646"
Oct 31 19:50:13.474720 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:13.474643    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-88eaf9-r9646" event={"ID":"231fb65d-49fe-497c-871c-0e15eafce2f9","Type":"ContainerDied","Data":"e6057760258541654de6ca8ba3250ce1673a66a6aa1d763c1ba2a995a43ea0f3"}
Oct 31 19:50:13.475137 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:13.475032    4645 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6057760258541654de6ca8ba3250ce1673a66a6aa1d763c1ba2a995a43ea0f3"
Oct 31 19:50:13.475303 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:13.474917    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-88eaf9-r9646"
Oct 31 19:50:13.613710 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:13.613675    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/231fb65d-49fe-497c-871c-0e15eafce2f9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/231fb65d-49fe-497c-871c-0e15eafce2f9-maas-api-openshift-ca\") pod \"231fb65d-49fe-497c-871c-0e15eafce2f9\" (UID: \"231fb65d-49fe-497c-871c-0e15eafce2f9\") "
Oct 31 19:50:13.613710 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:13.613712    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/231fb65d-49fe-497c-871c-0e15eafce2f9-kube-api-access-lszvv\" (UniqueName: \"kubernetes.io/projected/231fb65d-49fe-497c-871c-0e15eafce2f9-kube-api-access-lszvv\") pod \"231fb65d-49fe-497c-871c-0e15eafce2f9\" (UID: \"231fb65d-49fe-497c-871c-0e15eafce2f9\") "
Oct 31 19:50:13.613983 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:13.613956    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/231fb65d-49fe-497c-871c-0e15eafce2f9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/231fb65d-49fe-497c-871c-0e15eafce2f9-maas-api-xks-ca\") pod \"231fb65d-49fe-497c-871c-0e15eafce2f9\" (UID: \"231fb65d-49fe-497c-871c-0e15eafce2f9\") "
Oct 31 19:50:13.614101 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:13.614071    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/231fb65d-49fe-497c-871c-0e15eafce2f9-maas-api-openshift-ca" pod "231fb65d-49fe-497c-871c-0e15eafce2f9" (UID: "231fb65d-49fe-497c-871c-0e15eafce2f9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:50:13.615035 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:13.614991    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/231fb65d-49fe-497c-871c-0e15eafce2f9-maas-api-openshift-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:13.616722 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:13.616694    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/231fb65d-49fe-497c-871c-0e15eafce2f9-maas-api-xks-ca" pod "231fb65d-49fe-497c-871c-0e15eafce2f9" (UID: "231fb65d-49fe-497c-871c-0e15eafce2f9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:50:13.618049 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:13.617287    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/231fb65d-49fe-497c-871c-0e15eafce2f9-kube-api-access-lszvv" pod "231fb65d-49fe-497c-871c-0e15eafce2f9" (UID: "231fb65d-49fe-497c-871c-0e15eafce2f9"). InnerVolumeSpecName "kube-api-access-lszvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:50:13.627789 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:13.627752    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99" path="/var/lib/kubelet/pods/1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99/volumes"
Oct 31 19:50:13.716356 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:13.716300    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lszvv\" (UniqueName: \"kubernetes.io/projected/231fb65d-49fe-497c-871c-0e15eafce2f9-kube-api-access-lszvv\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:13.716356 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:13.716358    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/231fb65d-49fe-497c-871c-0e15eafce2f9-maas-api-xks-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:14.087129 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:14.087087    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:15.373066 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:15.373032    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:16.239464 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:16.239432    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:16.894054 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:16.894021    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:17.084216 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:17.084180    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:17.850721 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:17.850682    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:17.968351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:17.968103    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:18.260830 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:18.260800    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:18.705919 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:18.705886    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a4e312/test-model-a4e312-kserve-f4b656bdc-29rj9"]
Oct 31 19:50:18.736221 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:18.736134    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:18.865048 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:18.865004    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:19.326801 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:19.326770    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-88eaf9-56b45b6577-m674p"]
Oct 31 19:50:19.345700 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:19.345670    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-88eaf9-697ffc695-b9ds8"]
Oct 31 19:50:19.418135 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:19.418097    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:19.526745 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:19.526704    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:19.675440 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:19.675406    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:19.932606 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:19.932514    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:20.096351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:20.095863    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-9efeae/test-model-9efeae-kserve-7d97f79d8f-2nnrl"]
Oct 31 19:50:20.189601 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:20.189513    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-b1fae7/test-model-b1fae7-kserve-7c67554bff-4dwfc" podUID="e692773b-183e-4265-a650-edab04468a46" containerName="main" containerID="cri-o://80baa86f6f84f41f15fbf8e9e6569c2e9cfa1ad1e6dee063ae0d1fa9229fa792" gracePeriod=45
Oct 31 19:50:20.517982 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:20.517951    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b1fae7/test-model-b1fae7-kserve-7c67554bff-4dwfc"
Oct 31 19:50:20.550226 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:20.550192    4645 generic.go:356] "Generic (PLEG): container finished" podID="e692773b-183e-4265-a650-edab04468a46" containerID="80baa86f6f84f41f15fbf8e9e6569c2e9cfa1ad1e6dee063ae0d1fa9229fa792" exitCode=0
Oct 31 19:50:20.550683 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:20.550660    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b1fae7/test-model-b1fae7-kserve-7c67554bff-4dwfc" event={"ID":"e692773b-183e-4265-a650-edab04468a46","Type":"ContainerDied","Data":"80baa86f6f84f41f15fbf8e9e6569c2e9cfa1ad1e6dee063ae0d1fa9229fa792"}
Oct 31 19:50:20.550864 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:20.550846    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-b1fae7/test-model-b1fae7-kserve-7c67554bff-4dwfc" event={"ID":"e692773b-183e-4265-a650-edab04468a46","Type":"ContainerDied","Data":"e9c054d1c89802997af8a1028e49cbf954f573685ffb7bb88ca6dbc5f02d0c9a"}
Oct 31 19:50:20.551136 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:20.551119    4645 scope.go:122] "RemoveContainer" containerID="80baa86f6f84f41f15fbf8e9e6569c2e9cfa1ad1e6dee063ae0d1fa9229fa792"
Oct 31 19:50:20.551444 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:20.551428    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-b1fae7/test-model-b1fae7-kserve-7c67554bff-4dwfc"
Oct 31 19:50:20.551666 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:20.551635    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e692773b-183e-4265-a650-edab04468a46-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e692773b-183e-4265-a650-edab04468a46-dshm\") pod \"e692773b-183e-4265-a650-edab04468a46\" (UID: \"e692773b-183e-4265-a650-edab04468a46\") "
Oct 31 19:50:20.551892 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:20.551858    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e692773b-183e-4265-a650-edab04468a46-kube-api-access-7qhf2\" (UniqueName: \"kubernetes.io/projected/e692773b-183e-4265-a650-edab04468a46-kube-api-access-7qhf2\") pod \"e692773b-183e-4265-a650-edab04468a46\" (UID: \"e692773b-183e-4265-a650-edab04468a46\") "
Oct 31 19:50:20.552123 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:20.552104    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e692773b-183e-4265-a650-edab04468a46-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e692773b-183e-4265-a650-edab04468a46-model-cache\") pod \"e692773b-183e-4265-a650-edab04468a46\" (UID: \"e692773b-183e-4265-a650-edab04468a46\") "
Oct 31 19:50:20.552370 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:20.552350    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e692773b-183e-4265-a650-edab04468a46-home\" (UniqueName: \"kubernetes.io/empty-dir/e692773b-183e-4265-a650-edab04468a46-home\") pod \"e692773b-183e-4265-a650-edab04468a46\" (UID: \"e692773b-183e-4265-a650-edab04468a46\") "
Oct 31 19:50:20.552853 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:20.552813    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e692773b-183e-4265-a650-edab04468a46-tls-certs\" (UniqueName: \"kubernetes.io/secret/e692773b-183e-4265-a650-edab04468a46-tls-certs\") pod \"e692773b-183e-4265-a650-edab04468a46\" (UID: \"e692773b-183e-4265-a650-edab04468a46\") "
Oct 31 19:50:20.553250 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:20.553065    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e692773b-183e-4265-a650-edab04468a46-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e692773b-183e-4265-a650-edab04468a46-tmp-dir\") pod \"e692773b-183e-4265-a650-edab04468a46\" (UID: \"e692773b-183e-4265-a650-edab04468a46\") "
Oct 31 19:50:20.558725 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:20.553288    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e692773b-183e-4265-a650-edab04468a46-model-cache" pod "e692773b-183e-4265-a650-edab04468a46" (UID: "e692773b-183e-4265-a650-edab04468a46"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:20.558880 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:20.555975    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e692773b-183e-4265-a650-edab04468a46-kube-api-access-7qhf2" pod "e692773b-183e-4265-a650-edab04468a46" (UID: "e692773b-183e-4265-a650-edab04468a46"). InnerVolumeSpecName "kube-api-access-7qhf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:50:20.558880 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:20.557632    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e692773b-183e-4265-a650-edab04468a46-tmp-dir" pod "e692773b-183e-4265-a650-edab04468a46" (UID: "e692773b-183e-4265-a650-edab04468a46"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:20.559029 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:20.557798    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e692773b-183e-4265-a650-edab04468a46-home" pod "e692773b-183e-4265-a650-edab04468a46" (UID: "e692773b-183e-4265-a650-edab04468a46"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:20.578593 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:20.578568    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e692773b-183e-4265-a650-edab04468a46-tls-certs" pod "e692773b-183e-4265-a650-edab04468a46" (UID: "e692773b-183e-4265-a650-edab04468a46"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:50:20.580271 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:20.580241    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e692773b-183e-4265-a650-edab04468a46-dshm" pod "e692773b-183e-4265-a650-edab04468a46" (UID: "e692773b-183e-4265-a650-edab04468a46"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:20.606008 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:20.605985    4645 scope.go:122] "RemoveContainer" containerID="80baa86f6f84f41f15fbf8e9e6569c2e9cfa1ad1e6dee063ae0d1fa9229fa792"
Oct 31 19:50:20.606388 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:50:20.606362    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80baa86f6f84f41f15fbf8e9e6569c2e9cfa1ad1e6dee063ae0d1fa9229fa792\": container with ID starting with 80baa86f6f84f41f15fbf8e9e6569c2e9cfa1ad1e6dee063ae0d1fa9229fa792 not found: ID does not exist" containerID="80baa86f6f84f41f15fbf8e9e6569c2e9cfa1ad1e6dee063ae0d1fa9229fa792"
Oct 31 19:50:20.606475 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:20.606399    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80baa86f6f84f41f15fbf8e9e6569c2e9cfa1ad1e6dee063ae0d1fa9229fa792"} err="failed to get container status \"80baa86f6f84f41f15fbf8e9e6569c2e9cfa1ad1e6dee063ae0d1fa9229fa792\": rpc error: code = NotFound desc = could not find container \"80baa86f6f84f41f15fbf8e9e6569c2e9cfa1ad1e6dee063ae0d1fa9229fa792\": container with ID starting with 80baa86f6f84f41f15fbf8e9e6569c2e9cfa1ad1e6dee063ae0d1fa9229fa792 not found: ID does not exist"
Oct 31 19:50:20.660695 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:20.660670    4645 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e692773b-183e-4265-a650-edab04468a46-tls-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:20.660695 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:20.660693    4645 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e692773b-183e-4265-a650-edab04468a46-tmp-dir\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:20.660816 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:20.660703    4645 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e692773b-183e-4265-a650-edab04468a46-dshm\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:20.660816 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:20.660712    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qhf2\" (UniqueName: \"kubernetes.io/projected/e692773b-183e-4265-a650-edab04468a46-kube-api-access-7qhf2\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:20.660816 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:20.660722    4645 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e692773b-183e-4265-a650-edab04468a46-model-cache\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:20.660816 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:20.660735    4645 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e692773b-183e-4265-a650-edab04468a46-home\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:20.715956 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:20.715918    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-278c04/test-model-278c04-kserve-6b6f77c967-2f8mt" podUID="beb01fd0-e706-4d84-b1a6-c41614630f39" containerName="main" containerID="cri-o://7f9af928a1f875f399ff0791a342c36e0e1ab6fcf3ab396edf3cabc54d2fb54b" gracePeriod=45
Oct 31 19:50:20.878978 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:20.878948    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-b1fae7/test-model-b1fae7-kserve-7c67554bff-4dwfc"]
Oct 31 19:50:20.884085 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:20.884058    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-b1fae7/test-model-b1fae7-kserve-7c67554bff-4dwfc"]
Oct 31 19:50:20.986890 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:20.986830    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-278c04/test-model-278c04-kserve-6b6f77c967-2f8mt"
Oct 31 19:50:21.066141 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:21.066112    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/beb01fd0-e706-4d84-b1a6-c41614630f39-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/beb01fd0-e706-4d84-b1a6-c41614630f39-model-cache\") pod \"beb01fd0-e706-4d84-b1a6-c41614630f39\" (UID: \"beb01fd0-e706-4d84-b1a6-c41614630f39\") "
Oct 31 19:50:21.066304 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:21.066219    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/beb01fd0-e706-4d84-b1a6-c41614630f39-tls-certs\" (UniqueName: \"kubernetes.io/secret/beb01fd0-e706-4d84-b1a6-c41614630f39-tls-certs\") pod \"beb01fd0-e706-4d84-b1a6-c41614630f39\" (UID: \"beb01fd0-e706-4d84-b1a6-c41614630f39\") "
Oct 31 19:50:21.066460 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:21.066431    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beb01fd0-e706-4d84-b1a6-c41614630f39-model-cache" pod "beb01fd0-e706-4d84-b1a6-c41614630f39" (UID: "beb01fd0-e706-4d84-b1a6-c41614630f39"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:21.066670 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:21.066649    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beb01fd0-e706-4d84-b1a6-c41614630f39-tmp-dir" pod "beb01fd0-e706-4d84-b1a6-c41614630f39" (UID: "beb01fd0-e706-4d84-b1a6-c41614630f39"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:21.066762 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:21.066440    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/beb01fd0-e706-4d84-b1a6-c41614630f39-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/beb01fd0-e706-4d84-b1a6-c41614630f39-tmp-dir\") pod \"beb01fd0-e706-4d84-b1a6-c41614630f39\" (UID: \"beb01fd0-e706-4d84-b1a6-c41614630f39\") "
Oct 31 19:50:21.066835 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:21.066807    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/beb01fd0-e706-4d84-b1a6-c41614630f39-kube-api-access-wzdwj\" (UniqueName: \"kubernetes.io/projected/beb01fd0-e706-4d84-b1a6-c41614630f39-kube-api-access-wzdwj\") pod \"beb01fd0-e706-4d84-b1a6-c41614630f39\" (UID: \"beb01fd0-e706-4d84-b1a6-c41614630f39\") "
Oct 31 19:50:21.066898 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:21.066879    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/beb01fd0-e706-4d84-b1a6-c41614630f39-home\" (UniqueName: \"kubernetes.io/empty-dir/beb01fd0-e706-4d84-b1a6-c41614630f39-home\") pod \"beb01fd0-e706-4d84-b1a6-c41614630f39\" (UID: \"beb01fd0-e706-4d84-b1a6-c41614630f39\") "
Oct 31 19:50:21.066973 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:21.066957    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/beb01fd0-e706-4d84-b1a6-c41614630f39-dshm\" (UniqueName: \"kubernetes.io/empty-dir/beb01fd0-e706-4d84-b1a6-c41614630f39-dshm\") pod \"beb01fd0-e706-4d84-b1a6-c41614630f39\" (UID: \"beb01fd0-e706-4d84-b1a6-c41614630f39\") "
Oct 31 19:50:21.067228 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:21.067192    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beb01fd0-e706-4d84-b1a6-c41614630f39-home" pod "beb01fd0-e706-4d84-b1a6-c41614630f39" (UID: "beb01fd0-e706-4d84-b1a6-c41614630f39"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:21.067809 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:21.067784    4645 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/beb01fd0-e706-4d84-b1a6-c41614630f39-tmp-dir\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:21.067902 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:21.067813    4645 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/beb01fd0-e706-4d84-b1a6-c41614630f39-home\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:21.067902 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:21.067828    4645 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/beb01fd0-e706-4d84-b1a6-c41614630f39-model-cache\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:21.068951 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:21.068928    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/beb01fd0-e706-4d84-b1a6-c41614630f39-tls-certs" pod "beb01fd0-e706-4d84-b1a6-c41614630f39" (UID: "beb01fd0-e706-4d84-b1a6-c41614630f39"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:50:21.069409 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:21.069389    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/beb01fd0-e706-4d84-b1a6-c41614630f39-kube-api-access-wzdwj" pod "beb01fd0-e706-4d84-b1a6-c41614630f39" (UID: "beb01fd0-e706-4d84-b1a6-c41614630f39"). InnerVolumeSpecName "kube-api-access-wzdwj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:50:21.069587 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:21.069562    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beb01fd0-e706-4d84-b1a6-c41614630f39-dshm" pod "beb01fd0-e706-4d84-b1a6-c41614630f39" (UID: "beb01fd0-e706-4d84-b1a6-c41614630f39"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:21.078429 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:21.078405    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:21.169872 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:21.169838    4645 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/beb01fd0-e706-4d84-b1a6-c41614630f39-dshm\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:21.169872 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:21.169869    4645 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/beb01fd0-e706-4d84-b1a6-c41614630f39-tls-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:21.170096 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:21.169880    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzdwj\" (UniqueName: \"kubernetes.io/projected/beb01fd0-e706-4d84-b1a6-c41614630f39-kube-api-access-wzdwj\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:21.416926 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:21.416897    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-f8c3f7/test-model-f8c3f7-kserve-6fccbd6557-ts9sg"]
Oct 31 19:50:21.458508 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:21.458456    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:21.605237 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:21.605186    4645 generic.go:356] "Generic (PLEG): container finished" podID="beb01fd0-e706-4d84-b1a6-c41614630f39" containerID="7f9af928a1f875f399ff0791a342c36e0e1ab6fcf3ab396edf3cabc54d2fb54b" exitCode=0
Oct 31 19:50:21.605805 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:21.605534    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-278c04/test-model-278c04-kserve-6b6f77c967-2f8mt" event={"ID":"beb01fd0-e706-4d84-b1a6-c41614630f39","Type":"ContainerDied","Data":"7f9af928a1f875f399ff0791a342c36e0e1ab6fcf3ab396edf3cabc54d2fb54b"}
Oct 31 19:50:21.605805 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:21.605577    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-278c04/test-model-278c04-kserve-6b6f77c967-2f8mt" event={"ID":"beb01fd0-e706-4d84-b1a6-c41614630f39","Type":"ContainerDied","Data":"22bd3c8c0e4f06b877c905f60a1823febcc79934f392ca08e80c7cba92554e42"}
Oct 31 19:50:21.605805 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:21.605600    4645 scope.go:122] "RemoveContainer" containerID="7f9af928a1f875f399ff0791a342c36e0e1ab6fcf3ab396edf3cabc54d2fb54b"
Oct 31 19:50:21.605805 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:21.605718    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-278c04/test-model-278c04-kserve-6b6f77c967-2f8mt"
Oct 31 19:50:21.628151 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:21.627941    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e692773b-183e-4265-a650-edab04468a46" path="/var/lib/kubelet/pods/e692773b-183e-4265-a650-edab04468a46/volumes"
Oct 31 19:50:21.632781 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:21.632763    4645 scope.go:122] "RemoveContainer" containerID="7f9af928a1f875f399ff0791a342c36e0e1ab6fcf3ab396edf3cabc54d2fb54b"
Oct 31 19:50:21.633084 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:50:21.633064    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f9af928a1f875f399ff0791a342c36e0e1ab6fcf3ab396edf3cabc54d2fb54b\": container with ID starting with 7f9af928a1f875f399ff0791a342c36e0e1ab6fcf3ab396edf3cabc54d2fb54b not found: ID does not exist" containerID="7f9af928a1f875f399ff0791a342c36e0e1ab6fcf3ab396edf3cabc54d2fb54b"
Oct 31 19:50:21.633143 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:21.633092    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f9af928a1f875f399ff0791a342c36e0e1ab6fcf3ab396edf3cabc54d2fb54b"} err="failed to get container status \"7f9af928a1f875f399ff0791a342c36e0e1ab6fcf3ab396edf3cabc54d2fb54b\": rpc error: code = NotFound desc = could not find container \"7f9af928a1f875f399ff0791a342c36e0e1ab6fcf3ab396edf3cabc54d2fb54b\": container with ID starting with 7f9af928a1f875f399ff0791a342c36e0e1ab6fcf3ab396edf3cabc54d2fb54b not found: ID does not exist"
Oct 31 19:50:21.642909 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:21.642886    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-278c04/test-model-278c04-kserve-6b6f77c967-2f8mt"]
Oct 31 19:50:21.647682 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:21.647660    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-278c04/test-model-278c04-kserve-6b6f77c967-2f8mt"]
Oct 31 19:50:21.850971 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:21.850890    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b1fae7-t2chz"]
Oct 31 19:50:21.851746 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:21.851728    4645 cpu_manager.go:414] "containerMap: removing container" podUID="1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99" containerName="istio-proxy"
Oct 31 19:50:21.851746 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:21.851745    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99" containerName="istio-proxy"
Oct 31 19:50:21.851913 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:21.851771    4645 cpu_manager.go:414] "containerMap: removing container" podUID="beb01fd0-e706-4d84-b1a6-c41614630f39" containerName="main"
Oct 31 19:50:21.851913 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:21.851776    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="beb01fd0-e706-4d84-b1a6-c41614630f39" containerName="main"
Oct 31 19:50:21.851913 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:21.851785    4645 cpu_manager.go:414] "containerMap: removing container" podUID="e692773b-183e-4265-a650-edab04468a46" containerName="main"
Oct 31 19:50:21.851913 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:21.851790    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e692773b-183e-4265-a650-edab04468a46" containerName="main"
Oct 31 19:50:21.851913 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:21.851799    4645 cpu_manager.go:414] "containerMap: removing container" podUID="5c13db3a-c224-4a77-b079-5d491d2592a0" containerName="revoke-keys"
Oct 31 19:50:21.851913 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:21.851804    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c13db3a-c224-4a77-b079-5d491d2592a0" containerName="revoke-keys"
Oct 31 19:50:21.851913 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:21.851823    4645 cpu_manager.go:414] "containerMap: removing container" podUID="231fb65d-49fe-497c-871c-0e15eafce2f9" containerName="revoke-keys"
Oct 31 19:50:21.851913 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:21.851831    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="231fb65d-49fe-497c-871c-0e15eafce2f9" containerName="revoke-keys"
Oct 31 19:50:21.851913 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:21.851865    4645 cpu_manager.go:414] "containerMap: removing container" podUID="4bda9b5d-489e-4834-9676-1ed463ed81e0" containerName="revoke-keys"
Oct 31 19:50:21.851913 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:21.851873    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bda9b5d-489e-4834-9676-1ed463ed81e0" containerName="revoke-keys"
Oct 31 19:50:21.852293 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:21.852070    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bda9b5d-489e-4834-9676-1ed463ed81e0" containerName="revoke-keys"
Oct 31 19:50:21.852293 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:21.852088    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ba41e18-62b4-47d9-8a89-1b5a4e8a5a99" containerName="istio-proxy"
Oct 31 19:50:21.852293 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:21.852101    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="beb01fd0-e706-4d84-b1a6-c41614630f39" containerName="main"
Oct 31 19:50:21.852293 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:21.852111    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="e692773b-183e-4265-a650-edab04468a46" containerName="main"
Oct 31 19:50:21.852293 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:21.852119    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c13db3a-c224-4a77-b079-5d491d2592a0" containerName="revoke-keys"
Oct 31 19:50:21.852293 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:21.852128    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="231fb65d-49fe-497c-871c-0e15eafce2f9" containerName="revoke-keys"
Oct 31 19:50:21.857633 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:21.857615    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b1fae7-t2chz"
Oct 31 19:50:21.870125 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:21.865531    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:50:21.870125 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:21.865765    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 19:50:21.870125 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:21.869024    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b1fae7-t2chz"]
Oct 31 19:50:21.884447 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:21.882236    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxfcp\" (UniqueName: \"kubernetes.io/projected/18806293-fbe8-44b4-b4f4-1efeec8fb8d8-kube-api-access-zxfcp\") pod \"maas-api-revoke-keys-e2e-inf-a-b1fae7-t2chz\" (UID: \"18806293-fbe8-44b4-b4f4-1efeec8fb8d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b1fae7-t2chz"
Oct 31 19:50:21.884447 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:21.882368    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18806293-fbe8-44b4-b4f4-1efeec8fb8d8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b1fae7-t2chz\" (UID: \"18806293-fbe8-44b4-b4f4-1efeec8fb8d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b1fae7-t2chz"
Oct 31 19:50:21.884447 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:21.882447    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18806293-fbe8-44b4-b4f4-1efeec8fb8d8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b1fae7-t2chz\" (UID: \"18806293-fbe8-44b4-b4f4-1efeec8fb8d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b1fae7-t2chz"
Oct 31 19:50:21.983784 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:21.983758    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18806293-fbe8-44b4-b4f4-1efeec8fb8d8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b1fae7-t2chz\" (UID: \"18806293-fbe8-44b4-b4f4-1efeec8fb8d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b1fae7-t2chz"
Oct 31 19:50:21.983908 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:21.983863    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxfcp\" (UniqueName: \"kubernetes.io/projected/18806293-fbe8-44b4-b4f4-1efeec8fb8d8-kube-api-access-zxfcp\") pod \"maas-api-revoke-keys-e2e-inf-a-b1fae7-t2chz\" (UID: \"18806293-fbe8-44b4-b4f4-1efeec8fb8d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b1fae7-t2chz"
Oct 31 19:50:21.983962 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:21.983940    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18806293-fbe8-44b4-b4f4-1efeec8fb8d8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b1fae7-t2chz\" (UID: \"18806293-fbe8-44b4-b4f4-1efeec8fb8d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b1fae7-t2chz"
Oct 31 19:50:21.984564 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:21.984544    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18806293-fbe8-44b4-b4f4-1efeec8fb8d8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b1fae7-t2chz\" (UID: \"18806293-fbe8-44b4-b4f4-1efeec8fb8d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b1fae7-t2chz"
Oct 31 19:50:21.986117 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:21.986095    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18806293-fbe8-44b4-b4f4-1efeec8fb8d8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b1fae7-t2chz\" (UID: \"18806293-fbe8-44b4-b4f4-1efeec8fb8d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b1fae7-t2chz"
Oct 31 19:50:21.998474 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:21.998446    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxfcp\" (UniqueName: \"kubernetes.io/projected/18806293-fbe8-44b4-b4f4-1efeec8fb8d8-kube-api-access-zxfcp\") pod \"maas-api-revoke-keys-e2e-inf-a-b1fae7-t2chz\" (UID: \"18806293-fbe8-44b4-b4f4-1efeec8fb8d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b1fae7-t2chz"
Oct 31 19:50:22.179313 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:22.179292    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b1fae7-t2chz"
Oct 31 19:50:22.464380 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:50:22.464349    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18806293_fbe8_44b4_b4f4_1efeec8fb8d8.slice/crio-e0bfec57306b5d947f43c42eed41405be41b485dcb8de6d0a37d6e585e70d430 WatchSource:0}: Error finding container e0bfec57306b5d947f43c42eed41405be41b485dcb8de6d0a37d6e585e70d430: Status 404 returned error can't find the container with id e0bfec57306b5d947f43c42eed41405be41b485dcb8de6d0a37d6e585e70d430
Oct 31 19:50:22.464380 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:22.464359    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b1fae7-t2chz"]
Oct 31 19:50:22.468365 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:22.468306    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:22.631622 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:22.631588    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b1fae7-t2chz" event={"ID":"18806293-fbe8-44b4-b4f4-1efeec8fb8d8","Type":"ContainerStarted","Data":"e0bfec57306b5d947f43c42eed41405be41b485dcb8de6d0a37d6e585e70d430"}
Oct 31 19:50:23.620966 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:23.620936    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="beb01fd0-e706-4d84-b1a6-c41614630f39" path="/var/lib/kubelet/pods/beb01fd0-e706-4d84-b1a6-c41614630f39/volumes"
Oct 31 19:50:23.628463 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:23.628436    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:23.643140 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:23.643111    4645 generic.go:356] "Generic (PLEG): container finished" podID="18806293-fbe8-44b4-b4f4-1efeec8fb8d8" containerID="e597ec4feffc1554329c2f52f5e4185887c38ea0e679c1d77f726e6d98f66729" exitCode=0
Oct 31 19:50:23.643485 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:23.643201    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b1fae7-t2chz" event={"ID":"18806293-fbe8-44b4-b4f4-1efeec8fb8d8","Type":"ContainerDied","Data":"e597ec4feffc1554329c2f52f5e4185887c38ea0e679c1d77f726e6d98f66729"}
Oct 31 19:50:24.327117 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:24.327061    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-88eaf9-56b45b6577-m674p" podUID="7ae56bf5-1961-48fc-9c77-bb64e0d791cf" containerName="payload-processing" containerID="cri-o://622bd5e7b4d3fcab0d71e79b1c5c49e26fbf7866c1213da8d8181067d101fada" gracePeriod=25
Oct 31 19:50:24.346291 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:24.346258    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-88eaf9-697ffc695-b9ds8" podUID="f2c7c458-2fc5-4674-b080-7f0c9f29a3b9" containerName="payload-pre-processing" containerID="cri-o://210f25b91f3fcff0b9fda607493ebc2484d5d6ec1dd03de55ce184c9b5dbd59b" gracePeriod=25
Oct 31 19:50:24.482711 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:24.482675    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:24.834806 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:24.834783    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b1fae7-t2chz"
Oct 31 19:50:24.962321 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:24.962285    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18806293-fbe8-44b4-b4f4-1efeec8fb8d8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18806293-fbe8-44b4-b4f4-1efeec8fb8d8-maas-api-xks-ca\") pod \"18806293-fbe8-44b4-b4f4-1efeec8fb8d8\" (UID: \"18806293-fbe8-44b4-b4f4-1efeec8fb8d8\") "
Oct 31 19:50:24.962548 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:24.962358    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18806293-fbe8-44b4-b4f4-1efeec8fb8d8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18806293-fbe8-44b4-b4f4-1efeec8fb8d8-maas-api-openshift-ca\") pod \"18806293-fbe8-44b4-b4f4-1efeec8fb8d8\" (UID: \"18806293-fbe8-44b4-b4f4-1efeec8fb8d8\") "
Oct 31 19:50:24.962548 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:24.962498    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18806293-fbe8-44b4-b4f4-1efeec8fb8d8-kube-api-access-zxfcp\" (UniqueName: \"kubernetes.io/projected/18806293-fbe8-44b4-b4f4-1efeec8fb8d8-kube-api-access-zxfcp\") pod \"18806293-fbe8-44b4-b4f4-1efeec8fb8d8\" (UID: \"18806293-fbe8-44b4-b4f4-1efeec8fb8d8\") "
Oct 31 19:50:24.962734 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:24.962706    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18806293-fbe8-44b4-b4f4-1efeec8fb8d8-maas-api-openshift-ca" pod "18806293-fbe8-44b4-b4f4-1efeec8fb8d8" (UID: "18806293-fbe8-44b4-b4f4-1efeec8fb8d8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:50:24.963220 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:24.963197    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18806293-fbe8-44b4-b4f4-1efeec8fb8d8-maas-api-openshift-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:24.964430 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:24.964406    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18806293-fbe8-44b4-b4f4-1efeec8fb8d8-maas-api-xks-ca" pod "18806293-fbe8-44b4-b4f4-1efeec8fb8d8" (UID: "18806293-fbe8-44b4-b4f4-1efeec8fb8d8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:50:24.964574 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:24.964558    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18806293-fbe8-44b4-b4f4-1efeec8fb8d8-kube-api-access-zxfcp" pod "18806293-fbe8-44b4-b4f4-1efeec8fb8d8" (UID: "18806293-fbe8-44b4-b4f4-1efeec8fb8d8"). InnerVolumeSpecName "kube-api-access-zxfcp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:50:25.064913 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:25.064885    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxfcp\" (UniqueName: \"kubernetes.io/projected/18806293-fbe8-44b4-b4f4-1efeec8fb8d8-kube-api-access-zxfcp\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:25.064913 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:25.064913    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18806293-fbe8-44b4-b4f4-1efeec8fb8d8-maas-api-xks-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:25.253251 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:25.253154    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:25.664789 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:25.664758    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b1fae7-t2chz"
Oct 31 19:50:25.664939 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:25.664804    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b1fae7-t2chz" event={"ID":"18806293-fbe8-44b4-b4f4-1efeec8fb8d8","Type":"ContainerDied","Data":"e0bfec57306b5d947f43c42eed41405be41b485dcb8de6d0a37d6e585e70d430"}
Oct 31 19:50:25.664939 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:25.664848    4645 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0bfec57306b5d947f43c42eed41405be41b485dcb8de6d0a37d6e585e70d430"
Oct 31 19:50:25.959029 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:25.958994    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-88eaf9-56b45b6577-m674p" podUID="7ae56bf5-1961-48fc-9c77-bb64e0d791cf" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:50:25.959029 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:25.959014    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-88eaf9-697ffc695-b9ds8" podUID="f2c7c458-2fc5-4674-b080-7f0c9f29a3b9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:50:26.194260 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:26.194224    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:27.132353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:27.132209    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:28.076416 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:28.071504    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:28.906143 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:28.906105    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:29.865833 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:29.865797    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:30.086492 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:30.086461    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-d45f6c/test-model-d45f6c-kserve-784b7b884c-7mgtp"]
Oct 31 19:50:30.125699 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:30.125625    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:30.543486 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:30.543457    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-278c04-6969d7fcb9-252tn"]
Oct 31 19:50:30.579998 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:30.579965    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-278c04-86645f6fbc-p5qmg"]
Oct 31 19:50:30.644126 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:30.644096    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:30.757636 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:30.757609    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:30.984359 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:30.983589    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:32.126298 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:32.126262    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:32.259956 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:32.259921    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:32.408407 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:32.401920    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:32.887412 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:32.887378    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9"]
Oct 31 19:50:32.888248 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:32.888230    4645 cpu_manager.go:414] "containerMap: removing container" podUID="18806293-fbe8-44b4-b4f4-1efeec8fb8d8" containerName="revoke-keys"
Oct 31 19:50:32.888298 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:32.888249    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18806293-fbe8-44b4-b4f4-1efeec8fb8d8" containerName="revoke-keys"
Oct 31 19:50:32.888416 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:32.888392    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="18806293-fbe8-44b4-b4f4-1efeec8fb8d8" containerName="revoke-keys"
Oct 31 19:50:32.894036 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:32.894016    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9"
Oct 31 19:50:32.935078 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:32.935051    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aceb3312-c698-4f69-9b44-be9bd3ef0e6b-istio-podinfo\") pod \"e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9\" (UID: \"aceb3312-c698-4f69-9b44-be9bd3ef0e6b\") " pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9"
Oct 31 19:50:32.935201 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:32.935107    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vbwr\" (UniqueName: \"kubernetes.io/projected/aceb3312-c698-4f69-9b44-be9bd3ef0e6b-kube-api-access-2vbwr\") pod \"e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9\" (UID: \"aceb3312-c698-4f69-9b44-be9bd3ef0e6b\") " pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9"
Oct 31 19:50:32.935308 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:32.935294    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aceb3312-c698-4f69-9b44-be9bd3ef0e6b-workload-socket\") pod \"e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9\" (UID: \"aceb3312-c698-4f69-9b44-be9bd3ef0e6b\") " pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9"
Oct 31 19:50:32.935370 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:32.935322    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aceb3312-c698-4f69-9b44-be9bd3ef0e6b-credential-socket\") pod \"e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9\" (UID: \"aceb3312-c698-4f69-9b44-be9bd3ef0e6b\") " pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9"
Oct 31 19:50:32.935370 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:32.935363    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aceb3312-c698-4f69-9b44-be9bd3ef0e6b-workload-certs\") pod \"e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9\" (UID: \"aceb3312-c698-4f69-9b44-be9bd3ef0e6b\") " pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9"
Oct 31 19:50:32.935499 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:32.935473    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aceb3312-c698-4f69-9b44-be9bd3ef0e6b-istio-token\") pod \"e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9\" (UID: \"aceb3312-c698-4f69-9b44-be9bd3ef0e6b\") " pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9"
Oct 31 19:50:32.935546 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:32.935525    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aceb3312-c698-4f69-9b44-be9bd3ef0e6b-istio-data\") pod \"e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9\" (UID: \"aceb3312-c698-4f69-9b44-be9bd3ef0e6b\") " pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9"
Oct 31 19:50:32.935626 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:32.935606    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aceb3312-c698-4f69-9b44-be9bd3ef0e6b-istio-envoy\") pod \"e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9\" (UID: \"aceb3312-c698-4f69-9b44-be9bd3ef0e6b\") " pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9"
Oct 31 19:50:32.935757 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:32.935735    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aceb3312-c698-4f69-9b44-be9bd3ef0e6b-istiod-ca-cert\") pod \"e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9\" (UID: \"aceb3312-c698-4f69-9b44-be9bd3ef0e6b\") " pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9"
Oct 31 19:50:32.944519 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:32.944496    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-7c892a-openshift-default-dockercfg-zdlhk\""
Oct 31 19:50:32.946743 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:32.946718    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9"]
Oct 31 19:50:33.037306 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:33.037272    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vbwr\" (UniqueName: \"kubernetes.io/projected/aceb3312-c698-4f69-9b44-be9bd3ef0e6b-kube-api-access-2vbwr\") pod \"e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9\" (UID: \"aceb3312-c698-4f69-9b44-be9bd3ef0e6b\") " pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9"
Oct 31 19:50:33.037458 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:33.037449    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aceb3312-c698-4f69-9b44-be9bd3ef0e6b-workload-socket\") pod \"e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9\" (UID: \"aceb3312-c698-4f69-9b44-be9bd3ef0e6b\") " pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9"
Oct 31 19:50:33.037522 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:33.037475    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aceb3312-c698-4f69-9b44-be9bd3ef0e6b-credential-socket\") pod \"e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9\" (UID: \"aceb3312-c698-4f69-9b44-be9bd3ef0e6b\") " pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9"
Oct 31 19:50:33.037522 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:33.037514    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aceb3312-c698-4f69-9b44-be9bd3ef0e6b-workload-certs\") pod \"e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9\" (UID: \"aceb3312-c698-4f69-9b44-be9bd3ef0e6b\") " pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9"
Oct 31 19:50:33.037634 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:33.037567    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aceb3312-c698-4f69-9b44-be9bd3ef0e6b-istio-token\") pod \"e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9\" (UID: \"aceb3312-c698-4f69-9b44-be9bd3ef0e6b\") " pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9"
Oct 31 19:50:33.037634 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:33.037613    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aceb3312-c698-4f69-9b44-be9bd3ef0e6b-istio-data\") pod \"e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9\" (UID: \"aceb3312-c698-4f69-9b44-be9bd3ef0e6b\") " pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9"
Oct 31 19:50:33.037748 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:33.037659    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aceb3312-c698-4f69-9b44-be9bd3ef0e6b-istio-envoy\") pod \"e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9\" (UID: \"aceb3312-c698-4f69-9b44-be9bd3ef0e6b\") " pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9"
Oct 31 19:50:33.037899 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:33.037749    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aceb3312-c698-4f69-9b44-be9bd3ef0e6b-istiod-ca-cert\") pod \"e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9\" (UID: \"aceb3312-c698-4f69-9b44-be9bd3ef0e6b\") " pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9"
Oct 31 19:50:33.037899 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:33.037790    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aceb3312-c698-4f69-9b44-be9bd3ef0e6b-istio-podinfo\") pod \"e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9\" (UID: \"aceb3312-c698-4f69-9b44-be9bd3ef0e6b\") " pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9"
Oct 31 19:50:33.038025 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:33.037913    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aceb3312-c698-4f69-9b44-be9bd3ef0e6b-workload-socket\") pod \"e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9\" (UID: \"aceb3312-c698-4f69-9b44-be9bd3ef0e6b\") " pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9"
Oct 31 19:50:33.038025 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:33.037982    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aceb3312-c698-4f69-9b44-be9bd3ef0e6b-credential-socket\") pod \"e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9\" (UID: \"aceb3312-c698-4f69-9b44-be9bd3ef0e6b\") " pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9"
Oct 31 19:50:33.038097 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:33.038025    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aceb3312-c698-4f69-9b44-be9bd3ef0e6b-workload-certs\") pod \"e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9\" (UID: \"aceb3312-c698-4f69-9b44-be9bd3ef0e6b\") " pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9"
Oct 31 19:50:33.038444 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:33.038419    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aceb3312-c698-4f69-9b44-be9bd3ef0e6b-istio-data\") pod \"e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9\" (UID: \"aceb3312-c698-4f69-9b44-be9bd3ef0e6b\") " pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9"
Oct 31 19:50:33.038701 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:33.038677    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aceb3312-c698-4f69-9b44-be9bd3ef0e6b-istiod-ca-cert\") pod \"e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9\" (UID: \"aceb3312-c698-4f69-9b44-be9bd3ef0e6b\") " pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9"
Oct 31 19:50:33.040044 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:33.040015    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aceb3312-c698-4f69-9b44-be9bd3ef0e6b-istio-envoy\") pod \"e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9\" (UID: \"aceb3312-c698-4f69-9b44-be9bd3ef0e6b\") " pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9"
Oct 31 19:50:33.040383 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:33.040364    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aceb3312-c698-4f69-9b44-be9bd3ef0e6b-istio-podinfo\") pod \"e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9\" (UID: \"aceb3312-c698-4f69-9b44-be9bd3ef0e6b\") " pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9"
Oct 31 19:50:33.064450 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:33.064427    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aceb3312-c698-4f69-9b44-be9bd3ef0e6b-istio-token\") pod \"e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9\" (UID: \"aceb3312-c698-4f69-9b44-be9bd3ef0e6b\") " pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9"
Oct 31 19:50:33.065120 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:33.065101    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vbwr\" (UniqueName: \"kubernetes.io/projected/aceb3312-c698-4f69-9b44-be9bd3ef0e6b-kube-api-access-2vbwr\") pod \"e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9\" (UID: \"aceb3312-c698-4f69-9b44-be9bd3ef0e6b\") " pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9"
Oct 31 19:50:33.209137 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:33.209105    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9"
Oct 31 19:50:33.415577 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:33.415544    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:33.503388 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:33.503359    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9"]
Oct 31 19:50:33.503915 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:50:33.503889    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaceb3312_c698_4f69_9b44_be9bd3ef0e6b.slice/crio-43e44c03a13c066202b1bab74cafb0f8f407b0921ddd569f881b1afecee972d8 WatchSource:0}: Error finding container 43e44c03a13c066202b1bab74cafb0f8f407b0921ddd569f881b1afecee972d8: Status 404 returned error can't find the container with id 43e44c03a13c066202b1bab74cafb0f8f407b0921ddd569f881b1afecee972d8
Oct 31 19:50:33.507818 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:33.507780    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:50:33.507957 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:33.507867    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:50:33.507957 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:33.507914    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:50:33.728437 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:33.728399    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-a4e312/test-model-a4e312-kserve-f4b656bdc-29rj9" podUID="7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1" containerName="main" containerID="cri-o://cf2f7e92f053205e1c1e3a398ee0045de8bdee5bd3f08695cc94ed51ee666c19" gracePeriod=45
Oct 31 19:50:33.754193 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:33.754118    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9" event={"ID":"aceb3312-c698-4f69-9b44-be9bd3ef0e6b","Type":"ContainerStarted","Data":"ebb0038c362dad3b8d3f2bbbed721b338c570108d2d175d8e3474999a978b949"}
Oct 31 19:50:33.754193 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:33.754180    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9" event={"ID":"aceb3312-c698-4f69-9b44-be9bd3ef0e6b","Type":"ContainerStarted","Data":"43e44c03a13c066202b1bab74cafb0f8f407b0921ddd569f881b1afecee972d8"}
Oct 31 19:50:33.788386 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:33.788341    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9" podStartSLOduration=1.788312186 podStartE2EDuration="1.788312186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:50:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:50:33.784192601 +0000 UTC m=+1864.357426729" watchObservedRunningTime="2026-10-31 19:50:33.788312186 +0000 UTC m=+1864.361546314"
Oct 31 19:50:34.013797 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:34.013772    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a4e312/test-model-a4e312-kserve-f4b656bdc-29rj9"
Oct 31 19:50:34.055351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:34.055304    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1-tmp-dir\") pod \"7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1\" (UID: \"7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1\") "
Oct 31 19:50:34.055573 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:34.055553    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1-dshm\") pod \"7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1\" (UID: \"7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1\") "
Oct 31 19:50:34.055628 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:34.055610    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1-tmp-dir" pod "7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1" (UID: "7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:34.055669 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:34.055631    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1-model-cache\") pod \"7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1\" (UID: \"7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1\") "
Oct 31 19:50:34.055721 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:34.055667    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1-kube-api-access-v98bh\" (UniqueName: \"kubernetes.io/projected/7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1-kube-api-access-v98bh\") pod \"7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1\" (UID: \"7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1\") "
Oct 31 19:50:34.055780 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:34.055769    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1-home\" (UniqueName: \"kubernetes.io/empty-dir/7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1-home\") pod \"7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1\" (UID: \"7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1\") "
Oct 31 19:50:34.055899 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:34.055881    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1-tls-certs\" (UniqueName: \"kubernetes.io/secret/7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1-tls-certs\") pod \"7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1\" (UID: \"7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1\") "
Oct 31 19:50:34.055988 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:34.055918    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1-model-cache" pod "7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1" (UID: "7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:34.056081 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:34.056060    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1-home" pod "7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1" (UID: "7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:34.056897 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:34.056845    4645 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1-model-cache\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:34.056897 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:34.056872    4645 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1-home\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:34.056897 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:34.056889    4645 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1-tmp-dir\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:34.058683 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:34.058290    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1-dshm" pod "7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1" (UID: "7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:34.058683 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:34.058485    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1-kube-api-access-v98bh" pod "7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1" (UID: "7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1"). InnerVolumeSpecName "kube-api-access-v98bh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:50:34.058683 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:34.058613    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1-tls-certs" pod "7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1" (UID: "7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:50:34.158246 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:34.158212    4645 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1-dshm\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:34.158246 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:34.158243    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v98bh\" (UniqueName: \"kubernetes.io/projected/7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1-kube-api-access-v98bh\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:34.158476 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:34.158254    4645 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1-tls-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:34.209599 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:34.209549    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9"
Oct 31 19:50:34.211135 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:34.211108    4645 patch_prober.go:29] interesting pod/e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 31 19:50:34.211289 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:34.211165    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9" podUID="aceb3312-c698-4f69-9b44-be9bd3ef0e6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 31 19:50:34.257579 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:34.256668    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:34.789095 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:34.789063    4645 generic.go:356] "Generic (PLEG): container finished" podID="7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1" containerID="cf2f7e92f053205e1c1e3a398ee0045de8bdee5bd3f08695cc94ed51ee666c19" exitCode=0
Oct 31 19:50:34.789317 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:34.789109    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-a4e312/test-model-a4e312-kserve-f4b656bdc-29rj9"
Oct 31 19:50:34.789317 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:34.789137    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a4e312/test-model-a4e312-kserve-f4b656bdc-29rj9" event={"ID":"7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1","Type":"ContainerDied","Data":"cf2f7e92f053205e1c1e3a398ee0045de8bdee5bd3f08695cc94ed51ee666c19"}
Oct 31 19:50:34.789317 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:34.789188    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-a4e312/test-model-a4e312-kserve-f4b656bdc-29rj9" event={"ID":"7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1","Type":"ContainerDied","Data":"99efe634b2265a0741d868590d4b0e075b08ff94f2299ec43d51ed8dcaf5e23f"}
Oct 31 19:50:34.789317 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:34.789205    4645 scope.go:122] "RemoveContainer" containerID="cf2f7e92f053205e1c1e3a398ee0045de8bdee5bd3f08695cc94ed51ee666c19"
Oct 31 19:50:34.801957 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:34.801936    4645 scope.go:122] "RemoveContainer" containerID="cf2f7e92f053205e1c1e3a398ee0045de8bdee5bd3f08695cc94ed51ee666c19"
Oct 31 19:50:34.802211 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:50:34.802191    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf2f7e92f053205e1c1e3a398ee0045de8bdee5bd3f08695cc94ed51ee666c19\": container with ID starting with cf2f7e92f053205e1c1e3a398ee0045de8bdee5bd3f08695cc94ed51ee666c19 not found: ID does not exist" containerID="cf2f7e92f053205e1c1e3a398ee0045de8bdee5bd3f08695cc94ed51ee666c19"
Oct 31 19:50:34.802279 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:34.802217    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf2f7e92f053205e1c1e3a398ee0045de8bdee5bd3f08695cc94ed51ee666c19"} err="failed to get container status \"cf2f7e92f053205e1c1e3a398ee0045de8bdee5bd3f08695cc94ed51ee666c19\": rpc error: code = NotFound desc = could not find container \"cf2f7e92f053205e1c1e3a398ee0045de8bdee5bd3f08695cc94ed51ee666c19\": container with ID starting with cf2f7e92f053205e1c1e3a398ee0045de8bdee5bd3f08695cc94ed51ee666c19 not found: ID does not exist"
Oct 31 19:50:34.820500 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:34.820479    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-a4e312/test-model-a4e312-kserve-f4b656bdc-29rj9"]
Oct 31 19:50:34.825292 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:34.825268    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-a4e312/test-model-a4e312-kserve-f4b656bdc-29rj9"]
Oct 31 19:50:34.982550 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:34.982499    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:35.177408 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:35.152666    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-9efeae/test-model-9efeae-kserve-7d97f79d8f-2nnrl" podUID="72f178d8-b759-4bf3-90fa-1b0a1a8cb51e" containerName="main" containerID="cri-o://4982e92d7210d288d06cf8ec93fd057a6a03d5d70a801ab9afd8397eb88c0560" gracePeriod=45
Oct 31 19:50:35.234350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:35.221452    4645 patch_prober.go:29] interesting pod/e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 31 19:50:35.234350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:35.221512    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9" podUID="aceb3312-c698-4f69-9b44-be9bd3ef0e6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 31 19:50:35.234350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:35.231824    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:35.361359 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:35.278539    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-9efeae/test-model-9efeae-kserve-7d97f79d8f-2nnrl" podUID="72f178d8-b759-4bf3-90fa-1b0a1a8cb51e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.154:8000/ready\": read tcp 10.133.0.2:40778->10.133.0.154:8000: read: connection reset by peer"
Oct 31 19:50:35.483756 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:35.483536    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:35.553209 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:35.553157    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-278c04-6969d7fcb9-252tn" podUID="1bdc77fe-26fe-4f78-8242-4fed7e6f3c37" containerName="payload-processing" containerID="cri-o://0737d698d1ae9996feb82c0dc6627c55203dbda8a5e8425fdeabfdc2b56a1dd5" gracePeriod=25
Oct 31 19:50:35.584319 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:35.583403    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-278c04-86645f6fbc-p5qmg" podUID="12fd3774-779d-49d5-8fe2-6804ed6d50ab" containerName="payload-pre-processing" containerID="cri-o://9ee0280439447d0096853e72d3fea715a0b4d7e8367694f4b17124d87eb9a499" gracePeriod=25
Oct 31 19:50:35.651348 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:35.647518    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:35.718150 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:35.718113    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1" path="/var/lib/kubelet/pods/7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1/volumes"
Oct 31 19:50:35.808268 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:35.808235    4645 generic.go:356] "Generic (PLEG): container finished" podID="72f178d8-b759-4bf3-90fa-1b0a1a8cb51e" containerID="4982e92d7210d288d06cf8ec93fd057a6a03d5d70a801ab9afd8397eb88c0560" exitCode=0
Oct 31 19:50:35.809732 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:35.809703    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9efeae/test-model-9efeae-kserve-7d97f79d8f-2nnrl" event={"ID":"72f178d8-b759-4bf3-90fa-1b0a1a8cb51e","Type":"ContainerDied","Data":"4982e92d7210d288d06cf8ec93fd057a6a03d5d70a801ab9afd8397eb88c0560"}
Oct 31 19:50:35.941699 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:35.941672    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9efeae/test-model-9efeae-kserve-7d97f79d8f-2nnrl"
Oct 31 19:50:35.986715 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:35.986672    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-88eaf9-56b45b6577-m674p" podUID="7ae56bf5-1961-48fc-9c77-bb64e0d791cf" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:50:35.989763 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:35.989724    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-88eaf9-697ffc695-b9ds8" podUID="f2c7c458-2fc5-4674-b080-7f0c9f29a3b9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:50:36.012857 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:36.010183    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72f178d8-b759-4bf3-90fa-1b0a1a8cb51e-kube-api-access-pztm7\" (UniqueName: \"kubernetes.io/projected/72f178d8-b759-4bf3-90fa-1b0a1a8cb51e-kube-api-access-pztm7\") pod \"72f178d8-b759-4bf3-90fa-1b0a1a8cb51e\" (UID: \"72f178d8-b759-4bf3-90fa-1b0a1a8cb51e\") "
Oct 31 19:50:36.012857 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:36.010250    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72f178d8-b759-4bf3-90fa-1b0a1a8cb51e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72f178d8-b759-4bf3-90fa-1b0a1a8cb51e-tmp-dir\") pod \"72f178d8-b759-4bf3-90fa-1b0a1a8cb51e\" (UID: \"72f178d8-b759-4bf3-90fa-1b0a1a8cb51e\") "
Oct 31 19:50:36.012857 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:36.010287    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72f178d8-b759-4bf3-90fa-1b0a1a8cb51e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/72f178d8-b759-4bf3-90fa-1b0a1a8cb51e-dshm\") pod \"72f178d8-b759-4bf3-90fa-1b0a1a8cb51e\" (UID: \"72f178d8-b759-4bf3-90fa-1b0a1a8cb51e\") "
Oct 31 19:50:36.012857 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:36.010485    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72f178d8-b759-4bf3-90fa-1b0a1a8cb51e-home\" (UniqueName: \"kubernetes.io/empty-dir/72f178d8-b759-4bf3-90fa-1b0a1a8cb51e-home\") pod \"72f178d8-b759-4bf3-90fa-1b0a1a8cb51e\" (UID: \"72f178d8-b759-4bf3-90fa-1b0a1a8cb51e\") "
Oct 31 19:50:36.012857 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:36.010878    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72f178d8-b759-4bf3-90fa-1b0a1a8cb51e-tmp-dir" pod "72f178d8-b759-4bf3-90fa-1b0a1a8cb51e" (UID: "72f178d8-b759-4bf3-90fa-1b0a1a8cb51e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:36.012857 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:36.010925    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72f178d8-b759-4bf3-90fa-1b0a1a8cb51e-tls-certs\" (UniqueName: \"kubernetes.io/secret/72f178d8-b759-4bf3-90fa-1b0a1a8cb51e-tls-certs\") pod \"72f178d8-b759-4bf3-90fa-1b0a1a8cb51e\" (UID: \"72f178d8-b759-4bf3-90fa-1b0a1a8cb51e\") "
Oct 31 19:50:36.012857 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:36.011122    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72f178d8-b759-4bf3-90fa-1b0a1a8cb51e-home" pod "72f178d8-b759-4bf3-90fa-1b0a1a8cb51e" (UID: "72f178d8-b759-4bf3-90fa-1b0a1a8cb51e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:36.012857 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:36.011181    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72f178d8-b759-4bf3-90fa-1b0a1a8cb51e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72f178d8-b759-4bf3-90fa-1b0a1a8cb51e-model-cache\") pod \"72f178d8-b759-4bf3-90fa-1b0a1a8cb51e\" (UID: \"72f178d8-b759-4bf3-90fa-1b0a1a8cb51e\") "
Oct 31 19:50:36.012857 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:36.011878    4645 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72f178d8-b759-4bf3-90fa-1b0a1a8cb51e-tmp-dir\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:36.012857 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:36.011895    4645 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72f178d8-b759-4bf3-90fa-1b0a1a8cb51e-home\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:36.012857 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:36.012400    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72f178d8-b759-4bf3-90fa-1b0a1a8cb51e-model-cache" pod "72f178d8-b759-4bf3-90fa-1b0a1a8cb51e" (UID: "72f178d8-b759-4bf3-90fa-1b0a1a8cb51e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:36.017574 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:36.017546    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72f178d8-b759-4bf3-90fa-1b0a1a8cb51e-dshm" pod "72f178d8-b759-4bf3-90fa-1b0a1a8cb51e" (UID: "72f178d8-b759-4bf3-90fa-1b0a1a8cb51e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:36.023123 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:36.022940    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72f178d8-b759-4bf3-90fa-1b0a1a8cb51e-kube-api-access-pztm7" pod "72f178d8-b759-4bf3-90fa-1b0a1a8cb51e" (UID: "72f178d8-b759-4bf3-90fa-1b0a1a8cb51e"). InnerVolumeSpecName "kube-api-access-pztm7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:50:36.024102 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:36.023728    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72f178d8-b759-4bf3-90fa-1b0a1a8cb51e-tls-certs" pod "72f178d8-b759-4bf3-90fa-1b0a1a8cb51e" (UID: "72f178d8-b759-4bf3-90fa-1b0a1a8cb51e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:50:36.113309 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:36.113184    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pztm7\" (UniqueName: \"kubernetes.io/projected/72f178d8-b759-4bf3-90fa-1b0a1a8cb51e-kube-api-access-pztm7\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:36.113309 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:36.113215    4645 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72f178d8-b759-4bf3-90fa-1b0a1a8cb51e-dshm\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:36.113309 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:36.113225    4645 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72f178d8-b759-4bf3-90fa-1b0a1a8cb51e-tls-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:36.113309 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:36.113236    4645 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72f178d8-b759-4bf3-90fa-1b0a1a8cb51e-model-cache\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:36.148603 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:36.148493    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:36.214555 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:36.214512    4645 patch_prober.go:29] interesting pod/e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 31 19:50:36.214756 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:36.214606    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9" podUID="aceb3312-c698-4f69-9b44-be9bd3ef0e6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 31 19:50:36.283408 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:50:36.283167    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 19:50:36.438748 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:36.438704    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-f8c3f7/test-model-f8c3f7-kserve-6fccbd6557-ts9sg" podUID="2e6b6da3-9983-463c-a814-e07d9a22ed1f" containerName="main" containerID="cri-o://a47060f93eb207e0243ec596833623fd5c66cea327a8f338c70d52f0ab6fef78" gracePeriod=45
Oct 31 19:50:36.649408 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:36.649371    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:36.822838 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:36.822797    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9efeae/test-model-9efeae-kserve-7d97f79d8f-2nnrl" event={"ID":"72f178d8-b759-4bf3-90fa-1b0a1a8cb51e","Type":"ContainerDied","Data":"9e9320bc3f9c210608621b8bae4399ce5066c541f30c75e7bde9c0aa3abaa328"}
Oct 31 19:50:36.823047 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:36.822858    4645 scope.go:122] "RemoveContainer" containerID="4982e92d7210d288d06cf8ec93fd057a6a03d5d70a801ab9afd8397eb88c0560"
Oct 31 19:50:36.823114 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:36.823080    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9efeae/test-model-9efeae-kserve-7d97f79d8f-2nnrl"
Oct 31 19:50:36.826455 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:36.826420    4645 generic.go:356] "Generic (PLEG): container finished" podID="2e6b6da3-9983-463c-a814-e07d9a22ed1f" containerID="a47060f93eb207e0243ec596833623fd5c66cea327a8f338c70d52f0ab6fef78" exitCode=0
Oct 31 19:50:36.826664 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:36.826497    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f8c3f7/test-model-f8c3f7-kserve-6fccbd6557-ts9sg" event={"ID":"2e6b6da3-9983-463c-a814-e07d9a22ed1f","Type":"ContainerDied","Data":"a47060f93eb207e0243ec596833623fd5c66cea327a8f338c70d52f0ab6fef78"}
Oct 31 19:50:36.905844 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:36.905814    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-9efeae/test-model-9efeae-kserve-7d97f79d8f-2nnrl"]
Oct 31 19:50:36.911878 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:36.911849    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-9efeae/test-model-9efeae-kserve-7d97f79d8f-2nnrl"]
Oct 31 19:50:36.920131 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:36.920106    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f8c3f7/test-model-f8c3f7-kserve-6fccbd6557-ts9sg"
Oct 31 19:50:37.035949 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:37.035918    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e6b6da3-9983-463c-a814-e07d9a22ed1f-home\" (UniqueName: \"kubernetes.io/empty-dir/2e6b6da3-9983-463c-a814-e07d9a22ed1f-home\") pod \"2e6b6da3-9983-463c-a814-e07d9a22ed1f\" (UID: \"2e6b6da3-9983-463c-a814-e07d9a22ed1f\") "
Oct 31 19:50:37.036128 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:37.036000    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e6b6da3-9983-463c-a814-e07d9a22ed1f-tls-certs\" (UniqueName: \"kubernetes.io/secret/2e6b6da3-9983-463c-a814-e07d9a22ed1f-tls-certs\") pod \"2e6b6da3-9983-463c-a814-e07d9a22ed1f\" (UID: \"2e6b6da3-9983-463c-a814-e07d9a22ed1f\") "
Oct 31 19:50:37.036128 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:37.036030    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e6b6da3-9983-463c-a814-e07d9a22ed1f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e6b6da3-9983-463c-a814-e07d9a22ed1f-dshm\") pod \"2e6b6da3-9983-463c-a814-e07d9a22ed1f\" (UID: \"2e6b6da3-9983-463c-a814-e07d9a22ed1f\") "
Oct 31 19:50:37.036215 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:37.036191    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e6b6da3-9983-463c-a814-e07d9a22ed1f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e6b6da3-9983-463c-a814-e07d9a22ed1f-model-cache\") pod \"2e6b6da3-9983-463c-a814-e07d9a22ed1f\" (UID: \"2e6b6da3-9983-463c-a814-e07d9a22ed1f\") "
Oct 31 19:50:37.036249 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:37.036228    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e6b6da3-9983-463c-a814-e07d9a22ed1f-kube-api-access-m5fsp\" (UniqueName: \"kubernetes.io/projected/2e6b6da3-9983-463c-a814-e07d9a22ed1f-kube-api-access-m5fsp\") pod \"2e6b6da3-9983-463c-a814-e07d9a22ed1f\" (UID: \"2e6b6da3-9983-463c-a814-e07d9a22ed1f\") "
Oct 31 19:50:37.036285 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:37.036262    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e6b6da3-9983-463c-a814-e07d9a22ed1f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e6b6da3-9983-463c-a814-e07d9a22ed1f-tmp-dir\") pod \"2e6b6da3-9983-463c-a814-e07d9a22ed1f\" (UID: \"2e6b6da3-9983-463c-a814-e07d9a22ed1f\") "
Oct 31 19:50:37.036637 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:37.036608    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e6b6da3-9983-463c-a814-e07d9a22ed1f-home" pod "2e6b6da3-9983-463c-a814-e07d9a22ed1f" (UID: "2e6b6da3-9983-463c-a814-e07d9a22ed1f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:37.036855 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:37.036831    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e6b6da3-9983-463c-a814-e07d9a22ed1f-tmp-dir" pod "2e6b6da3-9983-463c-a814-e07d9a22ed1f" (UID: "2e6b6da3-9983-463c-a814-e07d9a22ed1f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:37.036945 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:37.036618    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e6b6da3-9983-463c-a814-e07d9a22ed1f-model-cache" pod "2e6b6da3-9983-463c-a814-e07d9a22ed1f" (UID: "2e6b6da3-9983-463c-a814-e07d9a22ed1f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:37.037281 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:37.037258    4645 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e6b6da3-9983-463c-a814-e07d9a22ed1f-home\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:37.037397 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:37.037287    4645 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e6b6da3-9983-463c-a814-e07d9a22ed1f-model-cache\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:37.037397 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:37.037304    4645 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e6b6da3-9983-463c-a814-e07d9a22ed1f-tmp-dir\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:37.042552 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:37.042515    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e6b6da3-9983-463c-a814-e07d9a22ed1f-kube-api-access-m5fsp" pod "2e6b6da3-9983-463c-a814-e07d9a22ed1f" (UID: "2e6b6da3-9983-463c-a814-e07d9a22ed1f"). InnerVolumeSpecName "kube-api-access-m5fsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:50:37.042666 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:37.042640    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e6b6da3-9983-463c-a814-e07d9a22ed1f-tls-certs" pod "2e6b6da3-9983-463c-a814-e07d9a22ed1f" (UID: "2e6b6da3-9983-463c-a814-e07d9a22ed1f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:50:37.042834 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:37.042802    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e6b6da3-9983-463c-a814-e07d9a22ed1f-dshm" pod "2e6b6da3-9983-463c-a814-e07d9a22ed1f" (UID: "2e6b6da3-9983-463c-a814-e07d9a22ed1f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:37.138911 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:37.138839    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5fsp\" (UniqueName: \"kubernetes.io/projected/2e6b6da3-9983-463c-a814-e07d9a22ed1f-kube-api-access-m5fsp\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:37.138911 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:37.138868    4645 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e6b6da3-9983-463c-a814-e07d9a22ed1f-tls-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:37.138911 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:37.138879    4645 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e6b6da3-9983-463c-a814-e07d9a22ed1f-dshm\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:37.209611 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:37.209571    4645 patch_prober.go:29] interesting pod/e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 31 19:50:37.209812 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:37.209637    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9" podUID="aceb3312-c698-4f69-9b44-be9bd3ef0e6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 31 19:50:37.425146 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:37.425108    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-88eaf9-9988c7848-fqszv"]
Oct 31 19:50:37.426413 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:37.425378    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-88eaf9-9988c7848-fqszv" podUID="4abee805-15d3-4b54-bd38-889f562f0c57" containerName="maas-api" containerID="cri-o://dc4c9f61fb81ea995f5ec6c2308a699a654fca7214253a53df32f301272b4449" gracePeriod=30
Oct 31 19:50:37.486867 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:37.486832    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:37.623240 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:37.623205    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72f178d8-b759-4bf3-90fa-1b0a1a8cb51e" path="/var/lib/kubelet/pods/72f178d8-b759-4bf3-90fa-1b0a1a8cb51e/volumes"
Oct 31 19:50:38.153101 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:38.141868    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:38.229875 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:38.229843    4645 patch_prober.go:29] interesting pod/e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 31 19:50:38.230389 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:38.230107    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9" podUID="aceb3312-c698-4f69-9b44-be9bd3ef0e6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 31 19:50:38.238650 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:38.238618    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np"]
Oct 31 19:50:38.240016 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:38.239992    4645 cpu_manager.go:414] "containerMap: removing container" podUID="2e6b6da3-9983-463c-a814-e07d9a22ed1f" containerName="main"
Oct 31 19:50:38.240132 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:38.240120    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e6b6da3-9983-463c-a814-e07d9a22ed1f" containerName="main"
Oct 31 19:50:38.240294 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:38.240283    4645 cpu_manager.go:414] "containerMap: removing container" podUID="7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1" containerName="main"
Oct 31 19:50:38.240397 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:38.240386    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1" containerName="main"
Oct 31 19:50:38.240490 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:38.240481    4645 cpu_manager.go:414] "containerMap: removing container" podUID="72f178d8-b759-4bf3-90fa-1b0a1a8cb51e" containerName="main"
Oct 31 19:50:38.240560 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:38.240551    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72f178d8-b759-4bf3-90fa-1b0a1a8cb51e" containerName="main"
Oct 31 19:50:38.240880 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:38.240868    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b61ab4a-9eb7-48d6-bc3a-7e3b4444a9b1" containerName="main"
Oct 31 19:50:38.240966 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:38.240958    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="72f178d8-b759-4bf3-90fa-1b0a1a8cb51e" containerName="main"
Oct 31 19:50:38.241049 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:38.241040    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e6b6da3-9983-463c-a814-e07d9a22ed1f" containerName="main"
Oct 31 19:50:38.245286 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:38.245265    4645 generic.go:356] "Generic (PLEG): container finished" podID="4abee805-15d3-4b54-bd38-889f562f0c57" containerID="dc4c9f61fb81ea995f5ec6c2308a699a654fca7214253a53df32f301272b4449" exitCode=0
Oct 31 19:50:38.246886 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:38.246867    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np"]
Oct 31 19:50:38.246988 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:38.246908    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-88eaf9-9988c7848-fqszv" event={"ID":"4abee805-15d3-4b54-bd38-889f562f0c57","Type":"ContainerDied","Data":"dc4c9f61fb81ea995f5ec6c2308a699a654fca7214253a53df32f301272b4449"}
Oct 31 19:50:38.246988 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:38.246940    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f8c3f7/test-model-f8c3f7-kserve-6fccbd6557-ts9sg" event={"ID":"2e6b6da3-9983-463c-a814-e07d9a22ed1f","Type":"ContainerDied","Data":"c3045f81894e4e24d257d936c668e52b3a319e50f52178e1a3bc19fd538ac3d7"}
Oct 31 19:50:38.246988 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:38.246968    4645 scope.go:122] "RemoveContainer" containerID="a47060f93eb207e0243ec596833623fd5c66cea327a8f338c70d52f0ab6fef78"
Oct 31 19:50:38.247162 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:38.247021    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f8c3f7/test-model-f8c3f7-kserve-6fccbd6557-ts9sg"
Oct 31 19:50:38.247614 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:38.247599    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np"
Oct 31 19:50:38.251711 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:38.251691    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-88eaf9-r9646"]
Oct 31 19:50:38.261484 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:38.259559    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-88eaf9-r9646"]
Oct 31 19:50:38.290496 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:38.290053    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-8dcfa8-openshift-default-dockercfg-jsqp5\""
Oct 31 19:50:38.292552 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:38.291953    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:38.292552 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:38.292373    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c712ccf6-e167-4c11-ba0a-df297a0caf05-credential-socket\") pod \"e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np\" (UID: \"c712ccf6-e167-4c11-ba0a-df297a0caf05\") " pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np"
Oct 31 19:50:38.292552 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:38.292549    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c712ccf6-e167-4c11-ba0a-df297a0caf05-istio-podinfo\") pod \"e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np\" (UID: \"c712ccf6-e167-4c11-ba0a-df297a0caf05\") " pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np"
Oct 31 19:50:38.292952 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:38.292674    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c712ccf6-e167-4c11-ba0a-df297a0caf05-istio-token\") pod \"e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np\" (UID: \"c712ccf6-e167-4c11-ba0a-df297a0caf05\") " pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np"
Oct 31 19:50:38.292952 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:38.292796    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c712ccf6-e167-4c11-ba0a-df297a0caf05-workload-certs\") pod \"e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np\" (UID: \"c712ccf6-e167-4c11-ba0a-df297a0caf05\") " pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np"
Oct 31 19:50:38.293140 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:38.293005    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c712ccf6-e167-4c11-ba0a-df297a0caf05-istio-data\") pod \"e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np\" (UID: \"c712ccf6-e167-4c11-ba0a-df297a0caf05\") " pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np"
Oct 31 19:50:38.293140 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:38.293105    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c712ccf6-e167-4c11-ba0a-df297a0caf05-istiod-ca-cert\") pod \"e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np\" (UID: \"c712ccf6-e167-4c11-ba0a-df297a0caf05\") " pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np"
Oct 31 19:50:38.293140 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:38.293131    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c712ccf6-e167-4c11-ba0a-df297a0caf05-istio-envoy\") pod \"e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np\" (UID: \"c712ccf6-e167-4c11-ba0a-df297a0caf05\") " pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np"
Oct 31 19:50:38.293316 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:38.293147    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbhq9\" (UniqueName: \"kubernetes.io/projected/c712ccf6-e167-4c11-ba0a-df297a0caf05-kube-api-access-vbhq9\") pod \"e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np\" (UID: \"c712ccf6-e167-4c11-ba0a-df297a0caf05\") " pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np"
Oct 31 19:50:38.293316 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:38.293168    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c712ccf6-e167-4c11-ba0a-df297a0caf05-workload-socket\") pod \"e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np\" (UID: \"c712ccf6-e167-4c11-ba0a-df297a0caf05\") " pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np"
Oct 31 19:50:38.296321 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:38.296303    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-88eaf9-9988c7848-fqszv"
Oct 31 19:50:38.390111 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:38.390072    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-f8c3f7/test-model-f8c3f7-kserve-6fccbd6557-ts9sg"]
Oct 31 19:50:38.395026 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:38.394993    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4abee805-15d3-4b54-bd38-889f562f0c57-kube-api-access-nknl4\" (UniqueName: \"kubernetes.io/projected/4abee805-15d3-4b54-bd38-889f562f0c57-kube-api-access-nknl4\") pod \"4abee805-15d3-4b54-bd38-889f562f0c57\" (UID: \"4abee805-15d3-4b54-bd38-889f562f0c57\") "
Oct 31 19:50:38.395215 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:38.395185    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4abee805-15d3-4b54-bd38-889f562f0c57-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4abee805-15d3-4b54-bd38-889f562f0c57-openshift-service-ca\") pod \"4abee805-15d3-4b54-bd38-889f562f0c57\" (UID: \"4abee805-15d3-4b54-bd38-889f562f0c57\") "
Oct 31 19:50:38.395720 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:38.395689    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4abee805-15d3-4b54-bd38-889f562f0c57-openshift-service-ca" pod "4abee805-15d3-4b54-bd38-889f562f0c57" (UID: "4abee805-15d3-4b54-bd38-889f562f0c57"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:50:38.395825 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:38.395740    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-f8c3f7/test-model-f8c3f7-kserve-6fccbd6557-ts9sg"]
Oct 31 19:50:38.395882 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:38.395840    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4abee805-15d3-4b54-bd38-889f562f0c57-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4abee805-15d3-4b54-bd38-889f562f0c57-metrics-certs\") pod \"4abee805-15d3-4b54-bd38-889f562f0c57\" (UID: \"4abee805-15d3-4b54-bd38-889f562f0c57\") "
Oct 31 19:50:38.395941 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:38.395882    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4abee805-15d3-4b54-bd38-889f562f0c57-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4abee805-15d3-4b54-bd38-889f562f0c57-maas-api-tls\") pod \"4abee805-15d3-4b54-bd38-889f562f0c57\" (UID: \"4abee805-15d3-4b54-bd38-889f562f0c57\") "
Oct 31 19:50:38.396580 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:38.396555    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c712ccf6-e167-4c11-ba0a-df297a0caf05-istio-data\") pod \"e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np\" (UID: \"c712ccf6-e167-4c11-ba0a-df297a0caf05\") " pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np"
Oct 31 19:50:38.396749 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:38.396726    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c712ccf6-e167-4c11-ba0a-df297a0caf05-istiod-ca-cert\") pod \"e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np\" (UID: \"c712ccf6-e167-4c11-ba0a-df297a0caf05\") " pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np"
Oct 31 19:50:38.396832 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:38.396769    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c712ccf6-e167-4c11-ba0a-df297a0caf05-istio-envoy\") pod \"e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np\" (UID: \"c712ccf6-e167-4c11-ba0a-df297a0caf05\") " pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np"
Oct 31 19:50:38.396832 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:38.396796    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbhq9\" (UniqueName: \"kubernetes.io/projected/c712ccf6-e167-4c11-ba0a-df297a0caf05-kube-api-access-vbhq9\") pod \"e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np\" (UID: \"c712ccf6-e167-4c11-ba0a-df297a0caf05\") " pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np"
Oct 31 19:50:38.396832 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:38.396824    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c712ccf6-e167-4c11-ba0a-df297a0caf05-workload-socket\") pod \"e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np\" (UID: \"c712ccf6-e167-4c11-ba0a-df297a0caf05\") " pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np"
Oct 31 19:50:38.397015 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:38.396881    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c712ccf6-e167-4c11-ba0a-df297a0caf05-istio-data\") pod \"e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np\" (UID: \"c712ccf6-e167-4c11-ba0a-df297a0caf05\") " pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np"
Oct 31 19:50:38.397015 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:38.396938    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c712ccf6-e167-4c11-ba0a-df297a0caf05-credential-socket\") pod \"e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np\" (UID: \"c712ccf6-e167-4c11-ba0a-df297a0caf05\") " pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np"
Oct 31 19:50:38.397015 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:38.396965    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c712ccf6-e167-4c11-ba0a-df297a0caf05-istio-podinfo\") pod \"e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np\" (UID: \"c712ccf6-e167-4c11-ba0a-df297a0caf05\") " pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np"
Oct 31 19:50:38.397183 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:38.397085    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c712ccf6-e167-4c11-ba0a-df297a0caf05-istio-token\") pod \"e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np\" (UID: \"c712ccf6-e167-4c11-ba0a-df297a0caf05\") " pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np"
Oct 31 19:50:38.397183 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:38.397147    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c712ccf6-e167-4c11-ba0a-df297a0caf05-workload-certs\") pod \"e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np\" (UID: \"c712ccf6-e167-4c11-ba0a-df297a0caf05\") " pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np"
Oct 31 19:50:38.397301 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:38.397223    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c712ccf6-e167-4c11-ba0a-df297a0caf05-workload-socket\") pod \"e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np\" (UID: \"c712ccf6-e167-4c11-ba0a-df297a0caf05\") " pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np"
Oct 31 19:50:38.397687 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:38.397662    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c712ccf6-e167-4c11-ba0a-df297a0caf05-workload-certs\") pod \"e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np\" (UID: \"c712ccf6-e167-4c11-ba0a-df297a0caf05\") " pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np"
Oct 31 19:50:38.397997 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:38.397978    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c712ccf6-e167-4c11-ba0a-df297a0caf05-istiod-ca-cert\") pod \"e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np\" (UID: \"c712ccf6-e167-4c11-ba0a-df297a0caf05\") " pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np"
Oct 31 19:50:38.398224 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:38.398198    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4abee805-15d3-4b54-bd38-889f562f0c57-kube-api-access-nknl4" pod "4abee805-15d3-4b54-bd38-889f562f0c57" (UID: "4abee805-15d3-4b54-bd38-889f562f0c57"). InnerVolumeSpecName "kube-api-access-nknl4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:50:38.398224 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:38.398206    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c712ccf6-e167-4c11-ba0a-df297a0caf05-credential-socket\") pod \"e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np\" (UID: \"c712ccf6-e167-4c11-ba0a-df297a0caf05\") " pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np"
Oct 31 19:50:38.398413 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:38.398394    4645 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4abee805-15d3-4b54-bd38-889f562f0c57-openshift-service-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:38.398946 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:38.398911    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4abee805-15d3-4b54-bd38-889f562f0c57-metrics-certs" pod "4abee805-15d3-4b54-bd38-889f562f0c57" (UID: "4abee805-15d3-4b54-bd38-889f562f0c57"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:50:38.400277 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:38.400253    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4abee805-15d3-4b54-bd38-889f562f0c57-maas-api-tls" pod "4abee805-15d3-4b54-bd38-889f562f0c57" (UID: "4abee805-15d3-4b54-bd38-889f562f0c57"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:50:38.400518 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:38.400498    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c712ccf6-e167-4c11-ba0a-df297a0caf05-istio-envoy\") pod \"e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np\" (UID: \"c712ccf6-e167-4c11-ba0a-df297a0caf05\") " pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np"
Oct 31 19:50:38.400582 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:38.400557    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c712ccf6-e167-4c11-ba0a-df297a0caf05-istio-podinfo\") pod \"e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np\" (UID: \"c712ccf6-e167-4c11-ba0a-df297a0caf05\") " pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np"
Oct 31 19:50:38.416496 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:38.416476    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c712ccf6-e167-4c11-ba0a-df297a0caf05-istio-token\") pod \"e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np\" (UID: \"c712ccf6-e167-4c11-ba0a-df297a0caf05\") " pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np"
Oct 31 19:50:38.421782 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:38.421751    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbhq9\" (UniqueName: \"kubernetes.io/projected/c712ccf6-e167-4c11-ba0a-df297a0caf05-kube-api-access-vbhq9\") pod \"e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np\" (UID: \"c712ccf6-e167-4c11-ba0a-df297a0caf05\") " pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np"
Oct 31 19:50:38.498668 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:38.479940    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:38.500846 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:38.500764    4645 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4abee805-15d3-4b54-bd38-889f562f0c57-metrics-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:38.500846 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:38.500795    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4abee805-15d3-4b54-bd38-889f562f0c57-maas-api-tls\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:38.500846 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:38.500814    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nknl4\" (UniqueName: \"kubernetes.io/projected/4abee805-15d3-4b54-bd38-889f562f0c57-kube-api-access-nknl4\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:38.605511 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:38.605479    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np"
Oct 31 19:50:38.934500 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:50:38.934470    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc712ccf6_e167_4c11_ba0a_df297a0caf05.slice/crio-6468e3048c4d991f11b3389294b5e0af03f8604984126ada12b2ad78044cd592 WatchSource:0}: Error finding container 6468e3048c4d991f11b3389294b5e0af03f8604984126ada12b2ad78044cd592: Status 404 returned error can't find the container with id 6468e3048c4d991f11b3389294b5e0af03f8604984126ada12b2ad78044cd592
Oct 31 19:50:38.936997 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:38.936973    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np"]
Oct 31 19:50:38.938274 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:38.938239    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:50:38.938375 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:38.938311    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:50:38.938375 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:38.938360    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:50:39.209614 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:39.209558    4645 patch_prober.go:29] interesting pod/e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 31 19:50:39.209796 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:39.209672    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9" podUID="aceb3312-c698-4f69-9b44-be9bd3ef0e6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 31 19:50:39.323153 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:39.323050    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-88eaf9-9988c7848-fqszv" event={"ID":"4abee805-15d3-4b54-bd38-889f562f0c57","Type":"ContainerDied","Data":"7a430cffc4e88ccce6a67d0dbaf4dfdaa1de7f714c09b19a4daa39048077e0a6"}
Oct 31 19:50:39.323441 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:39.323166    4645 scope.go:122] "RemoveContainer" containerID="dc4c9f61fb81ea995f5ec6c2308a699a654fca7214253a53df32f301272b4449"
Oct 31 19:50:39.323441 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:39.323366    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-88eaf9-9988c7848-fqszv"
Oct 31 19:50:39.331105 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:39.331077    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np" event={"ID":"c712ccf6-e167-4c11-ba0a-df297a0caf05","Type":"ContainerStarted","Data":"95a0a0e65faf9564ab4aff50b6e321570123169bad0ffad9f678ce4105a4a5df"}
Oct 31 19:50:39.331241 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:39.331129    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np" event={"ID":"c712ccf6-e167-4c11-ba0a-df297a0caf05","Type":"ContainerStarted","Data":"6468e3048c4d991f11b3389294b5e0af03f8604984126ada12b2ad78044cd592"}
Oct 31 19:50:39.369761 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:39.369606    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:39.393459 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:39.393398    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np" podStartSLOduration=2.3933779299999998 podStartE2EDuration="2.39337793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:50:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:50:39.386196233 +0000 UTC m=+1869.959430379" watchObservedRunningTime="2026-10-31 19:50:39.39337793 +0000 UTC m=+1869.966612061"
Oct 31 19:50:39.423542 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:39.423508    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-88eaf9-9988c7848-fqszv"]
Oct 31 19:50:39.433504 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:39.433455    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-88eaf9-9988c7848-fqszv"]
Oct 31 19:50:39.820386 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:39.803557    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="231fb65d-49fe-497c-871c-0e15eafce2f9" path="/var/lib/kubelet/pods/231fb65d-49fe-497c-871c-0e15eafce2f9/volumes"
Oct 31 19:50:39.820386 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:39.804473    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e6b6da3-9983-463c-a814-e07d9a22ed1f" path="/var/lib/kubelet/pods/2e6b6da3-9983-463c-a814-e07d9a22ed1f/volumes"
Oct 31 19:50:39.820386 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:39.805293    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4abee805-15d3-4b54-bd38-889f562f0c57" path="/var/lib/kubelet/pods/4abee805-15d3-4b54-bd38-889f562f0c57/volumes"
Oct 31 19:50:39.820386 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:39.805823    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np"
Oct 31 19:50:40.224119 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:40.224084    4645 patch_prober.go:29] interesting pod/e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 31 19:50:40.224485 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:40.224429    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9" podUID="aceb3312-c698-4f69-9b44-be9bd3ef0e6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 31 19:50:40.599405 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:40.596616    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:40.602386 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:40.601714    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a4e312-7b78fb8665-b9pn6"]
Oct 31 19:50:40.606836 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:40.606812    4645 patch_prober.go:29] interesting pod/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body=
Oct 31 19:50:40.606970 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:40.606889    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np" podUID="c712ccf6-e167-4c11-ba0a-df297a0caf05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused"
Oct 31 19:50:40.754384 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:40.754321    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:40.920773 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:40.920727    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:40.993963 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:40.993918    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b1fae7-t2chz"]
Oct 31 19:50:40.998939 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:40.998908    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b1fae7-t2chz"]
Oct 31 19:50:41.148352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:41.135937    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:41.301978 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:41.301893    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:41.550145 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:41.550118    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9"
Oct 31 19:50:41.550580 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:41.550550    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9"
Oct 31 19:50:41.551437 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:41.551418    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9"
Oct 31 19:50:41.576400 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:41.568854    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:41.605858 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:41.605833    4645 patch_prober.go:29] interesting pod/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body=
Oct 31 19:50:41.606061 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:41.606039    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np" podUID="c712ccf6-e167-4c11-ba0a-df297a0caf05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused"
Oct 31 19:50:41.623136 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:41.623107    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18806293-fbe8-44b4-b4f4-1efeec8fb8d8" path="/var/lib/kubelet/pods/18806293-fbe8-44b4-b4f4-1efeec8fb8d8/volumes"
Oct 31 19:50:42.392280 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:42.392244    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:42.606670 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:42.606633    4645 patch_prober.go:29] interesting pod/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body=
Oct 31 19:50:42.606835 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:42.606689    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np" podUID="c712ccf6-e167-4c11-ba0a-df297a0caf05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused"
Oct 31 19:50:43.381526 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:43.381488    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:43.606394 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:43.606324    4645 patch_prober.go:29] interesting pod/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body=
Oct 31 19:50:43.606775 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:43.606409    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np" podUID="c712ccf6-e167-4c11-ba0a-df297a0caf05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused"
Oct 31 19:50:44.181099 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:44.181062    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:44.595171 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:44.595135    4645 generic.go:356] "Generic (PLEG): container finished" podID="7ae56bf5-1961-48fc-9c77-bb64e0d791cf" containerID="622bd5e7b4d3fcab0d71e79b1c5c49e26fbf7866c1213da8d8181067d101fada" exitCode=0
Oct 31 19:50:44.595405 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:44.595286    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-88eaf9-56b45b6577-m674p" event={"ID":"7ae56bf5-1961-48fc-9c77-bb64e0d791cf","Type":"ContainerDied","Data":"622bd5e7b4d3fcab0d71e79b1c5c49e26fbf7866c1213da8d8181067d101fada"}
Oct 31 19:50:44.597725 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:44.597457    4645 generic.go:356] "Generic (PLEG): container finished" podID="f2c7c458-2fc5-4674-b080-7f0c9f29a3b9" containerID="210f25b91f3fcff0b9fda607493ebc2484d5d6ec1dd03de55ce184c9b5dbd59b" exitCode=0
Oct 31 19:50:44.597725 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:44.597628    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-88eaf9-697ffc695-b9ds8" event={"ID":"f2c7c458-2fc5-4674-b080-7f0c9f29a3b9","Type":"ContainerDied","Data":"210f25b91f3fcff0b9fda607493ebc2484d5d6ec1dd03de55ce184c9b5dbd59b"}
Oct 31 19:50:44.608320 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:44.608294    4645 patch_prober.go:29] interesting pod/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body=
Oct 31 19:50:44.608869 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:44.608830    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np" podUID="c712ccf6-e167-4c11-ba0a-df297a0caf05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused"
Oct 31 19:50:44.790777 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:44.790739    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-278c04-6969d7fcb9-252tn" podUID="1bdc77fe-26fe-4f78-8242-4fed7e6f3c37" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:50:44.793074 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:44.793013    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-278c04-86645f6fbc-p5qmg" podUID="12fd3774-779d-49d5-8fe2-6804ed6d50ab" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:50:44.845751 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:44.845709    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-88eaf9-56b45b6577-m674p"
Oct 31 19:50:44.934729 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:44.934699    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-88eaf9-697ffc695-b9ds8"
Oct 31 19:50:45.000236 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:45.000113    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ae56bf5-1961-48fc-9c77-bb64e0d791cf-kube-api-access-jc5dm\" (UniqueName: \"kubernetes.io/projected/7ae56bf5-1961-48fc-9c77-bb64e0d791cf-kube-api-access-jc5dm\") pod \"7ae56bf5-1961-48fc-9c77-bb64e0d791cf\" (UID: \"7ae56bf5-1961-48fc-9c77-bb64e0d791cf\") "
Oct 31 19:50:45.000631 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:45.000607    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ae56bf5-1961-48fc-9c77-bb64e0d791cf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ae56bf5-1961-48fc-9c77-bb64e0d791cf-plugins-config-volume\") pod \"7ae56bf5-1961-48fc-9c77-bb64e0d791cf\" (UID: \"7ae56bf5-1961-48fc-9c77-bb64e0d791cf\") "
Oct 31 19:50:45.004461 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:45.002493    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ae56bf5-1961-48fc-9c77-bb64e0d791cf-plugins-config-volume" pod "7ae56bf5-1961-48fc-9c77-bb64e0d791cf" (UID: "7ae56bf5-1961-48fc-9c77-bb64e0d791cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:50:45.009286 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:45.008162    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ae56bf5-1961-48fc-9c77-bb64e0d791cf-kube-api-access-jc5dm" pod "7ae56bf5-1961-48fc-9c77-bb64e0d791cf" (UID: "7ae56bf5-1961-48fc-9c77-bb64e0d791cf"). InnerVolumeSpecName "kube-api-access-jc5dm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:50:45.020526 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:45.020505    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:45.149351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:45.149084    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2c7c458-2fc5-4674-b080-7f0c9f29a3b9-kube-api-access-bb4mv\" (UniqueName: \"kubernetes.io/projected/f2c7c458-2fc5-4674-b080-7f0c9f29a3b9-kube-api-access-bb4mv\") pod \"f2c7c458-2fc5-4674-b080-7f0c9f29a3b9\" (UID: \"f2c7c458-2fc5-4674-b080-7f0c9f29a3b9\") "
Oct 31 19:50:45.149555 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:45.149434    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2c7c458-2fc5-4674-b080-7f0c9f29a3b9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2c7c458-2fc5-4674-b080-7f0c9f29a3b9-plugins-config-volume\") pod \"f2c7c458-2fc5-4674-b080-7f0c9f29a3b9\" (UID: \"f2c7c458-2fc5-4674-b080-7f0c9f29a3b9\") "
Oct 31 19:50:45.149555 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:45.149537    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-d45f6c/test-model-d45f6c-kserve-784b7b884c-7mgtp" podUID="16133210-4877-4f71-9d0d-fe90c29cca43" containerName="main" containerID="cri-o://927fee25bffc85ab95557c76ecacea961b89d56f08061c909db500df96cbf623" gracePeriod=45
Oct 31 19:50:45.150517 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:45.150470    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jc5dm\" (UniqueName: \"kubernetes.io/projected/7ae56bf5-1961-48fc-9c77-bb64e0d791cf-kube-api-access-jc5dm\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:45.150517 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:45.150501    4645 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ae56bf5-1961-48fc-9c77-bb64e0d791cf-plugins-config-volume\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:45.150832 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:45.150813    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2c7c458-2fc5-4674-b080-7f0c9f29a3b9-plugins-config-volume" pod "f2c7c458-2fc5-4674-b080-7f0c9f29a3b9" (UID: "f2c7c458-2fc5-4674-b080-7f0c9f29a3b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:50:45.152032 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:45.152002    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2c7c458-2fc5-4674-b080-7f0c9f29a3b9-kube-api-access-bb4mv" pod "f2c7c458-2fc5-4674-b080-7f0c9f29a3b9" (UID: "f2c7c458-2fc5-4674-b080-7f0c9f29a3b9"). InnerVolumeSpecName "kube-api-access-bb4mv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:50:45.154522 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:45.154485    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-d45f6c/test-model-d45f6c-kserve-784b7b884c-7mgtp" podUID="16133210-4877-4f71-9d0d-fe90c29cca43" containerName="main" probeResult="failure" output="Get \"https://10.133.0.150:8000/ready\": EOF"
Oct 31 19:50:45.186352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:45.184969    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:45.252816 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:45.252719    4645 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2c7c458-2fc5-4674-b080-7f0c9f29a3b9-plugins-config-volume\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:45.252816 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:45.252758    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bb4mv\" (UniqueName: \"kubernetes.io/projected/f2c7c458-2fc5-4674-b080-7f0c9f29a3b9-kube-api-access-bb4mv\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:45.499541 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:45.499507    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n"]
Oct 31 19:50:45.499877 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:45.499844    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n" podUID="72702097-1cf4-4d92-85b0-3b0b43aa3c51" containerName="istio-proxy" containerID="cri-o://28cc763a8511fedc7186e850d65c1981dbea0ecb650dd4e98440a0670b05689c" gracePeriod=30
Oct 31 19:50:45.504717 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:45.504641    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v"]
Oct 31 19:50:45.505424 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:45.505399    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v" podUID="14ca45a7-b42e-45d3-91bf-3698df3303c7" containerName="istio-proxy" containerID="cri-o://7e8510c621dbcba9d39a3f46ff71e3f92a47bffdd1af518ef26d926da7579781" gracePeriod=30
Oct 31 19:50:45.640376 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:45.606457    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-a4e312-7b78fb8665-b9pn6" podUID="ee5d853e-f677-4073-979c-3fe141091e4f" containerName="payload-processing" containerID="cri-o://3816993d0f891236223070cc61ac1c7eb7116ad4d7512b347b645dc280b7d861" gracePeriod=25
Oct 31 19:50:45.640376 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:45.632806    4645 patch_prober.go:29] interesting pod/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body=
Oct 31 19:50:45.640376 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:45.632871    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np" podUID="c712ccf6-e167-4c11-ba0a-df297a0caf05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused"
Oct 31 19:50:45.718676 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:45.710221    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:45.738459 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:45.723682    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d45f6c/test-model-d45f6c-kserve-784b7b884c-7mgtp"
Oct 31 19:50:45.821352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:45.821236    4645 generic.go:356] "Generic (PLEG): container finished" podID="16133210-4877-4f71-9d0d-fe90c29cca43" containerID="927fee25bffc85ab95557c76ecacea961b89d56f08061c909db500df96cbf623" exitCode=0
Oct 31 19:50:45.821795 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:45.821773    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d45f6c/test-model-d45f6c-kserve-784b7b884c-7mgtp"
Oct 31 19:50:45.836839 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:45.836804    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d45f6c/test-model-d45f6c-kserve-784b7b884c-7mgtp" event={"ID":"16133210-4877-4f71-9d0d-fe90c29cca43","Type":"ContainerDied","Data":"927fee25bffc85ab95557c76ecacea961b89d56f08061c909db500df96cbf623"}
Oct 31 19:50:45.837243 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:45.837215    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d45f6c/test-model-d45f6c-kserve-784b7b884c-7mgtp" event={"ID":"16133210-4877-4f71-9d0d-fe90c29cca43","Type":"ContainerDied","Data":"4555c0a45ece39f69018c79ce8cde0b77ace386759f150150437933fb0d4d9f0"}
Oct 31 19:50:45.837429 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:45.837415    4645 scope.go:122] "RemoveContainer" containerID="927fee25bffc85ab95557c76ecacea961b89d56f08061c909db500df96cbf623"
Oct 31 19:50:45.864735 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:45.863977    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16133210-4877-4f71-9d0d-fe90c29cca43-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16133210-4877-4f71-9d0d-fe90c29cca43-model-cache\") pod \"16133210-4877-4f71-9d0d-fe90c29cca43\" (UID: \"16133210-4877-4f71-9d0d-fe90c29cca43\") "
Oct 31 19:50:45.867731 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:45.867688    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16133210-4877-4f71-9d0d-fe90c29cca43-model-cache" pod "16133210-4877-4f71-9d0d-fe90c29cca43" (UID: "16133210-4877-4f71-9d0d-fe90c29cca43"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:45.892822 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:45.887026    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16133210-4877-4f71-9d0d-fe90c29cca43-home\" (UniqueName: \"kubernetes.io/empty-dir/16133210-4877-4f71-9d0d-fe90c29cca43-home\") pod \"16133210-4877-4f71-9d0d-fe90c29cca43\" (UID: \"16133210-4877-4f71-9d0d-fe90c29cca43\") "
Oct 31 19:50:45.892822 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:45.887092    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16133210-4877-4f71-9d0d-fe90c29cca43-tls-certs\" (UniqueName: \"kubernetes.io/secret/16133210-4877-4f71-9d0d-fe90c29cca43-tls-certs\") pod \"16133210-4877-4f71-9d0d-fe90c29cca43\" (UID: \"16133210-4877-4f71-9d0d-fe90c29cca43\") "
Oct 31 19:50:45.892822 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:45.887238    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16133210-4877-4f71-9d0d-fe90c29cca43-dshm\" (UniqueName: \"kubernetes.io/empty-dir/16133210-4877-4f71-9d0d-fe90c29cca43-dshm\") pod \"16133210-4877-4f71-9d0d-fe90c29cca43\" (UID: \"16133210-4877-4f71-9d0d-fe90c29cca43\") "
Oct 31 19:50:45.892822 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:45.887430    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16133210-4877-4f71-9d0d-fe90c29cca43-kube-api-access-llwmq\" (UniqueName: \"kubernetes.io/projected/16133210-4877-4f71-9d0d-fe90c29cca43-kube-api-access-llwmq\") pod \"16133210-4877-4f71-9d0d-fe90c29cca43\" (UID: \"16133210-4877-4f71-9d0d-fe90c29cca43\") "
Oct 31 19:50:45.892822 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:45.887674    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16133210-4877-4f71-9d0d-fe90c29cca43-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16133210-4877-4f71-9d0d-fe90c29cca43-tmp-dir\") pod \"16133210-4877-4f71-9d0d-fe90c29cca43\" (UID: \"16133210-4877-4f71-9d0d-fe90c29cca43\") "
Oct 31 19:50:45.892822 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:45.888952    4645 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16133210-4877-4f71-9d0d-fe90c29cca43-model-cache\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:45.892822 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:45.889171    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16133210-4877-4f71-9d0d-fe90c29cca43-tmp-dir" pod "16133210-4877-4f71-9d0d-fe90c29cca43" (UID: "16133210-4877-4f71-9d0d-fe90c29cca43"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:45.892822 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:45.889291    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16133210-4877-4f71-9d0d-fe90c29cca43-home" pod "16133210-4877-4f71-9d0d-fe90c29cca43" (UID: "16133210-4877-4f71-9d0d-fe90c29cca43"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:45.904754 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:45.893138    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-88eaf9-56b45b6577-m674p" event={"ID":"7ae56bf5-1961-48fc-9c77-bb64e0d791cf","Type":"ContainerDied","Data":"712e981cf64566e226e2470cb5a11f000409187b7a57ab42ef6de8227da08895"}
Oct 31 19:50:45.904754 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:45.893479    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-88eaf9-56b45b6577-m674p"
Oct 31 19:50:45.906614 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:45.905319    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-88eaf9-697ffc695-b9ds8"
Oct 31 19:50:45.932074 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:45.923751    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-88eaf9-697ffc695-b9ds8" event={"ID":"f2c7c458-2fc5-4674-b080-7f0c9f29a3b9","Type":"ContainerDied","Data":"b1d00891dd444836d1f16feb5196eae9802128075eae313b1677b8e30e2e93f0"}
Oct 31 19:50:45.932074 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:45.926946    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16133210-4877-4f71-9d0d-fe90c29cca43-tls-certs" pod "16133210-4877-4f71-9d0d-fe90c29cca43" (UID: "16133210-4877-4f71-9d0d-fe90c29cca43"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:50:45.957887 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:45.957836    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16133210-4877-4f71-9d0d-fe90c29cca43-dshm" pod "16133210-4877-4f71-9d0d-fe90c29cca43" (UID: "16133210-4877-4f71-9d0d-fe90c29cca43"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:45.960279 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:45.960240    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16133210-4877-4f71-9d0d-fe90c29cca43-kube-api-access-llwmq" pod "16133210-4877-4f71-9d0d-fe90c29cca43" (UID: "16133210-4877-4f71-9d0d-fe90c29cca43"). InnerVolumeSpecName "kube-api-access-llwmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:50:45.990558 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:45.990525    4645 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16133210-4877-4f71-9d0d-fe90c29cca43-tmp-dir\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:45.990558 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:45.990556    4645 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16133210-4877-4f71-9d0d-fe90c29cca43-home\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:45.990804 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:45.990568    4645 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16133210-4877-4f71-9d0d-fe90c29cca43-tls-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:45.990804 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:45.990579    4645 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16133210-4877-4f71-9d0d-fe90c29cca43-dshm\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:45.990804 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:45.990590    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llwmq\" (UniqueName: \"kubernetes.io/projected/16133210-4877-4f71-9d0d-fe90c29cca43-kube-api-access-llwmq\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:46.037860 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:46.037834    4645 scope.go:122] "RemoveContainer" containerID="927fee25bffc85ab95557c76ecacea961b89d56f08061c909db500df96cbf623"
Oct 31 19:50:46.038153 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:50:46.038133    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"927fee25bffc85ab95557c76ecacea961b89d56f08061c909db500df96cbf623\": container with ID starting with 927fee25bffc85ab95557c76ecacea961b89d56f08061c909db500df96cbf623 not found: ID does not exist" containerID="927fee25bffc85ab95557c76ecacea961b89d56f08061c909db500df96cbf623"
Oct 31 19:50:46.038237 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:46.038161    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"927fee25bffc85ab95557c76ecacea961b89d56f08061c909db500df96cbf623"} err="failed to get container status \"927fee25bffc85ab95557c76ecacea961b89d56f08061c909db500df96cbf623\": rpc error: code = NotFound desc = could not find container \"927fee25bffc85ab95557c76ecacea961b89d56f08061c909db500df96cbf623\": container with ID starting with 927fee25bffc85ab95557c76ecacea961b89d56f08061c909db500df96cbf623 not found: ID does not exist"
Oct 31 19:50:46.038237 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:46.038181    4645 scope.go:122] "RemoveContainer" containerID="622bd5e7b4d3fcab0d71e79b1c5c49e26fbf7866c1213da8d8181067d101fada"
Oct 31 19:50:46.050261 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:46.050237    4645 scope.go:122] "RemoveContainer" containerID="210f25b91f3fcff0b9fda607493ebc2484d5d6ec1dd03de55ce184c9b5dbd59b"
Oct 31 19:50:46.069508 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:46.069482    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-88eaf9-697ffc695-b9ds8"]
Oct 31 19:50:46.074881 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:46.074824    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-88eaf9-697ffc695-b9ds8"]
Oct 31 19:50:46.136721 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:46.136508    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-88eaf9-56b45b6577-m674p"]
Oct 31 19:50:46.148650 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:46.148612    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-88eaf9-56b45b6577-m674p"]
Oct 31 19:50:46.185893 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:46.185860    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-d45f6c/test-model-d45f6c-kserve-784b7b884c-7mgtp"]
Oct 31 19:50:46.195196 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:46.195170    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a4e312-7b78fb8665-b9pn6"
Oct 31 19:50:46.198028 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:46.198002    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-d45f6c/test-model-d45f6c-kserve-784b7b884c-7mgtp"]
Oct 31 19:50:46.295157 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:46.295114    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee5d853e-f677-4073-979c-3fe141091e4f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee5d853e-f677-4073-979c-3fe141091e4f-plugins-config-volume\") pod \"ee5d853e-f677-4073-979c-3fe141091e4f\" (UID: \"ee5d853e-f677-4073-979c-3fe141091e4f\") "
Oct 31 19:50:46.295451 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:46.295409    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee5d853e-f677-4073-979c-3fe141091e4f-kube-api-access-5zwm6\" (UniqueName: \"kubernetes.io/projected/ee5d853e-f677-4073-979c-3fe141091e4f-kube-api-access-5zwm6\") pod \"ee5d853e-f677-4073-979c-3fe141091e4f\" (UID: \"ee5d853e-f677-4073-979c-3fe141091e4f\") "
Oct 31 19:50:46.295625 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:46.295590    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee5d853e-f677-4073-979c-3fe141091e4f-plugins-config-volume" pod "ee5d853e-f677-4073-979c-3fe141091e4f" (UID: "ee5d853e-f677-4073-979c-3fe141091e4f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:50:46.296408 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:46.296382    4645 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee5d853e-f677-4073-979c-3fe141091e4f-plugins-config-volume\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:46.298706 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:46.298674    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee5d853e-f677-4073-979c-3fe141091e4f-kube-api-access-5zwm6" pod "ee5d853e-f677-4073-979c-3fe141091e4f" (UID: "ee5d853e-f677-4073-979c-3fe141091e4f"). InnerVolumeSpecName "kube-api-access-5zwm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:50:46.427899 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:46.427488    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zwm6\" (UniqueName: \"kubernetes.io/projected/ee5d853e-f677-4073-979c-3fe141091e4f-kube-api-access-5zwm6\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:46.480973 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:46.480933    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:46.615706 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:46.615674    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np"
Oct 31 19:50:46.616120 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:46.616084    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np"
Oct 31 19:50:46.616839 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:46.616815    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np"
Oct 31 19:50:46.956753 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:46.956721    4645 generic.go:356] "Generic (PLEG): container finished" podID="ee5d853e-f677-4073-979c-3fe141091e4f" containerID="3816993d0f891236223070cc61ac1c7eb7116ad4d7512b347b645dc280b7d861" exitCode=0
Oct 31 19:50:46.957253 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:46.956765    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a4e312-7b78fb8665-b9pn6"
Oct 31 19:50:46.957253 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:46.956807    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a4e312-7b78fb8665-b9pn6" event={"ID":"ee5d853e-f677-4073-979c-3fe141091e4f","Type":"ContainerDied","Data":"3816993d0f891236223070cc61ac1c7eb7116ad4d7512b347b645dc280b7d861"}
Oct 31 19:50:46.957253 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:46.956858    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a4e312-7b78fb8665-b9pn6" event={"ID":"ee5d853e-f677-4073-979c-3fe141091e4f","Type":"ContainerDied","Data":"307366be07f7b00625b380fe243fd00f25591391d47bb79815c2c03be2cde8db"}
Oct 31 19:50:46.957253 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:46.956888    4645 scope.go:122] "RemoveContainer" containerID="3816993d0f891236223070cc61ac1c7eb7116ad4d7512b347b645dc280b7d861"
Oct 31 19:50:46.970557 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:46.970539    4645 scope.go:122] "RemoveContainer" containerID="3816993d0f891236223070cc61ac1c7eb7116ad4d7512b347b645dc280b7d861"
Oct 31 19:50:46.970805 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:50:46.970784    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3816993d0f891236223070cc61ac1c7eb7116ad4d7512b347b645dc280b7d861\": container with ID starting with 3816993d0f891236223070cc61ac1c7eb7116ad4d7512b347b645dc280b7d861 not found: ID does not exist" containerID="3816993d0f891236223070cc61ac1c7eb7116ad4d7512b347b645dc280b7d861"
Oct 31 19:50:46.970883 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:46.970812    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3816993d0f891236223070cc61ac1c7eb7116ad4d7512b347b645dc280b7d861"} err="failed to get container status \"3816993d0f891236223070cc61ac1c7eb7116ad4d7512b347b645dc280b7d861\": rpc error: code = NotFound desc = could not find container \"3816993d0f891236223070cc61ac1c7eb7116ad4d7512b347b645dc280b7d861\": container with ID starting with 3816993d0f891236223070cc61ac1c7eb7116ad4d7512b347b645dc280b7d861 not found: ID does not exist"
Oct 31 19:50:47.003947 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:47.003925    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a4e312-7b78fb8665-b9pn6"]
Oct 31 19:50:47.011234 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:47.011203    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a4e312-7b78fb8665-b9pn6"]
Oct 31 19:50:47.621495 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:47.621465    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16133210-4877-4f71-9d0d-fe90c29cca43" path="/var/lib/kubelet/pods/16133210-4877-4f71-9d0d-fe90c29cca43/volumes"
Oct 31 19:50:47.621969 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:47.621954    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ae56bf5-1961-48fc-9c77-bb64e0d791cf" path="/var/lib/kubelet/pods/7ae56bf5-1961-48fc-9c77-bb64e0d791cf/volumes"
Oct 31 19:50:47.622392 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:47.622377    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee5d853e-f677-4073-979c-3fe141091e4f" path="/var/lib/kubelet/pods/ee5d853e-f677-4073-979c-3fe141091e4f/volumes"
Oct 31 19:50:47.623086 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:47.623070    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2c7c458-2fc5-4674-b080-7f0c9f29a3b9" path="/var/lib/kubelet/pods/f2c7c458-2fc5-4674-b080-7f0c9f29a3b9/volumes"
Oct 31 19:50:47.779516 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:47.779438    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:48.598338 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:48.598286    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-1e8cf0/test-model-1e8cf0-kserve-74747d94df-4gd76"]
Oct 31 19:50:48.666001 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:48.665968    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:48.868369 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:48.868239    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:49.161710 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:49.161676    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:49.298496 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:49.298442    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb"]
Oct 31 19:50:49.298758 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:49.298730    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb" podUID="892a87a2-b166-43fd-8e00-f5a7edd01857" containerName="istio-proxy" containerID="cri-o://a22258fa609d3dc229ac9d2df1c66c4fb141c12f2286cc386573ab78a162dc3a" gracePeriod=30
Oct 31 19:50:49.488461 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:49.487385    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:49.879785 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:49.879695    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:50.194352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:50.190201    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:50.369439 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:50.368936    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:50.544845 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:50.543648    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25"]
Oct 31 19:50:50.545745 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:50.544883    4645 cpu_manager.go:414] "containerMap: removing container" podUID="4abee805-15d3-4b54-bd38-889f562f0c57" containerName="maas-api"
Oct 31 19:50:50.545745 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:50.544903    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4abee805-15d3-4b54-bd38-889f562f0c57" containerName="maas-api"
Oct 31 19:50:50.545745 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:50.544957    4645 cpu_manager.go:414] "containerMap: removing container" podUID="16133210-4877-4f71-9d0d-fe90c29cca43" containerName="main"
Oct 31 19:50:50.545745 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:50.544965    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16133210-4877-4f71-9d0d-fe90c29cca43" containerName="main"
Oct 31 19:50:50.545745 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:50.544990    4645 cpu_manager.go:414] "containerMap: removing container" podUID="f2c7c458-2fc5-4674-b080-7f0c9f29a3b9" containerName="payload-pre-processing"
Oct 31 19:50:50.545745 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:50.544998    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2c7c458-2fc5-4674-b080-7f0c9f29a3b9" containerName="payload-pre-processing"
Oct 31 19:50:50.545745 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:50.545014    4645 cpu_manager.go:414] "containerMap: removing container" podUID="7ae56bf5-1961-48fc-9c77-bb64e0d791cf" containerName="payload-processing"
Oct 31 19:50:50.545745 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:50.545022    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ae56bf5-1961-48fc-9c77-bb64e0d791cf" containerName="payload-processing"
Oct 31 19:50:50.545745 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:50.545085    4645 cpu_manager.go:414] "containerMap: removing container" podUID="ee5d853e-f677-4073-979c-3fe141091e4f" containerName="payload-processing"
Oct 31 19:50:50.545745 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:50.545093    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee5d853e-f677-4073-979c-3fe141091e4f" containerName="payload-processing"
Oct 31 19:50:50.545745 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:50.545253    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ae56bf5-1961-48fc-9c77-bb64e0d791cf" containerName="payload-processing"
Oct 31 19:50:50.545745 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:50.545307    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee5d853e-f677-4073-979c-3fe141091e4f" containerName="payload-processing"
Oct 31 19:50:50.545745 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:50.545318    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="4abee805-15d3-4b54-bd38-889f562f0c57" containerName="maas-api"
Oct 31 19:50:50.545745 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:50.545348    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="16133210-4877-4f71-9d0d-fe90c29cca43" containerName="main"
Oct 31 19:50:50.545745 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:50.545360    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2c7c458-2fc5-4674-b080-7f0c9f29a3b9" containerName="payload-pre-processing"
Oct 31 19:50:50.552193 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:50.551103    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25"
Oct 31 19:50:50.582187 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:50.582151    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-c55cf1-openshift-default-dockercfg-jg5ls\""
Oct 31 19:50:50.591725 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:50.591697    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25"]
Oct 31 19:50:50.632616 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:50.632513    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2be79e8-0166-4066-8e48-fad5a7e85684-istiod-ca-cert\") pod \"e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25\" (UID: \"f2be79e8-0166-4066-8e48-fad5a7e85684\") " pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25"
Oct 31 19:50:50.632698 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:50.632645    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpqkf\" (UniqueName: \"kubernetes.io/projected/f2be79e8-0166-4066-8e48-fad5a7e85684-kube-api-access-xpqkf\") pod \"e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25\" (UID: \"f2be79e8-0166-4066-8e48-fad5a7e85684\") " pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25"
Oct 31 19:50:50.632762 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:50.632740    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2be79e8-0166-4066-8e48-fad5a7e85684-workload-socket\") pod \"e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25\" (UID: \"f2be79e8-0166-4066-8e48-fad5a7e85684\") " pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25"
Oct 31 19:50:50.633030 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:50.633000    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2be79e8-0166-4066-8e48-fad5a7e85684-credential-socket\") pod \"e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25\" (UID: \"f2be79e8-0166-4066-8e48-fad5a7e85684\") " pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25"
Oct 31 19:50:50.633123 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:50.633063    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2be79e8-0166-4066-8e48-fad5a7e85684-istio-data\") pod \"e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25\" (UID: \"f2be79e8-0166-4066-8e48-fad5a7e85684\") " pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25"
Oct 31 19:50:50.633123 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:50.633104    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2be79e8-0166-4066-8e48-fad5a7e85684-istio-podinfo\") pod \"e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25\" (UID: \"f2be79e8-0166-4066-8e48-fad5a7e85684\") " pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25"
Oct 31 19:50:50.633255 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:50.633188    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2be79e8-0166-4066-8e48-fad5a7e85684-workload-certs\") pod \"e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25\" (UID: \"f2be79e8-0166-4066-8e48-fad5a7e85684\") " pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25"
Oct 31 19:50:50.633255 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:50.633220    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2be79e8-0166-4066-8e48-fad5a7e85684-istio-token\") pod \"e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25\" (UID: \"f2be79e8-0166-4066-8e48-fad5a7e85684\") " pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25"
Oct 31 19:50:50.633455 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:50.633317    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2be79e8-0166-4066-8e48-fad5a7e85684-istio-envoy\") pod \"e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25\" (UID: \"f2be79e8-0166-4066-8e48-fad5a7e85684\") " pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25"
Oct 31 19:50:50.660949 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:50.660915    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:50.760356 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:50.758130    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2be79e8-0166-4066-8e48-fad5a7e85684-credential-socket\") pod \"e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25\" (UID: \"f2be79e8-0166-4066-8e48-fad5a7e85684\") " pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25"
Oct 31 19:50:50.760356 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:50.758200    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2be79e8-0166-4066-8e48-fad5a7e85684-istio-data\") pod \"e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25\" (UID: \"f2be79e8-0166-4066-8e48-fad5a7e85684\") " pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25"
Oct 31 19:50:50.760356 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:50.759213    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2be79e8-0166-4066-8e48-fad5a7e85684-istio-data\") pod \"e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25\" (UID: \"f2be79e8-0166-4066-8e48-fad5a7e85684\") " pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25"
Oct 31 19:50:50.760356 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:50.759288    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2be79e8-0166-4066-8e48-fad5a7e85684-istio-podinfo\") pod \"e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25\" (UID: \"f2be79e8-0166-4066-8e48-fad5a7e85684\") " pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25"
Oct 31 19:50:50.760356 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:50.759350    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2be79e8-0166-4066-8e48-fad5a7e85684-workload-certs\") pod \"e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25\" (UID: \"f2be79e8-0166-4066-8e48-fad5a7e85684\") " pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25"
Oct 31 19:50:50.760356 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:50.759380    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2be79e8-0166-4066-8e48-fad5a7e85684-istio-token\") pod \"e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25\" (UID: \"f2be79e8-0166-4066-8e48-fad5a7e85684\") " pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25"
Oct 31 19:50:50.760356 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:50.759464    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2be79e8-0166-4066-8e48-fad5a7e85684-istio-envoy\") pod \"e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25\" (UID: \"f2be79e8-0166-4066-8e48-fad5a7e85684\") " pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25"
Oct 31 19:50:50.760356 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:50.759516    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2be79e8-0166-4066-8e48-fad5a7e85684-istiod-ca-cert\") pod \"e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25\" (UID: \"f2be79e8-0166-4066-8e48-fad5a7e85684\") " pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25"
Oct 31 19:50:50.760356 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:50.759682    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpqkf\" (UniqueName: \"kubernetes.io/projected/f2be79e8-0166-4066-8e48-fad5a7e85684-kube-api-access-xpqkf\") pod \"e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25\" (UID: \"f2be79e8-0166-4066-8e48-fad5a7e85684\") " pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25"
Oct 31 19:50:50.760356 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:50.759797    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2be79e8-0166-4066-8e48-fad5a7e85684-workload-socket\") pod \"e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25\" (UID: \"f2be79e8-0166-4066-8e48-fad5a7e85684\") " pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25"
Oct 31 19:50:50.934404 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:50.869306    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2be79e8-0166-4066-8e48-fad5a7e85684-credential-socket\") pod \"e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25\" (UID: \"f2be79e8-0166-4066-8e48-fad5a7e85684\") " pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25"
Oct 31 19:50:50.934404 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:50.931808    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2be79e8-0166-4066-8e48-fad5a7e85684-workload-certs\") pod \"e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25\" (UID: \"f2be79e8-0166-4066-8e48-fad5a7e85684\") " pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25"
Oct 31 19:50:51.054350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:50.974960    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2be79e8-0166-4066-8e48-fad5a7e85684-istiod-ca-cert\") pod \"e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25\" (UID: \"f2be79e8-0166-4066-8e48-fad5a7e85684\") " pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25"
Oct 31 19:50:51.054350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:50.975547    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2be79e8-0166-4066-8e48-fad5a7e85684-workload-socket\") pod \"e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25\" (UID: \"f2be79e8-0166-4066-8e48-fad5a7e85684\") " pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25"
Oct 31 19:50:51.054350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:50.981820    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2be79e8-0166-4066-8e48-fad5a7e85684-istio-envoy\") pod \"e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25\" (UID: \"f2be79e8-0166-4066-8e48-fad5a7e85684\") " pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25"
Oct 31 19:50:51.071976 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:51.071707    4645 generic.go:356] "Generic (PLEG): container finished" podID="14ca45a7-b42e-45d3-91bf-3698df3303c7" containerID="7e8510c621dbcba9d39a3f46ff71e3f92a47bffdd1af518ef26d926da7579781" exitCode=0
Oct 31 19:50:51.071976 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:51.071908    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v" event={"ID":"14ca45a7-b42e-45d3-91bf-3698df3303c7","Type":"ContainerDied","Data":"7e8510c621dbcba9d39a3f46ff71e3f92a47bffdd1af518ef26d926da7579781"}
Oct 31 19:50:51.074261 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:51.074228    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpqkf\" (UniqueName: \"kubernetes.io/projected/f2be79e8-0166-4066-8e48-fad5a7e85684-kube-api-access-xpqkf\") pod \"e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25\" (UID: \"f2be79e8-0166-4066-8e48-fad5a7e85684\") " pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25"
Oct 31 19:50:51.075067 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:51.074769    4645 generic.go:356] "Generic (PLEG): container finished" podID="72702097-1cf4-4d92-85b0-3b0b43aa3c51" containerID="28cc763a8511fedc7186e850d65c1981dbea0ecb650dd4e98440a0670b05689c" exitCode=0
Oct 31 19:50:51.075067 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:51.075003    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n" event={"ID":"72702097-1cf4-4d92-85b0-3b0b43aa3c51","Type":"ContainerDied","Data":"28cc763a8511fedc7186e850d65c1981dbea0ecb650dd4e98440a0670b05689c"}
Oct 31 19:50:51.081009 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:51.080525    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2be79e8-0166-4066-8e48-fad5a7e85684-istio-token\") pod \"e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25\" (UID: \"f2be79e8-0166-4066-8e48-fad5a7e85684\") " pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25"
Oct 31 19:50:51.088772 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:51.088365    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:51.089184 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:51.089027    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2be79e8-0166-4066-8e48-fad5a7e85684-istio-podinfo\") pod \"e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25\" (UID: \"f2be79e8-0166-4066-8e48-fad5a7e85684\") " pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25"
Oct 31 19:50:51.146416 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:51.146385    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v"
Oct 31 19:50:51.177880 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:51.177850    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/14ca45a7-b42e-45d3-91bf-3698df3303c7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14ca45a7-b42e-45d3-91bf-3698df3303c7-istio-podinfo\") pod \"14ca45a7-b42e-45d3-91bf-3698df3303c7\" (UID: \"14ca45a7-b42e-45d3-91bf-3698df3303c7\") "
Oct 31 19:50:51.178189 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:51.178146    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14ca45a7-b42e-45d3-91bf-3698df3303c7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14ca45a7-b42e-45d3-91bf-3698df3303c7-istio-data\") pod \"14ca45a7-b42e-45d3-91bf-3698df3303c7\" (UID: \"14ca45a7-b42e-45d3-91bf-3698df3303c7\") "
Oct 31 19:50:51.180698 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:51.180582    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14ca45a7-b42e-45d3-91bf-3698df3303c7-istio-data" pod "14ca45a7-b42e-45d3-91bf-3698df3303c7" (UID: "14ca45a7-b42e-45d3-91bf-3698df3303c7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:51.180794 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:51.180686    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14ca45a7-b42e-45d3-91bf-3698df3303c7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14ca45a7-b42e-45d3-91bf-3698df3303c7-workload-socket\") pod \"14ca45a7-b42e-45d3-91bf-3698df3303c7\" (UID: \"14ca45a7-b42e-45d3-91bf-3698df3303c7\") "
Oct 31 19:50:51.180872 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:51.180806    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14ca45a7-b42e-45d3-91bf-3698df3303c7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14ca45a7-b42e-45d3-91bf-3698df3303c7-istio-envoy\") pod \"14ca45a7-b42e-45d3-91bf-3698df3303c7\" (UID: \"14ca45a7-b42e-45d3-91bf-3698df3303c7\") "
Oct 31 19:50:51.180932 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:51.180870    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14ca45a7-b42e-45d3-91bf-3698df3303c7-workload-socket" pod "14ca45a7-b42e-45d3-91bf-3698df3303c7" (UID: "14ca45a7-b42e-45d3-91bf-3698df3303c7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:51.180932 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:51.180923    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14ca45a7-b42e-45d3-91bf-3698df3303c7-kube-api-access-8t79d\" (UniqueName: \"kubernetes.io/projected/14ca45a7-b42e-45d3-91bf-3698df3303c7-kube-api-access-8t79d\") pod \"14ca45a7-b42e-45d3-91bf-3698df3303c7\" (UID: \"14ca45a7-b42e-45d3-91bf-3698df3303c7\") "
Oct 31 19:50:51.181259 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:51.181004    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14ca45a7-b42e-45d3-91bf-3698df3303c7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14ca45a7-b42e-45d3-91bf-3698df3303c7-workload-certs\") pod \"14ca45a7-b42e-45d3-91bf-3698df3303c7\" (UID: \"14ca45a7-b42e-45d3-91bf-3698df3303c7\") "
Oct 31 19:50:51.181259 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:51.181049    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14ca45a7-b42e-45d3-91bf-3698df3303c7-istio-token\" (UniqueName: \"kubernetes.io/projected/14ca45a7-b42e-45d3-91bf-3698df3303c7-istio-token\") pod \"14ca45a7-b42e-45d3-91bf-3698df3303c7\" (UID: \"14ca45a7-b42e-45d3-91bf-3698df3303c7\") "
Oct 31 19:50:51.181259 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:51.181051    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/14ca45a7-b42e-45d3-91bf-3698df3303c7-istio-podinfo" pod "14ca45a7-b42e-45d3-91bf-3698df3303c7" (UID: "14ca45a7-b42e-45d3-91bf-3698df3303c7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:50:51.181259 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:51.181099    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14ca45a7-b42e-45d3-91bf-3698df3303c7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14ca45a7-b42e-45d3-91bf-3698df3303c7-istiod-ca-cert\") pod \"14ca45a7-b42e-45d3-91bf-3698df3303c7\" (UID: \"14ca45a7-b42e-45d3-91bf-3698df3303c7\") "
Oct 31 19:50:51.181259 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:51.181168    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14ca45a7-b42e-45d3-91bf-3698df3303c7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14ca45a7-b42e-45d3-91bf-3698df3303c7-credential-socket\") pod \"14ca45a7-b42e-45d3-91bf-3698df3303c7\" (UID: \"14ca45a7-b42e-45d3-91bf-3698df3303c7\") "
Oct 31 19:50:51.181998 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:51.181811    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14ca45a7-b42e-45d3-91bf-3698df3303c7-istiod-ca-cert" pod "14ca45a7-b42e-45d3-91bf-3698df3303c7" (UID: "14ca45a7-b42e-45d3-91bf-3698df3303c7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:50:51.181998 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:51.181981    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14ca45a7-b42e-45d3-91bf-3698df3303c7-credential-socket" pod "14ca45a7-b42e-45d3-91bf-3698df3303c7" (UID: "14ca45a7-b42e-45d3-91bf-3698df3303c7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:51.182227 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:51.182204    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14ca45a7-b42e-45d3-91bf-3698df3303c7-workload-certs" pod "14ca45a7-b42e-45d3-91bf-3698df3303c7" (UID: "14ca45a7-b42e-45d3-91bf-3698df3303c7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:51.182286 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:51.182230    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14ca45a7-b42e-45d3-91bf-3698df3303c7-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:51.182286 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:51.182254    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14ca45a7-b42e-45d3-91bf-3698df3303c7-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:51.182286 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:51.182270    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14ca45a7-b42e-45d3-91bf-3698df3303c7-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:51.182439 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:51.182284    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14ca45a7-b42e-45d3-91bf-3698df3303c7-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:51.182439 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:51.182301    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14ca45a7-b42e-45d3-91bf-3698df3303c7-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:51.184259 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:51.184237    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14ca45a7-b42e-45d3-91bf-3698df3303c7-kube-api-access-8t79d" pod "14ca45a7-b42e-45d3-91bf-3698df3303c7" (UID: "14ca45a7-b42e-45d3-91bf-3698df3303c7"). InnerVolumeSpecName "kube-api-access-8t79d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:50:51.184519 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:51.184456    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14ca45a7-b42e-45d3-91bf-3698df3303c7-istio-envoy" pod "14ca45a7-b42e-45d3-91bf-3698df3303c7" (UID: "14ca45a7-b42e-45d3-91bf-3698df3303c7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:51.184667 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:51.184652    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14ca45a7-b42e-45d3-91bf-3698df3303c7-istio-token" pod "14ca45a7-b42e-45d3-91bf-3698df3303c7" (UID: "14ca45a7-b42e-45d3-91bf-3698df3303c7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:50:51.192904 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:51.192867    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n"
Oct 31 19:50:51.269198 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:51.262040    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25"
Oct 31 19:50:51.284559 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:51.284514    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72702097-1cf4-4d92-85b0-3b0b43aa3c51-kube-api-access-rfrwk\" (UniqueName: \"kubernetes.io/projected/72702097-1cf4-4d92-85b0-3b0b43aa3c51-kube-api-access-rfrwk\") pod \"72702097-1cf4-4d92-85b0-3b0b43aa3c51\" (UID: \"72702097-1cf4-4d92-85b0-3b0b43aa3c51\") "
Oct 31 19:50:51.284737 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:51.284717    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72702097-1cf4-4d92-85b0-3b0b43aa3c51-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72702097-1cf4-4d92-85b0-3b0b43aa3c51-workload-socket\") pod \"72702097-1cf4-4d92-85b0-3b0b43aa3c51\" (UID: \"72702097-1cf4-4d92-85b0-3b0b43aa3c51\") "
Oct 31 19:50:51.284802 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:51.284762    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72702097-1cf4-4d92-85b0-3b0b43aa3c51-istio-token\" (UniqueName: \"kubernetes.io/projected/72702097-1cf4-4d92-85b0-3b0b43aa3c51-istio-token\") pod \"72702097-1cf4-4d92-85b0-3b0b43aa3c51\" (UID: \"72702097-1cf4-4d92-85b0-3b0b43aa3c51\") "
Oct 31 19:50:51.284860 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:51.284800    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72702097-1cf4-4d92-85b0-3b0b43aa3c51-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72702097-1cf4-4d92-85b0-3b0b43aa3c51-istiod-ca-cert\") pod \"72702097-1cf4-4d92-85b0-3b0b43aa3c51\" (UID: \"72702097-1cf4-4d92-85b0-3b0b43aa3c51\") "
Oct 31 19:50:51.285034 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:51.285011    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72702097-1cf4-4d92-85b0-3b0b43aa3c51-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72702097-1cf4-4d92-85b0-3b0b43aa3c51-workload-certs\") pod \"72702097-1cf4-4d92-85b0-3b0b43aa3c51\" (UID: \"72702097-1cf4-4d92-85b0-3b0b43aa3c51\") "
Oct 31 19:50:51.285221 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:51.285188    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72702097-1cf4-4d92-85b0-3b0b43aa3c51-workload-socket" pod "72702097-1cf4-4d92-85b0-3b0b43aa3c51" (UID: "72702097-1cf4-4d92-85b0-3b0b43aa3c51"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:51.285355 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:51.285232    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72702097-1cf4-4d92-85b0-3b0b43aa3c51-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72702097-1cf4-4d92-85b0-3b0b43aa3c51-istio-data\") pod \"72702097-1cf4-4d92-85b0-3b0b43aa3c51\" (UID: \"72702097-1cf4-4d92-85b0-3b0b43aa3c51\") "
Oct 31 19:50:51.285432 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:51.285390    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/72702097-1cf4-4d92-85b0-3b0b43aa3c51-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72702097-1cf4-4d92-85b0-3b0b43aa3c51-istio-podinfo\") pod \"72702097-1cf4-4d92-85b0-3b0b43aa3c51\" (UID: \"72702097-1cf4-4d92-85b0-3b0b43aa3c51\") "
Oct 31 19:50:51.285432 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:51.285426    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72702097-1cf4-4d92-85b0-3b0b43aa3c51-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72702097-1cf4-4d92-85b0-3b0b43aa3c51-credential-socket\") pod \"72702097-1cf4-4d92-85b0-3b0b43aa3c51\" (UID: \"72702097-1cf4-4d92-85b0-3b0b43aa3c51\") "
Oct 31 19:50:51.285556 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:51.285453    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72702097-1cf4-4d92-85b0-3b0b43aa3c51-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72702097-1cf4-4d92-85b0-3b0b43aa3c51-istio-envoy\") pod \"72702097-1cf4-4d92-85b0-3b0b43aa3c51\" (UID: \"72702097-1cf4-4d92-85b0-3b0b43aa3c51\") "
Oct 31 19:50:51.285556 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:51.285542    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72702097-1cf4-4d92-85b0-3b0b43aa3c51-istiod-ca-cert" pod "72702097-1cf4-4d92-85b0-3b0b43aa3c51" (UID: "72702097-1cf4-4d92-85b0-3b0b43aa3c51"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:50:51.285664 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:51.285565    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72702097-1cf4-4d92-85b0-3b0b43aa3c51-workload-certs" pod "72702097-1cf4-4d92-85b0-3b0b43aa3c51" (UID: "72702097-1cf4-4d92-85b0-3b0b43aa3c51"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:51.286679 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:51.286281    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72702097-1cf4-4d92-85b0-3b0b43aa3c51-istio-data" pod "72702097-1cf4-4d92-85b0-3b0b43aa3c51" (UID: "72702097-1cf4-4d92-85b0-3b0b43aa3c51"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:51.286679 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:51.286548    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72702097-1cf4-4d92-85b0-3b0b43aa3c51-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:51.286679 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:51.286570    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72702097-1cf4-4d92-85b0-3b0b43aa3c51-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:51.286679 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:51.286584    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14ca45a7-b42e-45d3-91bf-3698df3303c7-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:51.286679 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:51.286599    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8t79d\" (UniqueName: \"kubernetes.io/projected/14ca45a7-b42e-45d3-91bf-3698df3303c7-kube-api-access-8t79d\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:51.286679 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:51.286614    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72702097-1cf4-4d92-85b0-3b0b43aa3c51-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:51.286679 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:51.286628    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14ca45a7-b42e-45d3-91bf-3698df3303c7-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:51.286679 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:51.286644    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72702097-1cf4-4d92-85b0-3b0b43aa3c51-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:51.286679 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:51.286657    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14ca45a7-b42e-45d3-91bf-3698df3303c7-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:51.287276 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:51.287253    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72702097-1cf4-4d92-85b0-3b0b43aa3c51-credential-socket" pod "72702097-1cf4-4d92-85b0-3b0b43aa3c51" (UID: "72702097-1cf4-4d92-85b0-3b0b43aa3c51"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:51.288088 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:51.288055    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72702097-1cf4-4d92-85b0-3b0b43aa3c51-kube-api-access-rfrwk" pod "72702097-1cf4-4d92-85b0-3b0b43aa3c51" (UID: "72702097-1cf4-4d92-85b0-3b0b43aa3c51"). InnerVolumeSpecName "kube-api-access-rfrwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:50:51.288186 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:51.288090    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72702097-1cf4-4d92-85b0-3b0b43aa3c51-istio-token" pod "72702097-1cf4-4d92-85b0-3b0b43aa3c51" (UID: "72702097-1cf4-4d92-85b0-3b0b43aa3c51"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:50:51.289769 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:51.289742    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/72702097-1cf4-4d92-85b0-3b0b43aa3c51-istio-podinfo" pod "72702097-1cf4-4d92-85b0-3b0b43aa3c51" (UID: "72702097-1cf4-4d92-85b0-3b0b43aa3c51"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:50:51.292773 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:51.292749    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72702097-1cf4-4d92-85b0-3b0b43aa3c51-istio-envoy" pod "72702097-1cf4-4d92-85b0-3b0b43aa3c51" (UID: "72702097-1cf4-4d92-85b0-3b0b43aa3c51"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:51.316493 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:51.316458    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:51.388836 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:51.388791    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72702097-1cf4-4d92-85b0-3b0b43aa3c51-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:51.388836 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:51.388836    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72702097-1cf4-4d92-85b0-3b0b43aa3c51-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:51.389045 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:51.388853    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72702097-1cf4-4d92-85b0-3b0b43aa3c51-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:51.389045 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:51.388867    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72702097-1cf4-4d92-85b0-3b0b43aa3c51-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:51.389045 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:51.388881    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfrwk\" (UniqueName: \"kubernetes.io/projected/72702097-1cf4-4d92-85b0-3b0b43aa3c51-kube-api-access-rfrwk\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:51.594529 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:51.594496    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25"]
Oct 31 19:50:51.595782 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:50:51.595752    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2be79e8_0166_4066_8e48_fad5a7e85684.slice/crio-76cfcafb84b0fa5fde104f0f1659a97b9c062279b3b167dbc969742401c49f7b WatchSource:0}: Error finding container 76cfcafb84b0fa5fde104f0f1659a97b9c062279b3b167dbc969742401c49f7b: Status 404 returned error can't find the container with id 76cfcafb84b0fa5fde104f0f1659a97b9c062279b3b167dbc969742401c49f7b
Oct 31 19:50:51.598817 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:51.598775    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:50:51.598938 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:51.598855    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:50:51.598938 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:51.598891    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:50:52.086807 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:52.086769    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25" event={"ID":"f2be79e8-0166-4066-8e48-fad5a7e85684","Type":"ContainerStarted","Data":"a3220b1028914edc39c544970b4bcd369fad5406860f38d28edd234b289d22d1"}
Oct 31 19:50:52.087268 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:52.086841    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25" event={"ID":"f2be79e8-0166-4066-8e48-fad5a7e85684","Type":"ContainerStarted","Data":"76cfcafb84b0fa5fde104f0f1659a97b9c062279b3b167dbc969742401c49f7b"}
Oct 31 19:50:52.088566 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:52.088541    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v"
Oct 31 19:50:52.088722 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:52.088586    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v" event={"ID":"14ca45a7-b42e-45d3-91bf-3698df3303c7","Type":"ContainerDied","Data":"1be67ee83d8301851b4f7d9b4fe126f44da3321c58625d56ecd12b01da4db5d1"}
Oct 31 19:50:52.088722 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:52.088647    4645 scope.go:122] "RemoveContainer" containerID="7e8510c621dbcba9d39a3f46ff71e3f92a47bffdd1af518ef26d926da7579781"
Oct 31 19:50:52.090537 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:52.090509    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n" event={"ID":"72702097-1cf4-4d92-85b0-3b0b43aa3c51","Type":"ContainerDied","Data":"07371aafea10e7d397fca60fff678d5bd031494300b7dcb70a9a4fa042125347"}
Oct 31 19:50:52.090673 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:52.090603    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n"
Oct 31 19:50:52.101179 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:52.101163    4645 scope.go:122] "RemoveContainer" containerID="28cc763a8511fedc7186e850d65c1981dbea0ecb650dd4e98440a0670b05689c"
Oct 31 19:50:52.196673 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:52.194996    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25" podStartSLOduration=2.19497467 podStartE2EDuration="2.19497467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:50:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:50:52.186507247 +0000 UTC m=+1882.759741380" watchObservedRunningTime="2026-10-31 19:50:52.19497467 +0000 UTC m=+1882.768208801"
Oct 31 19:50:52.240352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:52.239903    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n"]
Oct 31 19:50:52.247983 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:52.247610    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-j699n"]
Oct 31 19:50:52.250066 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:52.250038    4645 patch_prober.go:29] interesting pod/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 19:50:52.250272 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:52.250248    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb" podUID="892a87a2-b166-43fd-8e00-f5a7edd01857" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 19:50:52.263958 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:52.263934    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25"
Oct 31 19:50:52.270315 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:52.270293    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25"
Oct 31 19:50:52.281898 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:52.281872    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v"]
Oct 31 19:50:52.293404 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:52.293375    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-278c04-openshift-default-9cb8c9ff5-6sg8v"]
Oct 31 19:50:52.439365 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:52.437705    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:52.684495 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:52.684463    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:52.886411 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:52.885474    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:53.129352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:53.125179    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25"
Oct 31 19:50:53.129352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:53.129085    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25"
Oct 31 19:50:53.632315 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:53.632283    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14ca45a7-b42e-45d3-91bf-3698df3303c7" path="/var/lib/kubelet/pods/14ca45a7-b42e-45d3-91bf-3698df3303c7/volumes"
Oct 31 19:50:53.632865 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:53.632841    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72702097-1cf4-4d92-85b0-3b0b43aa3c51" path="/var/lib/kubelet/pods/72702097-1cf4-4d92-85b0-3b0b43aa3c51/volumes"
Oct 31 19:50:53.861353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:53.861120    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:54.601600 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:54.601574    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb"
Oct 31 19:50:54.669741 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:54.665420    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/892a87a2-b166-43fd-8e00-f5a7edd01857-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/892a87a2-b166-43fd-8e00-f5a7edd01857-workload-socket\") pod \"892a87a2-b166-43fd-8e00-f5a7edd01857\" (UID: \"892a87a2-b166-43fd-8e00-f5a7edd01857\") "
Oct 31 19:50:54.669741 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:54.665504    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/892a87a2-b166-43fd-8e00-f5a7edd01857-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/892a87a2-b166-43fd-8e00-f5a7edd01857-istio-podinfo\") pod \"892a87a2-b166-43fd-8e00-f5a7edd01857\" (UID: \"892a87a2-b166-43fd-8e00-f5a7edd01857\") "
Oct 31 19:50:54.669741 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:54.665596    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/892a87a2-b166-43fd-8e00-f5a7edd01857-istio-token\" (UniqueName: \"kubernetes.io/projected/892a87a2-b166-43fd-8e00-f5a7edd01857-istio-token\") pod \"892a87a2-b166-43fd-8e00-f5a7edd01857\" (UID: \"892a87a2-b166-43fd-8e00-f5a7edd01857\") "
Oct 31 19:50:54.669741 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:54.665736    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/892a87a2-b166-43fd-8e00-f5a7edd01857-workload-socket" pod "892a87a2-b166-43fd-8e00-f5a7edd01857" (UID: "892a87a2-b166-43fd-8e00-f5a7edd01857"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:54.669741 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:54.666086    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/892a87a2-b166-43fd-8e00-f5a7edd01857-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/892a87a2-b166-43fd-8e00-f5a7edd01857-workload-certs\") pod \"892a87a2-b166-43fd-8e00-f5a7edd01857\" (UID: \"892a87a2-b166-43fd-8e00-f5a7edd01857\") "
Oct 31 19:50:54.669741 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:54.666147    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/892a87a2-b166-43fd-8e00-f5a7edd01857-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/892a87a2-b166-43fd-8e00-f5a7edd01857-istiod-ca-cert\") pod \"892a87a2-b166-43fd-8e00-f5a7edd01857\" (UID: \"892a87a2-b166-43fd-8e00-f5a7edd01857\") "
Oct 31 19:50:54.669741 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:54.666211    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/892a87a2-b166-43fd-8e00-f5a7edd01857-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/892a87a2-b166-43fd-8e00-f5a7edd01857-istio-data\") pod \"892a87a2-b166-43fd-8e00-f5a7edd01857\" (UID: \"892a87a2-b166-43fd-8e00-f5a7edd01857\") "
Oct 31 19:50:54.669741 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:54.666269    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/892a87a2-b166-43fd-8e00-f5a7edd01857-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/892a87a2-b166-43fd-8e00-f5a7edd01857-istio-envoy\") pod \"892a87a2-b166-43fd-8e00-f5a7edd01857\" (UID: \"892a87a2-b166-43fd-8e00-f5a7edd01857\") "
Oct 31 19:50:54.669741 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:54.666412    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/892a87a2-b166-43fd-8e00-f5a7edd01857-kube-api-access-qs9qd\" (UniqueName: \"kubernetes.io/projected/892a87a2-b166-43fd-8e00-f5a7edd01857-kube-api-access-qs9qd\") pod \"892a87a2-b166-43fd-8e00-f5a7edd01857\" (UID: \"892a87a2-b166-43fd-8e00-f5a7edd01857\") "
Oct 31 19:50:54.669741 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:54.666508    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/892a87a2-b166-43fd-8e00-f5a7edd01857-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/892a87a2-b166-43fd-8e00-f5a7edd01857-credential-socket\") pod \"892a87a2-b166-43fd-8e00-f5a7edd01857\" (UID: \"892a87a2-b166-43fd-8e00-f5a7edd01857\") "
Oct 31 19:50:54.669741 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:54.667693    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/892a87a2-b166-43fd-8e00-f5a7edd01857-istiod-ca-cert" pod "892a87a2-b166-43fd-8e00-f5a7edd01857" (UID: "892a87a2-b166-43fd-8e00-f5a7edd01857"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:50:54.669741 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:54.667940    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/892a87a2-b166-43fd-8e00-f5a7edd01857-workload-certs" pod "892a87a2-b166-43fd-8e00-f5a7edd01857" (UID: "892a87a2-b166-43fd-8e00-f5a7edd01857"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:54.669741 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:54.668137    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/892a87a2-b166-43fd-8e00-f5a7edd01857-credential-socket" pod "892a87a2-b166-43fd-8e00-f5a7edd01857" (UID: "892a87a2-b166-43fd-8e00-f5a7edd01857"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:54.669741 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:54.668343    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/892a87a2-b166-43fd-8e00-f5a7edd01857-istio-data" pod "892a87a2-b166-43fd-8e00-f5a7edd01857" (UID: "892a87a2-b166-43fd-8e00-f5a7edd01857"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:54.669741 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:54.669642    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/892a87a2-b166-43fd-8e00-f5a7edd01857-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:54.673772 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:54.672669    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/892a87a2-b166-43fd-8e00-f5a7edd01857-istio-envoy" pod "892a87a2-b166-43fd-8e00-f5a7edd01857" (UID: "892a87a2-b166-43fd-8e00-f5a7edd01857"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:50:54.673772 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:54.672773    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/892a87a2-b166-43fd-8e00-f5a7edd01857-istio-podinfo" pod "892a87a2-b166-43fd-8e00-f5a7edd01857" (UID: "892a87a2-b166-43fd-8e00-f5a7edd01857"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:50:54.674527 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:54.674499    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/892a87a2-b166-43fd-8e00-f5a7edd01857-istio-token" pod "892a87a2-b166-43fd-8e00-f5a7edd01857" (UID: "892a87a2-b166-43fd-8e00-f5a7edd01857"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:50:54.675413 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:54.675139    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/892a87a2-b166-43fd-8e00-f5a7edd01857-kube-api-access-qs9qd" pod "892a87a2-b166-43fd-8e00-f5a7edd01857" (UID: "892a87a2-b166-43fd-8e00-f5a7edd01857"). InnerVolumeSpecName "kube-api-access-qs9qd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:50:54.771383 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:54.771318    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qs9qd\" (UniqueName: \"kubernetes.io/projected/892a87a2-b166-43fd-8e00-f5a7edd01857-kube-api-access-qs9qd\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:54.771530 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:54.771388    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/892a87a2-b166-43fd-8e00-f5a7edd01857-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:54.771530 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:54.771403    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/892a87a2-b166-43fd-8e00-f5a7edd01857-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:54.771530 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:54.771416    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/892a87a2-b166-43fd-8e00-f5a7edd01857-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:54.771530 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:54.771429    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/892a87a2-b166-43fd-8e00-f5a7edd01857-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:54.771530 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:54.771443    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/892a87a2-b166-43fd-8e00-f5a7edd01857-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:54.771530 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:54.771456    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/892a87a2-b166-43fd-8e00-f5a7edd01857-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:54.771530 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:54.771468    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/892a87a2-b166-43fd-8e00-f5a7edd01857-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:54.786276 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:54.786226    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-278c04-6969d7fcb9-252tn" podUID="1bdc77fe-26fe-4f78-8242-4fed7e6f3c37" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:50:54.786579 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:54.786551    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-278c04-86645f6fbc-p5qmg" podUID="12fd3774-779d-49d5-8fe2-6804ed6d50ab" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:50:54.835643 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:54.835554    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:55.045349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:55.044704    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:55.196355 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:55.194505    4645 generic.go:356] "Generic (PLEG): container finished" podID="892a87a2-b166-43fd-8e00-f5a7edd01857" containerID="a22258fa609d3dc229ac9d2df1c66c4fb141c12f2286cc386573ab78a162dc3a" exitCode=0
Oct 31 19:50:55.196355 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:55.195617    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb"
Oct 31 19:50:55.229983 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:55.229393    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb" event={"ID":"892a87a2-b166-43fd-8e00-f5a7edd01857","Type":"ContainerDied","Data":"a22258fa609d3dc229ac9d2df1c66c4fb141c12f2286cc386573ab78a162dc3a"}
Oct 31 19:50:55.229983 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:55.229481    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb" event={"ID":"892a87a2-b166-43fd-8e00-f5a7edd01857","Type":"ContainerDied","Data":"4a713b1438312e76b495f0932a3c8366c39c0644022d84946b319ee86cb915bd"}
Oct 31 19:50:55.229983 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:55.229511    4645 scope.go:122] "RemoveContainer" containerID="a22258fa609d3dc229ac9d2df1c66c4fb141c12f2286cc386573ab78a162dc3a"
Oct 31 19:50:55.264351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:55.264257    4645 scope.go:122] "RemoveContainer" containerID="a22258fa609d3dc229ac9d2df1c66c4fb141c12f2286cc386573ab78a162dc3a"
Oct 31 19:50:55.264797 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:50:55.264764    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a22258fa609d3dc229ac9d2df1c66c4fb141c12f2286cc386573ab78a162dc3a\": container with ID starting with a22258fa609d3dc229ac9d2df1c66c4fb141c12f2286cc386573ab78a162dc3a not found: ID does not exist" containerID="a22258fa609d3dc229ac9d2df1c66c4fb141c12f2286cc386573ab78a162dc3a"
Oct 31 19:50:55.265083 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:55.265045    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a22258fa609d3dc229ac9d2df1c66c4fb141c12f2286cc386573ab78a162dc3a"} err="failed to get container status \"a22258fa609d3dc229ac9d2df1c66c4fb141c12f2286cc386573ab78a162dc3a\": rpc error: code = NotFound desc = could not find container \"a22258fa609d3dc229ac9d2df1c66c4fb141c12f2286cc386573ab78a162dc3a\": container with ID starting with a22258fa609d3dc229ac9d2df1c66c4fb141c12f2286cc386573ab78a162dc3a not found: ID does not exist"
Oct 31 19:50:55.292026 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:55.291998    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb"]
Oct 31 19:50:55.302369 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:55.302347    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-88eaf9-openshift-default-6c9fd5847f-jz2sb"]
Oct 31 19:50:55.620702 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:55.620674    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="892a87a2-b166-43fd-8e00-f5a7edd01857" path="/var/lib/kubelet/pods/892a87a2-b166-43fd-8e00-f5a7edd01857/volumes"
Oct 31 19:50:55.807188 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:55.807159    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-278c04-86645f6fbc-p5qmg"
Oct 31 19:50:55.905625 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:55.905600    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12fd3774-779d-49d5-8fe2-6804ed6d50ab-kube-api-access-sn98s\" (UniqueName: \"kubernetes.io/projected/12fd3774-779d-49d5-8fe2-6804ed6d50ab-kube-api-access-sn98s\") pod \"12fd3774-779d-49d5-8fe2-6804ed6d50ab\" (UID: \"12fd3774-779d-49d5-8fe2-6804ed6d50ab\") "
Oct 31 19:50:55.905749 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:55.905687    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12fd3774-779d-49d5-8fe2-6804ed6d50ab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12fd3774-779d-49d5-8fe2-6804ed6d50ab-plugins-config-volume\") pod \"12fd3774-779d-49d5-8fe2-6804ed6d50ab\" (UID: \"12fd3774-779d-49d5-8fe2-6804ed6d50ab\") "
Oct 31 19:50:55.906091 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:55.906067    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12fd3774-779d-49d5-8fe2-6804ed6d50ab-plugins-config-volume" pod "12fd3774-779d-49d5-8fe2-6804ed6d50ab" (UID: "12fd3774-779d-49d5-8fe2-6804ed6d50ab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:50:55.906237 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:55.906220    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-278c04-6969d7fcb9-252tn"
Oct 31 19:50:55.906470 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:55.906449    4645 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12fd3774-779d-49d5-8fe2-6804ed6d50ab-plugins-config-volume\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:55.907673 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:55.907652    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12fd3774-779d-49d5-8fe2-6804ed6d50ab-kube-api-access-sn98s" pod "12fd3774-779d-49d5-8fe2-6804ed6d50ab" (UID: "12fd3774-779d-49d5-8fe2-6804ed6d50ab"). InnerVolumeSpecName "kube-api-access-sn98s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:50:56.008034 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:56.008002    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bdc77fe-26fe-4f78-8242-4fed7e6f3c37-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bdc77fe-26fe-4f78-8242-4fed7e6f3c37-plugins-config-volume\") pod \"1bdc77fe-26fe-4f78-8242-4fed7e6f3c37\" (UID: \"1bdc77fe-26fe-4f78-8242-4fed7e6f3c37\") "
Oct 31 19:50:56.008223 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:56.008114    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bdc77fe-26fe-4f78-8242-4fed7e6f3c37-kube-api-access-gvgw5\" (UniqueName: \"kubernetes.io/projected/1bdc77fe-26fe-4f78-8242-4fed7e6f3c37-kube-api-access-gvgw5\") pod \"1bdc77fe-26fe-4f78-8242-4fed7e6f3c37\" (UID: \"1bdc77fe-26fe-4f78-8242-4fed7e6f3c37\") "
Oct 31 19:50:56.008529 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:56.008502    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bdc77fe-26fe-4f78-8242-4fed7e6f3c37-plugins-config-volume" pod "1bdc77fe-26fe-4f78-8242-4fed7e6f3c37" (UID: "1bdc77fe-26fe-4f78-8242-4fed7e6f3c37"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:50:56.009121 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:56.009074    4645 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bdc77fe-26fe-4f78-8242-4fed7e6f3c37-plugins-config-volume\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:56.009312 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:56.009285    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sn98s\" (UniqueName: \"kubernetes.io/projected/12fd3774-779d-49d5-8fe2-6804ed6d50ab-kube-api-access-sn98s\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:56.010631 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:56.010610    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bdc77fe-26fe-4f78-8242-4fed7e6f3c37-kube-api-access-gvgw5" pod "1bdc77fe-26fe-4f78-8242-4fed7e6f3c37" (UID: "1bdc77fe-26fe-4f78-8242-4fed7e6f3c37"). InnerVolumeSpecName "kube-api-access-gvgw5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:50:56.111197 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:56.111166    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvgw5\" (UniqueName: \"kubernetes.io/projected/1bdc77fe-26fe-4f78-8242-4fed7e6f3c37-kube-api-access-gvgw5\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:50:56.191512 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:56.191479    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:56.204432 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:56.204401    4645 generic.go:356] "Generic (PLEG): container finished" podID="12fd3774-779d-49d5-8fe2-6804ed6d50ab" containerID="9ee0280439447d0096853e72d3fea715a0b4d7e8367694f4b17124d87eb9a499" exitCode=0
Oct 31 19:50:56.204538 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:56.204456    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-278c04-86645f6fbc-p5qmg"
Oct 31 19:50:56.204538 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:56.204488    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-278c04-86645f6fbc-p5qmg" event={"ID":"12fd3774-779d-49d5-8fe2-6804ed6d50ab","Type":"ContainerDied","Data":"9ee0280439447d0096853e72d3fea715a0b4d7e8367694f4b17124d87eb9a499"}
Oct 31 19:50:56.204623 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:56.204537    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-278c04-86645f6fbc-p5qmg" event={"ID":"12fd3774-779d-49d5-8fe2-6804ed6d50ab","Type":"ContainerDied","Data":"259be388723db631f48f8b4b2767ed7b86cf11de87172aea21a3e08b7839aab5"}
Oct 31 19:50:56.204623 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:56.204562    4645 scope.go:122] "RemoveContainer" containerID="9ee0280439447d0096853e72d3fea715a0b4d7e8367694f4b17124d87eb9a499"
Oct 31 19:50:56.206235 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:56.206213    4645 generic.go:356] "Generic (PLEG): container finished" podID="1bdc77fe-26fe-4f78-8242-4fed7e6f3c37" containerID="0737d698d1ae9996feb82c0dc6627c55203dbda8a5e8425fdeabfdc2b56a1dd5" exitCode=0
Oct 31 19:50:56.206383 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:56.206277    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-278c04-6969d7fcb9-252tn"
Oct 31 19:50:56.206383 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:56.206319    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-278c04-6969d7fcb9-252tn" event={"ID":"1bdc77fe-26fe-4f78-8242-4fed7e6f3c37","Type":"ContainerDied","Data":"0737d698d1ae9996feb82c0dc6627c55203dbda8a5e8425fdeabfdc2b56a1dd5"}
Oct 31 19:50:56.206506 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:56.206417    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-278c04-6969d7fcb9-252tn" event={"ID":"1bdc77fe-26fe-4f78-8242-4fed7e6f3c37","Type":"ContainerDied","Data":"31c3d412f6e8973b6f1ad22ef0f638e2f59ff82a0b8ad65c5b769b75c43ada9d"}
Oct 31 19:50:56.216830 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:56.216815    4645 scope.go:122] "RemoveContainer" containerID="9ee0280439447d0096853e72d3fea715a0b4d7e8367694f4b17124d87eb9a499"
Oct 31 19:50:56.217124 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:50:56.217104    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ee0280439447d0096853e72d3fea715a0b4d7e8367694f4b17124d87eb9a499\": container with ID starting with 9ee0280439447d0096853e72d3fea715a0b4d7e8367694f4b17124d87eb9a499 not found: ID does not exist" containerID="9ee0280439447d0096853e72d3fea715a0b4d7e8367694f4b17124d87eb9a499"
Oct 31 19:50:56.217183 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:56.217131    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ee0280439447d0096853e72d3fea715a0b4d7e8367694f4b17124d87eb9a499"} err="failed to get container status \"9ee0280439447d0096853e72d3fea715a0b4d7e8367694f4b17124d87eb9a499\": rpc error: code = NotFound desc = could not find container \"9ee0280439447d0096853e72d3fea715a0b4d7e8367694f4b17124d87eb9a499\": container with ID starting with 9ee0280439447d0096853e72d3fea715a0b4d7e8367694f4b17124d87eb9a499 not found: ID does not exist"
Oct 31 19:50:56.217183 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:56.217147    4645 scope.go:122] "RemoveContainer" containerID="0737d698d1ae9996feb82c0dc6627c55203dbda8a5e8425fdeabfdc2b56a1dd5"
Oct 31 19:50:56.227823 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:56.227807    4645 scope.go:122] "RemoveContainer" containerID="0737d698d1ae9996feb82c0dc6627c55203dbda8a5e8425fdeabfdc2b56a1dd5"
Oct 31 19:50:56.228056 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:50:56.228038    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0737d698d1ae9996feb82c0dc6627c55203dbda8a5e8425fdeabfdc2b56a1dd5\": container with ID starting with 0737d698d1ae9996feb82c0dc6627c55203dbda8a5e8425fdeabfdc2b56a1dd5 not found: ID does not exist" containerID="0737d698d1ae9996feb82c0dc6627c55203dbda8a5e8425fdeabfdc2b56a1dd5"
Oct 31 19:50:56.228102 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:56.228064    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0737d698d1ae9996feb82c0dc6627c55203dbda8a5e8425fdeabfdc2b56a1dd5"} err="failed to get container status \"0737d698d1ae9996feb82c0dc6627c55203dbda8a5e8425fdeabfdc2b56a1dd5\": rpc error: code = NotFound desc = could not find container \"0737d698d1ae9996feb82c0dc6627c55203dbda8a5e8425fdeabfdc2b56a1dd5\": container with ID starting with 0737d698d1ae9996feb82c0dc6627c55203dbda8a5e8425fdeabfdc2b56a1dd5 not found: ID does not exist"
Oct 31 19:50:56.251787 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:56.251761    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-278c04-6969d7fcb9-252tn"]
Oct 31 19:50:56.256669 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:56.256649    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-278c04-6969d7fcb9-252tn"]
Oct 31 19:50:56.267090 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:56.267072    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-278c04-86645f6fbc-p5qmg"]
Oct 31 19:50:56.274299 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:56.274261    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-278c04-86645f6fbc-p5qmg"]
Oct 31 19:50:56.374768 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:56.374739    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:56.786607 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:56.786200    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:56.925353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:56.925303    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:57.619399 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:57.619362    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12fd3774-779d-49d5-8fe2-6804ed6d50ab" path="/var/lib/kubelet/pods/12fd3774-779d-49d5-8fe2-6804ed6d50ab/volumes"
Oct 31 19:50:57.619805 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:57.619791    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bdc77fe-26fe-4f78-8242-4fed7e6f3c37" path="/var/lib/kubelet/pods/1bdc77fe-26fe-4f78-8242-4fed7e6f3c37/volumes"
Oct 31 19:50:57.759891 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:57.759858    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:50:59.107828 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:50:59.107787    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:00.124724 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:00.124675    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:00.823337 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:00.823292    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:00.991845 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:00.991813    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:01.199775 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:01.199741    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:01.372389 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:01.368720    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:02.398453 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:02.398424    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:03.545547 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:03.545503    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:03.571738 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:03.571690    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f8c3f7-bdc4d8f7-75mhm"]
Oct 31 19:51:03.621022 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:03.620973    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-1e8cf0/test-model-1e8cf0-kserve-74747d94df-4gd76" podUID="b673f934-bc6d-4ae8-8adf-4acedf4a8df6" containerName="main" containerID="cri-o://2853a7d35e514bd396357bfe4083025dd17e252065609eab05e30dd0eaacb8b2" gracePeriod=45
Oct 31 19:51:03.718497 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:03.718464    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:03.740630 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:03.740582    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-1e8cf0/test-model-1e8cf0-kserve-74747d94df-4gd76" podUID="b673f934-bc6d-4ae8-8adf-4acedf4a8df6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.158:8000/ready\": dial tcp 10.133.0.158:8000: connect: connection refused"
Oct 31 19:51:04.186246 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:04.186219    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1e8cf0/test-model-1e8cf0-kserve-74747d94df-4gd76"
Oct 31 19:51:04.274281 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:04.274255    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b673f934-bc6d-4ae8-8adf-4acedf4a8df6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b673f934-bc6d-4ae8-8adf-4acedf4a8df6-tmp-dir\") pod \"b673f934-bc6d-4ae8-8adf-4acedf4a8df6\" (UID: \"b673f934-bc6d-4ae8-8adf-4acedf4a8df6\") "
Oct 31 19:51:04.274460 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:04.274389    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b673f934-bc6d-4ae8-8adf-4acedf4a8df6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b673f934-bc6d-4ae8-8adf-4acedf4a8df6-dshm\") pod \"b673f934-bc6d-4ae8-8adf-4acedf4a8df6\" (UID: \"b673f934-bc6d-4ae8-8adf-4acedf4a8df6\") "
Oct 31 19:51:04.274519 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:04.274492    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b673f934-bc6d-4ae8-8adf-4acedf4a8df6-home\" (UniqueName: \"kubernetes.io/empty-dir/b673f934-bc6d-4ae8-8adf-4acedf4a8df6-home\") pod \"b673f934-bc6d-4ae8-8adf-4acedf4a8df6\" (UID: \"b673f934-bc6d-4ae8-8adf-4acedf4a8df6\") "
Oct 31 19:51:04.274567 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:04.274525    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b673f934-bc6d-4ae8-8adf-4acedf4a8df6-tmp-dir" pod "b673f934-bc6d-4ae8-8adf-4acedf4a8df6" (UID: "b673f934-bc6d-4ae8-8adf-4acedf4a8df6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:51:04.274567 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:04.274534    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b673f934-bc6d-4ae8-8adf-4acedf4a8df6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b673f934-bc6d-4ae8-8adf-4acedf4a8df6-model-cache\") pod \"b673f934-bc6d-4ae8-8adf-4acedf4a8df6\" (UID: \"b673f934-bc6d-4ae8-8adf-4acedf4a8df6\") "
Oct 31 19:51:04.274683 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:04.274664    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b673f934-bc6d-4ae8-8adf-4acedf4a8df6-kube-api-access-45gc7\" (UniqueName: \"kubernetes.io/projected/b673f934-bc6d-4ae8-8adf-4acedf4a8df6-kube-api-access-45gc7\") pod \"b673f934-bc6d-4ae8-8adf-4acedf4a8df6\" (UID: \"b673f934-bc6d-4ae8-8adf-4acedf4a8df6\") "
Oct 31 19:51:04.274743 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:04.274689    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b673f934-bc6d-4ae8-8adf-4acedf4a8df6-model-cache" pod "b673f934-bc6d-4ae8-8adf-4acedf4a8df6" (UID: "b673f934-bc6d-4ae8-8adf-4acedf4a8df6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:51:04.274743 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:04.274703    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b673f934-bc6d-4ae8-8adf-4acedf4a8df6-home" pod "b673f934-bc6d-4ae8-8adf-4acedf4a8df6" (UID: "b673f934-bc6d-4ae8-8adf-4acedf4a8df6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:51:04.274743 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:04.274710    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b673f934-bc6d-4ae8-8adf-4acedf4a8df6-tls-certs\" (UniqueName: \"kubernetes.io/secret/b673f934-bc6d-4ae8-8adf-4acedf4a8df6-tls-certs\") pod \"b673f934-bc6d-4ae8-8adf-4acedf4a8df6\" (UID: \"b673f934-bc6d-4ae8-8adf-4acedf4a8df6\") "
Oct 31 19:51:04.275310 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:04.275290    4645 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b673f934-bc6d-4ae8-8adf-4acedf4a8df6-tmp-dir\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:04.275310 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:04.275311    4645 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b673f934-bc6d-4ae8-8adf-4acedf4a8df6-home\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:04.275488 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:04.275357    4645 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b673f934-bc6d-4ae8-8adf-4acedf4a8df6-model-cache\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:04.276728 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:04.276700    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b673f934-bc6d-4ae8-8adf-4acedf4a8df6-dshm" pod "b673f934-bc6d-4ae8-8adf-4acedf4a8df6" (UID: "b673f934-bc6d-4ae8-8adf-4acedf4a8df6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:51:04.276828 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:04.276756    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b673f934-bc6d-4ae8-8adf-4acedf4a8df6-tls-certs" pod "b673f934-bc6d-4ae8-8adf-4acedf4a8df6" (UID: "b673f934-bc6d-4ae8-8adf-4acedf4a8df6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:51:04.276828 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:04.276770    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b673f934-bc6d-4ae8-8adf-4acedf4a8df6-kube-api-access-45gc7" pod "b673f934-bc6d-4ae8-8adf-4acedf4a8df6" (UID: "b673f934-bc6d-4ae8-8adf-4acedf4a8df6"). InnerVolumeSpecName "kube-api-access-45gc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:51:04.357168 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:04.357140    4645 generic.go:356] "Generic (PLEG): container finished" podID="b673f934-bc6d-4ae8-8adf-4acedf4a8df6" containerID="2853a7d35e514bd396357bfe4083025dd17e252065609eab05e30dd0eaacb8b2" exitCode=0
Oct 31 19:51:04.357287 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:04.357178    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1e8cf0/test-model-1e8cf0-kserve-74747d94df-4gd76"
Oct 31 19:51:04.357287 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:04.357229    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1e8cf0/test-model-1e8cf0-kserve-74747d94df-4gd76" event={"ID":"b673f934-bc6d-4ae8-8adf-4acedf4a8df6","Type":"ContainerDied","Data":"2853a7d35e514bd396357bfe4083025dd17e252065609eab05e30dd0eaacb8b2"}
Oct 31 19:51:04.357287 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:04.357277    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1e8cf0/test-model-1e8cf0-kserve-74747d94df-4gd76" event={"ID":"b673f934-bc6d-4ae8-8adf-4acedf4a8df6","Type":"ContainerDied","Data":"5042eddc43348d7c3ccefa1c22cb551754b3c15aa3a7d9ec57f4641fa1b89d6f"}
Oct 31 19:51:04.357435 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:04.357298    4645 scope.go:122] "RemoveContainer" containerID="2853a7d35e514bd396357bfe4083025dd17e252065609eab05e30dd0eaacb8b2"
Oct 31 19:51:04.368987 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:04.368967    4645 scope.go:122] "RemoveContainer" containerID="2853a7d35e514bd396357bfe4083025dd17e252065609eab05e30dd0eaacb8b2"
Oct 31 19:51:04.369272 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:51:04.369251    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2853a7d35e514bd396357bfe4083025dd17e252065609eab05e30dd0eaacb8b2\": container with ID starting with 2853a7d35e514bd396357bfe4083025dd17e252065609eab05e30dd0eaacb8b2 not found: ID does not exist" containerID="2853a7d35e514bd396357bfe4083025dd17e252065609eab05e30dd0eaacb8b2"
Oct 31 19:51:04.369365 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:04.369279    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2853a7d35e514bd396357bfe4083025dd17e252065609eab05e30dd0eaacb8b2"} err="failed to get container status \"2853a7d35e514bd396357bfe4083025dd17e252065609eab05e30dd0eaacb8b2\": rpc error: code = NotFound desc = could not find container \"2853a7d35e514bd396357bfe4083025dd17e252065609eab05e30dd0eaacb8b2\": container with ID starting with 2853a7d35e514bd396357bfe4083025dd17e252065609eab05e30dd0eaacb8b2 not found: ID does not exist"
Oct 31 19:51:04.377128 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:04.377103    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45gc7\" (UniqueName: \"kubernetes.io/projected/b673f934-bc6d-4ae8-8adf-4acedf4a8df6-kube-api-access-45gc7\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:04.377231 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:04.377130    4645 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b673f934-bc6d-4ae8-8adf-4acedf4a8df6-tls-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:04.377231 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:04.377145    4645 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b673f934-bc6d-4ae8-8adf-4acedf4a8df6-dshm\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:04.384993 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:04.384968    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-1e8cf0/test-model-1e8cf0-kserve-74747d94df-4gd76"]
Oct 31 19:51:04.389601 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:04.389579    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-1e8cf0/test-model-1e8cf0-kserve-74747d94df-4gd76"]
Oct 31 19:51:04.612128 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:04.612096    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:04.743499 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:04.743467    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:05.619462 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:05.619427    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b673f934-bc6d-4ae8-8adf-4acedf4a8df6" path="/var/lib/kubelet/pods/b673f934-bc6d-4ae8-8adf-4acedf4a8df6/volumes"
Oct 31 19:51:05.954094 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:05.954066    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:06.032801 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:06.032765    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs"]
Oct 31 19:51:06.033306 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:06.033269    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs" podUID="40240c3e-0f73-4ac9-80c1-4c581a6ee5a3" containerName="istio-proxy" containerID="cri-o://64b7e4f082bf7683be8f37b11ef727b2e7aa2978220a43025ff6a3deee35315a" gracePeriod=30
Oct 31 19:51:06.247053 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:51:06.246944    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 19:51:07.107022 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:07.106985    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:08.044690 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:08.044650    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:08.572460 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:08.572423    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-f8c3f7-bdc4d8f7-75mhm" podUID="33f3e8b8-6af6-4f4d-9ab9-ac00d1bbb9ce" containerName="payload-processing" containerID="cri-o://cefe3c3880a7157c8a023a3ce173f0524066c6ea79126f747ebea37e1beb1823" gracePeriod=25
Oct 31 19:51:08.941256 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:08.941217    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:09.685614 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:09.685578    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:09.907352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:09.906210    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:10.161562 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:10.161532    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:10.380309 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:10.380273    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:10.676017 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:10.675982    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:11.323699 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:11.323643    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs"
Oct 31 19:51:11.397415 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:11.397386    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/40240c3e-0f73-4ac9-80c1-4c581a6ee5a3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40240c3e-0f73-4ac9-80c1-4c581a6ee5a3-istio-podinfo\") pod \"40240c3e-0f73-4ac9-80c1-4c581a6ee5a3\" (UID: \"40240c3e-0f73-4ac9-80c1-4c581a6ee5a3\") "
Oct 31 19:51:11.397611 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:11.397421    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40240c3e-0f73-4ac9-80c1-4c581a6ee5a3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40240c3e-0f73-4ac9-80c1-4c581a6ee5a3-credential-socket\") pod \"40240c3e-0f73-4ac9-80c1-4c581a6ee5a3\" (UID: \"40240c3e-0f73-4ac9-80c1-4c581a6ee5a3\") "
Oct 31 19:51:11.397611 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:11.397445    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40240c3e-0f73-4ac9-80c1-4c581a6ee5a3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40240c3e-0f73-4ac9-80c1-4c581a6ee5a3-workload-certs\") pod \"40240c3e-0f73-4ac9-80c1-4c581a6ee5a3\" (UID: \"40240c3e-0f73-4ac9-80c1-4c581a6ee5a3\") "
Oct 31 19:51:11.397611 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:11.397576    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40240c3e-0f73-4ac9-80c1-4c581a6ee5a3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40240c3e-0f73-4ac9-80c1-4c581a6ee5a3-istiod-ca-cert\") pod \"40240c3e-0f73-4ac9-80c1-4c581a6ee5a3\" (UID: \"40240c3e-0f73-4ac9-80c1-4c581a6ee5a3\") "
Oct 31 19:51:11.397850 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:11.397624    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40240c3e-0f73-4ac9-80c1-4c581a6ee5a3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40240c3e-0f73-4ac9-80c1-4c581a6ee5a3-istio-envoy\") pod \"40240c3e-0f73-4ac9-80c1-4c581a6ee5a3\" (UID: \"40240c3e-0f73-4ac9-80c1-4c581a6ee5a3\") "
Oct 31 19:51:11.397850 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:11.397706    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40240c3e-0f73-4ac9-80c1-4c581a6ee5a3-istio-token\" (UniqueName: \"kubernetes.io/projected/40240c3e-0f73-4ac9-80c1-4c581a6ee5a3-istio-token\") pod \"40240c3e-0f73-4ac9-80c1-4c581a6ee5a3\" (UID: \"40240c3e-0f73-4ac9-80c1-4c581a6ee5a3\") "
Oct 31 19:51:11.397850 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:11.397724    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40240c3e-0f73-4ac9-80c1-4c581a6ee5a3-credential-socket" pod "40240c3e-0f73-4ac9-80c1-4c581a6ee5a3" (UID: "40240c3e-0f73-4ac9-80c1-4c581a6ee5a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:51:11.397850 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:11.397783    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40240c3e-0f73-4ac9-80c1-4c581a6ee5a3-kube-api-access-hs5dd\" (UniqueName: \"kubernetes.io/projected/40240c3e-0f73-4ac9-80c1-4c581a6ee5a3-kube-api-access-hs5dd\") pod \"40240c3e-0f73-4ac9-80c1-4c581a6ee5a3\" (UID: \"40240c3e-0f73-4ac9-80c1-4c581a6ee5a3\") "
Oct 31 19:51:11.398073 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:11.397869    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40240c3e-0f73-4ac9-80c1-4c581a6ee5a3-workload-certs" pod "40240c3e-0f73-4ac9-80c1-4c581a6ee5a3" (UID: "40240c3e-0f73-4ac9-80c1-4c581a6ee5a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:51:11.398073 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:11.397945    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40240c3e-0f73-4ac9-80c1-4c581a6ee5a3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40240c3e-0f73-4ac9-80c1-4c581a6ee5a3-istio-data\") pod \"40240c3e-0f73-4ac9-80c1-4c581a6ee5a3\" (UID: \"40240c3e-0f73-4ac9-80c1-4c581a6ee5a3\") "
Oct 31 19:51:11.398073 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:11.397997    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40240c3e-0f73-4ac9-80c1-4c581a6ee5a3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40240c3e-0f73-4ac9-80c1-4c581a6ee5a3-workload-socket\") pod \"40240c3e-0f73-4ac9-80c1-4c581a6ee5a3\" (UID: \"40240c3e-0f73-4ac9-80c1-4c581a6ee5a3\") "
Oct 31 19:51:11.398286 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:11.398263    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40240c3e-0f73-4ac9-80c1-4c581a6ee5a3-istiod-ca-cert" pod "40240c3e-0f73-4ac9-80c1-4c581a6ee5a3" (UID: "40240c3e-0f73-4ac9-80c1-4c581a6ee5a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:51:11.398491 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:11.398470    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40240c3e-0f73-4ac9-80c1-4c581a6ee5a3-workload-socket" pod "40240c3e-0f73-4ac9-80c1-4c581a6ee5a3" (UID: "40240c3e-0f73-4ac9-80c1-4c581a6ee5a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:51:11.398700 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:11.398675    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40240c3e-0f73-4ac9-80c1-4c581a6ee5a3-istio-data" pod "40240c3e-0f73-4ac9-80c1-4c581a6ee5a3" (UID: "40240c3e-0f73-4ac9-80c1-4c581a6ee5a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:51:11.398964 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:11.398945    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40240c3e-0f73-4ac9-80c1-4c581a6ee5a3-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:11.399039 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:11.398971    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40240c3e-0f73-4ac9-80c1-4c581a6ee5a3-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:11.399039 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:11.398986    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40240c3e-0f73-4ac9-80c1-4c581a6ee5a3-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:11.399039 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:11.398999    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40240c3e-0f73-4ac9-80c1-4c581a6ee5a3-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:11.399039 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:11.399012    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40240c3e-0f73-4ac9-80c1-4c581a6ee5a3-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:11.400784 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:11.400743    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/40240c3e-0f73-4ac9-80c1-4c581a6ee5a3-istio-podinfo" pod "40240c3e-0f73-4ac9-80c1-4c581a6ee5a3" (UID: "40240c3e-0f73-4ac9-80c1-4c581a6ee5a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:51:11.400891 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:11.400825    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40240c3e-0f73-4ac9-80c1-4c581a6ee5a3-istio-envoy" pod "40240c3e-0f73-4ac9-80c1-4c581a6ee5a3" (UID: "40240c3e-0f73-4ac9-80c1-4c581a6ee5a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:51:11.400982 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:11.400960    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40240c3e-0f73-4ac9-80c1-4c581a6ee5a3-kube-api-access-hs5dd" pod "40240c3e-0f73-4ac9-80c1-4c581a6ee5a3" (UID: "40240c3e-0f73-4ac9-80c1-4c581a6ee5a3"). InnerVolumeSpecName "kube-api-access-hs5dd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:51:11.401046 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:11.401008    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40240c3e-0f73-4ac9-80c1-4c581a6ee5a3-istio-token" pod "40240c3e-0f73-4ac9-80c1-4c581a6ee5a3" (UID: "40240c3e-0f73-4ac9-80c1-4c581a6ee5a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:51:11.426615 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:11.426586    4645 generic.go:356] "Generic (PLEG): container finished" podID="40240c3e-0f73-4ac9-80c1-4c581a6ee5a3" containerID="64b7e4f082bf7683be8f37b11ef727b2e7aa2978220a43025ff6a3deee35315a" exitCode=0
Oct 31 19:51:11.426727 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:11.426710    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs"
Oct 31 19:51:11.426773 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:11.426746    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs" event={"ID":"40240c3e-0f73-4ac9-80c1-4c581a6ee5a3","Type":"ContainerDied","Data":"64b7e4f082bf7683be8f37b11ef727b2e7aa2978220a43025ff6a3deee35315a"}
Oct 31 19:51:11.426860 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:11.426846    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs" event={"ID":"40240c3e-0f73-4ac9-80c1-4c581a6ee5a3","Type":"ContainerDied","Data":"01831d7db0338d667ad23d1fc3e3dc504b098dca8012500f1faefda26216fb3f"}
Oct 31 19:51:11.426909 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:11.426881    4645 scope.go:122] "RemoveContainer" containerID="64b7e4f082bf7683be8f37b11ef727b2e7aa2978220a43025ff6a3deee35315a"
Oct 31 19:51:11.443995 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:11.443977    4645 scope.go:122] "RemoveContainer" containerID="64b7e4f082bf7683be8f37b11ef727b2e7aa2978220a43025ff6a3deee35315a"
Oct 31 19:51:11.444278 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:51:11.444257    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64b7e4f082bf7683be8f37b11ef727b2e7aa2978220a43025ff6a3deee35315a\": container with ID starting with 64b7e4f082bf7683be8f37b11ef727b2e7aa2978220a43025ff6a3deee35315a not found: ID does not exist" containerID="64b7e4f082bf7683be8f37b11ef727b2e7aa2978220a43025ff6a3deee35315a"
Oct 31 19:51:11.444381 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:11.444288    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64b7e4f082bf7683be8f37b11ef727b2e7aa2978220a43025ff6a3deee35315a"} err="failed to get container status \"64b7e4f082bf7683be8f37b11ef727b2e7aa2978220a43025ff6a3deee35315a\": rpc error: code = NotFound desc = could not find container \"64b7e4f082bf7683be8f37b11ef727b2e7aa2978220a43025ff6a3deee35315a\": container with ID starting with 64b7e4f082bf7683be8f37b11ef727b2e7aa2978220a43025ff6a3deee35315a not found: ID does not exist"
Oct 31 19:51:11.457817 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:11.457796    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs"]
Oct 31 19:51:11.481560 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:11.481528    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs"]
Oct 31 19:51:11.500991 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:11.500963    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40240c3e-0f73-4ac9-80c1-4c581a6ee5a3-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:11.500991 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:11.500990    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40240c3e-0f73-4ac9-80c1-4c581a6ee5a3-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:11.501173 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:11.501001    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40240c3e-0f73-4ac9-80c1-4c581a6ee5a3-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:11.501173 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:11.501016    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hs5dd\" (UniqueName: \"kubernetes.io/projected/40240c3e-0f73-4ac9-80c1-4c581a6ee5a3-kube-api-access-hs5dd\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:11.622194 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:11.622120    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40240c3e-0f73-4ac9-80c1-4c581a6ee5a3" path="/var/lib/kubelet/pods/40240c3e-0f73-4ac9-80c1-4c581a6ee5a3/volumes"
Oct 31 19:51:11.838371 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:11.838271    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:12.040408 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:12.040374    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-f8c3f7-bdc4d8f7-75mhm" podUID="33f3e8b8-6af6-4f4d-9ab9-ac00d1bbb9ce" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:51:12.080235 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:12.080203    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d45f6c-55b6bc85f9-776mg"]
Oct 31 19:51:12.185415 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:12.185373    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:12.304361 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:12.304255    4645 patch_prober.go:29] interesting pod/e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:51:12.304546 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:12.304322    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-vh6zs" podUID="40240c3e-0f73-4ac9-80c1-4c581a6ee5a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:51:12.328181 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:12.328141    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:13.056982 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:13.056950    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:14.065737 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:14.065704    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:14.844407 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:14.844367    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:15.060256 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:15.060220    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1e8cf0-56bb54564c-t5hz7"]
Oct 31 19:51:15.137853 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:15.137761    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:15.271188 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:15.271153    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:16.024397 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:16.024356    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:16.857251 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:16.857218    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:16.959036 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:16.959002    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:17.080568 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:17.080536    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d45f6c-55b6bc85f9-776mg" podUID="d7fbab67-7316-40e5-ab5a-e848cd654c50" containerName="payload-pre-processing" containerID="cri-o://e7153bfff50994acdd846d9e5fa22f406abf093b81322f4bace69fe5366fe731" gracePeriod=25
Oct 31 19:51:17.580111 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:17.580076    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:17.753764 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:17.753684    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:17.911135 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:17.911097    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:17.934745 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:17.934717    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z"]
Oct 31 19:51:17.934958 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:17.934930    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z" podUID="8eee5947-9731-4194-aa80-7fbd619e551f" containerName="istio-proxy" containerID="cri-o://ff05706b16b2a500578c373d429e7fb217fc2519b8ee19ec35c709f2d08a75f5" gracePeriod=30
Oct 31 19:51:17.939638 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:17.939614    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt"]
Oct 31 19:51:17.939789 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:17.939771    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt" podUID="63076e22-91cb-460d-8d0f-579d4da51d8c" containerName="istio-proxy" containerID="cri-o://2ba6f95de5abd698e58dbde9e45f200890221940dc84b9deda9d3b2e70c54279" gracePeriod=30
Oct 31 19:51:18.118453 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:18.116819    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:18.352981 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:18.352774    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:18.513652 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:18.513614    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:18.665109 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:18.665080    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:18.804137 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:18.804103    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:19.054678 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:19.054600    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:19.546954 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:19.546915    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:19.719579 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:19.719550    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:20.060930 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:20.060894    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-1e8cf0-56bb54564c-t5hz7" podUID="788ad5fd-ae6a-4953-a113-1357e62f2b0b" containerName="payload-processing" containerID="cri-o://930b509ed77244d1252eeab03e45e11aefdee680e2b7680444ab0d49871fd277" gracePeriod=25
Oct 31 19:51:20.080876 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:20.080848    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:20.197423 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:20.197389    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d45f6c-55b6bc85f9-776mg" podUID="d7fbab67-7316-40e5-ab5a-e848cd654c50" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:51:20.763536 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:20.763501    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:21.017223 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:21.017144    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-fd1c91/auth-test-model-fd1c91-kserve-55d79cc5b5-svsm4"]
Oct 31 19:51:21.017969 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:21.017946    4645 cpu_manager.go:414] "containerMap: removing container" podUID="14ca45a7-b42e-45d3-91bf-3698df3303c7" containerName="istio-proxy"
Oct 31 19:51:21.017969 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:21.017970    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14ca45a7-b42e-45d3-91bf-3698df3303c7" containerName="istio-proxy"
Oct 31 19:51:21.018165 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:21.017987    4645 cpu_manager.go:414] "containerMap: removing container" podUID="72702097-1cf4-4d92-85b0-3b0b43aa3c51" containerName="istio-proxy"
Oct 31 19:51:21.018165 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:21.017995    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72702097-1cf4-4d92-85b0-3b0b43aa3c51" containerName="istio-proxy"
Oct 31 19:51:21.018165 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:21.018022    4645 cpu_manager.go:414] "containerMap: removing container" podUID="1bdc77fe-26fe-4f78-8242-4fed7e6f3c37" containerName="payload-processing"
Oct 31 19:51:21.018165 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:21.018029    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bdc77fe-26fe-4f78-8242-4fed7e6f3c37" containerName="payload-processing"
Oct 31 19:51:21.018165 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:21.018044    4645 cpu_manager.go:414] "containerMap: removing container" podUID="12fd3774-779d-49d5-8fe2-6804ed6d50ab" containerName="payload-pre-processing"
Oct 31 19:51:21.018165 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:21.018051    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12fd3774-779d-49d5-8fe2-6804ed6d50ab" containerName="payload-pre-processing"
Oct 31 19:51:21.018165 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:21.018068    4645 cpu_manager.go:414] "containerMap: removing container" podUID="b673f934-bc6d-4ae8-8adf-4acedf4a8df6" containerName="main"
Oct 31 19:51:21.018165 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:21.018074    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b673f934-bc6d-4ae8-8adf-4acedf4a8df6" containerName="main"
Oct 31 19:51:21.018165 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:21.018091    4645 cpu_manager.go:414] "containerMap: removing container" podUID="40240c3e-0f73-4ac9-80c1-4c581a6ee5a3" containerName="istio-proxy"
Oct 31 19:51:21.018165 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:21.018097    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40240c3e-0f73-4ac9-80c1-4c581a6ee5a3" containerName="istio-proxy"
Oct 31 19:51:21.018165 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:21.018139    4645 cpu_manager.go:414] "containerMap: removing container" podUID="892a87a2-b166-43fd-8e00-f5a7edd01857" containerName="istio-proxy"
Oct 31 19:51:21.018165 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:21.018150    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="892a87a2-b166-43fd-8e00-f5a7edd01857" containerName="istio-proxy"
Oct 31 19:51:21.018807 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:21.018351    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="892a87a2-b166-43fd-8e00-f5a7edd01857" containerName="istio-proxy"
Oct 31 19:51:21.018807 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:21.018370    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="14ca45a7-b42e-45d3-91bf-3698df3303c7" containerName="istio-proxy"
Oct 31 19:51:21.018807 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:21.018380    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="72702097-1cf4-4d92-85b0-3b0b43aa3c51" containerName="istio-proxy"
Oct 31 19:51:21.018807 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:21.018393    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bdc77fe-26fe-4f78-8242-4fed7e6f3c37" containerName="payload-processing"
Oct 31 19:51:21.018807 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:21.018406    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="12fd3774-779d-49d5-8fe2-6804ed6d50ab" containerName="payload-pre-processing"
Oct 31 19:51:21.018807 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:21.018416    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="b673f934-bc6d-4ae8-8adf-4acedf4a8df6" containerName="main"
Oct 31 19:51:21.018807 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:21.018427    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="40240c3e-0f73-4ac9-80c1-4c581a6ee5a3" containerName="istio-proxy"
Oct 31 19:51:21.023617 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:21.023596    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-fd1c91/auth-test-model-fd1c91-kserve-55d79cc5b5-svsm4"
Oct 31 19:51:21.035360 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:21.035314    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-fd1c91\"/\"auth-test-model-fd1c91-kserve-self-signed-certs\""
Oct 31 19:51:21.035512 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:21.035388    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-fd1c91\"/\"default-dockercfg-d2g45\""
Oct 31 19:51:21.035512 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:21.035485    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-fd1c91\"/\"kube-root-ca.crt\""
Oct 31 19:51:21.036019 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:21.036001    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-fd1c91\"/\"openshift-service-ca.crt\""
Oct 31 19:51:21.038317 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:21.038294    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-fd1c91/auth-test-model-fd1c91-kserve-55d79cc5b5-svsm4"]
Oct 31 19:51:21.089516 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:21.089484    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74a345a2-e822-4ba3-8fd9-0e8a41df4ffb-dshm\") pod \"auth-test-model-fd1c91-kserve-55d79cc5b5-svsm4\" (UID: \"74a345a2-e822-4ba3-8fd9-0e8a41df4ffb\") " pod="ai-tenant-e2e-shared-a-w0-fd1c91/auth-test-model-fd1c91-kserve-55d79cc5b5-svsm4"
Oct 31 19:51:21.089930 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:21.089672    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74a345a2-e822-4ba3-8fd9-0e8a41df4ffb-home\") pod \"auth-test-model-fd1c91-kserve-55d79cc5b5-svsm4\" (UID: \"74a345a2-e822-4ba3-8fd9-0e8a41df4ffb\") " pod="ai-tenant-e2e-shared-a-w0-fd1c91/auth-test-model-fd1c91-kserve-55d79cc5b5-svsm4"
Oct 31 19:51:21.089930 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:21.089731    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74a345a2-e822-4ba3-8fd9-0e8a41df4ffb-tls-certs\") pod \"auth-test-model-fd1c91-kserve-55d79cc5b5-svsm4\" (UID: \"74a345a2-e822-4ba3-8fd9-0e8a41df4ffb\") " pod="ai-tenant-e2e-shared-a-w0-fd1c91/auth-test-model-fd1c91-kserve-55d79cc5b5-svsm4"
Oct 31 19:51:21.089930 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:21.089840    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkg86\" (UniqueName: \"kubernetes.io/projected/74a345a2-e822-4ba3-8fd9-0e8a41df4ffb-kube-api-access-zkg86\") pod \"auth-test-model-fd1c91-kserve-55d79cc5b5-svsm4\" (UID: \"74a345a2-e822-4ba3-8fd9-0e8a41df4ffb\") " pod="ai-tenant-e2e-shared-a-w0-fd1c91/auth-test-model-fd1c91-kserve-55d79cc5b5-svsm4"
Oct 31 19:51:21.089930 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:21.089880    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74a345a2-e822-4ba3-8fd9-0e8a41df4ffb-tmp-dir\") pod \"auth-test-model-fd1c91-kserve-55d79cc5b5-svsm4\" (UID: \"74a345a2-e822-4ba3-8fd9-0e8a41df4ffb\") " pod="ai-tenant-e2e-shared-a-w0-fd1c91/auth-test-model-fd1c91-kserve-55d79cc5b5-svsm4"
Oct 31 19:51:21.089930 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:21.089901    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74a345a2-e822-4ba3-8fd9-0e8a41df4ffb-model-cache\") pod \"auth-test-model-fd1c91-kserve-55d79cc5b5-svsm4\" (UID: \"74a345a2-e822-4ba3-8fd9-0e8a41df4ffb\") " pod="ai-tenant-e2e-shared-a-w0-fd1c91/auth-test-model-fd1c91-kserve-55d79cc5b5-svsm4"
Oct 31 19:51:21.094429 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:21.094403    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:21.191552 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:21.191521    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74a345a2-e822-4ba3-8fd9-0e8a41df4ffb-model-cache\") pod \"auth-test-model-fd1c91-kserve-55d79cc5b5-svsm4\" (UID: \"74a345a2-e822-4ba3-8fd9-0e8a41df4ffb\") " pod="ai-tenant-e2e-shared-a-w0-fd1c91/auth-test-model-fd1c91-kserve-55d79cc5b5-svsm4"
Oct 31 19:51:21.191679 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:21.191603    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74a345a2-e822-4ba3-8fd9-0e8a41df4ffb-dshm\") pod \"auth-test-model-fd1c91-kserve-55d79cc5b5-svsm4\" (UID: \"74a345a2-e822-4ba3-8fd9-0e8a41df4ffb\") " pod="ai-tenant-e2e-shared-a-w0-fd1c91/auth-test-model-fd1c91-kserve-55d79cc5b5-svsm4"
Oct 31 19:51:21.191768 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:21.191752    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74a345a2-e822-4ba3-8fd9-0e8a41df4ffb-home\") pod \"auth-test-model-fd1c91-kserve-55d79cc5b5-svsm4\" (UID: \"74a345a2-e822-4ba3-8fd9-0e8a41df4ffb\") " pod="ai-tenant-e2e-shared-a-w0-fd1c91/auth-test-model-fd1c91-kserve-55d79cc5b5-svsm4"
Oct 31 19:51:21.191846 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:21.191831    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74a345a2-e822-4ba3-8fd9-0e8a41df4ffb-tls-certs\") pod \"auth-test-model-fd1c91-kserve-55d79cc5b5-svsm4\" (UID: \"74a345a2-e822-4ba3-8fd9-0e8a41df4ffb\") " pod="ai-tenant-e2e-shared-a-w0-fd1c91/auth-test-model-fd1c91-kserve-55d79cc5b5-svsm4"
Oct 31 19:51:21.191913 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:21.191873    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74a345a2-e822-4ba3-8fd9-0e8a41df4ffb-model-cache\") pod \"auth-test-model-fd1c91-kserve-55d79cc5b5-svsm4\" (UID: \"74a345a2-e822-4ba3-8fd9-0e8a41df4ffb\") " pod="ai-tenant-e2e-shared-a-w0-fd1c91/auth-test-model-fd1c91-kserve-55d79cc5b5-svsm4"
Oct 31 19:51:21.191986 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:21.191965    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkg86\" (UniqueName: \"kubernetes.io/projected/74a345a2-e822-4ba3-8fd9-0e8a41df4ffb-kube-api-access-zkg86\") pod \"auth-test-model-fd1c91-kserve-55d79cc5b5-svsm4\" (UID: \"74a345a2-e822-4ba3-8fd9-0e8a41df4ffb\") " pod="ai-tenant-e2e-shared-a-w0-fd1c91/auth-test-model-fd1c91-kserve-55d79cc5b5-svsm4"
Oct 31 19:51:21.192055 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:21.192038    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74a345a2-e822-4ba3-8fd9-0e8a41df4ffb-tmp-dir\") pod \"auth-test-model-fd1c91-kserve-55d79cc5b5-svsm4\" (UID: \"74a345a2-e822-4ba3-8fd9-0e8a41df4ffb\") " pod="ai-tenant-e2e-shared-a-w0-fd1c91/auth-test-model-fd1c91-kserve-55d79cc5b5-svsm4"
Oct 31 19:51:21.192115 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:21.192056    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74a345a2-e822-4ba3-8fd9-0e8a41df4ffb-home\") pod \"auth-test-model-fd1c91-kserve-55d79cc5b5-svsm4\" (UID: \"74a345a2-e822-4ba3-8fd9-0e8a41df4ffb\") " pod="ai-tenant-e2e-shared-a-w0-fd1c91/auth-test-model-fd1c91-kserve-55d79cc5b5-svsm4"
Oct 31 19:51:21.192319 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:21.192302    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74a345a2-e822-4ba3-8fd9-0e8a41df4ffb-tmp-dir\") pod \"auth-test-model-fd1c91-kserve-55d79cc5b5-svsm4\" (UID: \"74a345a2-e822-4ba3-8fd9-0e8a41df4ffb\") " pod="ai-tenant-e2e-shared-a-w0-fd1c91/auth-test-model-fd1c91-kserve-55d79cc5b5-svsm4"
Oct 31 19:51:21.193719 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:21.193699    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74a345a2-e822-4ba3-8fd9-0e8a41df4ffb-dshm\") pod \"auth-test-model-fd1c91-kserve-55d79cc5b5-svsm4\" (UID: \"74a345a2-e822-4ba3-8fd9-0e8a41df4ffb\") " pod="ai-tenant-e2e-shared-a-w0-fd1c91/auth-test-model-fd1c91-kserve-55d79cc5b5-svsm4"
Oct 31 19:51:21.194230 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:21.194213    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74a345a2-e822-4ba3-8fd9-0e8a41df4ffb-tls-certs\") pod \"auth-test-model-fd1c91-kserve-55d79cc5b5-svsm4\" (UID: \"74a345a2-e822-4ba3-8fd9-0e8a41df4ffb\") " pod="ai-tenant-e2e-shared-a-w0-fd1c91/auth-test-model-fd1c91-kserve-55d79cc5b5-svsm4"
Oct 31 19:51:21.204082 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:21.204060    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkg86\" (UniqueName: \"kubernetes.io/projected/74a345a2-e822-4ba3-8fd9-0e8a41df4ffb-kube-api-access-zkg86\") pod \"auth-test-model-fd1c91-kserve-55d79cc5b5-svsm4\" (UID: \"74a345a2-e822-4ba3-8fd9-0e8a41df4ffb\") " pod="ai-tenant-e2e-shared-a-w0-fd1c91/auth-test-model-fd1c91-kserve-55d79cc5b5-svsm4"
Oct 31 19:51:21.343536 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:21.343134    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-fd1c91/auth-test-model-fd1c91-kserve-55d79cc5b5-svsm4"
Oct 31 19:51:21.667779 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:51:21.667746    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod74a345a2_e822_4ba3_8fd9_0e8a41df4ffb.slice/crio-298d829391f260a4221c2e879f887d662e0893fa41751e23cfa58b6214d71ae3 WatchSource:0}: Error finding container 298d829391f260a4221c2e879f887d662e0893fa41751e23cfa58b6214d71ae3: Status 404 returned error can't find the container with id 298d829391f260a4221c2e879f887d662e0893fa41751e23cfa58b6214d71ae3
Oct 31 19:51:21.667941 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:21.667904    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-fd1c91/auth-test-model-fd1c91-kserve-55d79cc5b5-svsm4"]
Oct 31 19:51:21.689674 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:21.689645    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-fd1c91/auth-test-model-fd1c91-kserve-55d79cc5b5-svsm4" event={"ID":"74a345a2-e822-4ba3-8fd9-0e8a41df4ffb","Type":"ContainerStarted","Data":"298d829391f260a4221c2e879f887d662e0893fa41751e23cfa58b6214d71ae3"}
Oct 31 19:51:21.718950 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:21.718913    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:21.839036 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:21.839002    4645 patch_prober.go:29] interesting pod/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 19:51:21.839218 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:21.839069    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z" podUID="8eee5947-9731-4194-aa80-7fbd619e551f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 19:51:22.038990 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:22.038901    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-f8c3f7-bdc4d8f7-75mhm" podUID="33f3e8b8-6af6-4f4d-9ab9-ac00d1bbb9ce" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:51:22.142634 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:22.142604    4645 patch_prober.go:29] interesting pod/e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 19:51:22.143027 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:22.142658    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt" podUID="63076e22-91cb-460d-8d0f-579d4da51d8c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 19:51:22.640709 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:22.640667    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:22.748562 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:22.748522    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-fd1c91/auth-test-model-fd1c91-kserve-55d79cc5b5-svsm4" event={"ID":"74a345a2-e822-4ba3-8fd9-0e8a41df4ffb","Type":"ContainerStarted","Data":"ac4d000a63a29a7672c418ca586f440a375202969940eba9ae4d3cf6c8ba6bb3"}
Oct 31 19:51:22.772230 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:22.772183    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-fd1c91/auth-test-model-fd1c91-kserve-55d79cc5b5-svsm4" podStartSLOduration=2.7721682850000002 podStartE2EDuration="2.772168285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:51:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:51:22.766198286 +0000 UTC m=+1913.339432414" watchObservedRunningTime="2026-10-31 19:51:22.772168285 +0000 UTC m=+1913.345402410"
Oct 31 19:51:23.262137 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:23.262111    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z"
Oct 31 19:51:23.321887 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:23.321850    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt"
Oct 31 19:51:23.341994 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:23.341964    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8eee5947-9731-4194-aa80-7fbd619e551f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8eee5947-9731-4194-aa80-7fbd619e551f-istio-data\") pod \"8eee5947-9731-4194-aa80-7fbd619e551f\" (UID: \"8eee5947-9731-4194-aa80-7fbd619e551f\") "
Oct 31 19:51:23.341994 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:23.341998    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8eee5947-9731-4194-aa80-7fbd619e551f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8eee5947-9731-4194-aa80-7fbd619e551f-istio-podinfo\") pod \"8eee5947-9731-4194-aa80-7fbd619e551f\" (UID: \"8eee5947-9731-4194-aa80-7fbd619e551f\") "
Oct 31 19:51:23.342226 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:23.342185    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8eee5947-9731-4194-aa80-7fbd619e551f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eee5947-9731-4194-aa80-7fbd619e551f-credential-socket\") pod \"8eee5947-9731-4194-aa80-7fbd619e551f\" (UID: \"8eee5947-9731-4194-aa80-7fbd619e551f\") "
Oct 31 19:51:23.342303 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:23.342244    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8eee5947-9731-4194-aa80-7fbd619e551f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8eee5947-9731-4194-aa80-7fbd619e551f-istio-envoy\") pod \"8eee5947-9731-4194-aa80-7fbd619e551f\" (UID: \"8eee5947-9731-4194-aa80-7fbd619e551f\") "
Oct 31 19:51:23.342303 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:23.342282    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eee5947-9731-4194-aa80-7fbd619e551f-istio-data" pod "8eee5947-9731-4194-aa80-7fbd619e551f" (UID: "8eee5947-9731-4194-aa80-7fbd619e551f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:51:23.342303 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:23.342300    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8eee5947-9731-4194-aa80-7fbd619e551f-istio-token\" (UniqueName: \"kubernetes.io/projected/8eee5947-9731-4194-aa80-7fbd619e551f-istio-token\") pod \"8eee5947-9731-4194-aa80-7fbd619e551f\" (UID: \"8eee5947-9731-4194-aa80-7fbd619e551f\") "
Oct 31 19:51:23.342520 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:23.342342    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8eee5947-9731-4194-aa80-7fbd619e551f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eee5947-9731-4194-aa80-7fbd619e551f-workload-socket\") pod \"8eee5947-9731-4194-aa80-7fbd619e551f\" (UID: \"8eee5947-9731-4194-aa80-7fbd619e551f\") "
Oct 31 19:51:23.342520 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:23.342373    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8eee5947-9731-4194-aa80-7fbd619e551f-kube-api-access-f9pw9\" (UniqueName: \"kubernetes.io/projected/8eee5947-9731-4194-aa80-7fbd619e551f-kube-api-access-f9pw9\") pod \"8eee5947-9731-4194-aa80-7fbd619e551f\" (UID: \"8eee5947-9731-4194-aa80-7fbd619e551f\") "
Oct 31 19:51:23.342520 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:23.342448    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8eee5947-9731-4194-aa80-7fbd619e551f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8eee5947-9731-4194-aa80-7fbd619e551f-istiod-ca-cert\") pod \"8eee5947-9731-4194-aa80-7fbd619e551f\" (UID: \"8eee5947-9731-4194-aa80-7fbd619e551f\") "
Oct 31 19:51:23.342520 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:23.342473    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8eee5947-9731-4194-aa80-7fbd619e551f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8eee5947-9731-4194-aa80-7fbd619e551f-workload-certs\") pod \"8eee5947-9731-4194-aa80-7fbd619e551f\" (UID: \"8eee5947-9731-4194-aa80-7fbd619e551f\") "
Oct 31 19:51:23.342740 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:23.342532    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eee5947-9731-4194-aa80-7fbd619e551f-credential-socket" pod "8eee5947-9731-4194-aa80-7fbd619e551f" (UID: "8eee5947-9731-4194-aa80-7fbd619e551f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:51:23.343303 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:23.343276    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eee5947-9731-4194-aa80-7fbd619e551f-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:23.343488 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:23.343471    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8eee5947-9731-4194-aa80-7fbd619e551f-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:23.343611 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:23.343303    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eee5947-9731-4194-aa80-7fbd619e551f-workload-certs" pod "8eee5947-9731-4194-aa80-7fbd619e551f" (UID: "8eee5947-9731-4194-aa80-7fbd619e551f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:51:23.343732 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:23.343432    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8eee5947-9731-4194-aa80-7fbd619e551f-istiod-ca-cert" pod "8eee5947-9731-4194-aa80-7fbd619e551f" (UID: "8eee5947-9731-4194-aa80-7fbd619e551f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:51:23.343914 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:23.343809    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eee5947-9731-4194-aa80-7fbd619e551f-workload-socket" pod "8eee5947-9731-4194-aa80-7fbd619e551f" (UID: "8eee5947-9731-4194-aa80-7fbd619e551f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:51:23.344065 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:23.343862    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-fd1c91/auth-test-model-fd1c91-kserve-55d79cc5b5-svsm4"
Oct 31 19:51:23.344796 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:23.344751    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8eee5947-9731-4194-aa80-7fbd619e551f-istio-podinfo" pod "8eee5947-9731-4194-aa80-7fbd619e551f" (UID: "8eee5947-9731-4194-aa80-7fbd619e551f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:51:23.345009 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:23.344977    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8eee5947-9731-4194-aa80-7fbd619e551f-istio-envoy" pod "8eee5947-9731-4194-aa80-7fbd619e551f" (UID: "8eee5947-9731-4194-aa80-7fbd619e551f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:51:23.345775 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:23.345748    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8eee5947-9731-4194-aa80-7fbd619e551f-istio-token" pod "8eee5947-9731-4194-aa80-7fbd619e551f" (UID: "8eee5947-9731-4194-aa80-7fbd619e551f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:51:23.346245 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:23.346218    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8eee5947-9731-4194-aa80-7fbd619e551f-kube-api-access-f9pw9" pod "8eee5947-9731-4194-aa80-7fbd619e551f" (UID: "8eee5947-9731-4194-aa80-7fbd619e551f"). InnerVolumeSpecName "kube-api-access-f9pw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:51:23.444298 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:23.444263    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63076e22-91cb-460d-8d0f-579d4da51d8c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63076e22-91cb-460d-8d0f-579d4da51d8c-workload-socket\") pod \"63076e22-91cb-460d-8d0f-579d4da51d8c\" (UID: \"63076e22-91cb-460d-8d0f-579d4da51d8c\") "
Oct 31 19:51:23.444519 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:23.444315    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63076e22-91cb-460d-8d0f-579d4da51d8c-istio-token\" (UniqueName: \"kubernetes.io/projected/63076e22-91cb-460d-8d0f-579d4da51d8c-istio-token\") pod \"63076e22-91cb-460d-8d0f-579d4da51d8c\" (UID: \"63076e22-91cb-460d-8d0f-579d4da51d8c\") "
Oct 31 19:51:23.444519 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:23.444379    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63076e22-91cb-460d-8d0f-579d4da51d8c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63076e22-91cb-460d-8d0f-579d4da51d8c-istio-data\") pod \"63076e22-91cb-460d-8d0f-579d4da51d8c\" (UID: \"63076e22-91cb-460d-8d0f-579d4da51d8c\") "
Oct 31 19:51:23.444519 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:23.444430    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63076e22-91cb-460d-8d0f-579d4da51d8c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63076e22-91cb-460d-8d0f-579d4da51d8c-istio-envoy\") pod \"63076e22-91cb-460d-8d0f-579d4da51d8c\" (UID: \"63076e22-91cb-460d-8d0f-579d4da51d8c\") "
Oct 31 19:51:23.444664 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:23.444588    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63076e22-91cb-460d-8d0f-579d4da51d8c-workload-socket" pod "63076e22-91cb-460d-8d0f-579d4da51d8c" (UID: "63076e22-91cb-460d-8d0f-579d4da51d8c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:51:23.444664 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:23.444596    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/63076e22-91cb-460d-8d0f-579d4da51d8c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63076e22-91cb-460d-8d0f-579d4da51d8c-istio-podinfo\") pod \"63076e22-91cb-460d-8d0f-579d4da51d8c\" (UID: \"63076e22-91cb-460d-8d0f-579d4da51d8c\") "
Oct 31 19:51:23.444664 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:23.444637    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63076e22-91cb-460d-8d0f-579d4da51d8c-istio-data" pod "63076e22-91cb-460d-8d0f-579d4da51d8c" (UID: "63076e22-91cb-460d-8d0f-579d4da51d8c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:51:23.444770 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:23.444752    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63076e22-91cb-460d-8d0f-579d4da51d8c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63076e22-91cb-460d-8d0f-579d4da51d8c-istiod-ca-cert\") pod \"63076e22-91cb-460d-8d0f-579d4da51d8c\" (UID: \"63076e22-91cb-460d-8d0f-579d4da51d8c\") "
Oct 31 19:51:23.444806 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:23.444792    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63076e22-91cb-460d-8d0f-579d4da51d8c-kube-api-access-bl9l8\" (UniqueName: \"kubernetes.io/projected/63076e22-91cb-460d-8d0f-579d4da51d8c-kube-api-access-bl9l8\") pod \"63076e22-91cb-460d-8d0f-579d4da51d8c\" (UID: \"63076e22-91cb-460d-8d0f-579d4da51d8c\") "
Oct 31 19:51:23.444867 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:23.444852    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63076e22-91cb-460d-8d0f-579d4da51d8c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63076e22-91cb-460d-8d0f-579d4da51d8c-workload-certs\") pod \"63076e22-91cb-460d-8d0f-579d4da51d8c\" (UID: \"63076e22-91cb-460d-8d0f-579d4da51d8c\") "
Oct 31 19:51:23.444998 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:23.444979    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63076e22-91cb-460d-8d0f-579d4da51d8c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63076e22-91cb-460d-8d0f-579d4da51d8c-credential-socket\") pod \"63076e22-91cb-460d-8d0f-579d4da51d8c\" (UID: \"63076e22-91cb-460d-8d0f-579d4da51d8c\") "
Oct 31 19:51:23.445389 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:23.445236    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63076e22-91cb-460d-8d0f-579d4da51d8c-istiod-ca-cert" pod "63076e22-91cb-460d-8d0f-579d4da51d8c" (UID: "63076e22-91cb-460d-8d0f-579d4da51d8c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:51:23.445389 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:23.445304    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63076e22-91cb-460d-8d0f-579d4da51d8c-workload-certs" pod "63076e22-91cb-460d-8d0f-579d4da51d8c" (UID: "63076e22-91cb-460d-8d0f-579d4da51d8c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:51:23.445605 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:23.445537    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63076e22-91cb-460d-8d0f-579d4da51d8c-credential-socket" pod "63076e22-91cb-460d-8d0f-579d4da51d8c" (UID: "63076e22-91cb-460d-8d0f-579d4da51d8c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:51:23.445996 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:23.445972    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63076e22-91cb-460d-8d0f-579d4da51d8c-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:23.446115 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:23.445999    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8eee5947-9731-4194-aa80-7fbd619e551f-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:23.446115 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:23.446013    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63076e22-91cb-460d-8d0f-579d4da51d8c-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:23.446115 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:23.446027    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8eee5947-9731-4194-aa80-7fbd619e551f-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:23.446115 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:23.446042    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8eee5947-9731-4194-aa80-7fbd619e551f-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:23.446115 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:23.446056    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9pw9\" (UniqueName: \"kubernetes.io/projected/8eee5947-9731-4194-aa80-7fbd619e551f-kube-api-access-f9pw9\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:23.446115 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:23.446070    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63076e22-91cb-460d-8d0f-579d4da51d8c-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:23.446115 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:23.446084    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8eee5947-9731-4194-aa80-7fbd619e551f-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:23.446115 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:23.446099    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8eee5947-9731-4194-aa80-7fbd619e551f-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:23.446115 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:23.446111    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63076e22-91cb-460d-8d0f-579d4da51d8c-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:23.446673 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:23.446125    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63076e22-91cb-460d-8d0f-579d4da51d8c-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:23.446673 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:23.446138    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8eee5947-9731-4194-aa80-7fbd619e551f-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:23.446972 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:23.446952    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/63076e22-91cb-460d-8d0f-579d4da51d8c-istio-podinfo" pod "63076e22-91cb-460d-8d0f-579d4da51d8c" (UID: "63076e22-91cb-460d-8d0f-579d4da51d8c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:51:23.447111 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:23.447095    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63076e22-91cb-460d-8d0f-579d4da51d8c-istio-token" pod "63076e22-91cb-460d-8d0f-579d4da51d8c" (UID: "63076e22-91cb-460d-8d0f-579d4da51d8c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:51:23.447251 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:23.447231    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63076e22-91cb-460d-8d0f-579d4da51d8c-istio-envoy" pod "63076e22-91cb-460d-8d0f-579d4da51d8c" (UID: "63076e22-91cb-460d-8d0f-579d4da51d8c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:51:23.447644 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:23.447624    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63076e22-91cb-460d-8d0f-579d4da51d8c-kube-api-access-bl9l8" pod "63076e22-91cb-460d-8d0f-579d4da51d8c" (UID: "63076e22-91cb-460d-8d0f-579d4da51d8c"). InnerVolumeSpecName "kube-api-access-bl9l8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:51:23.560404 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:23.556112    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63076e22-91cb-460d-8d0f-579d4da51d8c-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:23.560404 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:23.556161    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63076e22-91cb-460d-8d0f-579d4da51d8c-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:23.560404 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:23.556178    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bl9l8\" (UniqueName: \"kubernetes.io/projected/63076e22-91cb-460d-8d0f-579d4da51d8c-kube-api-access-bl9l8\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:23.560404 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:23.556204    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63076e22-91cb-460d-8d0f-579d4da51d8c-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:23.701349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:23.699962    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:23.802521 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:23.797505    4645 generic.go:356] "Generic (PLEG): container finished" podID="8eee5947-9731-4194-aa80-7fbd619e551f" containerID="ff05706b16b2a500578c373d429e7fb217fc2519b8ee19ec35c709f2d08a75f5" exitCode=0
Oct 31 19:51:23.802521 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:23.797894    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z" event={"ID":"8eee5947-9731-4194-aa80-7fbd619e551f","Type":"ContainerDied","Data":"ff05706b16b2a500578c373d429e7fb217fc2519b8ee19ec35c709f2d08a75f5"}
Oct 31 19:51:23.802521 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:23.797964    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z" event={"ID":"8eee5947-9731-4194-aa80-7fbd619e551f","Type":"ContainerDied","Data":"055feac6c46e5ca11d8579f1e81525390030130b4d4dd7c6446abca5b3361b19"}
Oct 31 19:51:23.802521 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:23.797996    4645 scope.go:122] "RemoveContainer" containerID="ff05706b16b2a500578c373d429e7fb217fc2519b8ee19ec35c709f2d08a75f5"
Oct 31 19:51:23.802521 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:23.798162    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z"
Oct 31 19:51:23.820529 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:23.820486    4645 generic.go:356] "Generic (PLEG): container finished" podID="63076e22-91cb-460d-8d0f-579d4da51d8c" containerID="2ba6f95de5abd698e58dbde9e45f200890221940dc84b9deda9d3b2e70c54279" exitCode=0
Oct 31 19:51:23.822035 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:23.822003    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt"
Oct 31 19:51:23.822485 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:23.822446    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt" event={"ID":"63076e22-91cb-460d-8d0f-579d4da51d8c","Type":"ContainerDied","Data":"2ba6f95de5abd698e58dbde9e45f200890221940dc84b9deda9d3b2e70c54279"}
Oct 31 19:51:23.822716 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:23.822680    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt" event={"ID":"63076e22-91cb-460d-8d0f-579d4da51d8c","Type":"ContainerDied","Data":"18331802d8bb37b545bf019e23e0d113d53e127c1b14f7f5c5bab3ab4bbd011d"}
Oct 31 19:51:23.838173 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:23.838150    4645 scope.go:122] "RemoveContainer" containerID="db0770b5441fe07bb8782b88cf635a09301dc956518832c2b6371caf6bb70dd2"
Oct 31 19:51:23.862413 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:23.862386    4645 scope.go:122] "RemoveContainer" containerID="ff05706b16b2a500578c373d429e7fb217fc2519b8ee19ec35c709f2d08a75f5"
Oct 31 19:51:23.862916 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:51:23.862837    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff05706b16b2a500578c373d429e7fb217fc2519b8ee19ec35c709f2d08a75f5\": container with ID starting with ff05706b16b2a500578c373d429e7fb217fc2519b8ee19ec35c709f2d08a75f5 not found: ID does not exist" containerID="ff05706b16b2a500578c373d429e7fb217fc2519b8ee19ec35c709f2d08a75f5"
Oct 31 19:51:23.862916 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:23.862870    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff05706b16b2a500578c373d429e7fb217fc2519b8ee19ec35c709f2d08a75f5"} err="failed to get container status \"ff05706b16b2a500578c373d429e7fb217fc2519b8ee19ec35c709f2d08a75f5\": rpc error: code = NotFound desc = could not find container \"ff05706b16b2a500578c373d429e7fb217fc2519b8ee19ec35c709f2d08a75f5\": container with ID starting with ff05706b16b2a500578c373d429e7fb217fc2519b8ee19ec35c709f2d08a75f5 not found: ID does not exist"
Oct 31 19:51:23.862916 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:23.862892    4645 scope.go:122] "RemoveContainer" containerID="db0770b5441fe07bb8782b88cf635a09301dc956518832c2b6371caf6bb70dd2"
Oct 31 19:51:23.863297 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:51:23.863244    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db0770b5441fe07bb8782b88cf635a09301dc956518832c2b6371caf6bb70dd2\": container with ID starting with db0770b5441fe07bb8782b88cf635a09301dc956518832c2b6371caf6bb70dd2 not found: ID does not exist" containerID="db0770b5441fe07bb8782b88cf635a09301dc956518832c2b6371caf6bb70dd2"
Oct 31 19:51:23.863297 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:23.863279    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db0770b5441fe07bb8782b88cf635a09301dc956518832c2b6371caf6bb70dd2"} err="failed to get container status \"db0770b5441fe07bb8782b88cf635a09301dc956518832c2b6371caf6bb70dd2\": rpc error: code = NotFound desc = could not find container \"db0770b5441fe07bb8782b88cf635a09301dc956518832c2b6371caf6bb70dd2\": container with ID starting with db0770b5441fe07bb8782b88cf635a09301dc956518832c2b6371caf6bb70dd2 not found: ID does not exist"
Oct 31 19:51:23.863471 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:23.863300    4645 scope.go:122] "RemoveContainer" containerID="2ba6f95de5abd698e58dbde9e45f200890221940dc84b9deda9d3b2e70c54279"
Oct 31 19:51:23.889638 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:23.889607    4645 scope.go:122] "RemoveContainer" containerID="2ba6f95de5abd698e58dbde9e45f200890221940dc84b9deda9d3b2e70c54279"
Oct 31 19:51:23.889879 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:23.889842    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z"]
Oct 31 19:51:23.890069 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:51:23.890047    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ba6f95de5abd698e58dbde9e45f200890221940dc84b9deda9d3b2e70c54279\": container with ID starting with 2ba6f95de5abd698e58dbde9e45f200890221940dc84b9deda9d3b2e70c54279 not found: ID does not exist" containerID="2ba6f95de5abd698e58dbde9e45f200890221940dc84b9deda9d3b2e70c54279"
Oct 31 19:51:23.890140 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:23.890080    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ba6f95de5abd698e58dbde9e45f200890221940dc84b9deda9d3b2e70c54279"} err="failed to get container status \"2ba6f95de5abd698e58dbde9e45f200890221940dc84b9deda9d3b2e70c54279\": rpc error: code = NotFound desc = could not find container \"2ba6f95de5abd698e58dbde9e45f200890221940dc84b9deda9d3b2e70c54279\": container with ID starting with 2ba6f95de5abd698e58dbde9e45f200890221940dc84b9deda9d3b2e70c54279 not found: ID does not exist"
Oct 31 19:51:23.923872 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:23.923843    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-4xf5z"]
Oct 31 19:51:23.940458 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:23.940432    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt"]
Oct 31 19:51:23.951113 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:23.951088    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-9efeae-openshift-default-6df74d98cf-mhkkt"]
Oct 31 19:51:24.862792 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:24.862756    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:24.932884 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:24.932834    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-1e8cf0-56bb54564c-t5hz7" podUID="788ad5fd-ae6a-4953-a113-1357e62f2b0b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:51:25.621346 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:25.621303    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63076e22-91cb-460d-8d0f-579d4da51d8c" path="/var/lib/kubelet/pods/63076e22-91cb-460d-8d0f-579d4da51d8c/volumes"
Oct 31 19:51:25.621956 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:25.621931    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8eee5947-9731-4194-aa80-7fbd619e551f" path="/var/lib/kubelet/pods/8eee5947-9731-4194-aa80-7fbd619e551f/volumes"
Oct 31 19:51:25.763903 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:25.763867    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:26.430717 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:26.430663    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:26.655352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:26.649866    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:26.805404 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:26.801495    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:26.931417 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:26.927070    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:27.440569 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:27.440521    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:27.950708 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:27.950675    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:28.155823 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:28.155791    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:28.278956 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:28.278848    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:28.399321 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:28.399287    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:28.511436 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:28.511407    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:28.759350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:28.757246    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:28.774226 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:28.770756    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2"]
Oct 31 19:51:28.774226 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:28.771030    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2" podUID="76b3b657-6b14-43ff-b7ce-1e9e3b615141" containerName="istio-proxy" containerID="cri-o://fbec275cf9320e13f885a9f0ff45095d5e7c5b2aac8ba10f9d7c72adbf5e195a" gracePeriod=30
Oct 31 19:51:28.785270 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:28.785224    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks"]
Oct 31 19:51:28.788190 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:28.787132    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks" podUID="1cf00cb6-1d65-42f3-9c91-f4622454a913" containerName="istio-proxy" containerID="cri-o://5a69f40deb4941134268160b1b5f8ce1a1c5a1ef5d30b28b7d699aeca47d4285" gracePeriod=30
Oct 31 19:51:29.075352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:29.068003    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:29.098742 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:29.096535    4645 generic.go:356] "Generic (PLEG): container finished" podID="33f3e8b8-6af6-4f4d-9ab9-ac00d1bbb9ce" containerID="cefe3c3880a7157c8a023a3ce173f0524066c6ea79126f747ebea37e1beb1823" exitCode=0
Oct 31 19:51:29.098742 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:29.096648    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f8c3f7-bdc4d8f7-75mhm" event={"ID":"33f3e8b8-6af6-4f4d-9ab9-ac00d1bbb9ce","Type":"ContainerDied","Data":"cefe3c3880a7157c8a023a3ce173f0524066c6ea79126f747ebea37e1beb1823"}
Oct 31 19:51:29.306544 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:29.306515    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f8c3f7-bdc4d8f7-75mhm"
Oct 31 19:51:29.388043 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:29.387979    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33f3e8b8-6af6-4f4d-9ab9-ac00d1bbb9ce-kube-api-access-ht62f\" (UniqueName: \"kubernetes.io/projected/33f3e8b8-6af6-4f4d-9ab9-ac00d1bbb9ce-kube-api-access-ht62f\") pod \"33f3e8b8-6af6-4f4d-9ab9-ac00d1bbb9ce\" (UID: \"33f3e8b8-6af6-4f4d-9ab9-ac00d1bbb9ce\") "
Oct 31 19:51:29.388256 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:29.388240    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33f3e8b8-6af6-4f4d-9ab9-ac00d1bbb9ce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33f3e8b8-6af6-4f4d-9ab9-ac00d1bbb9ce-plugins-config-volume\") pod \"33f3e8b8-6af6-4f4d-9ab9-ac00d1bbb9ce\" (UID: \"33f3e8b8-6af6-4f4d-9ab9-ac00d1bbb9ce\") "
Oct 31 19:51:29.388694 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:29.388670    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33f3e8b8-6af6-4f4d-9ab9-ac00d1bbb9ce-plugins-config-volume" pod "33f3e8b8-6af6-4f4d-9ab9-ac00d1bbb9ce" (UID: "33f3e8b8-6af6-4f4d-9ab9-ac00d1bbb9ce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:51:29.388812 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:29.388798    4645 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33f3e8b8-6af6-4f4d-9ab9-ac00d1bbb9ce-plugins-config-volume\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:29.390015 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:29.389988    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33f3e8b8-6af6-4f4d-9ab9-ac00d1bbb9ce-kube-api-access-ht62f" pod "33f3e8b8-6af6-4f4d-9ab9-ac00d1bbb9ce" (UID: "33f3e8b8-6af6-4f4d-9ab9-ac00d1bbb9ce"). InnerVolumeSpecName "kube-api-access-ht62f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:51:29.490384 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:29.490351    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ht62f\" (UniqueName: \"kubernetes.io/projected/33f3e8b8-6af6-4f4d-9ab9-ac00d1bbb9ce-kube-api-access-ht62f\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:30.119791 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:30.119757    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f8c3f7-bdc4d8f7-75mhm"
Oct 31 19:51:30.119791 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:30.119770    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f8c3f7-bdc4d8f7-75mhm" event={"ID":"33f3e8b8-6af6-4f4d-9ab9-ac00d1bbb9ce","Type":"ContainerDied","Data":"5c3b5e9ccdc4571e245f76b92826c2dde0a828bce10f09a3b4279fa2111d8284"}
Oct 31 19:51:30.120296 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:30.119859    4645 scope.go:122] "RemoveContainer" containerID="cefe3c3880a7157c8a023a3ce173f0524066c6ea79126f747ebea37e1beb1823"
Oct 31 19:51:30.139196 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:30.139173    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f8c3f7-bdc4d8f7-75mhm"]
Oct 31 19:51:30.146531 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:30.146509    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f8c3f7-bdc4d8f7-75mhm"]
Oct 31 19:51:30.197675 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:30.197648    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d45f6c-55b6bc85f9-776mg" podUID="d7fbab67-7316-40e5-ab5a-e848cd654c50" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:51:30.342604 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:30.342575    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:30.588840 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:30.588808    4645 patch_prober.go:29] interesting pod/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 19:51:30.588993 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:30.588876    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2" podUID="76b3b657-6b14-43ff-b7ce-1e9e3b615141" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 19:51:31.128201 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:31.128126    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:31.344074 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:31.344041    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-fd1c91/auth-test-model-fd1c91-kserve-55d79cc5b5-svsm4"
Oct 31 19:51:31.356079 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:31.356053    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-fd1c91/auth-test-model-fd1c91-kserve-55d79cc5b5-svsm4"
Oct 31 19:51:31.620036 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:31.620005    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33f3e8b8-6af6-4f4d-9ab9-ac00d1bbb9ce" path="/var/lib/kubelet/pods/33f3e8b8-6af6-4f4d-9ab9-ac00d1bbb9ce/volumes"
Oct 31 19:51:32.013292 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:32.013260    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:32.145069 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:32.145039    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-fd1c91/auth-test-model-fd1c91-kserve-55d79cc5b5-svsm4"
Oct 31 19:51:32.944383 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:32.944348    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:33.951510 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:33.951452    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:34.247269 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:34.247207    4645 generic.go:356] "Generic (PLEG): container finished" podID="1cf00cb6-1d65-42f3-9c91-f4622454a913" containerID="5a69f40deb4941134268160b1b5f8ce1a1c5a1ef5d30b28b7d699aeca47d4285" exitCode=0
Oct 31 19:51:34.247656 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:34.247287    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks" event={"ID":"1cf00cb6-1d65-42f3-9c91-f4622454a913","Type":"ContainerDied","Data":"5a69f40deb4941134268160b1b5f8ce1a1c5a1ef5d30b28b7d699aeca47d4285"}
Oct 31 19:51:34.250418 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:34.250396    4645 generic.go:356] "Generic (PLEG): container finished" podID="76b3b657-6b14-43ff-b7ce-1e9e3b615141" containerID="fbec275cf9320e13f885a9f0ff45095d5e7c5b2aac8ba10f9d7c72adbf5e195a" exitCode=0
Oct 31 19:51:34.250507 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:34.250437    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2" event={"ID":"76b3b657-6b14-43ff-b7ce-1e9e3b615141","Type":"ContainerDied","Data":"fbec275cf9320e13f885a9f0ff45095d5e7c5b2aac8ba10f9d7c72adbf5e195a"}
Oct 31 19:51:34.250507 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:34.250475    4645 scope.go:122] "RemoveContainer" containerID="de1325c6e76563fc630e2eec234bff76121d6e736f691e783aa15922c2597480"
Oct 31 19:51:34.293230 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:34.293181    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks"
Oct 31 19:51:34.373951 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:34.373927    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cf00cb6-1d65-42f3-9c91-f4622454a913-kube-api-access-s66vw\" (UniqueName: \"kubernetes.io/projected/1cf00cb6-1d65-42f3-9c91-f4622454a913-kube-api-access-s66vw\") pod \"1cf00cb6-1d65-42f3-9c91-f4622454a913\" (UID: \"1cf00cb6-1d65-42f3-9c91-f4622454a913\") "
Oct 31 19:51:34.374055 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:34.373980    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cf00cb6-1d65-42f3-9c91-f4622454a913-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cf00cb6-1d65-42f3-9c91-f4622454a913-istio-data\") pod \"1cf00cb6-1d65-42f3-9c91-f4622454a913\" (UID: \"1cf00cb6-1d65-42f3-9c91-f4622454a913\") "
Oct 31 19:51:34.374055 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:34.374007    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cf00cb6-1d65-42f3-9c91-f4622454a913-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cf00cb6-1d65-42f3-9c91-f4622454a913-istiod-ca-cert\") pod \"1cf00cb6-1d65-42f3-9c91-f4622454a913\" (UID: \"1cf00cb6-1d65-42f3-9c91-f4622454a913\") "
Oct 31 19:51:34.374149 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:34.374076    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cf00cb6-1d65-42f3-9c91-f4622454a913-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cf00cb6-1d65-42f3-9c91-f4622454a913-workload-certs\") pod \"1cf00cb6-1d65-42f3-9c91-f4622454a913\" (UID: \"1cf00cb6-1d65-42f3-9c91-f4622454a913\") "
Oct 31 19:51:34.374149 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:34.374142    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cf00cb6-1d65-42f3-9c91-f4622454a913-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cf00cb6-1d65-42f3-9c91-f4622454a913-credential-socket\") pod \"1cf00cb6-1d65-42f3-9c91-f4622454a913\" (UID: \"1cf00cb6-1d65-42f3-9c91-f4622454a913\") "
Oct 31 19:51:34.374223 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:34.374170    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1cf00cb6-1d65-42f3-9c91-f4622454a913-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cf00cb6-1d65-42f3-9c91-f4622454a913-istio-podinfo\") pod \"1cf00cb6-1d65-42f3-9c91-f4622454a913\" (UID: \"1cf00cb6-1d65-42f3-9c91-f4622454a913\") "
Oct 31 19:51:34.374223 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:34.374217    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cf00cb6-1d65-42f3-9c91-f4622454a913-istio-token\" (UniqueName: \"kubernetes.io/projected/1cf00cb6-1d65-42f3-9c91-f4622454a913-istio-token\") pod \"1cf00cb6-1d65-42f3-9c91-f4622454a913\" (UID: \"1cf00cb6-1d65-42f3-9c91-f4622454a913\") "
Oct 31 19:51:34.374353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:34.374315    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cf00cb6-1d65-42f3-9c91-f4622454a913-istio-data" pod "1cf00cb6-1d65-42f3-9c91-f4622454a913" (UID: "1cf00cb6-1d65-42f3-9c91-f4622454a913"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:51:34.374424 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:34.374382    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cf00cb6-1d65-42f3-9c91-f4622454a913-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cf00cb6-1d65-42f3-9c91-f4622454a913-istio-envoy\") pod \"1cf00cb6-1d65-42f3-9c91-f4622454a913\" (UID: \"1cf00cb6-1d65-42f3-9c91-f4622454a913\") "
Oct 31 19:51:34.374490 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:34.374424    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cf00cb6-1d65-42f3-9c91-f4622454a913-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cf00cb6-1d65-42f3-9c91-f4622454a913-workload-socket\") pod \"1cf00cb6-1d65-42f3-9c91-f4622454a913\" (UID: \"1cf00cb6-1d65-42f3-9c91-f4622454a913\") "
Oct 31 19:51:34.374541 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:34.374526    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cf00cb6-1d65-42f3-9c91-f4622454a913-credential-socket" pod "1cf00cb6-1d65-42f3-9c91-f4622454a913" (UID: "1cf00cb6-1d65-42f3-9c91-f4622454a913"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:51:34.374600 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:34.374572    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cf00cb6-1d65-42f3-9c91-f4622454a913-istiod-ca-cert" pod "1cf00cb6-1d65-42f3-9c91-f4622454a913" (UID: "1cf00cb6-1d65-42f3-9c91-f4622454a913"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:51:34.374744 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:34.374725    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cf00cb6-1d65-42f3-9c91-f4622454a913-workload-certs" pod "1cf00cb6-1d65-42f3-9c91-f4622454a913" (UID: "1cf00cb6-1d65-42f3-9c91-f4622454a913"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:51:34.374744 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:34.374737    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cf00cb6-1d65-42f3-9c91-f4622454a913-workload-socket" pod "1cf00cb6-1d65-42f3-9c91-f4622454a913" (UID: "1cf00cb6-1d65-42f3-9c91-f4622454a913"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:51:34.375368 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:34.375310    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cf00cb6-1d65-42f3-9c91-f4622454a913-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:34.375487 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:34.375371    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cf00cb6-1d65-42f3-9c91-f4622454a913-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:34.375487 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:34.375382    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cf00cb6-1d65-42f3-9c91-f4622454a913-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:34.375487 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:34.375393    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cf00cb6-1d65-42f3-9c91-f4622454a913-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:34.375487 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:34.375404    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cf00cb6-1d65-42f3-9c91-f4622454a913-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:34.376663 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:34.376641    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cf00cb6-1d65-42f3-9c91-f4622454a913-kube-api-access-s66vw" pod "1cf00cb6-1d65-42f3-9c91-f4622454a913" (UID: "1cf00cb6-1d65-42f3-9c91-f4622454a913"). InnerVolumeSpecName "kube-api-access-s66vw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:51:34.376746 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:34.376664    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1cf00cb6-1d65-42f3-9c91-f4622454a913-istio-podinfo" pod "1cf00cb6-1d65-42f3-9c91-f4622454a913" (UID: "1cf00cb6-1d65-42f3-9c91-f4622454a913"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:51:34.376865 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:34.376849    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cf00cb6-1d65-42f3-9c91-f4622454a913-istio-token" pod "1cf00cb6-1d65-42f3-9c91-f4622454a913" (UID: "1cf00cb6-1d65-42f3-9c91-f4622454a913"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:51:34.377243 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:34.377228    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cf00cb6-1d65-42f3-9c91-f4622454a913-istio-envoy" pod "1cf00cb6-1d65-42f3-9c91-f4622454a913" (UID: "1cf00cb6-1d65-42f3-9c91-f4622454a913"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:51:34.384935 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:34.384916    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2"
Oct 31 19:51:34.476678 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:34.476647    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76b3b657-6b14-43ff-b7ce-1e9e3b615141-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76b3b657-6b14-43ff-b7ce-1e9e3b615141-istio-data\") pod \"76b3b657-6b14-43ff-b7ce-1e9e3b615141\" (UID: \"76b3b657-6b14-43ff-b7ce-1e9e3b615141\") "
Oct 31 19:51:34.476833 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:34.476776    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76b3b657-6b14-43ff-b7ce-1e9e3b615141-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76b3b657-6b14-43ff-b7ce-1e9e3b615141-credential-socket\") pod \"76b3b657-6b14-43ff-b7ce-1e9e3b615141\" (UID: \"76b3b657-6b14-43ff-b7ce-1e9e3b615141\") "
Oct 31 19:51:34.476833 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:34.476814    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76b3b657-6b14-43ff-b7ce-1e9e3b615141-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76b3b657-6b14-43ff-b7ce-1e9e3b615141-workload-certs\") pod \"76b3b657-6b14-43ff-b7ce-1e9e3b615141\" (UID: \"76b3b657-6b14-43ff-b7ce-1e9e3b615141\") "
Oct 31 19:51:34.476938 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:34.476851    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76b3b657-6b14-43ff-b7ce-1e9e3b615141-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76b3b657-6b14-43ff-b7ce-1e9e3b615141-workload-socket\") pod \"76b3b657-6b14-43ff-b7ce-1e9e3b615141\" (UID: \"76b3b657-6b14-43ff-b7ce-1e9e3b615141\") "
Oct 31 19:51:34.476938 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:34.476894    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76b3b657-6b14-43ff-b7ce-1e9e3b615141-kube-api-access-8qxbj\" (UniqueName: \"kubernetes.io/projected/76b3b657-6b14-43ff-b7ce-1e9e3b615141-kube-api-access-8qxbj\") pod \"76b3b657-6b14-43ff-b7ce-1e9e3b615141\" (UID: \"76b3b657-6b14-43ff-b7ce-1e9e3b615141\") "
Oct 31 19:51:34.477026 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:34.476937    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76b3b657-6b14-43ff-b7ce-1e9e3b615141-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76b3b657-6b14-43ff-b7ce-1e9e3b615141-istio-envoy\") pod \"76b3b657-6b14-43ff-b7ce-1e9e3b615141\" (UID: \"76b3b657-6b14-43ff-b7ce-1e9e3b615141\") "
Oct 31 19:51:34.477026 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:34.476949    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76b3b657-6b14-43ff-b7ce-1e9e3b615141-istio-data" pod "76b3b657-6b14-43ff-b7ce-1e9e3b615141" (UID: "76b3b657-6b14-43ff-b7ce-1e9e3b615141"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:51:34.477026 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:34.476984    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76b3b657-6b14-43ff-b7ce-1e9e3b615141-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76b3b657-6b14-43ff-b7ce-1e9e3b615141-istiod-ca-cert\") pod \"76b3b657-6b14-43ff-b7ce-1e9e3b615141\" (UID: \"76b3b657-6b14-43ff-b7ce-1e9e3b615141\") "
Oct 31 19:51:34.477026 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:34.477019    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76b3b657-6b14-43ff-b7ce-1e9e3b615141-istio-token\" (UniqueName: \"kubernetes.io/projected/76b3b657-6b14-43ff-b7ce-1e9e3b615141-istio-token\") pod \"76b3b657-6b14-43ff-b7ce-1e9e3b615141\" (UID: \"76b3b657-6b14-43ff-b7ce-1e9e3b615141\") "
Oct 31 19:51:34.477244 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:34.477055    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76b3b657-6b14-43ff-b7ce-1e9e3b615141-credential-socket" pod "76b3b657-6b14-43ff-b7ce-1e9e3b615141" (UID: "76b3b657-6b14-43ff-b7ce-1e9e3b615141"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:51:34.477244 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:34.477082    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76b3b657-6b14-43ff-b7ce-1e9e3b615141-workload-certs" pod "76b3b657-6b14-43ff-b7ce-1e9e3b615141" (UID: "76b3b657-6b14-43ff-b7ce-1e9e3b615141"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:51:34.477244 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:34.477127    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/76b3b657-6b14-43ff-b7ce-1e9e3b615141-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76b3b657-6b14-43ff-b7ce-1e9e3b615141-istio-podinfo\") pod \"76b3b657-6b14-43ff-b7ce-1e9e3b615141\" (UID: \"76b3b657-6b14-43ff-b7ce-1e9e3b615141\") "
Oct 31 19:51:34.477244 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:34.477221    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76b3b657-6b14-43ff-b7ce-1e9e3b615141-workload-socket" pod "76b3b657-6b14-43ff-b7ce-1e9e3b615141" (UID: "76b3b657-6b14-43ff-b7ce-1e9e3b615141"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:51:34.477487 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:34.477459    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76b3b657-6b14-43ff-b7ce-1e9e3b615141-istiod-ca-cert" pod "76b3b657-6b14-43ff-b7ce-1e9e3b615141" (UID: "76b3b657-6b14-43ff-b7ce-1e9e3b615141"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:51:34.477966 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:34.477921    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cf00cb6-1d65-42f3-9c91-f4622454a913-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:34.477966 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:34.477946    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76b3b657-6b14-43ff-b7ce-1e9e3b615141-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:34.477966 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:34.477962    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76b3b657-6b14-43ff-b7ce-1e9e3b615141-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:34.478209 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:34.477976    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76b3b657-6b14-43ff-b7ce-1e9e3b615141-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:34.478209 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:34.477990    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76b3b657-6b14-43ff-b7ce-1e9e3b615141-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:34.478209 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:34.478004    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s66vw\" (UniqueName: \"kubernetes.io/projected/1cf00cb6-1d65-42f3-9c91-f4622454a913-kube-api-access-s66vw\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:34.478209 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:34.478019    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cf00cb6-1d65-42f3-9c91-f4622454a913-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:34.478209 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:34.478034    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cf00cb6-1d65-42f3-9c91-f4622454a913-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:34.478209 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:34.478048    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76b3b657-6b14-43ff-b7ce-1e9e3b615141-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:34.479513 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:34.479488    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76b3b657-6b14-43ff-b7ce-1e9e3b615141-istio-token" pod "76b3b657-6b14-43ff-b7ce-1e9e3b615141" (UID: "76b3b657-6b14-43ff-b7ce-1e9e3b615141"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:51:34.479601 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:34.479546    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76b3b657-6b14-43ff-b7ce-1e9e3b615141-istio-envoy" pod "76b3b657-6b14-43ff-b7ce-1e9e3b615141" (UID: "76b3b657-6b14-43ff-b7ce-1e9e3b615141"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:51:34.479695 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:34.479677    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76b3b657-6b14-43ff-b7ce-1e9e3b615141-kube-api-access-8qxbj" pod "76b3b657-6b14-43ff-b7ce-1e9e3b615141" (UID: "76b3b657-6b14-43ff-b7ce-1e9e3b615141"). InnerVolumeSpecName "kube-api-access-8qxbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:51:34.479932 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:34.479911    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/76b3b657-6b14-43ff-b7ce-1e9e3b615141-istio-podinfo" pod "76b3b657-6b14-43ff-b7ce-1e9e3b615141" (UID: "76b3b657-6b14-43ff-b7ce-1e9e3b615141"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:51:34.579194 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:34.579138    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qxbj\" (UniqueName: \"kubernetes.io/projected/76b3b657-6b14-43ff-b7ce-1e9e3b615141-kube-api-access-8qxbj\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:34.579194 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:34.579163    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76b3b657-6b14-43ff-b7ce-1e9e3b615141-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:34.579194 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:34.579173    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76b3b657-6b14-43ff-b7ce-1e9e3b615141-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:34.579194 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:34.579181    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76b3b657-6b14-43ff-b7ce-1e9e3b615141-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:34.690917 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:34.690888    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:34.825071 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:34.822936    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:34.933521 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:34.933482    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-1e8cf0-56bb54564c-t5hz7" podUID="788ad5fd-ae6a-4953-a113-1357e62f2b0b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:51:35.026374 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:35.024833    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp"]
Oct 31 19:51:35.026374 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:35.025082    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp" podUID="b73e8ddc-c45d-4537-a4c4-f22747469ca8" containerName="istio-proxy" containerID="cri-o://c5b432470850c5b9ec1435a151d8d255b18704cc2e8d732ec21f426793c16b00" gracePeriod=30
Oct 31 19:51:35.031899 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:35.031868    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq"]
Oct 31 19:51:35.032103 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:35.032078    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq" podUID="37f24530-822f-4186-acfa-a7912c710c56" containerName="istio-proxy" containerID="cri-o://5fa2823567167438a072ca570e805fcb42cf724e81caee7226a2805694a07e70" gracePeriod=30
Oct 31 19:51:35.050128 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:35.050097    4645 patch_prober.go:29] interesting pod/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 19:51:35.050238 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:35.050149    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp" podUID="b73e8ddc-c45d-4537-a4c4-f22747469ca8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 19:51:35.068812 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:35.068785    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:35.272644 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:35.269360    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2" event={"ID":"76b3b657-6b14-43ff-b7ce-1e9e3b615141","Type":"ContainerDied","Data":"7ea647d7c238754ee6c2ba9d3a8fc04031ab61763d5327cdaf84b3c90983c87e"}
Oct 31 19:51:35.272644 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:35.269444    4645 scope.go:122] "RemoveContainer" containerID="fbec275cf9320e13f885a9f0ff45095d5e7c5b2aac8ba10f9d7c72adbf5e195a"
Oct 31 19:51:35.272644 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:35.269633    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2"
Oct 31 19:51:35.284454 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:35.283778    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks" event={"ID":"1cf00cb6-1d65-42f3-9c91-f4622454a913","Type":"ContainerDied","Data":"781193ce99a79a2b6eda0e2093dac7e0181c7a1261ba7e93eb7c7b288f4c0c11"}
Oct 31 19:51:35.284454 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:35.283919    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks"
Oct 31 19:51:35.321548 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:35.321515    4645 scope.go:122] "RemoveContainer" containerID="5a69f40deb4941134268160b1b5f8ce1a1c5a1ef5d30b28b7d699aeca47d4285"
Oct 31 19:51:35.325532 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:35.325505    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2"]
Oct 31 19:51:35.330371 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:35.330348    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-8z2f2"]
Oct 31 19:51:35.338233 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:35.338208    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks"]
Oct 31 19:51:35.347850 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:35.347827    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-f8c3f7-openshift-default-78b8d845bc-46xks"]
Oct 31 19:51:35.620968 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:35.620879    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cf00cb6-1d65-42f3-9c91-f4622454a913" path="/var/lib/kubelet/pods/1cf00cb6-1d65-42f3-9c91-f4622454a913/volumes"
Oct 31 19:51:35.621487 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:35.621465    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76b3b657-6b14-43ff-b7ce-1e9e3b615141" path="/var/lib/kubelet/pods/76b3b657-6b14-43ff-b7ce-1e9e3b615141/volumes"
Oct 31 19:51:35.749030 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:35.748999    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:35.891687 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:35.891590    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:36.247627 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:51:36.247589    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 19:51:36.342977 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:36.342941    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8"]
Oct 31 19:51:36.344009 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:36.343988    4645 cpu_manager.go:414] "containerMap: removing container" podUID="1cf00cb6-1d65-42f3-9c91-f4622454a913" containerName="istio-proxy"
Oct 31 19:51:36.344116 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:36.344007    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cf00cb6-1d65-42f3-9c91-f4622454a913" containerName="istio-proxy"
Oct 31 19:51:36.344116 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:36.344029    4645 cpu_manager.go:414] "containerMap: removing container" podUID="76b3b657-6b14-43ff-b7ce-1e9e3b615141" containerName="istio-proxy"
Oct 31 19:51:36.344116 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:36.344037    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76b3b657-6b14-43ff-b7ce-1e9e3b615141" containerName="istio-proxy"
Oct 31 19:51:36.344116 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:36.344057    4645 cpu_manager.go:414] "containerMap: removing container" podUID="76b3b657-6b14-43ff-b7ce-1e9e3b615141" containerName="istio-proxy"
Oct 31 19:51:36.344116 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:36.344064    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76b3b657-6b14-43ff-b7ce-1e9e3b615141" containerName="istio-proxy"
Oct 31 19:51:36.344116 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:36.344086    4645 cpu_manager.go:414] "containerMap: removing container" podUID="8eee5947-9731-4194-aa80-7fbd619e551f" containerName="istio-proxy"
Oct 31 19:51:36.344116 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:36.344093    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8eee5947-9731-4194-aa80-7fbd619e551f" containerName="istio-proxy"
Oct 31 19:51:36.344116 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:36.344105    4645 cpu_manager.go:414] "containerMap: removing container" podUID="33f3e8b8-6af6-4f4d-9ab9-ac00d1bbb9ce" containerName="payload-processing"
Oct 31 19:51:36.344116 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:36.344112    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33f3e8b8-6af6-4f4d-9ab9-ac00d1bbb9ce" containerName="payload-processing"
Oct 31 19:51:36.345047 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:36.344134    4645 cpu_manager.go:414] "containerMap: removing container" podUID="63076e22-91cb-460d-8d0f-579d4da51d8c" containerName="istio-proxy"
Oct 31 19:51:36.345047 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:36.344141    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63076e22-91cb-460d-8d0f-579d4da51d8c" containerName="istio-proxy"
Oct 31 19:51:36.345047 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:36.344166    4645 cpu_manager.go:414] "containerMap: removing container" podUID="8eee5947-9731-4194-aa80-7fbd619e551f" containerName="istio-proxy"
Oct 31 19:51:36.345047 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:36.344173    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8eee5947-9731-4194-aa80-7fbd619e551f" containerName="istio-proxy"
Oct 31 19:51:36.345047 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:36.344319    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="76b3b657-6b14-43ff-b7ce-1e9e3b615141" containerName="istio-proxy"
Oct 31 19:51:36.345047 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:36.344348    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="8eee5947-9731-4194-aa80-7fbd619e551f" containerName="istio-proxy"
Oct 31 19:51:36.345047 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:36.344357    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="33f3e8b8-6af6-4f4d-9ab9-ac00d1bbb9ce" containerName="payload-processing"
Oct 31 19:51:36.345047 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:36.344369    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="63076e22-91cb-460d-8d0f-579d4da51d8c" containerName="istio-proxy"
Oct 31 19:51:36.345047 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:36.344378    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="8eee5947-9731-4194-aa80-7fbd619e551f" containerName="istio-proxy"
Oct 31 19:51:36.345047 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:36.344394    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cf00cb6-1d65-42f3-9c91-f4622454a913" containerName="istio-proxy"
Oct 31 19:51:36.350431 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:36.350411    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8"
Oct 31 19:51:36.353793 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:36.353774    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-e618e1-openshift-default-dockercfg-h496q\""
Oct 31 19:51:36.371380 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:36.371359    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8"]
Oct 31 19:51:36.410611 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:36.410584    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:36.518413 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:36.517409    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ede30242-d4ae-4caa-9c4b-8918314a1ead-istio-data\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8\" (UID: \"ede30242-d4ae-4caa-9c4b-8918314a1ead\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8"
Oct 31 19:51:36.518413 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:36.517502    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ede30242-d4ae-4caa-9c4b-8918314a1ead-istio-token\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8\" (UID: \"ede30242-d4ae-4caa-9c4b-8918314a1ead\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8"
Oct 31 19:51:36.518413 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:36.517559    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8gnc\" (UniqueName: \"kubernetes.io/projected/ede30242-d4ae-4caa-9c4b-8918314a1ead-kube-api-access-x8gnc\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8\" (UID: \"ede30242-d4ae-4caa-9c4b-8918314a1ead\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8"
Oct 31 19:51:36.518413 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:36.517679    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede30242-d4ae-4caa-9c4b-8918314a1ead-credential-socket\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8\" (UID: \"ede30242-d4ae-4caa-9c4b-8918314a1ead\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8"
Oct 31 19:51:36.518413 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:36.517717    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ede30242-d4ae-4caa-9c4b-8918314a1ead-istio-envoy\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8\" (UID: \"ede30242-d4ae-4caa-9c4b-8918314a1ead\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8"
Oct 31 19:51:36.518413 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:36.517752    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ede30242-d4ae-4caa-9c4b-8918314a1ead-istiod-ca-cert\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8\" (UID: \"ede30242-d4ae-4caa-9c4b-8918314a1ead\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8"
Oct 31 19:51:36.518413 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:36.517793    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede30242-d4ae-4caa-9c4b-8918314a1ead-workload-socket\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8\" (UID: \"ede30242-d4ae-4caa-9c4b-8918314a1ead\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8"
Oct 31 19:51:36.518413 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:36.517850    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ede30242-d4ae-4caa-9c4b-8918314a1ead-workload-certs\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8\" (UID: \"ede30242-d4ae-4caa-9c4b-8918314a1ead\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8"
Oct 31 19:51:36.518413 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:36.517943    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ede30242-d4ae-4caa-9c4b-8918314a1ead-istio-podinfo\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8\" (UID: \"ede30242-d4ae-4caa-9c4b-8918314a1ead\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8"
Oct 31 19:51:36.560508 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:36.557258    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:36.624361 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:36.621778    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede30242-d4ae-4caa-9c4b-8918314a1ead-workload-socket\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8\" (UID: \"ede30242-d4ae-4caa-9c4b-8918314a1ead\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8"
Oct 31 19:51:36.624361 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:36.621858    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ede30242-d4ae-4caa-9c4b-8918314a1ead-workload-certs\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8\" (UID: \"ede30242-d4ae-4caa-9c4b-8918314a1ead\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8"
Oct 31 19:51:36.624361 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:36.621946    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ede30242-d4ae-4caa-9c4b-8918314a1ead-istio-podinfo\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8\" (UID: \"ede30242-d4ae-4caa-9c4b-8918314a1ead\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8"
Oct 31 19:51:36.624361 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:36.622088    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ede30242-d4ae-4caa-9c4b-8918314a1ead-istio-data\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8\" (UID: \"ede30242-d4ae-4caa-9c4b-8918314a1ead\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8"
Oct 31 19:51:36.624361 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:36.622156    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ede30242-d4ae-4caa-9c4b-8918314a1ead-istio-token\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8\" (UID: \"ede30242-d4ae-4caa-9c4b-8918314a1ead\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8"
Oct 31 19:51:36.624361 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:36.622206    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8gnc\" (UniqueName: \"kubernetes.io/projected/ede30242-d4ae-4caa-9c4b-8918314a1ead-kube-api-access-x8gnc\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8\" (UID: \"ede30242-d4ae-4caa-9c4b-8918314a1ead\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8"
Oct 31 19:51:36.624361 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:36.622319    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede30242-d4ae-4caa-9c4b-8918314a1ead-credential-socket\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8\" (UID: \"ede30242-d4ae-4caa-9c4b-8918314a1ead\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8"
Oct 31 19:51:36.624361 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:36.622371    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ede30242-d4ae-4caa-9c4b-8918314a1ead-istio-envoy\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8\" (UID: \"ede30242-d4ae-4caa-9c4b-8918314a1ead\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8"
Oct 31 19:51:36.624361 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:36.622409    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ede30242-d4ae-4caa-9c4b-8918314a1ead-istiod-ca-cert\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8\" (UID: \"ede30242-d4ae-4caa-9c4b-8918314a1ead\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8"
Oct 31 19:51:36.624361 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:36.623344    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ede30242-d4ae-4caa-9c4b-8918314a1ead-istiod-ca-cert\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8\" (UID: \"ede30242-d4ae-4caa-9c4b-8918314a1ead\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8"
Oct 31 19:51:36.624361 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:36.623694    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede30242-d4ae-4caa-9c4b-8918314a1ead-workload-socket\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8\" (UID: \"ede30242-d4ae-4caa-9c4b-8918314a1ead\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8"
Oct 31 19:51:36.624361 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:36.624023    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ede30242-d4ae-4caa-9c4b-8918314a1ead-workload-certs\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8\" (UID: \"ede30242-d4ae-4caa-9c4b-8918314a1ead\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8"
Oct 31 19:51:36.625299 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:36.624768    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede30242-d4ae-4caa-9c4b-8918314a1ead-credential-socket\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8\" (UID: \"ede30242-d4ae-4caa-9c4b-8918314a1ead\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8"
Oct 31 19:51:36.625299 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:36.625039    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ede30242-d4ae-4caa-9c4b-8918314a1ead-istio-data\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8\" (UID: \"ede30242-d4ae-4caa-9c4b-8918314a1ead\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8"
Oct 31 19:51:36.659360 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:36.649180    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ede30242-d4ae-4caa-9c4b-8918314a1ead-istio-token\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8\" (UID: \"ede30242-d4ae-4caa-9c4b-8918314a1ead\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8"
Oct 31 19:51:36.659360 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:36.649475    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ede30242-d4ae-4caa-9c4b-8918314a1ead-istio-envoy\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8\" (UID: \"ede30242-d4ae-4caa-9c4b-8918314a1ead\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8"
Oct 31 19:51:36.659360 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:36.653935    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ede30242-d4ae-4caa-9c4b-8918314a1ead-istio-podinfo\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8\" (UID: \"ede30242-d4ae-4caa-9c4b-8918314a1ead\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8"
Oct 31 19:51:36.659360 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:36.654377    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8gnc\" (UniqueName: \"kubernetes.io/projected/ede30242-d4ae-4caa-9c4b-8918314a1ead-kube-api-access-x8gnc\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8\" (UID: \"ede30242-d4ae-4caa-9c4b-8918314a1ead\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8"
Oct 31 19:51:36.669581 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:36.669552    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8"
Oct 31 19:51:36.929971 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:36.929931    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:37.014294 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:37.013637    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8"]
Oct 31 19:51:37.014294 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:51:37.014151    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podede30242_d4ae_4caa_9c4b_8918314a1ead.slice/crio-af3f9275b982225bbbb8387f3810b4073305442900f79b67a498b1965515c87a WatchSource:0}: Error finding container af3f9275b982225bbbb8387f3810b4073305442900f79b67a498b1965515c87a: Status 404 returned error can't find the container with id af3f9275b982225bbbb8387f3810b4073305442900f79b67a498b1965515c87a
Oct 31 19:51:37.019738 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:37.019697    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:51:37.020045 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:37.020015    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:51:37.020189 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:37.020116    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:51:37.044483 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:37.044450    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:37.309536 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:37.309464    4645 generic.go:356] "Generic (PLEG): container finished" podID="d7fbab67-7316-40e5-ab5a-e848cd654c50" containerID="e7153bfff50994acdd846d9e5fa22f406abf093b81322f4bace69fe5366fe731" exitCode=0
Oct 31 19:51:37.309536 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:37.309501    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d45f6c-55b6bc85f9-776mg" event={"ID":"d7fbab67-7316-40e5-ab5a-e848cd654c50","Type":"ContainerDied","Data":"e7153bfff50994acdd846d9e5fa22f406abf093b81322f4bace69fe5366fe731"}
Oct 31 19:51:37.311994 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:37.311963    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8" event={"ID":"ede30242-d4ae-4caa-9c4b-8918314a1ead","Type":"ContainerStarted","Data":"22276da2e572fbe9afeabcf73cd849af3e094556271c14915e031ea48e2f852a"}
Oct 31 19:51:37.312182 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:37.312013    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8" event={"ID":"ede30242-d4ae-4caa-9c4b-8918314a1ead","Type":"ContainerStarted","Data":"af3f9275b982225bbbb8387f3810b4073305442900f79b67a498b1965515c87a"}
Oct 31 19:51:37.340406 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:37.340353    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8" podStartSLOduration=1.340316914 podStartE2EDuration="1.340316914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:51:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:51:37.333301539 +0000 UTC m=+1927.906535670" watchObservedRunningTime="2026-10-31 19:51:37.340316914 +0000 UTC m=+1927.913551046"
Oct 31 19:51:37.358996 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:37.358972    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d45f6c-55b6bc85f9-776mg"
Oct 31 19:51:37.437651 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:37.437615    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7fbab67-7316-40e5-ab5a-e848cd654c50-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7fbab67-7316-40e5-ab5a-e848cd654c50-plugins-config-volume\") pod \"d7fbab67-7316-40e5-ab5a-e848cd654c50\" (UID: \"d7fbab67-7316-40e5-ab5a-e848cd654c50\") "
Oct 31 19:51:37.437852 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:37.437684    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7fbab67-7316-40e5-ab5a-e848cd654c50-kube-api-access-xsqdv\" (UniqueName: \"kubernetes.io/projected/d7fbab67-7316-40e5-ab5a-e848cd654c50-kube-api-access-xsqdv\") pod \"d7fbab67-7316-40e5-ab5a-e848cd654c50\" (UID: \"d7fbab67-7316-40e5-ab5a-e848cd654c50\") "
Oct 31 19:51:37.438600 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:37.438568    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7fbab67-7316-40e5-ab5a-e848cd654c50-plugins-config-volume" pod "d7fbab67-7316-40e5-ab5a-e848cd654c50" (UID: "d7fbab67-7316-40e5-ab5a-e848cd654c50"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:51:37.439281 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:37.439259    4645 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7fbab67-7316-40e5-ab5a-e848cd654c50-plugins-config-volume\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:37.441568 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:37.441544    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7fbab67-7316-40e5-ab5a-e848cd654c50-kube-api-access-xsqdv" pod "d7fbab67-7316-40e5-ab5a-e848cd654c50" (UID: "d7fbab67-7316-40e5-ab5a-e848cd654c50"). InnerVolumeSpecName "kube-api-access-xsqdv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:51:37.541545 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:37.541008    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsqdv\" (UniqueName: \"kubernetes.io/projected/d7fbab67-7316-40e5-ab5a-e848cd654c50-kube-api-access-xsqdv\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:37.669829 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:37.669803    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8"
Oct 31 19:51:37.741223 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:37.741187    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:38.321528 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:38.321499    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d45f6c-55b6bc85f9-776mg"
Oct 31 19:51:38.322038 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:38.321520    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d45f6c-55b6bc85f9-776mg" event={"ID":"d7fbab67-7316-40e5-ab5a-e848cd654c50","Type":"ContainerDied","Data":"29f6b7579270e50a5c84056d49a2202a943cf25e307dbf2bd1949c6caad1541b"}
Oct 31 19:51:38.322038 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:38.321628    4645 scope.go:122] "RemoveContainer" containerID="e7153bfff50994acdd846d9e5fa22f406abf093b81322f4bace69fe5366fe731"
Oct 31 19:51:38.340597 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:38.340575    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d45f6c-55b6bc85f9-776mg"]
Oct 31 19:51:38.344622 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:38.344598    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d45f6c-55b6bc85f9-776mg"]
Oct 31 19:51:38.495927 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:38.495888    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:38.677393 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:38.677359    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8"
Oct 31 19:51:39.187685 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:39.187651    4645 patch_prober.go:29] interesting pod/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 19:51:39.187837 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:39.187713    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq" podUID="37f24530-822f-4186-acfa-a7912c710c56" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 19:51:39.252445 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:39.252417    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8dcfa8-86d859f49f-j5f54"]
Oct 31 19:51:39.270507 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:39.270481    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8dcfa8-68fb9b44f5-6dmlv"]
Oct 31 19:51:39.333311 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:39.333278    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:39.337785 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:39.337759    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8"
Oct 31 19:51:39.339073 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:39.339045    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8"
Oct 31 19:51:39.439838 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:39.439758    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:39.682349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:39.677621    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7fbab67-7316-40e5-ab5a-e848cd654c50" path="/var/lib/kubelet/pods/d7fbab67-7316-40e5-ab5a-e848cd654c50/volumes"
Oct 31 19:51:39.795870 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:39.795795    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:39.917859 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:39.917817    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:40.351822 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:40.351738    4645 generic.go:356] "Generic (PLEG): container finished" podID="788ad5fd-ae6a-4953-a113-1357e62f2b0b" containerID="930b509ed77244d1252eeab03e45e11aefdee680e2b7680444ab0d49871fd277" exitCode=0
Oct 31 19:51:40.352229 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:40.351819    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1e8cf0-56bb54564c-t5hz7" event={"ID":"788ad5fd-ae6a-4953-a113-1357e62f2b0b","Type":"ContainerDied","Data":"930b509ed77244d1252eeab03e45e11aefdee680e2b7680444ab0d49871fd277"}
Oct 31 19:51:40.354133 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:40.354108    4645 generic.go:356] "Generic (PLEG): container finished" podID="37f24530-822f-4186-acfa-a7912c710c56" containerID="5fa2823567167438a072ca570e805fcb42cf724e81caee7226a2805694a07e70" exitCode=0
Oct 31 19:51:40.354259 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:40.354145    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq" event={"ID":"37f24530-822f-4186-acfa-a7912c710c56","Type":"ContainerDied","Data":"5fa2823567167438a072ca570e805fcb42cf724e81caee7226a2805694a07e70"}
Oct 31 19:51:40.356840 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:40.356815    4645 generic.go:356] "Generic (PLEG): container finished" podID="b73e8ddc-c45d-4537-a4c4-f22747469ca8" containerID="c5b432470850c5b9ec1435a151d8d255b18704cc2e8d732ec21f426793c16b00" exitCode=0
Oct 31 19:51:40.357051 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:40.357022    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp" event={"ID":"b73e8ddc-c45d-4537-a4c4-f22747469ca8","Type":"ContainerDied","Data":"c5b432470850c5b9ec1435a151d8d255b18704cc2e8d732ec21f426793c16b00"}
Oct 31 19:51:40.357134 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:40.357086    4645 scope.go:122] "RemoveContainer" containerID="dad871fc77704439f1b4795a36263cdb6a1e934a5c0fb333d92e9145bb9b7db1"
Oct 31 19:51:40.405804 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:40.400238    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1e8cf0-56bb54564c-t5hz7"
Oct 31 19:51:40.476182 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:40.476146    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp"
Oct 31 19:51:40.480623 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:40.480601    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq"
Oct 31 19:51:40.511419 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:40.511394    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/788ad5fd-ae6a-4953-a113-1357e62f2b0b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/788ad5fd-ae6a-4953-a113-1357e62f2b0b-plugins-config-volume\") pod \"788ad5fd-ae6a-4953-a113-1357e62f2b0b\" (UID: \"788ad5fd-ae6a-4953-a113-1357e62f2b0b\") "
Oct 31 19:51:40.511555 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:40.511460    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/788ad5fd-ae6a-4953-a113-1357e62f2b0b-kube-api-access-2vxd9\" (UniqueName: \"kubernetes.io/projected/788ad5fd-ae6a-4953-a113-1357e62f2b0b-kube-api-access-2vxd9\") pod \"788ad5fd-ae6a-4953-a113-1357e62f2b0b\" (UID: \"788ad5fd-ae6a-4953-a113-1357e62f2b0b\") "
Oct 31 19:51:40.511779 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:40.511756    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/788ad5fd-ae6a-4953-a113-1357e62f2b0b-plugins-config-volume" pod "788ad5fd-ae6a-4953-a113-1357e62f2b0b" (UID: "788ad5fd-ae6a-4953-a113-1357e62f2b0b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:51:40.512322 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:40.512301    4645 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/788ad5fd-ae6a-4953-a113-1357e62f2b0b-plugins-config-volume\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:40.514056 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:40.514036    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/788ad5fd-ae6a-4953-a113-1357e62f2b0b-kube-api-access-2vxd9" pod "788ad5fd-ae6a-4953-a113-1357e62f2b0b" (UID: "788ad5fd-ae6a-4953-a113-1357e62f2b0b"). InnerVolumeSpecName "kube-api-access-2vxd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:51:40.612822 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:40.612743    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37f24530-822f-4186-acfa-a7912c710c56-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37f24530-822f-4186-acfa-a7912c710c56-istio-envoy\") pod \"37f24530-822f-4186-acfa-a7912c710c56\" (UID: \"37f24530-822f-4186-acfa-a7912c710c56\") "
Oct 31 19:51:40.612822 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:40.612792    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b73e8ddc-c45d-4537-a4c4-f22747469ca8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b73e8ddc-c45d-4537-a4c4-f22747469ca8-istiod-ca-cert\") pod \"b73e8ddc-c45d-4537-a4c4-f22747469ca8\" (UID: \"b73e8ddc-c45d-4537-a4c4-f22747469ca8\") "
Oct 31 19:51:40.613047 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:40.612855    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b73e8ddc-c45d-4537-a4c4-f22747469ca8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b73e8ddc-c45d-4537-a4c4-f22747469ca8-credential-socket\") pod \"b73e8ddc-c45d-4537-a4c4-f22747469ca8\" (UID: \"b73e8ddc-c45d-4537-a4c4-f22747469ca8\") "
Oct 31 19:51:40.613047 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:40.612894    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37f24530-822f-4186-acfa-a7912c710c56-istio-token\" (UniqueName: \"kubernetes.io/projected/37f24530-822f-4186-acfa-a7912c710c56-istio-token\") pod \"37f24530-822f-4186-acfa-a7912c710c56\" (UID: \"37f24530-822f-4186-acfa-a7912c710c56\") "
Oct 31 19:51:40.613047 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:40.612956    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37f24530-822f-4186-acfa-a7912c710c56-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37f24530-822f-4186-acfa-a7912c710c56-istio-data\") pod \"37f24530-822f-4186-acfa-a7912c710c56\" (UID: \"37f24530-822f-4186-acfa-a7912c710c56\") "
Oct 31 19:51:40.613047 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:40.613002    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b73e8ddc-c45d-4537-a4c4-f22747469ca8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b73e8ddc-c45d-4537-a4c4-f22747469ca8-istio-data\") pod \"b73e8ddc-c45d-4537-a4c4-f22747469ca8\" (UID: \"b73e8ddc-c45d-4537-a4c4-f22747469ca8\") "
Oct 31 19:51:40.613278 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:40.613051    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b73e8ddc-c45d-4537-a4c4-f22747469ca8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b73e8ddc-c45d-4537-a4c4-f22747469ca8-workload-certs\") pod \"b73e8ddc-c45d-4537-a4c4-f22747469ca8\" (UID: \"b73e8ddc-c45d-4537-a4c4-f22747469ca8\") "
Oct 31 19:51:40.613278 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:40.613073    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37f24530-822f-4186-acfa-a7912c710c56-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37f24530-822f-4186-acfa-a7912c710c56-workload-socket\") pod \"37f24530-822f-4186-acfa-a7912c710c56\" (UID: \"37f24530-822f-4186-acfa-a7912c710c56\") "
Oct 31 19:51:40.613278 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:40.613101    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b73e8ddc-c45d-4537-a4c4-f22747469ca8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b73e8ddc-c45d-4537-a4c4-f22747469ca8-istio-podinfo\") pod \"b73e8ddc-c45d-4537-a4c4-f22747469ca8\" (UID: \"b73e8ddc-c45d-4537-a4c4-f22747469ca8\") "
Oct 31 19:51:40.613278 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:40.613177    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b73e8ddc-c45d-4537-a4c4-f22747469ca8-credential-socket" pod "b73e8ddc-c45d-4537-a4c4-f22747469ca8" (UID: "b73e8ddc-c45d-4537-a4c4-f22747469ca8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:51:40.613278 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:40.613182    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37f24530-822f-4186-acfa-a7912c710c56-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37f24530-822f-4186-acfa-a7912c710c56-credential-socket\") pod \"37f24530-822f-4186-acfa-a7912c710c56\" (UID: \"37f24530-822f-4186-acfa-a7912c710c56\") "
Oct 31 19:51:40.613278 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:40.613226    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b73e8ddc-c45d-4537-a4c4-f22747469ca8-istiod-ca-cert" pod "b73e8ddc-c45d-4537-a4c4-f22747469ca8" (UID: "b73e8ddc-c45d-4537-a4c4-f22747469ca8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:51:40.613278 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:40.613257    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37f24530-822f-4186-acfa-a7912c710c56-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37f24530-822f-4186-acfa-a7912c710c56-istiod-ca-cert\") pod \"37f24530-822f-4186-acfa-a7912c710c56\" (UID: \"37f24530-822f-4186-acfa-a7912c710c56\") "
Oct 31 19:51:40.613658 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:40.613304    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b73e8ddc-c45d-4537-a4c4-f22747469ca8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b73e8ddc-c45d-4537-a4c4-f22747469ca8-workload-socket\") pod \"b73e8ddc-c45d-4537-a4c4-f22747469ca8\" (UID: \"b73e8ddc-c45d-4537-a4c4-f22747469ca8\") "
Oct 31 19:51:40.613658 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:40.613420    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37f24530-822f-4186-acfa-a7912c710c56-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37f24530-822f-4186-acfa-a7912c710c56-workload-certs\") pod \"37f24530-822f-4186-acfa-a7912c710c56\" (UID: \"37f24530-822f-4186-acfa-a7912c710c56\") "
Oct 31 19:51:40.613658 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:40.613488    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/37f24530-822f-4186-acfa-a7912c710c56-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37f24530-822f-4186-acfa-a7912c710c56-istio-podinfo\") pod \"37f24530-822f-4186-acfa-a7912c710c56\" (UID: \"37f24530-822f-4186-acfa-a7912c710c56\") "
Oct 31 19:51:40.613658 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:40.613526    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b73e8ddc-c45d-4537-a4c4-f22747469ca8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b73e8ddc-c45d-4537-a4c4-f22747469ca8-istio-envoy\") pod \"b73e8ddc-c45d-4537-a4c4-f22747469ca8\" (UID: \"b73e8ddc-c45d-4537-a4c4-f22747469ca8\") "
Oct 31 19:51:40.613658 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:40.613602    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b73e8ddc-c45d-4537-a4c4-f22747469ca8-kube-api-access-9gj68\" (UniqueName: \"kubernetes.io/projected/b73e8ddc-c45d-4537-a4c4-f22747469ca8-kube-api-access-9gj68\") pod \"b73e8ddc-c45d-4537-a4c4-f22747469ca8\" (UID: \"b73e8ddc-c45d-4537-a4c4-f22747469ca8\") "
Oct 31 19:51:40.613658 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:40.613646    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b73e8ddc-c45d-4537-a4c4-f22747469ca8-istio-token\" (UniqueName: \"kubernetes.io/projected/b73e8ddc-c45d-4537-a4c4-f22747469ca8-istio-token\") pod \"b73e8ddc-c45d-4537-a4c4-f22747469ca8\" (UID: \"b73e8ddc-c45d-4537-a4c4-f22747469ca8\") "
Oct 31 19:51:40.613999 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:40.613420    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37f24530-822f-4186-acfa-a7912c710c56-credential-socket" pod "37f24530-822f-4186-acfa-a7912c710c56" (UID: "37f24530-822f-4186-acfa-a7912c710c56"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:51:40.613999 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:40.613433    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37f24530-822f-4186-acfa-a7912c710c56-istio-data" pod "37f24530-822f-4186-acfa-a7912c710c56" (UID: "37f24530-822f-4186-acfa-a7912c710c56"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:51:40.613999 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:40.613677    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37f24530-822f-4186-acfa-a7912c710c56-kube-api-access-9jjrm\" (UniqueName: \"kubernetes.io/projected/37f24530-822f-4186-acfa-a7912c710c56-kube-api-access-9jjrm\") pod \"37f24530-822f-4186-acfa-a7912c710c56\" (UID: \"37f24530-822f-4186-acfa-a7912c710c56\") "
Oct 31 19:51:40.614612 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:40.614587    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37f24530-822f-4186-acfa-a7912c710c56-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:40.614735 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:40.614617    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b73e8ddc-c45d-4537-a4c4-f22747469ca8-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:40.614735 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:40.614632    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b73e8ddc-c45d-4537-a4c4-f22747469ca8-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:40.614735 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:40.614646    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vxd9\" (UniqueName: \"kubernetes.io/projected/788ad5fd-ae6a-4953-a113-1357e62f2b0b-kube-api-access-2vxd9\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:40.614735 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:40.614660    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37f24530-822f-4186-acfa-a7912c710c56-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:40.614735 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:40.613556    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b73e8ddc-c45d-4537-a4c4-f22747469ca8-workload-certs" pod "b73e8ddc-c45d-4537-a4c4-f22747469ca8" (UID: "b73e8ddc-c45d-4537-a4c4-f22747469ca8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:51:40.614735 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:40.613620    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37f24530-822f-4186-acfa-a7912c710c56-workload-socket" pod "37f24530-822f-4186-acfa-a7912c710c56" (UID: "37f24530-822f-4186-acfa-a7912c710c56"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:51:40.614735 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:40.613654    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b73e8ddc-c45d-4537-a4c4-f22747469ca8-istio-data" pod "b73e8ddc-c45d-4537-a4c4-f22747469ca8" (UID: "b73e8ddc-c45d-4537-a4c4-f22747469ca8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:51:40.614735 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:40.613902    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37f24530-822f-4186-acfa-a7912c710c56-workload-certs" pod "37f24530-822f-4186-acfa-a7912c710c56" (UID: "37f24530-822f-4186-acfa-a7912c710c56"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:51:40.614735 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:40.614274    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b73e8ddc-c45d-4537-a4c4-f22747469ca8-workload-socket" pod "b73e8ddc-c45d-4537-a4c4-f22747469ca8" (UID: "b73e8ddc-c45d-4537-a4c4-f22747469ca8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:51:40.614735 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:40.614521    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37f24530-822f-4186-acfa-a7912c710c56-istiod-ca-cert" pod "37f24530-822f-4186-acfa-a7912c710c56" (UID: "37f24530-822f-4186-acfa-a7912c710c56"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:51:40.615570 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:40.615548    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37f24530-822f-4186-acfa-a7912c710c56-istio-envoy" pod "37f24530-822f-4186-acfa-a7912c710c56" (UID: "37f24530-822f-4186-acfa-a7912c710c56"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:51:40.616720 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:40.616695    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/37f24530-822f-4186-acfa-a7912c710c56-istio-podinfo" pod "37f24530-822f-4186-acfa-a7912c710c56" (UID: "37f24530-822f-4186-acfa-a7912c710c56"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:51:40.616804 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:40.616732    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b73e8ddc-c45d-4537-a4c4-f22747469ca8-istio-podinfo" pod "b73e8ddc-c45d-4537-a4c4-f22747469ca8" (UID: "b73e8ddc-c45d-4537-a4c4-f22747469ca8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:51:40.616848 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:40.616810    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b73e8ddc-c45d-4537-a4c4-f22747469ca8-istio-envoy" pod "b73e8ddc-c45d-4537-a4c4-f22747469ca8" (UID: "b73e8ddc-c45d-4537-a4c4-f22747469ca8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:51:40.616966 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:40.616945    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b73e8ddc-c45d-4537-a4c4-f22747469ca8-istio-token" pod "b73e8ddc-c45d-4537-a4c4-f22747469ca8" (UID: "b73e8ddc-c45d-4537-a4c4-f22747469ca8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:51:40.617062 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:40.616972    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b73e8ddc-c45d-4537-a4c4-f22747469ca8-kube-api-access-9gj68" pod "b73e8ddc-c45d-4537-a4c4-f22747469ca8" (UID: "b73e8ddc-c45d-4537-a4c4-f22747469ca8"). InnerVolumeSpecName "kube-api-access-9gj68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:51:40.617556 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:40.617531    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37f24530-822f-4186-acfa-a7912c710c56-kube-api-access-9jjrm" pod "37f24530-822f-4186-acfa-a7912c710c56" (UID: "37f24530-822f-4186-acfa-a7912c710c56"). InnerVolumeSpecName "kube-api-access-9jjrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:51:40.618551 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:40.618511    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37f24530-822f-4186-acfa-a7912c710c56-istio-token" pod "37f24530-822f-4186-acfa-a7912c710c56" (UID: "37f24530-822f-4186-acfa-a7912c710c56"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:51:40.716061 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:40.716033    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37f24530-822f-4186-acfa-a7912c710c56-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:40.716061 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:40.716063    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b73e8ddc-c45d-4537-a4c4-f22747469ca8-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:40.716269 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:40.716076    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b73e8ddc-c45d-4537-a4c4-f22747469ca8-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:40.716269 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:40.716090    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37f24530-822f-4186-acfa-a7912c710c56-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:40.716269 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:40.716102    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b73e8ddc-c45d-4537-a4c4-f22747469ca8-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:40.716269 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:40.716116    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37f24530-822f-4186-acfa-a7912c710c56-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:40.716269 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:40.716128    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b73e8ddc-c45d-4537-a4c4-f22747469ca8-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:40.716269 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:40.716141    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37f24530-822f-4186-acfa-a7912c710c56-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:40.716269 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:40.716161    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37f24530-822f-4186-acfa-a7912c710c56-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:40.716269 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:40.716174    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b73e8ddc-c45d-4537-a4c4-f22747469ca8-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:40.716269 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:40.716187    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gj68\" (UniqueName: \"kubernetes.io/projected/b73e8ddc-c45d-4537-a4c4-f22747469ca8-kube-api-access-9gj68\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:40.716269 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:40.716201    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b73e8ddc-c45d-4537-a4c4-f22747469ca8-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:40.716269 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:40.716214    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jjrm\" (UniqueName: \"kubernetes.io/projected/37f24530-822f-4186-acfa-a7912c710c56-kube-api-access-9jjrm\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:40.716269 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:40.716227    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37f24530-822f-4186-acfa-a7912c710c56-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:40.840674 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:40.840635    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:41.288324 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:41.288169    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:41.367209 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:41.367144    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq"
Oct 31 19:51:41.367640 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:41.367144    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq" event={"ID":"37f24530-822f-4186-acfa-a7912c710c56","Type":"ContainerDied","Data":"556a80c364930ecf78a02105780d7b7ee1883f649979a2a96eb7e2f5bbad38a1"}
Oct 31 19:51:41.367640 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:41.367311    4645 scope.go:122] "RemoveContainer" containerID="5fa2823567167438a072ca570e805fcb42cf724e81caee7226a2805694a07e70"
Oct 31 19:51:41.369148 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:41.369019    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp"
Oct 31 19:51:41.369148 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:41.369099    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp" event={"ID":"b73e8ddc-c45d-4537-a4c4-f22747469ca8","Type":"ContainerDied","Data":"461d2269ee4e677af6a3a40392c9f1e43071098f2c90cb583d66f6f51f356dc0"}
Oct 31 19:51:41.373479 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:41.373458    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1e8cf0-56bb54564c-t5hz7"
Oct 31 19:51:41.373597 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:41.373542    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1e8cf0-56bb54564c-t5hz7" event={"ID":"788ad5fd-ae6a-4953-a113-1357e62f2b0b","Type":"ContainerDied","Data":"86e9445306c8e88a8bbb5bfd1c80d849e12382bc3eb62a0131dd84d1a2d887cd"}
Oct 31 19:51:41.383224 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:41.383208    4645 scope.go:122] "RemoveContainer" containerID="c5b432470850c5b9ec1435a151d8d255b18704cc2e8d732ec21f426793c16b00"
Oct 31 19:51:41.394778 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:41.394761    4645 scope.go:122] "RemoveContainer" containerID="930b509ed77244d1252eeab03e45e11aefdee680e2b7680444ab0d49871fd277"
Oct 31 19:51:41.414904 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:41.414880    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:41.416688 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:41.416665    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp"]
Oct 31 19:51:41.421829 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:41.421805    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-6vqjp"]
Oct 31 19:51:41.439714 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:41.439695    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq"]
Oct 31 19:51:41.443885 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:41.443864    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-d45f6c-openshift-default-7b5486ffcc-q2mtq"]
Oct 31 19:51:41.455589 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:41.455546    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1e8cf0-56bb54564c-t5hz7"]
Oct 31 19:51:41.486502 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:41.477104    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1e8cf0-56bb54564c-t5hz7"]
Oct 31 19:51:41.706356 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:41.706178    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37f24530-822f-4186-acfa-a7912c710c56" path="/var/lib/kubelet/pods/37f24530-822f-4186-acfa-a7912c710c56/volumes"
Oct 31 19:51:41.707366 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:41.707312    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="788ad5fd-ae6a-4953-a113-1357e62f2b0b" path="/var/lib/kubelet/pods/788ad5fd-ae6a-4953-a113-1357e62f2b0b/volumes"
Oct 31 19:51:41.708356 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:41.708315    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b73e8ddc-c45d-4537-a4c4-f22747469ca8" path="/var/lib/kubelet/pods/b73e8ddc-c45d-4537-a4c4-f22747469ca8/volumes"
Oct 31 19:51:41.846386 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:41.846350    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:42.068000 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:42.067881    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-cd9729/auth-test-model-cd9729-kserve-65f895f85c-pcprs"]
Oct 31 19:51:42.068987 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:42.068958    4645 cpu_manager.go:414] "containerMap: removing container" podUID="d7fbab67-7316-40e5-ab5a-e848cd654c50" containerName="payload-pre-processing"
Oct 31 19:51:42.068987 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:42.068984    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7fbab67-7316-40e5-ab5a-e848cd654c50" containerName="payload-pre-processing"
Oct 31 19:51:42.069190 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:42.069022    4645 cpu_manager.go:414] "containerMap: removing container" podUID="b73e8ddc-c45d-4537-a4c4-f22747469ca8" containerName="istio-proxy"
Oct 31 19:51:42.069190 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:42.069030    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b73e8ddc-c45d-4537-a4c4-f22747469ca8" containerName="istio-proxy"
Oct 31 19:51:42.069190 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:42.069052    4645 cpu_manager.go:414] "containerMap: removing container" podUID="b73e8ddc-c45d-4537-a4c4-f22747469ca8" containerName="istio-proxy"
Oct 31 19:51:42.069190 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:42.069060    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b73e8ddc-c45d-4537-a4c4-f22747469ca8" containerName="istio-proxy"
Oct 31 19:51:42.069190 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:42.069103    4645 cpu_manager.go:414] "containerMap: removing container" podUID="37f24530-822f-4186-acfa-a7912c710c56" containerName="istio-proxy"
Oct 31 19:51:42.069190 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:42.069110    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37f24530-822f-4186-acfa-a7912c710c56" containerName="istio-proxy"
Oct 31 19:51:42.069190 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:42.069128    4645 cpu_manager.go:414] "containerMap: removing container" podUID="788ad5fd-ae6a-4953-a113-1357e62f2b0b" containerName="payload-processing"
Oct 31 19:51:42.069190 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:42.069136    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="788ad5fd-ae6a-4953-a113-1357e62f2b0b" containerName="payload-processing"
Oct 31 19:51:42.069623 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:42.069302    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="76b3b657-6b14-43ff-b7ce-1e9e3b615141" containerName="istio-proxy"
Oct 31 19:51:42.069623 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:42.069321    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="37f24530-822f-4186-acfa-a7912c710c56" containerName="istio-proxy"
Oct 31 19:51:42.069623 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:42.069347    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="788ad5fd-ae6a-4953-a113-1357e62f2b0b" containerName="payload-processing"
Oct 31 19:51:42.069623 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:42.069362    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7fbab67-7316-40e5-ab5a-e848cd654c50" containerName="payload-pre-processing"
Oct 31 19:51:42.069623 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:42.069376    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="b73e8ddc-c45d-4537-a4c4-f22747469ca8" containerName="istio-proxy"
Oct 31 19:51:42.069623 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:42.069386    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="b73e8ddc-c45d-4537-a4c4-f22747469ca8" containerName="istio-proxy"
Oct 31 19:51:42.075636 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:42.075613    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-cd9729/auth-test-model-cd9729-kserve-65f895f85c-pcprs"
Oct 31 19:51:42.082657 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:42.082633    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-cd9729/auth-test-model-cd9729-kserve-65f895f85c-pcprs"]
Oct 31 19:51:42.082803 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:42.082756    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-cd9729\"/\"auth-test-model-cd9729-kserve-self-signed-certs\""
Oct 31 19:51:42.082873 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:42.082782    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-cd9729\"/\"openshift-service-ca.crt\""
Oct 31 19:51:42.082940 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:42.082924    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-cd9729\"/\"default-dockercfg-fb9l5\""
Oct 31 19:51:42.083039 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:42.083018    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-cd9729\"/\"kube-root-ca.crt\""
Oct 31 19:51:42.127843 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:42.127821    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:42.176934 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:42.176908    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96edc0b2-28d0-4058-8aff-c0e179fe5f2e-model-cache\") pod \"auth-test-model-cd9729-kserve-65f895f85c-pcprs\" (UID: \"96edc0b2-28d0-4058-8aff-c0e179fe5f2e\") " pod="ai-tenant-e2e-shared-a-w1-cd9729/auth-test-model-cd9729-kserve-65f895f85c-pcprs"
Oct 31 19:51:42.177051 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:42.176942    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96edc0b2-28d0-4058-8aff-c0e179fe5f2e-dshm\") pod \"auth-test-model-cd9729-kserve-65f895f85c-pcprs\" (UID: \"96edc0b2-28d0-4058-8aff-c0e179fe5f2e\") " pod="ai-tenant-e2e-shared-a-w1-cd9729/auth-test-model-cd9729-kserve-65f895f85c-pcprs"
Oct 31 19:51:42.177051 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:42.176959    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96edc0b2-28d0-4058-8aff-c0e179fe5f2e-tls-certs\") pod \"auth-test-model-cd9729-kserve-65f895f85c-pcprs\" (UID: \"96edc0b2-28d0-4058-8aff-c0e179fe5f2e\") " pod="ai-tenant-e2e-shared-a-w1-cd9729/auth-test-model-cd9729-kserve-65f895f85c-pcprs"
Oct 31 19:51:42.177156 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:42.177137    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqcgj\" (UniqueName: \"kubernetes.io/projected/96edc0b2-28d0-4058-8aff-c0e179fe5f2e-kube-api-access-rqcgj\") pod \"auth-test-model-cd9729-kserve-65f895f85c-pcprs\" (UID: \"96edc0b2-28d0-4058-8aff-c0e179fe5f2e\") " pod="ai-tenant-e2e-shared-a-w1-cd9729/auth-test-model-cd9729-kserve-65f895f85c-pcprs"
Oct 31 19:51:42.177194 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:42.177176    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96edc0b2-28d0-4058-8aff-c0e179fe5f2e-tmp-dir\") pod \"auth-test-model-cd9729-kserve-65f895f85c-pcprs\" (UID: \"96edc0b2-28d0-4058-8aff-c0e179fe5f2e\") " pod="ai-tenant-e2e-shared-a-w1-cd9729/auth-test-model-cd9729-kserve-65f895f85c-pcprs"
Oct 31 19:51:42.177252 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:42.177235    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96edc0b2-28d0-4058-8aff-c0e179fe5f2e-home\") pod \"auth-test-model-cd9729-kserve-65f895f85c-pcprs\" (UID: \"96edc0b2-28d0-4058-8aff-c0e179fe5f2e\") " pod="ai-tenant-e2e-shared-a-w1-cd9729/auth-test-model-cd9729-kserve-65f895f85c-pcprs"
Oct 31 19:51:42.287360 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:42.283910    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96edc0b2-28d0-4058-8aff-c0e179fe5f2e-tmp-dir\") pod \"auth-test-model-cd9729-kserve-65f895f85c-pcprs\" (UID: \"96edc0b2-28d0-4058-8aff-c0e179fe5f2e\") " pod="ai-tenant-e2e-shared-a-w1-cd9729/auth-test-model-cd9729-kserve-65f895f85c-pcprs"
Oct 31 19:51:42.287360 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:42.284440    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96edc0b2-28d0-4058-8aff-c0e179fe5f2e-home\") pod \"auth-test-model-cd9729-kserve-65f895f85c-pcprs\" (UID: \"96edc0b2-28d0-4058-8aff-c0e179fe5f2e\") " pod="ai-tenant-e2e-shared-a-w1-cd9729/auth-test-model-cd9729-kserve-65f895f85c-pcprs"
Oct 31 19:51:42.287360 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:42.284956    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96edc0b2-28d0-4058-8aff-c0e179fe5f2e-home\") pod \"auth-test-model-cd9729-kserve-65f895f85c-pcprs\" (UID: \"96edc0b2-28d0-4058-8aff-c0e179fe5f2e\") " pod="ai-tenant-e2e-shared-a-w1-cd9729/auth-test-model-cd9729-kserve-65f895f85c-pcprs"
Oct 31 19:51:42.287360 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:42.284342    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96edc0b2-28d0-4058-8aff-c0e179fe5f2e-tmp-dir\") pod \"auth-test-model-cd9729-kserve-65f895f85c-pcprs\" (UID: \"96edc0b2-28d0-4058-8aff-c0e179fe5f2e\") " pod="ai-tenant-e2e-shared-a-w1-cd9729/auth-test-model-cd9729-kserve-65f895f85c-pcprs"
Oct 31 19:51:42.287360 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:42.285023    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96edc0b2-28d0-4058-8aff-c0e179fe5f2e-model-cache\") pod \"auth-test-model-cd9729-kserve-65f895f85c-pcprs\" (UID: \"96edc0b2-28d0-4058-8aff-c0e179fe5f2e\") " pod="ai-tenant-e2e-shared-a-w1-cd9729/auth-test-model-cd9729-kserve-65f895f85c-pcprs"
Oct 31 19:51:42.287360 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:42.285287    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96edc0b2-28d0-4058-8aff-c0e179fe5f2e-model-cache\") pod \"auth-test-model-cd9729-kserve-65f895f85c-pcprs\" (UID: \"96edc0b2-28d0-4058-8aff-c0e179fe5f2e\") " pod="ai-tenant-e2e-shared-a-w1-cd9729/auth-test-model-cd9729-kserve-65f895f85c-pcprs"
Oct 31 19:51:42.287360 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:42.285376    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96edc0b2-28d0-4058-8aff-c0e179fe5f2e-dshm\") pod \"auth-test-model-cd9729-kserve-65f895f85c-pcprs\" (UID: \"96edc0b2-28d0-4058-8aff-c0e179fe5f2e\") " pod="ai-tenant-e2e-shared-a-w1-cd9729/auth-test-model-cd9729-kserve-65f895f85c-pcprs"
Oct 31 19:51:42.287360 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:42.285402    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96edc0b2-28d0-4058-8aff-c0e179fe5f2e-tls-certs\") pod \"auth-test-model-cd9729-kserve-65f895f85c-pcprs\" (UID: \"96edc0b2-28d0-4058-8aff-c0e179fe5f2e\") " pod="ai-tenant-e2e-shared-a-w1-cd9729/auth-test-model-cd9729-kserve-65f895f85c-pcprs"
Oct 31 19:51:42.287360 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:42.285530    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqcgj\" (UniqueName: \"kubernetes.io/projected/96edc0b2-28d0-4058-8aff-c0e179fe5f2e-kube-api-access-rqcgj\") pod \"auth-test-model-cd9729-kserve-65f895f85c-pcprs\" (UID: \"96edc0b2-28d0-4058-8aff-c0e179fe5f2e\") " pod="ai-tenant-e2e-shared-a-w1-cd9729/auth-test-model-cd9729-kserve-65f895f85c-pcprs"
Oct 31 19:51:42.308913 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:42.308875    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96edc0b2-28d0-4058-8aff-c0e179fe5f2e-dshm\") pod \"auth-test-model-cd9729-kserve-65f895f85c-pcprs\" (UID: \"96edc0b2-28d0-4058-8aff-c0e179fe5f2e\") " pod="ai-tenant-e2e-shared-a-w1-cd9729/auth-test-model-cd9729-kserve-65f895f85c-pcprs"
Oct 31 19:51:42.334732 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:42.333655    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqcgj\" (UniqueName: \"kubernetes.io/projected/96edc0b2-28d0-4058-8aff-c0e179fe5f2e-kube-api-access-rqcgj\") pod \"auth-test-model-cd9729-kserve-65f895f85c-pcprs\" (UID: \"96edc0b2-28d0-4058-8aff-c0e179fe5f2e\") " pod="ai-tenant-e2e-shared-a-w1-cd9729/auth-test-model-cd9729-kserve-65f895f85c-pcprs"
Oct 31 19:51:42.345363 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:42.339249    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:42.345363 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:42.343910    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96edc0b2-28d0-4058-8aff-c0e179fe5f2e-tls-certs\") pod \"auth-test-model-cd9729-kserve-65f895f85c-pcprs\" (UID: \"96edc0b2-28d0-4058-8aff-c0e179fe5f2e\") " pod="ai-tenant-e2e-shared-a-w1-cd9729/auth-test-model-cd9729-kserve-65f895f85c-pcprs"
Oct 31 19:51:42.392757 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:42.392716    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-cd9729/auth-test-model-cd9729-kserve-65f895f85c-pcprs"
Oct 31 19:51:42.493955 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:42.493926    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:42.530371 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:42.530311    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8dcfa8-6c7cb4c95f-cws7q"]
Oct 31 19:51:42.530583 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:42.530560    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8dcfa8-6c7cb4c95f-cws7q" podUID="be41f707-30c3-49f1-91c6-bea2e72c172a" containerName="maas-api" containerID="cri-o://97254e810ef08368367cb3f4cb56afcd1c2227c7a7a5cf37093c12a9228dd64c" gracePeriod=30
Oct 31 19:51:42.605989 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:42.605891    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:42.679611 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:51:42.679581    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod96edc0b2_28d0_4058_8aff_c0e179fe5f2e.slice/crio-5d5ba015191ee8ae81c5cb298a225bcd10b7aaa7c500f7f367b350d8d2cd7a84 WatchSource:0}: Error finding container 5d5ba015191ee8ae81c5cb298a225bcd10b7aaa7c500f7f367b350d8d2cd7a84: Status 404 returned error can't find the container with id 5d5ba015191ee8ae81c5cb298a225bcd10b7aaa7c500f7f367b350d8d2cd7a84
Oct 31 19:51:42.680172 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:42.680137    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-cd9729/auth-test-model-cd9729-kserve-65f895f85c-pcprs"]
Oct 31 19:51:42.887408 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:42.887030    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:42.992418 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:42.992100    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8dcfa8-6c7cb4c95f-cws7q"
Oct 31 19:51:43.072978 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:43.072945    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:43.100453 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:43.100422    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be41f707-30c3-49f1-91c6-bea2e72c172a-kube-api-access-54dwh\" (UniqueName: \"kubernetes.io/projected/be41f707-30c3-49f1-91c6-bea2e72c172a-kube-api-access-54dwh\") pod \"be41f707-30c3-49f1-91c6-bea2e72c172a\" (UID: \"be41f707-30c3-49f1-91c6-bea2e72c172a\") "
Oct 31 19:51:43.100642 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:43.100469    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be41f707-30c3-49f1-91c6-bea2e72c172a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be41f707-30c3-49f1-91c6-bea2e72c172a-openshift-service-ca\") pod \"be41f707-30c3-49f1-91c6-bea2e72c172a\" (UID: \"be41f707-30c3-49f1-91c6-bea2e72c172a\") "
Oct 31 19:51:43.100642 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:43.100567    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be41f707-30c3-49f1-91c6-bea2e72c172a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/be41f707-30c3-49f1-91c6-bea2e72c172a-metrics-certs\") pod \"be41f707-30c3-49f1-91c6-bea2e72c172a\" (UID: \"be41f707-30c3-49f1-91c6-bea2e72c172a\") "
Oct 31 19:51:43.100764 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:43.100729    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be41f707-30c3-49f1-91c6-bea2e72c172a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be41f707-30c3-49f1-91c6-bea2e72c172a-maas-api-tls\") pod \"be41f707-30c3-49f1-91c6-bea2e72c172a\" (UID: \"be41f707-30c3-49f1-91c6-bea2e72c172a\") "
Oct 31 19:51:43.101284 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:43.100976    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be41f707-30c3-49f1-91c6-bea2e72c172a-openshift-service-ca" pod "be41f707-30c3-49f1-91c6-bea2e72c172a" (UID: "be41f707-30c3-49f1-91c6-bea2e72c172a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:51:43.101496 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:43.101463    4645 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be41f707-30c3-49f1-91c6-bea2e72c172a-openshift-service-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:43.102944 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:43.102922    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be41f707-30c3-49f1-91c6-bea2e72c172a-maas-api-tls" pod "be41f707-30c3-49f1-91c6-bea2e72c172a" (UID: "be41f707-30c3-49f1-91c6-bea2e72c172a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:51:43.103166 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:43.103138    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be41f707-30c3-49f1-91c6-bea2e72c172a-metrics-certs" pod "be41f707-30c3-49f1-91c6-bea2e72c172a" (UID: "be41f707-30c3-49f1-91c6-bea2e72c172a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:51:43.103166 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:43.103154    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be41f707-30c3-49f1-91c6-bea2e72c172a-kube-api-access-54dwh" pod "be41f707-30c3-49f1-91c6-bea2e72c172a" (UID: "be41f707-30c3-49f1-91c6-bea2e72c172a"). InnerVolumeSpecName "kube-api-access-54dwh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:51:43.215185 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:43.214695    4645 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be41f707-30c3-49f1-91c6-bea2e72c172a-metrics-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:43.215185 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:43.214734    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be41f707-30c3-49f1-91c6-bea2e72c172a-maas-api-tls\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:43.215185 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:43.214810    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54dwh\" (UniqueName: \"kubernetes.io/projected/be41f707-30c3-49f1-91c6-bea2e72c172a-kube-api-access-54dwh\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:43.216975 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:43.216553    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:43.398405 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:43.398363    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-cd9729/auth-test-model-cd9729-kserve-65f895f85c-pcprs" event={"ID":"96edc0b2-28d0-4058-8aff-c0e179fe5f2e","Type":"ContainerStarted","Data":"18e085ad912a1794d78b216d685983baa6868317b6299200b768d5b15dc54cec"}
Oct 31 19:51:43.398837 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:43.398417    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-cd9729/auth-test-model-cd9729-kserve-65f895f85c-pcprs" event={"ID":"96edc0b2-28d0-4058-8aff-c0e179fe5f2e","Type":"ContainerStarted","Data":"5d5ba015191ee8ae81c5cb298a225bcd10b7aaa7c500f7f367b350d8d2cd7a84"}
Oct 31 19:51:43.399850 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:43.399825    4645 generic.go:356] "Generic (PLEG): container finished" podID="be41f707-30c3-49f1-91c6-bea2e72c172a" containerID="97254e810ef08368367cb3f4cb56afcd1c2227c7a7a5cf37093c12a9228dd64c" exitCode=0
Oct 31 19:51:43.399964 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:43.399863    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8dcfa8-6c7cb4c95f-cws7q"
Oct 31 19:51:43.399964 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:43.399903    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8dcfa8-6c7cb4c95f-cws7q" event={"ID":"be41f707-30c3-49f1-91c6-bea2e72c172a","Type":"ContainerDied","Data":"97254e810ef08368367cb3f4cb56afcd1c2227c7a7a5cf37093c12a9228dd64c"}
Oct 31 19:51:43.399964 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:43.399953    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8dcfa8-6c7cb4c95f-cws7q" event={"ID":"be41f707-30c3-49f1-91c6-bea2e72c172a","Type":"ContainerDied","Data":"701ac01b7d0826ad77372bb75eea56acf3fe2f9466544c5df01486dd79bcb11f"}
Oct 31 19:51:43.400072 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:43.399976    4645 scope.go:122] "RemoveContainer" containerID="97254e810ef08368367cb3f4cb56afcd1c2227c7a7a5cf37093c12a9228dd64c"
Oct 31 19:51:43.411897 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:43.411876    4645 scope.go:122] "RemoveContainer" containerID="97254e810ef08368367cb3f4cb56afcd1c2227c7a7a5cf37093c12a9228dd64c"
Oct 31 19:51:43.412199 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:51:43.412174    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97254e810ef08368367cb3f4cb56afcd1c2227c7a7a5cf37093c12a9228dd64c\": container with ID starting with 97254e810ef08368367cb3f4cb56afcd1c2227c7a7a5cf37093c12a9228dd64c not found: ID does not exist" containerID="97254e810ef08368367cb3f4cb56afcd1c2227c7a7a5cf37093c12a9228dd64c"
Oct 31 19:51:43.412269 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:43.412225    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97254e810ef08368367cb3f4cb56afcd1c2227c7a7a5cf37093c12a9228dd64c"} err="failed to get container status \"97254e810ef08368367cb3f4cb56afcd1c2227c7a7a5cf37093c12a9228dd64c\": rpc error: code = NotFound desc = could not find container \"97254e810ef08368367cb3f4cb56afcd1c2227c7a7a5cf37093c12a9228dd64c\": container with ID starting with 97254e810ef08368367cb3f4cb56afcd1c2227c7a7a5cf37093c12a9228dd64c not found: ID does not exist"
Oct 31 19:51:43.419570 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:43.419530    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-cd9729/auth-test-model-cd9729-kserve-65f895f85c-pcprs" podStartSLOduration=1.419517379 podStartE2EDuration="1.419517379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:51:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:51:43.415869428 +0000 UTC m=+1933.989103557" watchObservedRunningTime="2026-10-31 19:51:43.419517379 +0000 UTC m=+1933.992751508"
Oct 31 19:51:43.433829 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:43.433806    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8dcfa8-6c7cb4c95f-cws7q"]
Oct 31 19:51:43.437655 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:43.437635    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8dcfa8-6c7cb4c95f-cws7q"]
Oct 31 19:51:43.601081 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:43.600996    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:43.619178 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:43.619145    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be41f707-30c3-49f1-91c6-bea2e72c172a" path="/var/lib/kubelet/pods/be41f707-30c3-49f1-91c6-bea2e72c172a/volumes"
Oct 31 19:51:44.035198 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:44.035155    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:44.252919 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:44.252881    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-8dcfa8-86d859f49f-j5f54" podUID="d8315528-0430-4e79-9f94-82df4e73befa" containerName="payload-processing" containerID="cri-o://13afe4d98f3967936393bbdcb7595c7f58c5d1ebd79c33561aa881c8147d4a77" gracePeriod=25
Oct 31 19:51:44.271593 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:44.271564    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8dcfa8-68fb9b44f5-6dmlv" podUID="0ec69924-e842-432a-a9fa-05db6c0d0fbf" containerName="payload-pre-processing" containerID="cri-o://cda7edf66edde4444772acf54632eeefb09f20a62d103ac4ab89940c0abfa530" gracePeriod=25
Oct 31 19:51:44.393589 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:44.393564    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-cd9729/auth-test-model-cd9729-kserve-65f895f85c-pcprs"
Oct 31 19:51:44.411383 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:44.411345    4645 generic.go:356] "Generic (PLEG): container finished" podID="d8315528-0430-4e79-9f94-82df4e73befa" containerID="13afe4d98f3967936393bbdcb7595c7f58c5d1ebd79c33561aa881c8147d4a77" exitCode=0
Oct 31 19:51:44.411847 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:44.411423    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8dcfa8-86d859f49f-j5f54" event={"ID":"d8315528-0430-4e79-9f94-82df4e73befa","Type":"ContainerDied","Data":"13afe4d98f3967936393bbdcb7595c7f58c5d1ebd79c33561aa881c8147d4a77"}
Oct 31 19:51:44.545952 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:44.545925    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8dcfa8-86d859f49f-j5f54"
Oct 31 19:51:44.733306 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:44.733272    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8315528-0430-4e79-9f94-82df4e73befa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8315528-0430-4e79-9f94-82df4e73befa-plugins-config-volume\") pod \"d8315528-0430-4e79-9f94-82df4e73befa\" (UID: \"d8315528-0430-4e79-9f94-82df4e73befa\") "
Oct 31 19:51:44.733553 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:44.733377    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8315528-0430-4e79-9f94-82df4e73befa-kube-api-access-lplc9\" (UniqueName: \"kubernetes.io/projected/d8315528-0430-4e79-9f94-82df4e73befa-kube-api-access-lplc9\") pod \"d8315528-0430-4e79-9f94-82df4e73befa\" (UID: \"d8315528-0430-4e79-9f94-82df4e73befa\") "
Oct 31 19:51:44.733728 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:44.733703    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8315528-0430-4e79-9f94-82df4e73befa-plugins-config-volume" pod "d8315528-0430-4e79-9f94-82df4e73befa" (UID: "d8315528-0430-4e79-9f94-82df4e73befa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:51:44.734182 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:44.734159    4645 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8315528-0430-4e79-9f94-82df4e73befa-plugins-config-volume\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:44.735636 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:44.735614    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8315528-0430-4e79-9f94-82df4e73befa-kube-api-access-lplc9" pod "d8315528-0430-4e79-9f94-82df4e73befa" (UID: "d8315528-0430-4e79-9f94-82df4e73befa"). InnerVolumeSpecName "kube-api-access-lplc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:51:44.835971 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:44.835938    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lplc9\" (UniqueName: \"kubernetes.io/projected/d8315528-0430-4e79-9f94-82df4e73befa-kube-api-access-lplc9\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:45.250723 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:45.250691    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:45.422289 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:45.422246    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8dcfa8-86d859f49f-j5f54" event={"ID":"d8315528-0430-4e79-9f94-82df4e73befa","Type":"ContainerDied","Data":"1694cafc1245e3393ed2f21ff5a86d4c421227794817cd0e0473f42df2b0b06f"}
Oct 31 19:51:45.422785 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:45.422314    4645 scope.go:122] "RemoveContainer" containerID="13afe4d98f3967936393bbdcb7595c7f58c5d1ebd79c33561aa881c8147d4a77"
Oct 31 19:51:45.422785 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:45.422262    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8dcfa8-86d859f49f-j5f54"
Oct 31 19:51:45.448925 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:45.448900    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8dcfa8-86d859f49f-j5f54"]
Oct 31 19:51:45.452854 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:45.452833    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8dcfa8-86d859f49f-j5f54"]
Oct 31 19:51:45.620788 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:45.620726    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8315528-0430-4e79-9f94-82df4e73befa" path="/var/lib/kubelet/pods/d8315528-0430-4e79-9f94-82df4e73befa/volumes"
Oct 31 19:51:45.696424 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:45.696390    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:46.147623 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:46.147590    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:46.422784 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:46.422752    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:46.633004 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:46.632966    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4"]
Oct 31 19:51:46.633824 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:46.633760    4645 cpu_manager.go:414] "containerMap: removing container" podUID="d8315528-0430-4e79-9f94-82df4e73befa" containerName="payload-processing"
Oct 31 19:51:46.633824 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:46.633784    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8315528-0430-4e79-9f94-82df4e73befa" containerName="payload-processing"
Oct 31 19:51:46.634045 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:46.633839    4645 cpu_manager.go:414] "containerMap: removing container" podUID="be41f707-30c3-49f1-91c6-bea2e72c172a" containerName="maas-api"
Oct 31 19:51:46.634045 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:46.633847    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be41f707-30c3-49f1-91c6-bea2e72c172a" containerName="maas-api"
Oct 31 19:51:46.634045 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:46.634027    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="be41f707-30c3-49f1-91c6-bea2e72c172a" containerName="maas-api"
Oct 31 19:51:46.634203 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:46.634051    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8315528-0430-4e79-9f94-82df4e73befa" containerName="payload-processing"
Oct 31 19:51:46.639180 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:46.639158    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4"
Oct 31 19:51:46.642998 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:46.642975    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-ae4ce2-openshift-default-dockercfg-btj6n\""
Oct 31 19:51:46.656744 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:46.656724    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4"]
Oct 31 19:51:46.661837 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:46.660961    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c010664-b8c0-4a2c-91d7-3f66e006b4ec-istiod-ca-cert\") pod \"e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4\" (UID: \"5c010664-b8c0-4a2c-91d7-3f66e006b4ec\") " pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4"
Oct 31 19:51:46.661837 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:46.661041    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c010664-b8c0-4a2c-91d7-3f66e006b4ec-istio-token\") pod \"e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4\" (UID: \"5c010664-b8c0-4a2c-91d7-3f66e006b4ec\") " pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4"
Oct 31 19:51:46.661837 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:46.661482    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5f66\" (UniqueName: \"kubernetes.io/projected/5c010664-b8c0-4a2c-91d7-3f66e006b4ec-kube-api-access-n5f66\") pod \"e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4\" (UID: \"5c010664-b8c0-4a2c-91d7-3f66e006b4ec\") " pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4"
Oct 31 19:51:46.661837 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:46.661554    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c010664-b8c0-4a2c-91d7-3f66e006b4ec-istio-podinfo\") pod \"e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4\" (UID: \"5c010664-b8c0-4a2c-91d7-3f66e006b4ec\") " pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4"
Oct 31 19:51:46.661837 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:46.661609    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c010664-b8c0-4a2c-91d7-3f66e006b4ec-workload-socket\") pod \"e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4\" (UID: \"5c010664-b8c0-4a2c-91d7-3f66e006b4ec\") " pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4"
Oct 31 19:51:46.661837 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:46.661667    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c010664-b8c0-4a2c-91d7-3f66e006b4ec-istio-data\") pod \"e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4\" (UID: \"5c010664-b8c0-4a2c-91d7-3f66e006b4ec\") " pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4"
Oct 31 19:51:46.661837 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:46.661749    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c010664-b8c0-4a2c-91d7-3f66e006b4ec-istio-envoy\") pod \"e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4\" (UID: \"5c010664-b8c0-4a2c-91d7-3f66e006b4ec\") " pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4"
Oct 31 19:51:46.687302 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:46.686417    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:46.687578 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:46.687557    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c010664-b8c0-4a2c-91d7-3f66e006b4ec-workload-certs\") pod \"e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4\" (UID: \"5c010664-b8c0-4a2c-91d7-3f66e006b4ec\") " pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4"
Oct 31 19:51:46.687700 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:46.687681    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c010664-b8c0-4a2c-91d7-3f66e006b4ec-credential-socket\") pod \"e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4\" (UID: \"5c010664-b8c0-4a2c-91d7-3f66e006b4ec\") " pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4"
Oct 31 19:51:46.826349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:46.814541    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c010664-b8c0-4a2c-91d7-3f66e006b4ec-credential-socket\") pod \"e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4\" (UID: \"5c010664-b8c0-4a2c-91d7-3f66e006b4ec\") " pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4"
Oct 31 19:51:46.826349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:46.814654    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c010664-b8c0-4a2c-91d7-3f66e006b4ec-istiod-ca-cert\") pod \"e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4\" (UID: \"5c010664-b8c0-4a2c-91d7-3f66e006b4ec\") " pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4"
Oct 31 19:51:46.826349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:46.814718    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c010664-b8c0-4a2c-91d7-3f66e006b4ec-istio-token\") pod \"e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4\" (UID: \"5c010664-b8c0-4a2c-91d7-3f66e006b4ec\") " pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4"
Oct 31 19:51:46.826349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:46.814769    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5f66\" (UniqueName: \"kubernetes.io/projected/5c010664-b8c0-4a2c-91d7-3f66e006b4ec-kube-api-access-n5f66\") pod \"e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4\" (UID: \"5c010664-b8c0-4a2c-91d7-3f66e006b4ec\") " pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4"
Oct 31 19:51:46.826349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:46.814826    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c010664-b8c0-4a2c-91d7-3f66e006b4ec-istio-podinfo\") pod \"e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4\" (UID: \"5c010664-b8c0-4a2c-91d7-3f66e006b4ec\") " pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4"
Oct 31 19:51:46.826349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:46.814883    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c010664-b8c0-4a2c-91d7-3f66e006b4ec-workload-socket\") pod \"e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4\" (UID: \"5c010664-b8c0-4a2c-91d7-3f66e006b4ec\") " pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4"
Oct 31 19:51:46.826349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:46.814947    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c010664-b8c0-4a2c-91d7-3f66e006b4ec-istio-data\") pod \"e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4\" (UID: \"5c010664-b8c0-4a2c-91d7-3f66e006b4ec\") " pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4"
Oct 31 19:51:46.826349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:46.815043    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c010664-b8c0-4a2c-91d7-3f66e006b4ec-istio-envoy\") pod \"e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4\" (UID: \"5c010664-b8c0-4a2c-91d7-3f66e006b4ec\") " pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4"
Oct 31 19:51:46.826349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:46.815092    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c010664-b8c0-4a2c-91d7-3f66e006b4ec-workload-certs\") pod \"e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4\" (UID: \"5c010664-b8c0-4a2c-91d7-3f66e006b4ec\") " pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4"
Oct 31 19:51:46.826349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:46.815695    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c010664-b8c0-4a2c-91d7-3f66e006b4ec-workload-certs\") pod \"e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4\" (UID: \"5c010664-b8c0-4a2c-91d7-3f66e006b4ec\") " pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4"
Oct 31 19:51:46.826349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:46.821231    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c010664-b8c0-4a2c-91d7-3f66e006b4ec-istiod-ca-cert\") pod \"e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4\" (UID: \"5c010664-b8c0-4a2c-91d7-3f66e006b4ec\") " pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4"
Oct 31 19:51:46.826349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:46.821516    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c010664-b8c0-4a2c-91d7-3f66e006b4ec-istio-data\") pod \"e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4\" (UID: \"5c010664-b8c0-4a2c-91d7-3f66e006b4ec\") " pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4"
Oct 31 19:51:46.826349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:46.821760    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c010664-b8c0-4a2c-91d7-3f66e006b4ec-workload-socket\") pod \"e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4\" (UID: \"5c010664-b8c0-4a2c-91d7-3f66e006b4ec\") " pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4"
Oct 31 19:51:46.834517 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:46.834490    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c010664-b8c0-4a2c-91d7-3f66e006b4ec-istio-envoy\") pod \"e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4\" (UID: \"5c010664-b8c0-4a2c-91d7-3f66e006b4ec\") " pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4"
Oct 31 19:51:46.841033 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:46.839300    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c010664-b8c0-4a2c-91d7-3f66e006b4ec-credential-socket\") pod \"e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4\" (UID: \"5c010664-b8c0-4a2c-91d7-3f66e006b4ec\") " pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4"
Oct 31 19:51:46.960166 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:46.960052    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c010664-b8c0-4a2c-91d7-3f66e006b4ec-istio-podinfo\") pod \"e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4\" (UID: \"5c010664-b8c0-4a2c-91d7-3f66e006b4ec\") " pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4"
Oct 31 19:51:46.974447 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:46.973909    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c010664-b8c0-4a2c-91d7-3f66e006b4ec-istio-token\") pod \"e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4\" (UID: \"5c010664-b8c0-4a2c-91d7-3f66e006b4ec\") " pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4"
Oct 31 19:51:46.977355 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:46.975322    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5f66\" (UniqueName: \"kubernetes.io/projected/5c010664-b8c0-4a2c-91d7-3f66e006b4ec-kube-api-access-n5f66\") pod \"e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4\" (UID: \"5c010664-b8c0-4a2c-91d7-3f66e006b4ec\") " pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4"
Oct 31 19:51:47.102096 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:47.102063    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:47.258379 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:47.258287    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4"
Oct 31 19:51:47.461237 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:47.461191    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:47.549112 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:47.549081    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4"]
Oct 31 19:51:47.552116 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:51:47.552086    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c010664_b8c0_4a2c_91d7_3f66e006b4ec.slice/crio-72217eb365e7188264d60bd0b756c7ab89a3b7b88d21144e727a1255d35b6292 WatchSource:0}: Error finding container 72217eb365e7188264d60bd0b756c7ab89a3b7b88d21144e727a1255d35b6292: Status 404 returned error can't find the container with id 72217eb365e7188264d60bd0b756c7ab89a3b7b88d21144e727a1255d35b6292
Oct 31 19:51:47.555246 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:47.555208    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:51:47.555361 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:47.555294    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:51:47.555450 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:47.555388    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:51:48.232348 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:48.232303    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:48.456179 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:48.456133    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4" event={"ID":"5c010664-b8c0-4a2c-91d7-3f66e006b4ec","Type":"ContainerStarted","Data":"1e4c85c9543fa3181e0d806d168f058638109986d6fdb39ba16812d43c75a0a6"}
Oct 31 19:51:48.456344 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:48.456198    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4" event={"ID":"5c010664-b8c0-4a2c-91d7-3f66e006b4ec","Type":"ContainerStarted","Data":"72217eb365e7188264d60bd0b756c7ab89a3b7b88d21144e727a1255d35b6292"}
Oct 31 19:51:48.481075 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:48.481013    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4" podStartSLOduration=2.480994622 podStartE2EDuration="2.480994622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:51:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:51:48.478045219 +0000 UTC m=+1939.051279348" watchObservedRunningTime="2026-10-31 19:51:48.480994622 +0000 UTC m=+1939.054228752"
Oct 31 19:51:48.543007 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:48.542945    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:49.071023 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:49.070993    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:49.258934 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:49.258898    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4"
Oct 31 19:51:49.265067 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:49.265044    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4"
Oct 31 19:51:49.463671 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:49.463639    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4"
Oct 31 19:51:49.464638 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:49.464617    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4"
Oct 31 19:51:49.962975 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:49.962940    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:50.747769 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:50.747730    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8dcfa8-68fb9b44f5-6dmlv" podUID="0ec69924-e842-432a-a9fa-05db6c0d0fbf" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:51:50.936212 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:50.936177    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:51.085735 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:51.085646    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:51.207072 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:51.207035    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:51.630554 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:51.630508    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:51.754407 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:51.754354    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:51.780426 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:51.780394    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-f5c1ab/auth-test-model-f5c1ab-kserve-77f7d9ccc-4ljhr"]
Oct 31 19:51:51.785407 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:51.785385    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-f5c1ab/auth-test-model-f5c1ab-kserve-77f7d9ccc-4ljhr"
Oct 31 19:51:51.799694 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:51.797621    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-f5c1ab\"/\"auth-test-model-f5c1ab-kserve-self-signed-certs\""
Oct 31 19:51:51.799694 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:51.798253    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-f5c1ab/auth-test-model-f5c1ab-kserve-77f7d9ccc-4ljhr"]
Oct 31 19:51:51.807135 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:51.807113    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-f5c1ab\"/\"openshift-service-ca.crt\""
Oct 31 19:51:51.807287 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:51.807228    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-f5c1ab\"/\"kube-root-ca.crt\""
Oct 31 19:51:51.837803 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:51.837641    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-f5c1ab\"/\"default-dockercfg-tz8jd\""
Oct 31 19:51:51.913581 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:51.913553    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:51.959780 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:51.959748    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q229g\" (UniqueName: \"kubernetes.io/projected/d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b-kube-api-access-q229g\") pod \"auth-test-model-f5c1ab-kserve-77f7d9ccc-4ljhr\" (UID: \"d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b\") " pod="ai-tenant-e2e-shared-b-w0-f5c1ab/auth-test-model-f5c1ab-kserve-77f7d9ccc-4ljhr"
Oct 31 19:51:51.959938 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:51.959899    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b-dshm\") pod \"auth-test-model-f5c1ab-kserve-77f7d9ccc-4ljhr\" (UID: \"d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b\") " pod="ai-tenant-e2e-shared-b-w0-f5c1ab/auth-test-model-f5c1ab-kserve-77f7d9ccc-4ljhr"
Oct 31 19:51:51.959999 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:51.959952    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b-model-cache\") pod \"auth-test-model-f5c1ab-kserve-77f7d9ccc-4ljhr\" (UID: \"d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b\") " pod="ai-tenant-e2e-shared-b-w0-f5c1ab/auth-test-model-f5c1ab-kserve-77f7d9ccc-4ljhr"
Oct 31 19:51:51.959999 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:51.959974    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b-tls-certs\") pod \"auth-test-model-f5c1ab-kserve-77f7d9ccc-4ljhr\" (UID: \"d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b\") " pod="ai-tenant-e2e-shared-b-w0-f5c1ab/auth-test-model-f5c1ab-kserve-77f7d9ccc-4ljhr"
Oct 31 19:51:51.960189 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:51.960007    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b-tmp-dir\") pod \"auth-test-model-f5c1ab-kserve-77f7d9ccc-4ljhr\" (UID: \"d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b\") " pod="ai-tenant-e2e-shared-b-w0-f5c1ab/auth-test-model-f5c1ab-kserve-77f7d9ccc-4ljhr"
Oct 31 19:51:51.960245 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:51.960218    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b-home\") pod \"auth-test-model-f5c1ab-kserve-77f7d9ccc-4ljhr\" (UID: \"d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b\") " pod="ai-tenant-e2e-shared-b-w0-f5c1ab/auth-test-model-f5c1ab-kserve-77f7d9ccc-4ljhr"
Oct 31 19:51:52.026972 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:52.026940    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np"]
Oct 31 19:51:52.061752 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:52.061717    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b-home\") pod \"auth-test-model-f5c1ab-kserve-77f7d9ccc-4ljhr\" (UID: \"d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b\") " pod="ai-tenant-e2e-shared-b-w0-f5c1ab/auth-test-model-f5c1ab-kserve-77f7d9ccc-4ljhr"
Oct 31 19:51:52.061923 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:52.061867    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q229g\" (UniqueName: \"kubernetes.io/projected/d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b-kube-api-access-q229g\") pod \"auth-test-model-f5c1ab-kserve-77f7d9ccc-4ljhr\" (UID: \"d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b\") " pod="ai-tenant-e2e-shared-b-w0-f5c1ab/auth-test-model-f5c1ab-kserve-77f7d9ccc-4ljhr"
Oct 31 19:51:52.062020 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:52.062005    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b-dshm\") pod \"auth-test-model-f5c1ab-kserve-77f7d9ccc-4ljhr\" (UID: \"d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b\") " pod="ai-tenant-e2e-shared-b-w0-f5c1ab/auth-test-model-f5c1ab-kserve-77f7d9ccc-4ljhr"
Oct 31 19:51:52.062087 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:52.062043    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b-model-cache\") pod \"auth-test-model-f5c1ab-kserve-77f7d9ccc-4ljhr\" (UID: \"d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b\") " pod="ai-tenant-e2e-shared-b-w0-f5c1ab/auth-test-model-f5c1ab-kserve-77f7d9ccc-4ljhr"
Oct 31 19:51:52.062087 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:52.062067    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b-home\") pod \"auth-test-model-f5c1ab-kserve-77f7d9ccc-4ljhr\" (UID: \"d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b\") " pod="ai-tenant-e2e-shared-b-w0-f5c1ab/auth-test-model-f5c1ab-kserve-77f7d9ccc-4ljhr"
Oct 31 19:51:52.062087 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:52.062079    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b-tls-certs\") pod \"auth-test-model-f5c1ab-kserve-77f7d9ccc-4ljhr\" (UID: \"d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b\") " pod="ai-tenant-e2e-shared-b-w0-f5c1ab/auth-test-model-f5c1ab-kserve-77f7d9ccc-4ljhr"
Oct 31 19:51:52.062250 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:52.062125    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b-tmp-dir\") pod \"auth-test-model-f5c1ab-kserve-77f7d9ccc-4ljhr\" (UID: \"d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b\") " pod="ai-tenant-e2e-shared-b-w0-f5c1ab/auth-test-model-f5c1ab-kserve-77f7d9ccc-4ljhr"
Oct 31 19:51:52.062466 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:52.062436    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b-model-cache\") pod \"auth-test-model-f5c1ab-kserve-77f7d9ccc-4ljhr\" (UID: \"d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b\") " pod="ai-tenant-e2e-shared-b-w0-f5c1ab/auth-test-model-f5c1ab-kserve-77f7d9ccc-4ljhr"
Oct 31 19:51:52.062466 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:52.062462    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b-tmp-dir\") pod \"auth-test-model-f5c1ab-kserve-77f7d9ccc-4ljhr\" (UID: \"d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b\") " pod="ai-tenant-e2e-shared-b-w0-f5c1ab/auth-test-model-f5c1ab-kserve-77f7d9ccc-4ljhr"
Oct 31 19:51:52.064317 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:52.064290    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b-dshm\") pod \"auth-test-model-f5c1ab-kserve-77f7d9ccc-4ljhr\" (UID: \"d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b\") " pod="ai-tenant-e2e-shared-b-w0-f5c1ab/auth-test-model-f5c1ab-kserve-77f7d9ccc-4ljhr"
Oct 31 19:51:52.064608 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:52.064587    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b-tls-certs\") pod \"auth-test-model-f5c1ab-kserve-77f7d9ccc-4ljhr\" (UID: \"d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b\") " pod="ai-tenant-e2e-shared-b-w0-f5c1ab/auth-test-model-f5c1ab-kserve-77f7d9ccc-4ljhr"
Oct 31 19:51:52.080224 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:52.080199    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q229g\" (UniqueName: \"kubernetes.io/projected/d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b-kube-api-access-q229g\") pod \"auth-test-model-f5c1ab-kserve-77f7d9ccc-4ljhr\" (UID: \"d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b\") " pod="ai-tenant-e2e-shared-b-w0-f5c1ab/auth-test-model-f5c1ab-kserve-77f7d9ccc-4ljhr"
Oct 31 19:51:52.103799 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:52.103774    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-f5c1ab/auth-test-model-f5c1ab-kserve-77f7d9ccc-4ljhr"
Oct 31 19:51:52.167088 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:52.166890    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:52.395258 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:52.394860    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-cd9729/auth-test-model-cd9729-kserve-65f895f85c-pcprs"
Oct 31 19:51:52.412612 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:52.412587    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-cd9729/auth-test-model-cd9729-kserve-65f895f85c-pcprs"
Oct 31 19:51:52.496218 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:52.496134    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np" podUID="c712ccf6-e167-4c11-ba0a-df297a0caf05" containerName="istio-proxy" containerID="cri-o://95a0a0e65faf9564ab4aff50b6e321570123169bad0ffad9f678ce4105a4a5df" gracePeriod=30
Oct 31 19:51:52.508342 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:52.508309    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-cd9729/auth-test-model-cd9729-kserve-65f895f85c-pcprs"
Oct 31 19:51:52.598856 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:52.598828    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-f5c1ab/auth-test-model-f5c1ab-kserve-77f7d9ccc-4ljhr"]
Oct 31 19:51:52.599153 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:51:52.599129    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd1063ddb_50eb_4c9e_a37c_1e84e5e1da9b.slice/crio-8aa949bf63335ac9ad85a2b7fc2dbd51eb0988be8535055a3f394ed444d6bc31 WatchSource:0}: Error finding container 8aa949bf63335ac9ad85a2b7fc2dbd51eb0988be8535055a3f394ed444d6bc31: Status 404 returned error can't find the container with id 8aa949bf63335ac9ad85a2b7fc2dbd51eb0988be8535055a3f394ed444d6bc31
Oct 31 19:51:52.644802 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:52.644776    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:52.908191 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:52.908094    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:53.045954 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:53.045919    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:53.507869 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:53.507831    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-f5c1ab/auth-test-model-f5c1ab-kserve-77f7d9ccc-4ljhr" event={"ID":"d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b","Type":"ContainerStarted","Data":"a0b2d0bbde02722938141bebe4836a59e1776fd4bd726caa9d65ae7ab6c3896d"}
Oct 31 19:51:53.508303 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:53.507893    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-f5c1ab/auth-test-model-f5c1ab-kserve-77f7d9ccc-4ljhr" event={"ID":"d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b","Type":"ContainerStarted","Data":"8aa949bf63335ac9ad85a2b7fc2dbd51eb0988be8535055a3f394ed444d6bc31"}
Oct 31 19:51:53.531425 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:53.531377    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-f5c1ab/auth-test-model-f5c1ab-kserve-77f7d9ccc-4ljhr" podStartSLOduration=2.531362584 podStartE2EDuration="2.531362584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:51:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:51:53.527580533 +0000 UTC m=+1944.100814662" watchObservedRunningTime="2026-10-31 19:51:53.531362584 +0000 UTC m=+1944.104596709"
Oct 31 19:51:53.841761 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:53.841661    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:54.104618 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:54.104552    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-f5c1ab/auth-test-model-f5c1ab-kserve-77f7d9ccc-4ljhr"
Oct 31 19:51:54.738404 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:54.738353    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:55.228994 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:55.228961    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2"]
Oct 31 19:51:55.232618 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:55.232598    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2"
Oct 31 19:51:55.237251 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:55.237230    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-d8c398-openshift-default-dockercfg-6tdvv\""
Oct 31 19:51:55.249800 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:55.249779    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2"]
Oct 31 19:51:55.420301 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:55.420273    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90bf19b5-9c42-4481-873e-c55d96f717eb-workload-socket\") pod \"e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2\" (UID: \"90bf19b5-9c42-4481-873e-c55d96f717eb\") " pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2"
Oct 31 19:51:55.420471 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:55.420364    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90bf19b5-9c42-4481-873e-c55d96f717eb-istio-podinfo\") pod \"e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2\" (UID: \"90bf19b5-9c42-4481-873e-c55d96f717eb\") " pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2"
Oct 31 19:51:55.420471 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:55.420434    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z49st\" (UniqueName: \"kubernetes.io/projected/90bf19b5-9c42-4481-873e-c55d96f717eb-kube-api-access-z49st\") pod \"e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2\" (UID: \"90bf19b5-9c42-4481-873e-c55d96f717eb\") " pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2"
Oct 31 19:51:55.420545 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:55.420498    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90bf19b5-9c42-4481-873e-c55d96f717eb-istio-envoy\") pod \"e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2\" (UID: \"90bf19b5-9c42-4481-873e-c55d96f717eb\") " pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2"
Oct 31 19:51:55.420614 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:55.420591    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90bf19b5-9c42-4481-873e-c55d96f717eb-istio-data\") pod \"e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2\" (UID: \"90bf19b5-9c42-4481-873e-c55d96f717eb\") " pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2"
Oct 31 19:51:55.420652 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:55.420642    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90bf19b5-9c42-4481-873e-c55d96f717eb-istio-token\") pod \"e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2\" (UID: \"90bf19b5-9c42-4481-873e-c55d96f717eb\") " pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2"
Oct 31 19:51:55.420730 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:55.420716    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90bf19b5-9c42-4481-873e-c55d96f717eb-workload-certs\") pod \"e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2\" (UID: \"90bf19b5-9c42-4481-873e-c55d96f717eb\") " pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2"
Oct 31 19:51:55.420762 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:55.420750    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90bf19b5-9c42-4481-873e-c55d96f717eb-credential-socket\") pod \"e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2\" (UID: \"90bf19b5-9c42-4481-873e-c55d96f717eb\") " pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2"
Oct 31 19:51:55.420906 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:55.420888    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90bf19b5-9c42-4481-873e-c55d96f717eb-istiod-ca-cert\") pod \"e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2\" (UID: \"90bf19b5-9c42-4481-873e-c55d96f717eb\") " pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2"
Oct 31 19:51:55.521644 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:55.521582    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90bf19b5-9c42-4481-873e-c55d96f717eb-istio-data\") pod \"e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2\" (UID: \"90bf19b5-9c42-4481-873e-c55d96f717eb\") " pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2"
Oct 31 19:51:55.521644 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:55.521610    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90bf19b5-9c42-4481-873e-c55d96f717eb-istio-token\") pod \"e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2\" (UID: \"90bf19b5-9c42-4481-873e-c55d96f717eb\") " pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2"
Oct 31 19:51:55.521834 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:55.521650    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90bf19b5-9c42-4481-873e-c55d96f717eb-workload-certs\") pod \"e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2\" (UID: \"90bf19b5-9c42-4481-873e-c55d96f717eb\") " pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2"
Oct 31 19:51:55.521834 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:55.521682    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90bf19b5-9c42-4481-873e-c55d96f717eb-credential-socket\") pod \"e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2\" (UID: \"90bf19b5-9c42-4481-873e-c55d96f717eb\") " pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2"
Oct 31 19:51:55.521834 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:55.521767    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90bf19b5-9c42-4481-873e-c55d96f717eb-istiod-ca-cert\") pod \"e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2\" (UID: \"90bf19b5-9c42-4481-873e-c55d96f717eb\") " pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2"
Oct 31 19:51:55.521834 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:55.521822    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90bf19b5-9c42-4481-873e-c55d96f717eb-workload-socket\") pod \"e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2\" (UID: \"90bf19b5-9c42-4481-873e-c55d96f717eb\") " pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2"
Oct 31 19:51:55.522053 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:55.521904    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90bf19b5-9c42-4481-873e-c55d96f717eb-istio-podinfo\") pod \"e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2\" (UID: \"90bf19b5-9c42-4481-873e-c55d96f717eb\") " pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2"
Oct 31 19:51:55.522053 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:55.521939    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z49st\" (UniqueName: \"kubernetes.io/projected/90bf19b5-9c42-4481-873e-c55d96f717eb-kube-api-access-z49st\") pod \"e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2\" (UID: \"90bf19b5-9c42-4481-873e-c55d96f717eb\") " pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2"
Oct 31 19:51:55.522053 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:55.522021    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90bf19b5-9c42-4481-873e-c55d96f717eb-istio-envoy\") pod \"e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2\" (UID: \"90bf19b5-9c42-4481-873e-c55d96f717eb\") " pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2"
Oct 31 19:51:55.522213 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:55.522114    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90bf19b5-9c42-4481-873e-c55d96f717eb-workload-certs\") pod \"e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2\" (UID: \"90bf19b5-9c42-4481-873e-c55d96f717eb\") " pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2"
Oct 31 19:51:55.522213 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:55.522152    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90bf19b5-9c42-4481-873e-c55d96f717eb-credential-socket\") pod \"e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2\" (UID: \"90bf19b5-9c42-4481-873e-c55d96f717eb\") " pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2"
Oct 31 19:51:55.522350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:55.522206    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90bf19b5-9c42-4481-873e-c55d96f717eb-workload-socket\") pod \"e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2\" (UID: \"90bf19b5-9c42-4481-873e-c55d96f717eb\") " pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2"
Oct 31 19:51:55.522411 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:55.522321    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90bf19b5-9c42-4481-873e-c55d96f717eb-istio-data\") pod \"e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2\" (UID: \"90bf19b5-9c42-4481-873e-c55d96f717eb\") " pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2"
Oct 31 19:51:55.522573 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:55.522553    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90bf19b5-9c42-4481-873e-c55d96f717eb-istiod-ca-cert\") pod \"e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2\" (UID: \"90bf19b5-9c42-4481-873e-c55d96f717eb\") " pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2"
Oct 31 19:51:55.524373 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:55.524351    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90bf19b5-9c42-4481-873e-c55d96f717eb-istio-envoy\") pod \"e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2\" (UID: \"90bf19b5-9c42-4481-873e-c55d96f717eb\") " pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2"
Oct 31 19:51:55.524755 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:55.524737    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90bf19b5-9c42-4481-873e-c55d96f717eb-istio-podinfo\") pod \"e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2\" (UID: \"90bf19b5-9c42-4481-873e-c55d96f717eb\") " pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2"
Oct 31 19:51:55.533646 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:55.533620    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90bf19b5-9c42-4481-873e-c55d96f717eb-istio-token\") pod \"e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2\" (UID: \"90bf19b5-9c42-4481-873e-c55d96f717eb\") " pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2"
Oct 31 19:51:55.533854 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:55.533834    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z49st\" (UniqueName: \"kubernetes.io/projected/90bf19b5-9c42-4481-873e-c55d96f717eb-kube-api-access-z49st\") pod \"e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2\" (UID: \"90bf19b5-9c42-4481-873e-c55d96f717eb\") " pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2"
Oct 31 19:51:55.549650 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:55.549631    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2"
Oct 31 19:51:55.681383 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:55.680308    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:55.842218 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:55.842190    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2"]
Oct 31 19:51:55.842550 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:51:55.842194    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90bf19b5_9c42_4481_873e_c55d96f717eb.slice/crio-53650f939076865059f5e8a0782ceefe323d69f763d077b826fc656e218f159c WatchSource:0}: Error finding container 53650f939076865059f5e8a0782ceefe323d69f763d077b826fc656e218f159c: Status 404 returned error can't find the container with id 53650f939076865059f5e8a0782ceefe323d69f763d077b826fc656e218f159c
Oct 31 19:51:55.845248 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:55.845208    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:51:55.845361 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:55.845280    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:51:55.845361 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:55.845312    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:51:56.535586 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:56.535550    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2" event={"ID":"90bf19b5-9c42-4481-873e-c55d96f717eb","Type":"ContainerStarted","Data":"b944ec729e9867c907806253346871b7687f9df0a792c3b7bcd79b72df4d7138"}
Oct 31 19:51:56.535734 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:56.535604    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2" event={"ID":"90bf19b5-9c42-4481-873e-c55d96f717eb","Type":"ContainerStarted","Data":"53650f939076865059f5e8a0782ceefe323d69f763d077b826fc656e218f159c"}
Oct 31 19:51:56.550119 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:56.550093    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2"
Oct 31 19:51:56.552081 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:56.552055    4645 patch_prober.go:29] interesting pod/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body=
Oct 31 19:51:56.552349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:56.552110    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2" podUID="90bf19b5-9c42-4481-873e-c55d96f717eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused"
Oct 31 19:51:56.553203 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:56.553138    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:56.600368 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:56.586374    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2" podStartSLOduration=1.5863535 podStartE2EDuration="1.5863535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:51:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:51:56.580223829 +0000 UTC m=+1947.153457960" watchObservedRunningTime="2026-10-31 19:51:56.5863535 +0000 UTC m=+1947.159587653"
Oct 31 19:51:57.320565 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:57.320526    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:57.550246 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:57.550216    4645 patch_prober.go:29] interesting pod/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body=
Oct 31 19:51:57.550405 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:57.550270    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2" podUID="90bf19b5-9c42-4481-873e-c55d96f717eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused"
Oct 31 19:51:57.835184 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:57.835162    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np"
Oct 31 19:51:57.902426 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:57.902352    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c712ccf6-e167-4c11-ba0a-df297a0caf05-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c712ccf6-e167-4c11-ba0a-df297a0caf05-istiod-ca-cert\") pod \"c712ccf6-e167-4c11-ba0a-df297a0caf05\" (UID: \"c712ccf6-e167-4c11-ba0a-df297a0caf05\") "
Oct 31 19:51:57.902426 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:57.902394    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c712ccf6-e167-4c11-ba0a-df297a0caf05-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c712ccf6-e167-4c11-ba0a-df297a0caf05-istio-data\") pod \"c712ccf6-e167-4c11-ba0a-df297a0caf05\" (UID: \"c712ccf6-e167-4c11-ba0a-df297a0caf05\") "
Oct 31 19:51:57.902646 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:57.902426    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c712ccf6-e167-4c11-ba0a-df297a0caf05-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c712ccf6-e167-4c11-ba0a-df297a0caf05-workload-socket\") pod \"c712ccf6-e167-4c11-ba0a-df297a0caf05\" (UID: \"c712ccf6-e167-4c11-ba0a-df297a0caf05\") "
Oct 31 19:51:57.902646 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:57.902514    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c712ccf6-e167-4c11-ba0a-df297a0caf05-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c712ccf6-e167-4c11-ba0a-df297a0caf05-credential-socket\") pod \"c712ccf6-e167-4c11-ba0a-df297a0caf05\" (UID: \"c712ccf6-e167-4c11-ba0a-df297a0caf05\") "
Oct 31 19:51:57.902646 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:57.902560    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c712ccf6-e167-4c11-ba0a-df297a0caf05-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c712ccf6-e167-4c11-ba0a-df297a0caf05-istio-podinfo\") pod \"c712ccf6-e167-4c11-ba0a-df297a0caf05\" (UID: \"c712ccf6-e167-4c11-ba0a-df297a0caf05\") "
Oct 31 19:51:57.902860 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:57.902728    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c712ccf6-e167-4c11-ba0a-df297a0caf05-kube-api-access-vbhq9\" (UniqueName: \"kubernetes.io/projected/c712ccf6-e167-4c11-ba0a-df297a0caf05-kube-api-access-vbhq9\") pod \"c712ccf6-e167-4c11-ba0a-df297a0caf05\" (UID: \"c712ccf6-e167-4c11-ba0a-df297a0caf05\") "
Oct 31 19:51:57.902860 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:57.902762    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c712ccf6-e167-4c11-ba0a-df297a0caf05-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c712ccf6-e167-4c11-ba0a-df297a0caf05-istio-envoy\") pod \"c712ccf6-e167-4c11-ba0a-df297a0caf05\" (UID: \"c712ccf6-e167-4c11-ba0a-df297a0caf05\") "
Oct 31 19:51:57.902964 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:57.902865    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c712ccf6-e167-4c11-ba0a-df297a0caf05-istio-token\" (UniqueName: \"kubernetes.io/projected/c712ccf6-e167-4c11-ba0a-df297a0caf05-istio-token\") pod \"c712ccf6-e167-4c11-ba0a-df297a0caf05\" (UID: \"c712ccf6-e167-4c11-ba0a-df297a0caf05\") "
Oct 31 19:51:57.902964 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:57.902911    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c712ccf6-e167-4c11-ba0a-df297a0caf05-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c712ccf6-e167-4c11-ba0a-df297a0caf05-workload-certs\") pod \"c712ccf6-e167-4c11-ba0a-df297a0caf05\" (UID: \"c712ccf6-e167-4c11-ba0a-df297a0caf05\") "
Oct 31 19:51:57.904035 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:57.902754    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c712ccf6-e167-4c11-ba0a-df297a0caf05-istiod-ca-cert" pod "c712ccf6-e167-4c11-ba0a-df297a0caf05" (UID: "c712ccf6-e167-4c11-ba0a-df297a0caf05"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:51:57.904035 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:57.903023    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c712ccf6-e167-4c11-ba0a-df297a0caf05-credential-socket" pod "c712ccf6-e167-4c11-ba0a-df297a0caf05" (UID: "c712ccf6-e167-4c11-ba0a-df297a0caf05"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:51:57.904035 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:57.903193    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c712ccf6-e167-4c11-ba0a-df297a0caf05-istio-data" pod "c712ccf6-e167-4c11-ba0a-df297a0caf05" (UID: "c712ccf6-e167-4c11-ba0a-df297a0caf05"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:51:57.904035 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:57.903366    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c712ccf6-e167-4c11-ba0a-df297a0caf05-workload-socket" pod "c712ccf6-e167-4c11-ba0a-df297a0caf05" (UID: "c712ccf6-e167-4c11-ba0a-df297a0caf05"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:51:57.904035 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:57.903778    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c712ccf6-e167-4c11-ba0a-df297a0caf05-workload-certs" pod "c712ccf6-e167-4c11-ba0a-df297a0caf05" (UID: "c712ccf6-e167-4c11-ba0a-df297a0caf05"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:51:57.905948 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:57.905923    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c712ccf6-e167-4c11-ba0a-df297a0caf05-istio-envoy" pod "c712ccf6-e167-4c11-ba0a-df297a0caf05" (UID: "c712ccf6-e167-4c11-ba0a-df297a0caf05"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:51:57.906189 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:57.906087    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c712ccf6-e167-4c11-ba0a-df297a0caf05-istio-podinfo" pod "c712ccf6-e167-4c11-ba0a-df297a0caf05" (UID: "c712ccf6-e167-4c11-ba0a-df297a0caf05"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:51:57.906189 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:57.906172    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c712ccf6-e167-4c11-ba0a-df297a0caf05-istio-token" pod "c712ccf6-e167-4c11-ba0a-df297a0caf05" (UID: "c712ccf6-e167-4c11-ba0a-df297a0caf05"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:51:57.907825 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:57.907799    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c712ccf6-e167-4c11-ba0a-df297a0caf05-kube-api-access-vbhq9" pod "c712ccf6-e167-4c11-ba0a-df297a0caf05" (UID: "c712ccf6-e167-4c11-ba0a-df297a0caf05"). InnerVolumeSpecName "kube-api-access-vbhq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:51:58.005324 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:58.005292    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbhq9\" (UniqueName: \"kubernetes.io/projected/c712ccf6-e167-4c11-ba0a-df297a0caf05-kube-api-access-vbhq9\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:58.005491 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:58.005322    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c712ccf6-e167-4c11-ba0a-df297a0caf05-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:58.005491 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:58.005367    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c712ccf6-e167-4c11-ba0a-df297a0caf05-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:58.005491 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:58.005393    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c712ccf6-e167-4c11-ba0a-df297a0caf05-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:58.005491 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:58.005406    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c712ccf6-e167-4c11-ba0a-df297a0caf05-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:58.005491 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:58.005418    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c712ccf6-e167-4c11-ba0a-df297a0caf05-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:58.005491 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:58.005432    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c712ccf6-e167-4c11-ba0a-df297a0caf05-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:58.005491 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:58.005444    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c712ccf6-e167-4c11-ba0a-df297a0caf05-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:58.005491 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:58.005452    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c712ccf6-e167-4c11-ba0a-df297a0caf05-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:51:58.148837 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:58.148809    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:58.550834 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:58.550804    4645 patch_prober.go:29] interesting pod/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body=
Oct 31 19:51:58.551257 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:58.550857    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2" podUID="90bf19b5-9c42-4481-873e-c55d96f717eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused"
Oct 31 19:51:58.553091 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:58.553065    4645 generic.go:356] "Generic (PLEG): container finished" podID="c712ccf6-e167-4c11-ba0a-df297a0caf05" containerID="95a0a0e65faf9564ab4aff50b6e321570123169bad0ffad9f678ce4105a4a5df" exitCode=0
Oct 31 19:51:58.553220 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:58.553125    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np"
Oct 31 19:51:58.553220 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:58.553150    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np" event={"ID":"c712ccf6-e167-4c11-ba0a-df297a0caf05","Type":"ContainerDied","Data":"95a0a0e65faf9564ab4aff50b6e321570123169bad0ffad9f678ce4105a4a5df"}
Oct 31 19:51:58.553220 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:58.553209    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np" event={"ID":"c712ccf6-e167-4c11-ba0a-df297a0caf05","Type":"ContainerDied","Data":"6468e3048c4d991f11b3389294b5e0af03f8604984126ada12b2ad78044cd592"}
Oct 31 19:51:58.553395 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:58.553237    4645 scope.go:122] "RemoveContainer" containerID="95a0a0e65faf9564ab4aff50b6e321570123169bad0ffad9f678ce4105a4a5df"
Oct 31 19:51:58.592507 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:58.592408    4645 scope.go:122] "RemoveContainer" containerID="95a0a0e65faf9564ab4aff50b6e321570123169bad0ffad9f678ce4105a4a5df"
Oct 31 19:51:58.592784 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:51:58.592759    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95a0a0e65faf9564ab4aff50b6e321570123169bad0ffad9f678ce4105a4a5df\": container with ID starting with 95a0a0e65faf9564ab4aff50b6e321570123169bad0ffad9f678ce4105a4a5df not found: ID does not exist" containerID="95a0a0e65faf9564ab4aff50b6e321570123169bad0ffad9f678ce4105a4a5df"
Oct 31 19:51:58.592874 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:58.592797    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95a0a0e65faf9564ab4aff50b6e321570123169bad0ffad9f678ce4105a4a5df"} err="failed to get container status \"95a0a0e65faf9564ab4aff50b6e321570123169bad0ffad9f678ce4105a4a5df\": rpc error: code = NotFound desc = could not find container \"95a0a0e65faf9564ab4aff50b6e321570123169bad0ffad9f678ce4105a4a5df\": container with ID starting with 95a0a0e65faf9564ab4aff50b6e321570123169bad0ffad9f678ce4105a4a5df not found: ID does not exist"
Oct 31 19:51:58.611120 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:58.611088    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np"]
Oct 31 19:51:58.616889 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:58.616866    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-8dcfa8-openshift-default-b5bfd9bf-cz2np"]
Oct 31 19:51:59.081262 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:59.081214    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:51:59.550500 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:59.550465    4645 patch_prober.go:29] interesting pod/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body=
Oct 31 19:51:59.550706 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:59.550522    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2" podUID="90bf19b5-9c42-4481-873e-c55d96f717eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused"
Oct 31 19:51:59.620185 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:59.620147    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c712ccf6-e167-4c11-ba0a-df297a0caf05" path="/var/lib/kubelet/pods/c712ccf6-e167-4c11-ba0a-df297a0caf05/volumes"
Oct 31 19:51:59.848731 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:51:59.848634    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:00.550417 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:00.550383    4645 patch_prober.go:29] interesting pod/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body=
Oct 31 19:52:00.550632 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:00.550443    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2" podUID="90bf19b5-9c42-4481-873e-c55d96f717eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused"
Oct 31 19:52:00.639267 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:00.639230    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:00.737682 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:00.737646    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8dcfa8-68fb9b44f5-6dmlv" podUID="0ec69924-e842-432a-a9fa-05db6c0d0fbf" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:52:00.788362 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:00.784349    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:01.550927 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:01.550899    4645 patch_prober.go:29] interesting pod/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body=
Oct 31 19:52:01.551093 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:01.550947    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2" podUID="90bf19b5-9c42-4481-873e-c55d96f717eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused"
Oct 31 19:52:01.745671 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:01.745641    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:02.104264 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:02.104229    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-f5c1ab/auth-test-model-f5c1ab-kserve-77f7d9ccc-4ljhr"
Oct 31 19:52:02.124115 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:02.124088    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-f5c1ab/auth-test-model-f5c1ab-kserve-77f7d9ccc-4ljhr"
Oct 31 19:52:02.555818 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:02.555785    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2"
Oct 31 19:52:02.556792 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:02.556763    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2"
Oct 31 19:52:02.557478 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:02.557455    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2"
Oct 31 19:52:02.625003 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:02.624933    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:02.648065 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:02.648038    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-f5c1ab/auth-test-model-f5c1ab-kserve-77f7d9ccc-4ljhr"
Oct 31 19:52:03.449890 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:03.449854    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:04.288785 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:04.288755    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w6-a891ce/auth-test-model-a891ce-kserve-c6cd86cc7-vckq2"]
Oct 31 19:52:04.289718 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:04.289692    4645 cpu_manager.go:414] "containerMap: removing container" podUID="c712ccf6-e167-4c11-ba0a-df297a0caf05" containerName="istio-proxy"
Oct 31 19:52:04.289718 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:04.289716    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c712ccf6-e167-4c11-ba0a-df297a0caf05" containerName="istio-proxy"
Oct 31 19:52:04.289915 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:04.289836    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="c712ccf6-e167-4c11-ba0a-df297a0caf05" containerName="istio-proxy"
Oct 31 19:52:04.293499 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:04.293480    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-a891ce/auth-test-model-a891ce-kserve-c6cd86cc7-vckq2"
Oct 31 19:52:04.310948 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:04.310918    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w6-a891ce/auth-test-model-a891ce-kserve-c6cd86cc7-vckq2"]
Oct 31 19:52:04.314024 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:04.314000    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w6-a891ce\"/\"default-dockercfg-m26fx\""
Oct 31 19:52:04.314140 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:04.314072    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w6-a891ce\"/\"openshift-service-ca.crt\""
Oct 31 19:52:04.314564 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:04.314540    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w6-a891ce\"/\"kube-root-ca.crt\""
Oct 31 19:52:04.314953 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:04.314929    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w6-a891ce\"/\"auth-test-model-a891ce-kserve-self-signed-certs\""
Oct 31 19:52:04.384577 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:04.384549    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:04.447849 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:04.447820    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfqh7\" (UniqueName: \"kubernetes.io/projected/5f6f43f0-380a-45d9-a8ab-324872927234-kube-api-access-sfqh7\") pod \"auth-test-model-a891ce-kserve-c6cd86cc7-vckq2\" (UID: \"5f6f43f0-380a-45d9-a8ab-324872927234\") " pod="ai-tenant-e2e-shared-a-w6-a891ce/auth-test-model-a891ce-kserve-c6cd86cc7-vckq2"
Oct 31 19:52:04.448007 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:04.447869    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f6f43f0-380a-45d9-a8ab-324872927234-dshm\") pod \"auth-test-model-a891ce-kserve-c6cd86cc7-vckq2\" (UID: \"5f6f43f0-380a-45d9-a8ab-324872927234\") " pod="ai-tenant-e2e-shared-a-w6-a891ce/auth-test-model-a891ce-kserve-c6cd86cc7-vckq2"
Oct 31 19:52:04.448007 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:04.447935    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f6f43f0-380a-45d9-a8ab-324872927234-tmp-dir\") pod \"auth-test-model-a891ce-kserve-c6cd86cc7-vckq2\" (UID: \"5f6f43f0-380a-45d9-a8ab-324872927234\") " pod="ai-tenant-e2e-shared-a-w6-a891ce/auth-test-model-a891ce-kserve-c6cd86cc7-vckq2"
Oct 31 19:52:04.448103 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:04.448088    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f6f43f0-380a-45d9-a8ab-324872927234-home\") pod \"auth-test-model-a891ce-kserve-c6cd86cc7-vckq2\" (UID: \"5f6f43f0-380a-45d9-a8ab-324872927234\") " pod="ai-tenant-e2e-shared-a-w6-a891ce/auth-test-model-a891ce-kserve-c6cd86cc7-vckq2"
Oct 31 19:52:04.448292 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:04.448279    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f6f43f0-380a-45d9-a8ab-324872927234-model-cache\") pod \"auth-test-model-a891ce-kserve-c6cd86cc7-vckq2\" (UID: \"5f6f43f0-380a-45d9-a8ab-324872927234\") " pod="ai-tenant-e2e-shared-a-w6-a891ce/auth-test-model-a891ce-kserve-c6cd86cc7-vckq2"
Oct 31 19:52:04.448372 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:04.448309    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f6f43f0-380a-45d9-a8ab-324872927234-tls-certs\") pod \"auth-test-model-a891ce-kserve-c6cd86cc7-vckq2\" (UID: \"5f6f43f0-380a-45d9-a8ab-324872927234\") " pod="ai-tenant-e2e-shared-a-w6-a891ce/auth-test-model-a891ce-kserve-c6cd86cc7-vckq2"
Oct 31 19:52:04.511793 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:04.511759    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:04.616151 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:04.553489    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f6f43f0-380a-45d9-a8ab-324872927234-model-cache\") pod \"auth-test-model-a891ce-kserve-c6cd86cc7-vckq2\" (UID: \"5f6f43f0-380a-45d9-a8ab-324872927234\") " pod="ai-tenant-e2e-shared-a-w6-a891ce/auth-test-model-a891ce-kserve-c6cd86cc7-vckq2"
Oct 31 19:52:04.616151 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:04.553553    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f6f43f0-380a-45d9-a8ab-324872927234-tls-certs\") pod \"auth-test-model-a891ce-kserve-c6cd86cc7-vckq2\" (UID: \"5f6f43f0-380a-45d9-a8ab-324872927234\") " pod="ai-tenant-e2e-shared-a-w6-a891ce/auth-test-model-a891ce-kserve-c6cd86cc7-vckq2"
Oct 31 19:52:04.616151 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:04.553642    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfqh7\" (UniqueName: \"kubernetes.io/projected/5f6f43f0-380a-45d9-a8ab-324872927234-kube-api-access-sfqh7\") pod \"auth-test-model-a891ce-kserve-c6cd86cc7-vckq2\" (UID: \"5f6f43f0-380a-45d9-a8ab-324872927234\") " pod="ai-tenant-e2e-shared-a-w6-a891ce/auth-test-model-a891ce-kserve-c6cd86cc7-vckq2"
Oct 31 19:52:04.616151 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:04.553696    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f6f43f0-380a-45d9-a8ab-324872927234-dshm\") pod \"auth-test-model-a891ce-kserve-c6cd86cc7-vckq2\" (UID: \"5f6f43f0-380a-45d9-a8ab-324872927234\") " pod="ai-tenant-e2e-shared-a-w6-a891ce/auth-test-model-a891ce-kserve-c6cd86cc7-vckq2"
Oct 31 19:52:04.616151 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:04.553720    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f6f43f0-380a-45d9-a8ab-324872927234-tmp-dir\") pod \"auth-test-model-a891ce-kserve-c6cd86cc7-vckq2\" (UID: \"5f6f43f0-380a-45d9-a8ab-324872927234\") " pod="ai-tenant-e2e-shared-a-w6-a891ce/auth-test-model-a891ce-kserve-c6cd86cc7-vckq2"
Oct 31 19:52:04.616151 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:04.553890    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f6f43f0-380a-45d9-a8ab-324872927234-home\") pod \"auth-test-model-a891ce-kserve-c6cd86cc7-vckq2\" (UID: \"5f6f43f0-380a-45d9-a8ab-324872927234\") " pod="ai-tenant-e2e-shared-a-w6-a891ce/auth-test-model-a891ce-kserve-c6cd86cc7-vckq2"
Oct 31 19:52:04.616151 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:04.554529    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f6f43f0-380a-45d9-a8ab-324872927234-home\") pod \"auth-test-model-a891ce-kserve-c6cd86cc7-vckq2\" (UID: \"5f6f43f0-380a-45d9-a8ab-324872927234\") " pod="ai-tenant-e2e-shared-a-w6-a891ce/auth-test-model-a891ce-kserve-c6cd86cc7-vckq2"
Oct 31 19:52:04.616151 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:04.554796    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f6f43f0-380a-45d9-a8ab-324872927234-model-cache\") pod \"auth-test-model-a891ce-kserve-c6cd86cc7-vckq2\" (UID: \"5f6f43f0-380a-45d9-a8ab-324872927234\") " pod="ai-tenant-e2e-shared-a-w6-a891ce/auth-test-model-a891ce-kserve-c6cd86cc7-vckq2"
Oct 31 19:52:04.616151 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:04.580111    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f6f43f0-380a-45d9-a8ab-324872927234-tmp-dir\") pod \"auth-test-model-a891ce-kserve-c6cd86cc7-vckq2\" (UID: \"5f6f43f0-380a-45d9-a8ab-324872927234\") " pod="ai-tenant-e2e-shared-a-w6-a891ce/auth-test-model-a891ce-kserve-c6cd86cc7-vckq2"
Oct 31 19:52:04.653393 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:04.641843    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f6f43f0-380a-45d9-a8ab-324872927234-dshm\") pod \"auth-test-model-a891ce-kserve-c6cd86cc7-vckq2\" (UID: \"5f6f43f0-380a-45d9-a8ab-324872927234\") " pod="ai-tenant-e2e-shared-a-w6-a891ce/auth-test-model-a891ce-kserve-c6cd86cc7-vckq2"
Oct 31 19:52:04.664312 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:04.660795    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfqh7\" (UniqueName: \"kubernetes.io/projected/5f6f43f0-380a-45d9-a8ab-324872927234-kube-api-access-sfqh7\") pod \"auth-test-model-a891ce-kserve-c6cd86cc7-vckq2\" (UID: \"5f6f43f0-380a-45d9-a8ab-324872927234\") " pod="ai-tenant-e2e-shared-a-w6-a891ce/auth-test-model-a891ce-kserve-c6cd86cc7-vckq2"
Oct 31 19:52:04.684460 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:04.669901    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f6f43f0-380a-45d9-a8ab-324872927234-tls-certs\") pod \"auth-test-model-a891ce-kserve-c6cd86cc7-vckq2\" (UID: \"5f6f43f0-380a-45d9-a8ab-324872927234\") " pod="ai-tenant-e2e-shared-a-w6-a891ce/auth-test-model-a891ce-kserve-c6cd86cc7-vckq2"
Oct 31 19:52:04.713144 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:04.711495    4645 generic.go:356] "Generic (PLEG): container finished" podID="0ec69924-e842-432a-a9fa-05db6c0d0fbf" containerID="cda7edf66edde4444772acf54632eeefb09f20a62d103ac4ab89940c0abfa530" exitCode=0
Oct 31 19:52:04.713144 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:04.711579    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8dcfa8-68fb9b44f5-6dmlv" event={"ID":"0ec69924-e842-432a-a9fa-05db6c0d0fbf","Type":"ContainerDied","Data":"cda7edf66edde4444772acf54632eeefb09f20a62d103ac4ab89940c0abfa530"}
Oct 31 19:52:04.727882 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:04.727858    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8dcfa8-68fb9b44f5-6dmlv"
Oct 31 19:52:04.858722 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:04.858693    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ec69924-e842-432a-a9fa-05db6c0d0fbf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ec69924-e842-432a-a9fa-05db6c0d0fbf-plugins-config-volume\") pod \"0ec69924-e842-432a-a9fa-05db6c0d0fbf\" (UID: \"0ec69924-e842-432a-a9fa-05db6c0d0fbf\") "
Oct 31 19:52:04.858865 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:04.858783    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ec69924-e842-432a-a9fa-05db6c0d0fbf-kube-api-access-bfmjk\" (UniqueName: \"kubernetes.io/projected/0ec69924-e842-432a-a9fa-05db6c0d0fbf-kube-api-access-bfmjk\") pod \"0ec69924-e842-432a-a9fa-05db6c0d0fbf\" (UID: \"0ec69924-e842-432a-a9fa-05db6c0d0fbf\") "
Oct 31 19:52:04.859044 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:04.859024    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ec69924-e842-432a-a9fa-05db6c0d0fbf-plugins-config-volume" pod "0ec69924-e842-432a-a9fa-05db6c0d0fbf" (UID: "0ec69924-e842-432a-a9fa-05db6c0d0fbf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:52:04.859262 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:04.859239    4645 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ec69924-e842-432a-a9fa-05db6c0d0fbf-plugins-config-volume\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:52:04.860803 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:04.860779    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ec69924-e842-432a-a9fa-05db6c0d0fbf-kube-api-access-bfmjk" pod "0ec69924-e842-432a-a9fa-05db6c0d0fbf" (UID: "0ec69924-e842-432a-a9fa-05db6c0d0fbf"). InnerVolumeSpecName "kube-api-access-bfmjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:52:04.955110 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:04.955085    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-a891ce/auth-test-model-a891ce-kserve-c6cd86cc7-vckq2"
Oct 31 19:52:04.960826 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:04.960801    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfmjk\" (UniqueName: \"kubernetes.io/projected/0ec69924-e842-432a-a9fa-05db6c0d0fbf-kube-api-access-bfmjk\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:52:05.240406 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:05.240376    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w6-a891ce/auth-test-model-a891ce-kserve-c6cd86cc7-vckq2"]
Oct 31 19:52:05.240662 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:52:05.240643    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5f6f43f0_380a_45d9_a8ab_324872927234.slice/crio-b6db8efcb580fb5bda9909cb12a1c38debd1a3903bb302860f33061355060a5a WatchSource:0}: Error finding container b6db8efcb580fb5bda9909cb12a1c38debd1a3903bb302860f33061355060a5a: Status 404 returned error can't find the container with id b6db8efcb580fb5bda9909cb12a1c38debd1a3903bb302860f33061355060a5a
Oct 31 19:52:05.607007 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:05.606923    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:05.723518 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:05.723478    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8dcfa8-68fb9b44f5-6dmlv" event={"ID":"0ec69924-e842-432a-a9fa-05db6c0d0fbf","Type":"ContainerDied","Data":"79bc132b69b670f0cc12c147ae27cb8ab0025eba0d3467cc0aec5c39deaab2ea"}
Oct 31 19:52:05.723741 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:05.723588    4645 scope.go:122] "RemoveContainer" containerID="cda7edf66edde4444772acf54632eeefb09f20a62d103ac4ab89940c0abfa530"
Oct 31 19:52:05.723896 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:05.723871    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8dcfa8-68fb9b44f5-6dmlv"
Oct 31 19:52:05.725868 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:05.725815    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-a891ce/auth-test-model-a891ce-kserve-c6cd86cc7-vckq2" event={"ID":"5f6f43f0-380a-45d9-a8ab-324872927234","Type":"ContainerStarted","Data":"e6425167d186986409fc9f751f86fc01d76cb8e70fc4c95b8e65b6e3ff58da83"}
Oct 31 19:52:05.725868 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:05.725862    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-a891ce/auth-test-model-a891ce-kserve-c6cd86cc7-vckq2" event={"ID":"5f6f43f0-380a-45d9-a8ab-324872927234","Type":"ContainerStarted","Data":"b6db8efcb580fb5bda9909cb12a1c38debd1a3903bb302860f33061355060a5a"}
Oct 31 19:52:05.762702 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:05.762658    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w6-a891ce/auth-test-model-a891ce-kserve-c6cd86cc7-vckq2" podStartSLOduration=1.7626442610000002 podStartE2EDuration="1.762644261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:52:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:52:05.758720089 +0000 UTC m=+1956.331954219" watchObservedRunningTime="2026-10-31 19:52:05.762644261 +0000 UTC m=+1956.335878429"
Oct 31 19:52:05.788680 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:05.788643    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8dcfa8-68fb9b44f5-6dmlv"]
Oct 31 19:52:05.792988 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:05.792964    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8dcfa8-68fb9b44f5-6dmlv"]
Oct 31 19:52:05.956028 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:05.956003    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w6-a891ce/auth-test-model-a891ce-kserve-c6cd86cc7-vckq2"
Oct 31 19:52:05.961896 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:05.961860    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:06.247035 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:52:06.246958    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 19:52:06.976811 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:06.974192    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:07.619843 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:07.619801    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ec69924-e842-432a-a9fa-05db6c0d0fbf" path="/var/lib/kubelet/pods/0ec69924-e842-432a-a9fa-05db6c0d0fbf/volumes"
Oct 31 19:52:07.916555 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:07.908008    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:08.630618 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:08.630580    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:09.537220 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:09.537186    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:10.482401 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:10.482365    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:11.259398 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:11.259360    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:11.365738 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:11.365696    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:11.561060 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:11.560969    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:12.358040 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:12.358007    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:13.189843 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:13.187981    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:14.182681 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:14.182646    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:14.955393 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:14.955360    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w6-a891ce/auth-test-model-a891ce-kserve-c6cd86cc7-vckq2"
Oct 31 19:52:14.967543 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:14.967522    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w6-a891ce/auth-test-model-a891ce-kserve-c6cd86cc7-vckq2"
Oct 31 19:52:15.050893 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:15.050866    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:15.864830 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:15.864800    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w6-a891ce/auth-test-model-a891ce-kserve-c6cd86cc7-vckq2"
Oct 31 19:52:15.921943 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:15.921902    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:16.392427 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:16.392400    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm"]
Oct 31 19:52:16.392842 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:16.392577    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm" podUID="940375be-4b29-43ca-b0de-1643eaa82341" containerName="istio-proxy" containerID="cri-o://33261bccefb45b8390c33d027c8835feecd482139c806933f79afd02a02e4d37" gracePeriod=30
Oct 31 19:52:16.743736 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:16.743705    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:17.000549 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:17.000436    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:17.590911 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:17.590867    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:18.116212 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:18.116157    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:18.554991 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:18.554961    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:18.789688 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:18.789657    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:19.379016 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:19.378985    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:20.164813 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:20.164776    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:20.300900 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:20.300862    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-435728/auth-test-model-435728-kserve-76df7cb4d9-t5bn2"]
Oct 31 19:52:20.302010 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:20.301982    4645 cpu_manager.go:414] "containerMap: removing container" podUID="0ec69924-e842-432a-a9fa-05db6c0d0fbf" containerName="payload-pre-processing"
Oct 31 19:52:20.302142 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:20.302026    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ec69924-e842-432a-a9fa-05db6c0d0fbf" containerName="payload-pre-processing"
Oct 31 19:52:20.302295 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:20.302279    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ec69924-e842-432a-a9fa-05db6c0d0fbf" containerName="payload-pre-processing"
Oct 31 19:52:20.306517 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:20.306494    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-435728/auth-test-model-435728-kserve-76df7cb4d9-t5bn2"
Oct 31 19:52:20.318965 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:20.318942    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-435728\"/\"auth-test-model-435728-kserve-self-signed-certs\""
Oct 31 19:52:20.319304 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:20.319193    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-435728\"/\"openshift-service-ca.crt\""
Oct 31 19:52:20.319490 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:20.319227    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-435728\"/\"kube-root-ca.crt\""
Oct 31 19:52:20.319490 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:20.319302    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-435728\"/\"default-dockercfg-7q8xn\""
Oct 31 19:52:20.320550 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:20.320526    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-435728/auth-test-model-435728-kserve-76df7cb4d9-t5bn2"]
Oct 31 19:52:20.377450 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:20.377422    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:20.398738 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:20.398716    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa08927f-11e4-458e-beac-c96cfa4de425-tmp-dir\") pod \"auth-test-model-435728-kserve-76df7cb4d9-t5bn2\" (UID: \"fa08927f-11e4-458e-beac-c96cfa4de425\") " pod="ai-tenant-e2e-shared-b-w1-435728/auth-test-model-435728-kserve-76df7cb4d9-t5bn2"
Oct 31 19:52:20.398863 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:20.398846    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa08927f-11e4-458e-beac-c96cfa4de425-dshm\") pod \"auth-test-model-435728-kserve-76df7cb4d9-t5bn2\" (UID: \"fa08927f-11e4-458e-beac-c96cfa4de425\") " pod="ai-tenant-e2e-shared-b-w1-435728/auth-test-model-435728-kserve-76df7cb4d9-t5bn2"
Oct 31 19:52:20.399036 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:20.398941    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa08927f-11e4-458e-beac-c96cfa4de425-home\") pod \"auth-test-model-435728-kserve-76df7cb4d9-t5bn2\" (UID: \"fa08927f-11e4-458e-beac-c96cfa4de425\") " pod="ai-tenant-e2e-shared-b-w1-435728/auth-test-model-435728-kserve-76df7cb4d9-t5bn2"
Oct 31 19:52:20.399095 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:20.399078    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa08927f-11e4-458e-beac-c96cfa4de425-model-cache\") pod \"auth-test-model-435728-kserve-76df7cb4d9-t5bn2\" (UID: \"fa08927f-11e4-458e-beac-c96cfa4de425\") " pod="ai-tenant-e2e-shared-b-w1-435728/auth-test-model-435728-kserve-76df7cb4d9-t5bn2"
Oct 31 19:52:20.399139 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:20.399108    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcshj\" (UniqueName: \"kubernetes.io/projected/fa08927f-11e4-458e-beac-c96cfa4de425-kube-api-access-jcshj\") pod \"auth-test-model-435728-kserve-76df7cb4d9-t5bn2\" (UID: \"fa08927f-11e4-458e-beac-c96cfa4de425\") " pod="ai-tenant-e2e-shared-b-w1-435728/auth-test-model-435728-kserve-76df7cb4d9-t5bn2"
Oct 31 19:52:20.399200 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:20.399148    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa08927f-11e4-458e-beac-c96cfa4de425-tls-certs\") pod \"auth-test-model-435728-kserve-76df7cb4d9-t5bn2\" (UID: \"fa08927f-11e4-458e-beac-c96cfa4de425\") " pod="ai-tenant-e2e-shared-b-w1-435728/auth-test-model-435728-kserve-76df7cb4d9-t5bn2"
Oct 31 19:52:20.505633 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:20.505551    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa08927f-11e4-458e-beac-c96cfa4de425-tmp-dir\") pod \"auth-test-model-435728-kserve-76df7cb4d9-t5bn2\" (UID: \"fa08927f-11e4-458e-beac-c96cfa4de425\") " pod="ai-tenant-e2e-shared-b-w1-435728/auth-test-model-435728-kserve-76df7cb4d9-t5bn2"
Oct 31 19:52:20.505979 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:20.505954    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa08927f-11e4-458e-beac-c96cfa4de425-dshm\") pod \"auth-test-model-435728-kserve-76df7cb4d9-t5bn2\" (UID: \"fa08927f-11e4-458e-beac-c96cfa4de425\") " pod="ai-tenant-e2e-shared-b-w1-435728/auth-test-model-435728-kserve-76df7cb4d9-t5bn2"
Oct 31 19:52:20.506159 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:20.506142    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa08927f-11e4-458e-beac-c96cfa4de425-home\") pod \"auth-test-model-435728-kserve-76df7cb4d9-t5bn2\" (UID: \"fa08927f-11e4-458e-beac-c96cfa4de425\") " pod="ai-tenant-e2e-shared-b-w1-435728/auth-test-model-435728-kserve-76df7cb4d9-t5bn2"
Oct 31 19:52:20.506448 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:20.506427    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa08927f-11e4-458e-beac-c96cfa4de425-model-cache\") pod \"auth-test-model-435728-kserve-76df7cb4d9-t5bn2\" (UID: \"fa08927f-11e4-458e-beac-c96cfa4de425\") " pod="ai-tenant-e2e-shared-b-w1-435728/auth-test-model-435728-kserve-76df7cb4d9-t5bn2"
Oct 31 19:52:20.506629 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:20.506596    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcshj\" (UniqueName: \"kubernetes.io/projected/fa08927f-11e4-458e-beac-c96cfa4de425-kube-api-access-jcshj\") pod \"auth-test-model-435728-kserve-76df7cb4d9-t5bn2\" (UID: \"fa08927f-11e4-458e-beac-c96cfa4de425\") " pod="ai-tenant-e2e-shared-b-w1-435728/auth-test-model-435728-kserve-76df7cb4d9-t5bn2"
Oct 31 19:52:20.506752 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:20.506177    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa08927f-11e4-458e-beac-c96cfa4de425-tmp-dir\") pod \"auth-test-model-435728-kserve-76df7cb4d9-t5bn2\" (UID: \"fa08927f-11e4-458e-beac-c96cfa4de425\") " pod="ai-tenant-e2e-shared-b-w1-435728/auth-test-model-435728-kserve-76df7cb4d9-t5bn2"
Oct 31 19:52:20.506871 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:20.506723    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa08927f-11e4-458e-beac-c96cfa4de425-home\") pod \"auth-test-model-435728-kserve-76df7cb4d9-t5bn2\" (UID: \"fa08927f-11e4-458e-beac-c96cfa4de425\") " pod="ai-tenant-e2e-shared-b-w1-435728/auth-test-model-435728-kserve-76df7cb4d9-t5bn2"
Oct 31 19:52:20.507024 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:20.506926    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa08927f-11e4-458e-beac-c96cfa4de425-tls-certs\") pod \"auth-test-model-435728-kserve-76df7cb4d9-t5bn2\" (UID: \"fa08927f-11e4-458e-beac-c96cfa4de425\") " pod="ai-tenant-e2e-shared-b-w1-435728/auth-test-model-435728-kserve-76df7cb4d9-t5bn2"
Oct 31 19:52:20.507961 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:20.507941    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa08927f-11e4-458e-beac-c96cfa4de425-model-cache\") pod \"auth-test-model-435728-kserve-76df7cb4d9-t5bn2\" (UID: \"fa08927f-11e4-458e-beac-c96cfa4de425\") " pod="ai-tenant-e2e-shared-b-w1-435728/auth-test-model-435728-kserve-76df7cb4d9-t5bn2"
Oct 31 19:52:20.512512 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:20.512465    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa08927f-11e4-458e-beac-c96cfa4de425-dshm\") pod \"auth-test-model-435728-kserve-76df7cb4d9-t5bn2\" (UID: \"fa08927f-11e4-458e-beac-c96cfa4de425\") " pod="ai-tenant-e2e-shared-b-w1-435728/auth-test-model-435728-kserve-76df7cb4d9-t5bn2"
Oct 31 19:52:20.542095 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:20.540669    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa08927f-11e4-458e-beac-c96cfa4de425-tls-certs\") pod \"auth-test-model-435728-kserve-76df7cb4d9-t5bn2\" (UID: \"fa08927f-11e4-458e-beac-c96cfa4de425\") " pod="ai-tenant-e2e-shared-b-w1-435728/auth-test-model-435728-kserve-76df7cb4d9-t5bn2"
Oct 31 19:52:20.547554 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:20.547477    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcshj\" (UniqueName: \"kubernetes.io/projected/fa08927f-11e4-458e-beac-c96cfa4de425-kube-api-access-jcshj\") pod \"auth-test-model-435728-kserve-76df7cb4d9-t5bn2\" (UID: \"fa08927f-11e4-458e-beac-c96cfa4de425\") " pod="ai-tenant-e2e-shared-b-w1-435728/auth-test-model-435728-kserve-76df7cb4d9-t5bn2"
Oct 31 19:52:20.622350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:20.621304    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-435728/auth-test-model-435728-kserve-76df7cb4d9-t5bn2"
Oct 31 19:52:20.959258 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:52:20.959227    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfa08927f_11e4_458e_beac_c96cfa4de425.slice/crio-c5d25a7313f77443448697588368790a33ded3909fe91a8138c72fe3ad6682fb WatchSource:0}: Error finding container c5d25a7313f77443448697588368790a33ded3909fe91a8138c72fe3ad6682fb: Status 404 returned error can't find the container with id c5d25a7313f77443448697588368790a33ded3909fe91a8138c72fe3ad6682fb
Oct 31 19:52:20.959765 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:20.959742    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-435728/auth-test-model-435728-kserve-76df7cb4d9-t5bn2"]
Oct 31 19:52:21.117187 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:21.117080    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:21.607691 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:21.607656    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-d8c398/test-model-d8c398-kserve-57476967b9-v6ld4"]
Oct 31 19:52:21.682745 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:21.682700    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:21.823354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:21.820261    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm"
Oct 31 19:52:21.912840 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:21.912798    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-435728/auth-test-model-435728-kserve-76df7cb4d9-t5bn2" event={"ID":"fa08927f-11e4-458e-beac-c96cfa4de425","Type":"ContainerStarted","Data":"f5828703961f47a98c706588e1954631d1fbd12aa89119a568e04c13bca9bc77"}
Oct 31 19:52:21.913050 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:21.912855    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-435728/auth-test-model-435728-kserve-76df7cb4d9-t5bn2" event={"ID":"fa08927f-11e4-458e-beac-c96cfa4de425","Type":"ContainerStarted","Data":"c5d25a7313f77443448697588368790a33ded3909fe91a8138c72fe3ad6682fb"}
Oct 31 19:52:21.914488 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:21.914458    4645 generic.go:356] "Generic (PLEG): container finished" podID="940375be-4b29-43ca-b0de-1643eaa82341" containerID="33261bccefb45b8390c33d027c8835feecd482139c806933f79afd02a02e4d37" exitCode=0
Oct 31 19:52:21.914611 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:21.914491    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm" event={"ID":"940375be-4b29-43ca-b0de-1643eaa82341","Type":"ContainerDied","Data":"33261bccefb45b8390c33d027c8835feecd482139c806933f79afd02a02e4d37"}
Oct 31 19:52:21.914611 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:21.914542    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm" event={"ID":"940375be-4b29-43ca-b0de-1643eaa82341","Type":"ContainerDied","Data":"8df25cfd4d3a9c7af50b381668759ec5847688a778ef448463883a3654c97c38"}
Oct 31 19:52:21.914611 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:21.914501    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm"
Oct 31 19:52:21.914725 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:21.914565    4645 scope.go:122] "RemoveContainer" containerID="33261bccefb45b8390c33d027c8835feecd482139c806933f79afd02a02e4d37"
Oct 31 19:52:21.926905 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:21.926882    4645 scope.go:122] "RemoveContainer" containerID="33261bccefb45b8390c33d027c8835feecd482139c806933f79afd02a02e4d37"
Oct 31 19:52:21.927222 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:52:21.927196    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33261bccefb45b8390c33d027c8835feecd482139c806933f79afd02a02e4d37\": container with ID starting with 33261bccefb45b8390c33d027c8835feecd482139c806933f79afd02a02e4d37 not found: ID does not exist" containerID="33261bccefb45b8390c33d027c8835feecd482139c806933f79afd02a02e4d37"
Oct 31 19:52:21.927379 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:21.927236    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33261bccefb45b8390c33d027c8835feecd482139c806933f79afd02a02e4d37"} err="failed to get container status \"33261bccefb45b8390c33d027c8835feecd482139c806933f79afd02a02e4d37\": rpc error: code = NotFound desc = could not find container \"33261bccefb45b8390c33d027c8835feecd482139c806933f79afd02a02e4d37\": container with ID starting with 33261bccefb45b8390c33d027c8835feecd482139c806933f79afd02a02e4d37 not found: ID does not exist"
Oct 31 19:52:21.935099 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:21.935019    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-435728/auth-test-model-435728-kserve-76df7cb4d9-t5bn2" podStartSLOduration=1.935003562 podStartE2EDuration="1.935003562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:52:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:52:21.932486556 +0000 UTC m=+1972.505720682" watchObservedRunningTime="2026-10-31 19:52:21.935003562 +0000 UTC m=+1972.508237690"
Oct 31 19:52:21.964171 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:21.964146    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/940375be-4b29-43ca-b0de-1643eaa82341-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/940375be-4b29-43ca-b0de-1643eaa82341-istio-podinfo\") pod \"940375be-4b29-43ca-b0de-1643eaa82341\" (UID: \"940375be-4b29-43ca-b0de-1643eaa82341\") "
Oct 31 19:52:21.964313 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:21.964272    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/940375be-4b29-43ca-b0de-1643eaa82341-kube-api-access-qhwjb\" (UniqueName: \"kubernetes.io/projected/940375be-4b29-43ca-b0de-1643eaa82341-kube-api-access-qhwjb\") pod \"940375be-4b29-43ca-b0de-1643eaa82341\" (UID: \"940375be-4b29-43ca-b0de-1643eaa82341\") "
Oct 31 19:52:21.964313 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:21.964306    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/940375be-4b29-43ca-b0de-1643eaa82341-istio-token\" (UniqueName: \"kubernetes.io/projected/940375be-4b29-43ca-b0de-1643eaa82341-istio-token\") pod \"940375be-4b29-43ca-b0de-1643eaa82341\" (UID: \"940375be-4b29-43ca-b0de-1643eaa82341\") "
Oct 31 19:52:21.964490 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:21.964408    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/940375be-4b29-43ca-b0de-1643eaa82341-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/940375be-4b29-43ca-b0de-1643eaa82341-istiod-ca-cert\") pod \"940375be-4b29-43ca-b0de-1643eaa82341\" (UID: \"940375be-4b29-43ca-b0de-1643eaa82341\") "
Oct 31 19:52:21.964490 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:21.964448    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/940375be-4b29-43ca-b0de-1643eaa82341-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/940375be-4b29-43ca-b0de-1643eaa82341-istio-envoy\") pod \"940375be-4b29-43ca-b0de-1643eaa82341\" (UID: \"940375be-4b29-43ca-b0de-1643eaa82341\") "
Oct 31 19:52:21.964490 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:21.964467    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/940375be-4b29-43ca-b0de-1643eaa82341-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/940375be-4b29-43ca-b0de-1643eaa82341-workload-certs\") pod \"940375be-4b29-43ca-b0de-1643eaa82341\" (UID: \"940375be-4b29-43ca-b0de-1643eaa82341\") "
Oct 31 19:52:21.964661 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:21.964514    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/940375be-4b29-43ca-b0de-1643eaa82341-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/940375be-4b29-43ca-b0de-1643eaa82341-istio-data\") pod \"940375be-4b29-43ca-b0de-1643eaa82341\" (UID: \"940375be-4b29-43ca-b0de-1643eaa82341\") "
Oct 31 19:52:21.964727 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:21.964698    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/940375be-4b29-43ca-b0de-1643eaa82341-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/940375be-4b29-43ca-b0de-1643eaa82341-credential-socket\") pod \"940375be-4b29-43ca-b0de-1643eaa82341\" (UID: \"940375be-4b29-43ca-b0de-1643eaa82341\") "
Oct 31 19:52:21.964783 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:21.964731    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/940375be-4b29-43ca-b0de-1643eaa82341-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/940375be-4b29-43ca-b0de-1643eaa82341-workload-socket\") pod \"940375be-4b29-43ca-b0de-1643eaa82341\" (UID: \"940375be-4b29-43ca-b0de-1643eaa82341\") "
Oct 31 19:52:21.964834 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:21.964807    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/940375be-4b29-43ca-b0de-1643eaa82341-workload-certs" pod "940375be-4b29-43ca-b0de-1643eaa82341" (UID: "940375be-4b29-43ca-b0de-1643eaa82341"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:52:21.964938 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:21.964915    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/940375be-4b29-43ca-b0de-1643eaa82341-istio-data" pod "940375be-4b29-43ca-b0de-1643eaa82341" (UID: "940375be-4b29-43ca-b0de-1643eaa82341"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:52:21.965050 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:21.965030    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/940375be-4b29-43ca-b0de-1643eaa82341-credential-socket" pod "940375be-4b29-43ca-b0de-1643eaa82341" (UID: "940375be-4b29-43ca-b0de-1643eaa82341"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:52:21.965111 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:21.965101    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/940375be-4b29-43ca-b0de-1643eaa82341-istiod-ca-cert" pod "940375be-4b29-43ca-b0de-1643eaa82341" (UID: "940375be-4b29-43ca-b0de-1643eaa82341"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:52:21.965266 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:21.965242    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/940375be-4b29-43ca-b0de-1643eaa82341-workload-socket" pod "940375be-4b29-43ca-b0de-1643eaa82341" (UID: "940375be-4b29-43ca-b0de-1643eaa82341"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:52:21.965553 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:21.965533    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/940375be-4b29-43ca-b0de-1643eaa82341-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:52:21.965634 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:21.965559    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/940375be-4b29-43ca-b0de-1643eaa82341-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:52:21.965634 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:21.965572    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/940375be-4b29-43ca-b0de-1643eaa82341-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:52:21.965634 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:21.965585    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/940375be-4b29-43ca-b0de-1643eaa82341-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:52:21.965634 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:21.965594    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/940375be-4b29-43ca-b0de-1643eaa82341-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:52:21.966599 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:21.966579    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/940375be-4b29-43ca-b0de-1643eaa82341-istio-envoy" pod "940375be-4b29-43ca-b0de-1643eaa82341" (UID: "940375be-4b29-43ca-b0de-1643eaa82341"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:52:21.966720 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:21.966706    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/940375be-4b29-43ca-b0de-1643eaa82341-istio-podinfo" pod "940375be-4b29-43ca-b0de-1643eaa82341" (UID: "940375be-4b29-43ca-b0de-1643eaa82341"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:52:21.966924 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:21.966909    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/940375be-4b29-43ca-b0de-1643eaa82341-istio-token" pod "940375be-4b29-43ca-b0de-1643eaa82341" (UID: "940375be-4b29-43ca-b0de-1643eaa82341"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:52:21.966972 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:21.966942    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/940375be-4b29-43ca-b0de-1643eaa82341-kube-api-access-qhwjb" pod "940375be-4b29-43ca-b0de-1643eaa82341" (UID: "940375be-4b29-43ca-b0de-1643eaa82341"). InnerVolumeSpecName "kube-api-access-qhwjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:52:22.083403 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:22.072462    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhwjb\" (UniqueName: \"kubernetes.io/projected/940375be-4b29-43ca-b0de-1643eaa82341-kube-api-access-qhwjb\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:52:22.083403 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:22.072499    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/940375be-4b29-43ca-b0de-1643eaa82341-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:52:22.083403 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:22.072515    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/940375be-4b29-43ca-b0de-1643eaa82341-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:52:22.083403 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:22.072529    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/940375be-4b29-43ca-b0de-1643eaa82341-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:52:22.241488 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:22.241459    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm"]
Oct 31 19:52:22.245356 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:22.245314    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-fd1c91-openshift-default-7f584ff8ff-lvxjm"]
Oct 31 19:52:22.283599 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:22.283574    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:22.622206 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:22.622130    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-435728/auth-test-model-435728-kserve-76df7cb4d9-t5bn2"
Oct 31 19:52:23.284427 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:23.284389    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:23.650752 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:23.650629    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="940375be-4b29-43ca-b0de-1643eaa82341" path="/var/lib/kubelet/pods/940375be-4b29-43ca-b0de-1643eaa82341/volumes"
Oct 31 19:52:24.659049 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:24.659010    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:24.773753 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:24.773716    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:25.618353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:25.618022    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:25.925828 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:25.917782    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:26.216515 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:26.207829    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:26.802959 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:26.802919    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:27.511401 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:27.499076    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:27.639553 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:27.639432    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:28.608354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:28.607958    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:29.593352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:29.592895    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:30.621529 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:30.621496    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-435728/auth-test-model-435728-kserve-76df7cb4d9-t5bn2"
Oct 31 19:52:30.640613 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:30.640580    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-435728/auth-test-model-435728-kserve-76df7cb4d9-t5bn2"
Oct 31 19:52:30.709287 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:30.709255    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:30.989720 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:30.989691    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:31.120775 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:31.120737    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:31.424480 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:31.423722    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-435728/auth-test-model-435728-kserve-76df7cb4d9-t5bn2"
Oct 31 19:52:31.785412 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:31.784925    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:32.224289 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:32.224254    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:32.288374 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:32.287317    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-d9c048-79ff7865f5-qq5qp"]
Oct 31 19:52:32.288374 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:32.288183    4645 cpu_manager.go:414] "containerMap: removing container" podUID="940375be-4b29-43ca-b0de-1643eaa82341" containerName="istio-proxy"
Oct 31 19:52:32.288374 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:32.288220    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="940375be-4b29-43ca-b0de-1643eaa82341" containerName="istio-proxy"
Oct 31 19:52:32.288374 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:32.288348    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="940375be-4b29-43ca-b0de-1643eaa82341" containerName="istio-proxy"
Oct 31 19:52:32.294142 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:32.294120    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d9c048-79ff7865f5-qq5qp"
Oct 31 19:52:32.294925 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:32.294896    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d9c048-5459f7f8dc-rhrkn"]
Oct 31 19:52:32.300025 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:32.300007    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d9c048-5459f7f8dc-rhrkn"
Oct 31 19:52:32.309374 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:32.308072    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-d9c048-79ff7865f5-qq5qp"]
Oct 31 19:52:32.309374 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:32.308136    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-d9c048\""
Oct 31 19:52:32.309374 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:32.308681    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-d9c048-dockercfg-4x27r\""
Oct 31 19:52:32.341926 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:32.340646    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d9c048-5459f7f8dc-rhrkn"]
Oct 31 19:52:32.360567 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:32.359796    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4677f66-6fc4-4c4a-8539-ede1dfe8f503-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-d9c048-79ff7865f5-qq5qp\" (UID: \"b4677f66-6fc4-4c4a-8539-ede1dfe8f503\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d9c048-79ff7865f5-qq5qp"
Oct 31 19:52:32.360567 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:32.359885    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d33bec5-7276-4b99-9a02-6ac2eaaf770f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-d9c048-5459f7f8dc-rhrkn\" (UID: \"7d33bec5-7276-4b99-9a02-6ac2eaaf770f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d9c048-5459f7f8dc-rhrkn"
Oct 31 19:52:32.360567 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:32.359927    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wdwr\" (UniqueName: \"kubernetes.io/projected/b4677f66-6fc4-4c4a-8539-ede1dfe8f503-kube-api-access-2wdwr\") pod \"payload-processing-e2e-shared-b-w4-d9c048-79ff7865f5-qq5qp\" (UID: \"b4677f66-6fc4-4c4a-8539-ede1dfe8f503\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d9c048-79ff7865f5-qq5qp"
Oct 31 19:52:32.360567 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:32.360037    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4pxv\" (UniqueName: \"kubernetes.io/projected/7d33bec5-7276-4b99-9a02-6ac2eaaf770f-kube-api-access-b4pxv\") pod \"payload-pre-processing-e2e-shared-b-w4-d9c048-5459f7f8dc-rhrkn\" (UID: \"7d33bec5-7276-4b99-9a02-6ac2eaaf770f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d9c048-5459f7f8dc-rhrkn"
Oct 31 19:52:32.384112 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:32.380142    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:32.492401 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:32.486622    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4pxv\" (UniqueName: \"kubernetes.io/projected/7d33bec5-7276-4b99-9a02-6ac2eaaf770f-kube-api-access-b4pxv\") pod \"payload-pre-processing-e2e-shared-b-w4-d9c048-5459f7f8dc-rhrkn\" (UID: \"7d33bec5-7276-4b99-9a02-6ac2eaaf770f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d9c048-5459f7f8dc-rhrkn"
Oct 31 19:52:32.493310 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:32.493274    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4677f66-6fc4-4c4a-8539-ede1dfe8f503-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-d9c048-79ff7865f5-qq5qp\" (UID: \"b4677f66-6fc4-4c4a-8539-ede1dfe8f503\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d9c048-79ff7865f5-qq5qp"
Oct 31 19:52:32.493606 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:32.493585    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d33bec5-7276-4b99-9a02-6ac2eaaf770f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-d9c048-5459f7f8dc-rhrkn\" (UID: \"7d33bec5-7276-4b99-9a02-6ac2eaaf770f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d9c048-5459f7f8dc-rhrkn"
Oct 31 19:52:32.493765 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:32.493750    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wdwr\" (UniqueName: \"kubernetes.io/projected/b4677f66-6fc4-4c4a-8539-ede1dfe8f503-kube-api-access-2wdwr\") pod \"payload-processing-e2e-shared-b-w4-d9c048-79ff7865f5-qq5qp\" (UID: \"b4677f66-6fc4-4c4a-8539-ede1dfe8f503\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d9c048-79ff7865f5-qq5qp"
Oct 31 19:52:32.494996 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:32.494972    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4677f66-6fc4-4c4a-8539-ede1dfe8f503-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-d9c048-79ff7865f5-qq5qp\" (UID: \"b4677f66-6fc4-4c4a-8539-ede1dfe8f503\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d9c048-79ff7865f5-qq5qp"
Oct 31 19:52:32.594616 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:32.594561    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d33bec5-7276-4b99-9a02-6ac2eaaf770f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-d9c048-5459f7f8dc-rhrkn\" (UID: \"7d33bec5-7276-4b99-9a02-6ac2eaaf770f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d9c048-5459f7f8dc-rhrkn"
Oct 31 19:52:32.600116 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:32.600058    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4pxv\" (UniqueName: \"kubernetes.io/projected/7d33bec5-7276-4b99-9a02-6ac2eaaf770f-kube-api-access-b4pxv\") pod \"payload-pre-processing-e2e-shared-b-w4-d9c048-5459f7f8dc-rhrkn\" (UID: \"7d33bec5-7276-4b99-9a02-6ac2eaaf770f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d9c048-5459f7f8dc-rhrkn"
Oct 31 19:52:32.605265 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:32.605238    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wdwr\" (UniqueName: \"kubernetes.io/projected/b4677f66-6fc4-4c4a-8539-ede1dfe8f503-kube-api-access-2wdwr\") pod \"payload-processing-e2e-shared-b-w4-d9c048-79ff7865f5-qq5qp\" (UID: \"b4677f66-6fc4-4c4a-8539-ede1dfe8f503\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d9c048-79ff7865f5-qq5qp"
Oct 31 19:52:32.655531 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:32.655501    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d9c048-79ff7865f5-qq5qp"
Oct 31 19:52:32.674339 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:32.674298    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d9c048-5459f7f8dc-rhrkn"
Oct 31 19:52:32.865173 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:32.865135    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:32.977284 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:32.977249    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-d9c048-79ff7865f5-qq5qp"]
Oct 31 19:52:32.984374 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:32.984339    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-785d26-6467945f49-6xsr7"]
Oct 31 19:52:33.020942 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:33.020911    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-785d26-589cf5bf5f-fmrnc"]
Oct 31 19:52:33.027754 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:33.027730    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d9c048-5459f7f8dc-rhrkn"]
Oct 31 19:52:33.092956 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:33.092906    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:33.250490 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:33.250420    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:33.621412 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:33.621379    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d9c048-5459f7f8dc-rhrkn" event={"ID":"7d33bec5-7276-4b99-9a02-6ac2eaaf770f","Type":"ContainerStarted","Data":"93e922dc887ee5e527ed9b2837c331ca8a034671b2ddbd9fa2eb01e78364c7d3"}
Oct 31 19:52:33.621586 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:33.621444    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d9c048-5459f7f8dc-rhrkn" event={"ID":"7d33bec5-7276-4b99-9a02-6ac2eaaf770f","Type":"ContainerStarted","Data":"60ada9036ba294374cfc05df9265a7308622326c8a8031498a5b9eb91f932abe"}
Oct 31 19:52:33.621586 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:33.621459    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d9c048-79ff7865f5-qq5qp" event={"ID":"b4677f66-6fc4-4c4a-8539-ede1dfe8f503","Type":"ContainerStarted","Data":"4c0a2d2968e63a1cbf5e9c590794fbb50ff640b4998c60b08ceb9dc5aed8bff6"}
Oct 31 19:52:33.621586 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:33.621474    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d9c048-79ff7865f5-qq5qp" event={"ID":"b4677f66-6fc4-4c4a-8539-ede1dfe8f503","Type":"ContainerStarted","Data":"36c1a19e36d791dfc728bf70fcde9e4dbef4c44e056c56c402d16988d1e35d25"}
Oct 31 19:52:33.621586 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:33.621490    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d9c048-5459f7f8dc-rhrkn"
Oct 31 19:52:33.621586 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:33.621505    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d9c048-79ff7865f5-qq5qp"
Oct 31 19:52:33.671293 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:33.671239    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d9c048-5459f7f8dc-rhrkn" podStartSLOduration=1.671221595 podStartE2EDuration="1.671221595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:52:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:52:33.668165946 +0000 UTC m=+1984.241400064" watchObservedRunningTime="2026-10-31 19:52:33.671221595 +0000 UTC m=+1984.244455724"
Oct 31 19:52:33.748050 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:33.748006    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d9c048-79ff7865f5-qq5qp" podStartSLOduration=1.7479955870000001 podStartE2EDuration="1.747995587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:52:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:52:33.747927628 +0000 UTC m=+1984.321161757" watchObservedRunningTime="2026-10-31 19:52:33.747995587 +0000 UTC m=+1984.321229737"
Oct 31 19:52:34.181217 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:34.181182    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:35.070924 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:35.070874    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:36.052393 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:36.052356    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:36.247286 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:52:36.247248    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 19:52:36.637233 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:36.637139    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-d8c398/test-model-d8c398-kserve-57476967b9-v6ld4" podUID="15c43438-d0c9-41d4-92dd-788371b30f8f" containerName="main" containerID="cri-o://99755302a08703e09405870dc95679697e366ff6bb8c71587119701327f23d70" gracePeriod=45
Oct 31 19:52:36.658538 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:36.658496    4645 generic.go:356] "Generic (PLEG): container finished" podID="ede30242-d4ae-4caa-9c4b-8918314a1ead" containerID="22276da2e572fbe9afeabcf73cd849af3e094556271c14915e031ea48e2f852a" exitCode=0
Oct 31 19:52:36.658650 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:36.658560    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8" event={"ID":"ede30242-d4ae-4caa-9c4b-8918314a1ead","Type":"ContainerDied","Data":"22276da2e572fbe9afeabcf73cd849af3e094556271c14915e031ea48e2f852a"}
Oct 31 19:52:36.658972 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:36.658957    4645 scope.go:122] "RemoveContainer" containerID="22276da2e572fbe9afeabcf73cd849af3e094556271c14915e031ea48e2f852a"
Oct 31 19:52:36.660782 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:36.660753    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:52:36.660881 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:36.660825    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:52:36.660881 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:36.660858    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:52:36.665719 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:36.665701    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:37.248636 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:37.247973    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:37.249110 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:37.248761    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d8c398/test-model-d8c398-kserve-57476967b9-v6ld4"
Oct 31 19:52:37.286275 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:37.286237    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15c43438-d0c9-41d4-92dd-788371b30f8f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15c43438-d0c9-41d4-92dd-788371b30f8f-tmp-dir\") pod \"15c43438-d0c9-41d4-92dd-788371b30f8f\" (UID: \"15c43438-d0c9-41d4-92dd-788371b30f8f\") "
Oct 31 19:52:37.286490 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:37.286322    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15c43438-d0c9-41d4-92dd-788371b30f8f-kube-api-access-q4zgl\" (UniqueName: \"kubernetes.io/projected/15c43438-d0c9-41d4-92dd-788371b30f8f-kube-api-access-q4zgl\") pod \"15c43438-d0c9-41d4-92dd-788371b30f8f\" (UID: \"15c43438-d0c9-41d4-92dd-788371b30f8f\") "
Oct 31 19:52:37.286490 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:37.286438    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15c43438-d0c9-41d4-92dd-788371b30f8f-home\" (UniqueName: \"kubernetes.io/empty-dir/15c43438-d0c9-41d4-92dd-788371b30f8f-home\") pod \"15c43438-d0c9-41d4-92dd-788371b30f8f\" (UID: \"15c43438-d0c9-41d4-92dd-788371b30f8f\") "
Oct 31 19:52:37.286620 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:37.286532    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15c43438-d0c9-41d4-92dd-788371b30f8f-tmp-dir" pod "15c43438-d0c9-41d4-92dd-788371b30f8f" (UID: "15c43438-d0c9-41d4-92dd-788371b30f8f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:52:37.286620 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:37.286574    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15c43438-d0c9-41d4-92dd-788371b30f8f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/15c43438-d0c9-41d4-92dd-788371b30f8f-dshm\") pod \"15c43438-d0c9-41d4-92dd-788371b30f8f\" (UID: \"15c43438-d0c9-41d4-92dd-788371b30f8f\") "
Oct 31 19:52:37.286798 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:37.286771    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15c43438-d0c9-41d4-92dd-788371b30f8f-tls-certs\" (UniqueName: \"kubernetes.io/secret/15c43438-d0c9-41d4-92dd-788371b30f8f-tls-certs\") pod \"15c43438-d0c9-41d4-92dd-788371b30f8f\" (UID: \"15c43438-d0c9-41d4-92dd-788371b30f8f\") "
Oct 31 19:52:37.286944 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:37.286826    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15c43438-d0c9-41d4-92dd-788371b30f8f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15c43438-d0c9-41d4-92dd-788371b30f8f-model-cache\") pod \"15c43438-d0c9-41d4-92dd-788371b30f8f\" (UID: \"15c43438-d0c9-41d4-92dd-788371b30f8f\") "
Oct 31 19:52:37.287666 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:37.287207    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15c43438-d0c9-41d4-92dd-788371b30f8f-home" pod "15c43438-d0c9-41d4-92dd-788371b30f8f" (UID: "15c43438-d0c9-41d4-92dd-788371b30f8f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:52:37.287666 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:37.287502    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15c43438-d0c9-41d4-92dd-788371b30f8f-model-cache" pod "15c43438-d0c9-41d4-92dd-788371b30f8f" (UID: "15c43438-d0c9-41d4-92dd-788371b30f8f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:52:37.287949 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:37.287744    4645 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15c43438-d0c9-41d4-92dd-788371b30f8f-home\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:52:37.287949 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:37.287768    4645 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15c43438-d0c9-41d4-92dd-788371b30f8f-model-cache\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:52:37.287949 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:37.287784    4645 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15c43438-d0c9-41d4-92dd-788371b30f8f-tmp-dir\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:52:37.293745 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:37.292764    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15c43438-d0c9-41d4-92dd-788371b30f8f-dshm" pod "15c43438-d0c9-41d4-92dd-788371b30f8f" (UID: "15c43438-d0c9-41d4-92dd-788371b30f8f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:52:37.293745 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:37.292948    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15c43438-d0c9-41d4-92dd-788371b30f8f-kube-api-access-q4zgl" pod "15c43438-d0c9-41d4-92dd-788371b30f8f" (UID: "15c43438-d0c9-41d4-92dd-788371b30f8f"). InnerVolumeSpecName "kube-api-access-q4zgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:52:37.299557 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:37.299524    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15c43438-d0c9-41d4-92dd-788371b30f8f-tls-certs" pod "15c43438-d0c9-41d4-92dd-788371b30f8f" (UID: "15c43438-d0c9-41d4-92dd-788371b30f8f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:52:37.389010 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:37.388977    4645 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15c43438-d0c9-41d4-92dd-788371b30f8f-tls-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:52:37.389010 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:37.389007    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4zgl\" (UniqueName: \"kubernetes.io/projected/15c43438-d0c9-41d4-92dd-788371b30f8f-kube-api-access-q4zgl\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:52:37.389010 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:37.389017    4645 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15c43438-d0c9-41d4-92dd-788371b30f8f-dshm\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:52:37.667979 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:37.667941    4645 generic.go:356] "Generic (PLEG): container finished" podID="15c43438-d0c9-41d4-92dd-788371b30f8f" containerID="99755302a08703e09405870dc95679697e366ff6bb8c71587119701327f23d70" exitCode=0
Oct 31 19:52:37.668149 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:37.667985    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-d8c398/test-model-d8c398-kserve-57476967b9-v6ld4"
Oct 31 19:52:37.668149 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:37.668026    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d8c398/test-model-d8c398-kserve-57476967b9-v6ld4" event={"ID":"15c43438-d0c9-41d4-92dd-788371b30f8f","Type":"ContainerDied","Data":"99755302a08703e09405870dc95679697e366ff6bb8c71587119701327f23d70"}
Oct 31 19:52:37.668149 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:37.668074    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-d8c398/test-model-d8c398-kserve-57476967b9-v6ld4" event={"ID":"15c43438-d0c9-41d4-92dd-788371b30f8f","Type":"ContainerDied","Data":"5044e280505a6e32d2b513ac8b484fc1fbf636818d36861e8c1d69cd0b581e06"}
Oct 31 19:52:37.668149 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:37.668114    4645 scope.go:122] "RemoveContainer" containerID="99755302a08703e09405870dc95679697e366ff6bb8c71587119701327f23d70"
Oct 31 19:52:37.670777 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:37.670750    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8" event={"ID":"ede30242-d4ae-4caa-9c4b-8918314a1ead","Type":"ContainerStarted","Data":"85880c2b644d01275aa9674a5d67f076e83cac0aa9206136cca068e69f29023f"}
Oct 31 19:52:37.689007 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:37.688981    4645 scope.go:122] "RemoveContainer" containerID="99755302a08703e09405870dc95679697e366ff6bb8c71587119701327f23d70"
Oct 31 19:52:37.689301 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:52:37.689278    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99755302a08703e09405870dc95679697e366ff6bb8c71587119701327f23d70\": container with ID starting with 99755302a08703e09405870dc95679697e366ff6bb8c71587119701327f23d70 not found: ID does not exist" containerID="99755302a08703e09405870dc95679697e366ff6bb8c71587119701327f23d70"
Oct 31 19:52:37.689392 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:37.689311    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99755302a08703e09405870dc95679697e366ff6bb8c71587119701327f23d70"} err="failed to get container status \"99755302a08703e09405870dc95679697e366ff6bb8c71587119701327f23d70\": rpc error: code = NotFound desc = could not find container \"99755302a08703e09405870dc95679697e366ff6bb8c71587119701327f23d70\": container with ID starting with 99755302a08703e09405870dc95679697e366ff6bb8c71587119701327f23d70 not found: ID does not exist"
Oct 31 19:52:37.695982 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:37.695959    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-d8c398/test-model-d8c398-kserve-57476967b9-v6ld4"]
Oct 31 19:52:37.707304 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:37.707259    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-d8c398/test-model-d8c398-kserve-57476967b9-v6ld4"]
Oct 31 19:52:37.985606 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:37.985555    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-785d26-6467945f49-6xsr7" podUID="dfb5bd91-de17-4a3b-afd7-08cb9d538c50" containerName="payload-processing" containerID="cri-o://d64c55bd9ce7ec704d4cc217aed7dfa483df19e03ec11f1ff5ac94c22cf0099f" gracePeriod=25
Oct 31 19:52:38.021554 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:38.021519    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-785d26-589cf5bf5f-fmrnc" podUID="cdd7678b-849b-4a59-8cf5-227831aa2037" containerName="payload-pre-processing" containerID="cri-o://4a81ca6acd86ce80ca101ee87b197875a7b6133afca7c8d1c4e42e459e1275af" gracePeriod=25
Oct 31 19:52:38.029056 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:38.029022    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-d8c398/test-model-d8c398-kserve-57476967b9-v6ld4" podUID="15c43438-d0c9-41d4-92dd-788371b30f8f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.178:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:52:38.300548 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:38.297700    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:38.680929 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:38.680693    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8"
Oct 31 19:52:38.708715 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:38.682542    4645 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 31 19:52:38.708715 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:38.682609    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8" podUID="ede30242-d4ae-4caa-9c4b-8918314a1ead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 31 19:52:38.793893 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:38.793863    4645 patch_prober.go:29] interesting pod/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": read tcp 10.133.0.2:43698->10.133.0.199:15021: read: connection reset by peer" start-of-body=
Oct 31 19:52:38.794043 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:38.793926    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25" podUID="f2be79e8-0166-4066-8e48-fad5a7e85684" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": read tcp 10.133.0.2:43698->10.133.0.199:15021: read: connection reset by peer"
Oct 31 19:52:39.111922 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:39.111844    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:39.338476 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:39.338445    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8"
Oct 31 19:52:39.619601 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:39.619566    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15c43438-d0c9-41d4-92dd-788371b30f8f" path="/var/lib/kubelet/pods/15c43438-d0c9-41d4-92dd-788371b30f8f/volumes"
Oct 31 19:52:39.680937 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:39.676903    4645 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 31 19:52:39.680937 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:39.676957    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8" podUID="ede30242-d4ae-4caa-9c4b-8918314a1ead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 31 19:52:39.700465 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:39.700438    4645 generic.go:356] "Generic (PLEG): container finished" podID="f2be79e8-0166-4066-8e48-fad5a7e85684" containerID="a3220b1028914edc39c544970b4bcd369fad5406860f38d28edd234b289d22d1" exitCode=0
Oct 31 19:52:39.700607 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:39.700504    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25" event={"ID":"f2be79e8-0166-4066-8e48-fad5a7e85684","Type":"ContainerDied","Data":"a3220b1028914edc39c544970b4bcd369fad5406860f38d28edd234b289d22d1"}
Oct 31 19:52:39.700958 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:39.700940    4645 scope.go:122] "RemoveContainer" containerID="a3220b1028914edc39c544970b4bcd369fad5406860f38d28edd234b289d22d1"
Oct 31 19:52:39.702723 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:39.702691    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:52:39.702823 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:39.702778    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:52:39.702823 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:39.702812    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:52:39.886968 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:39.886852    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:40.672354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:40.671695    4645 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 31 19:52:40.672354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:40.671755    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8" podUID="ede30242-d4ae-4caa-9c4b-8918314a1ead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 31 19:52:40.691104 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:40.691068    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:40.783134 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:40.783093    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25" event={"ID":"f2be79e8-0166-4066-8e48-fad5a7e85684","Type":"ContainerStarted","Data":"4170efc426099cb7ba25f4eef3e13a78a9d5c05bb5a28a1033bff39c14611b04"}
Oct 31 19:52:41.279352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:41.278725    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25"
Oct 31 19:52:41.279352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:41.278851    4645 patch_prober.go:29] interesting pod/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Oct 31 19:52:41.279352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:41.278896    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25" podUID="f2be79e8-0166-4066-8e48-fad5a7e85684" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Oct 31 19:52:41.594404 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:41.579011    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:41.672146 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:41.671695    4645 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 31 19:52:41.672146 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:41.671757    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8" podUID="ede30242-d4ae-4caa-9c4b-8918314a1ead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 31 19:52:42.276353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:42.274480    4645 patch_prober.go:29] interesting pod/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Oct 31 19:52:42.276353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:42.274554    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25" podUID="f2be79e8-0166-4066-8e48-fad5a7e85684" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Oct 31 19:52:42.492350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:42.486962    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:42.677409 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:42.677043    4645 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body=
Oct 31 19:52:42.677409 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:42.677112    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8" podUID="ede30242-d4ae-4caa-9c4b-8918314a1ead" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused"
Oct 31 19:52:43.026737 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:43.026646    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-785d26-6467945f49-6xsr7" podUID="dfb5bd91-de17-4a3b-afd7-08cb9d538c50" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:52:43.245187 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:43.245153    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:43.272350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:43.270837    4645 patch_prober.go:29] interesting pod/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Oct 31 19:52:43.272350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:43.270928    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25" podUID="f2be79e8-0166-4066-8e48-fad5a7e85684" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Oct 31 19:52:43.546815 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:43.544466    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w6-e1d060/auth-test-model-e1d060-kserve-589d89d7b9-2vkt5"]
Oct 31 19:52:43.546815 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:43.545142    4645 cpu_manager.go:414] "containerMap: removing container" podUID="15c43438-d0c9-41d4-92dd-788371b30f8f" containerName="main"
Oct 31 19:52:43.546815 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:43.545154    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15c43438-d0c9-41d4-92dd-788371b30f8f" containerName="main"
Oct 31 19:52:43.546815 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:43.545299    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="15c43438-d0c9-41d4-92dd-788371b30f8f" containerName="main"
Oct 31 19:52:43.549172 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:43.549149    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-e1d060/auth-test-model-e1d060-kserve-589d89d7b9-2vkt5"
Oct 31 19:52:43.553533 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:43.553509    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w6-e1d060\"/\"auth-test-model-e1d060-kserve-self-signed-certs\""
Oct 31 19:52:43.553825 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:43.553808    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w6-e1d060\"/\"default-dockercfg-ckzqx\""
Oct 31 19:52:43.554026 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:43.554009    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w6-e1d060\"/\"openshift-service-ca.crt\""
Oct 31 19:52:43.554147 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:43.554131    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w6-e1d060\"/\"kube-root-ca.crt\""
Oct 31 19:52:43.610237 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:43.610204    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:43.624724 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:43.624702    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w6-e1d060/auth-test-model-e1d060-kserve-589d89d7b9-2vkt5"]
Oct 31 19:52:43.639804 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:43.639780    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/093bfd46-77e5-45d4-aa2e-d07fd8dafe33-dshm\") pod \"auth-test-model-e1d060-kserve-589d89d7b9-2vkt5\" (UID: \"093bfd46-77e5-45d4-aa2e-d07fd8dafe33\") " pod="ai-tenant-e2e-shared-b-w6-e1d060/auth-test-model-e1d060-kserve-589d89d7b9-2vkt5"
Oct 31 19:52:43.639947 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:43.639822    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh6k6\" (UniqueName: \"kubernetes.io/projected/093bfd46-77e5-45d4-aa2e-d07fd8dafe33-kube-api-access-rh6k6\") pod \"auth-test-model-e1d060-kserve-589d89d7b9-2vkt5\" (UID: \"093bfd46-77e5-45d4-aa2e-d07fd8dafe33\") " pod="ai-tenant-e2e-shared-b-w6-e1d060/auth-test-model-e1d060-kserve-589d89d7b9-2vkt5"
Oct 31 19:52:43.640029 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:43.640005    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/093bfd46-77e5-45d4-aa2e-d07fd8dafe33-model-cache\") pod \"auth-test-model-e1d060-kserve-589d89d7b9-2vkt5\" (UID: \"093bfd46-77e5-45d4-aa2e-d07fd8dafe33\") " pod="ai-tenant-e2e-shared-b-w6-e1d060/auth-test-model-e1d060-kserve-589d89d7b9-2vkt5"
Oct 31 19:52:43.640084 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:43.640052    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/093bfd46-77e5-45d4-aa2e-d07fd8dafe33-home\") pod \"auth-test-model-e1d060-kserve-589d89d7b9-2vkt5\" (UID: \"093bfd46-77e5-45d4-aa2e-d07fd8dafe33\") " pod="ai-tenant-e2e-shared-b-w6-e1d060/auth-test-model-e1d060-kserve-589d89d7b9-2vkt5"
Oct 31 19:52:43.640290 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:43.640272    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/093bfd46-77e5-45d4-aa2e-d07fd8dafe33-tmp-dir\") pod \"auth-test-model-e1d060-kserve-589d89d7b9-2vkt5\" (UID: \"093bfd46-77e5-45d4-aa2e-d07fd8dafe33\") " pod="ai-tenant-e2e-shared-b-w6-e1d060/auth-test-model-e1d060-kserve-589d89d7b9-2vkt5"
Oct 31 19:52:43.640368 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:43.640302    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/093bfd46-77e5-45d4-aa2e-d07fd8dafe33-tls-certs\") pod \"auth-test-model-e1d060-kserve-589d89d7b9-2vkt5\" (UID: \"093bfd46-77e5-45d4-aa2e-d07fd8dafe33\") " pod="ai-tenant-e2e-shared-b-w6-e1d060/auth-test-model-e1d060-kserve-589d89d7b9-2vkt5"
Oct 31 19:52:43.680649 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:43.680590    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8"
Oct 31 19:52:43.683255 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:43.683230    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8"
Oct 31 19:52:43.764352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:43.756399    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/093bfd46-77e5-45d4-aa2e-d07fd8dafe33-dshm\") pod \"auth-test-model-e1d060-kserve-589d89d7b9-2vkt5\" (UID: \"093bfd46-77e5-45d4-aa2e-d07fd8dafe33\") " pod="ai-tenant-e2e-shared-b-w6-e1d060/auth-test-model-e1d060-kserve-589d89d7b9-2vkt5"
Oct 31 19:52:43.764352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:43.756491    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh6k6\" (UniqueName: \"kubernetes.io/projected/093bfd46-77e5-45d4-aa2e-d07fd8dafe33-kube-api-access-rh6k6\") pod \"auth-test-model-e1d060-kserve-589d89d7b9-2vkt5\" (UID: \"093bfd46-77e5-45d4-aa2e-d07fd8dafe33\") " pod="ai-tenant-e2e-shared-b-w6-e1d060/auth-test-model-e1d060-kserve-589d89d7b9-2vkt5"
Oct 31 19:52:43.764352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:43.756644    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/093bfd46-77e5-45d4-aa2e-d07fd8dafe33-model-cache\") pod \"auth-test-model-e1d060-kserve-589d89d7b9-2vkt5\" (UID: \"093bfd46-77e5-45d4-aa2e-d07fd8dafe33\") " pod="ai-tenant-e2e-shared-b-w6-e1d060/auth-test-model-e1d060-kserve-589d89d7b9-2vkt5"
Oct 31 19:52:43.764352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:43.756683    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/093bfd46-77e5-45d4-aa2e-d07fd8dafe33-home\") pod \"auth-test-model-e1d060-kserve-589d89d7b9-2vkt5\" (UID: \"093bfd46-77e5-45d4-aa2e-d07fd8dafe33\") " pod="ai-tenant-e2e-shared-b-w6-e1d060/auth-test-model-e1d060-kserve-589d89d7b9-2vkt5"
Oct 31 19:52:43.764352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:43.756943    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/093bfd46-77e5-45d4-aa2e-d07fd8dafe33-tmp-dir\") pod \"auth-test-model-e1d060-kserve-589d89d7b9-2vkt5\" (UID: \"093bfd46-77e5-45d4-aa2e-d07fd8dafe33\") " pod="ai-tenant-e2e-shared-b-w6-e1d060/auth-test-model-e1d060-kserve-589d89d7b9-2vkt5"
Oct 31 19:52:43.764352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:43.756972    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/093bfd46-77e5-45d4-aa2e-d07fd8dafe33-tls-certs\") pod \"auth-test-model-e1d060-kserve-589d89d7b9-2vkt5\" (UID: \"093bfd46-77e5-45d4-aa2e-d07fd8dafe33\") " pod="ai-tenant-e2e-shared-b-w6-e1d060/auth-test-model-e1d060-kserve-589d89d7b9-2vkt5"
Oct 31 19:52:43.787593 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:43.787559    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/093bfd46-77e5-45d4-aa2e-d07fd8dafe33-model-cache\") pod \"auth-test-model-e1d060-kserve-589d89d7b9-2vkt5\" (UID: \"093bfd46-77e5-45d4-aa2e-d07fd8dafe33\") " pod="ai-tenant-e2e-shared-b-w6-e1d060/auth-test-model-e1d060-kserve-589d89d7b9-2vkt5"
Oct 31 19:52:43.882620 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:43.788066    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/093bfd46-77e5-45d4-aa2e-d07fd8dafe33-home\") pod \"auth-test-model-e1d060-kserve-589d89d7b9-2vkt5\" (UID: \"093bfd46-77e5-45d4-aa2e-d07fd8dafe33\") " pod="ai-tenant-e2e-shared-b-w6-e1d060/auth-test-model-e1d060-kserve-589d89d7b9-2vkt5"
Oct 31 19:52:43.886417 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:43.788995    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/093bfd46-77e5-45d4-aa2e-d07fd8dafe33-tmp-dir\") pod \"auth-test-model-e1d060-kserve-589d89d7b9-2vkt5\" (UID: \"093bfd46-77e5-45d4-aa2e-d07fd8dafe33\") " pod="ai-tenant-e2e-shared-b-w6-e1d060/auth-test-model-e1d060-kserve-589d89d7b9-2vkt5"
Oct 31 19:52:43.896650 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:43.812642    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/093bfd46-77e5-45d4-aa2e-d07fd8dafe33-dshm\") pod \"auth-test-model-e1d060-kserve-589d89d7b9-2vkt5\" (UID: \"093bfd46-77e5-45d4-aa2e-d07fd8dafe33\") " pod="ai-tenant-e2e-shared-b-w6-e1d060/auth-test-model-e1d060-kserve-589d89d7b9-2vkt5"
Oct 31 19:52:43.993235 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:43.989686    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/093bfd46-77e5-45d4-aa2e-d07fd8dafe33-tls-certs\") pod \"auth-test-model-e1d060-kserve-589d89d7b9-2vkt5\" (UID: \"093bfd46-77e5-45d4-aa2e-d07fd8dafe33\") " pod="ai-tenant-e2e-shared-b-w6-e1d060/auth-test-model-e1d060-kserve-589d89d7b9-2vkt5"
Oct 31 19:52:43.993235 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:43.993060    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh6k6\" (UniqueName: \"kubernetes.io/projected/093bfd46-77e5-45d4-aa2e-d07fd8dafe33-kube-api-access-rh6k6\") pod \"auth-test-model-e1d060-kserve-589d89d7b9-2vkt5\" (UID: \"093bfd46-77e5-45d4-aa2e-d07fd8dafe33\") " pod="ai-tenant-e2e-shared-b-w6-e1d060/auth-test-model-e1d060-kserve-589d89d7b9-2vkt5"
Oct 31 19:52:44.035791 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:44.035759    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-785d26-589cf5bf5f-fmrnc" podUID="cdd7678b-849b-4a59-8cf5-227831aa2037" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:52:44.180443 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:44.175692    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:44.219368 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:44.218707    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-e1d060/auth-test-model-e1d060-kserve-589d89d7b9-2vkt5"
Oct 31 19:52:44.283572 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:44.276515    4645 patch_prober.go:29] interesting pod/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Oct 31 19:52:44.283572 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:44.276585    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25" podUID="f2be79e8-0166-4066-8e48-fad5a7e85684" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Oct 31 19:52:44.596308 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:44.596279    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w6-e1d060/auth-test-model-e1d060-kserve-589d89d7b9-2vkt5"]
Oct 31 19:52:44.596820 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:52:44.596800    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod093bfd46_77e5_45d4_aa2e_d07fd8dafe33.slice/crio-49894448c9647abda62f0169d4c690a4ac03fed55f641f0c38c0afe921effd6b WatchSource:0}: Error finding container 49894448c9647abda62f0169d4c690a4ac03fed55f641f0c38c0afe921effd6b: Status 404 returned error can't find the container with id 49894448c9647abda62f0169d4c690a4ac03fed55f641f0c38c0afe921effd6b
Oct 31 19:52:44.644458 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:44.644435    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d9c048-5459f7f8dc-rhrkn"
Oct 31 19:52:44.989454 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:44.989418    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-e1d060/auth-test-model-e1d060-kserve-589d89d7b9-2vkt5" event={"ID":"093bfd46-77e5-45d4-aa2e-d07fd8dafe33","Type":"ContainerStarted","Data":"935cd178a0f34fbf2f41990252513cd18c71ed1dbbc9fda6bafa11c08daf08ea"}
Oct 31 19:52:44.989836 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:44.989460    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-e1d060/auth-test-model-e1d060-kserve-589d89d7b9-2vkt5" event={"ID":"093bfd46-77e5-45d4-aa2e-d07fd8dafe33","Type":"ContainerStarted","Data":"49894448c9647abda62f0169d4c690a4ac03fed55f641f0c38c0afe921effd6b"}
Oct 31 19:52:45.012125 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:45.012072    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w6-e1d060/auth-test-model-e1d060-kserve-589d89d7b9-2vkt5" podStartSLOduration=2.012059193 podStartE2EDuration="2.012059193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:52:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:52:45.00847015 +0000 UTC m=+1995.581704304" watchObservedRunningTime="2026-10-31 19:52:45.012059193 +0000 UTC m=+1995.585293322"
Oct 31 19:52:45.038575 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:45.038540    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:45.219291 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:45.219258    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w6-e1d060/auth-test-model-e1d060-kserve-589d89d7b9-2vkt5"
Oct 31 19:52:45.278792 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:45.278706    4645 patch_prober.go:29] interesting pod/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body=
Oct 31 19:52:45.279041 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:45.279010    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25" podUID="f2be79e8-0166-4066-8e48-fad5a7e85684" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused"
Oct 31 19:52:45.652522 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:45.652449    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d9c048-79ff7865f5-qq5qp"
Oct 31 19:52:45.879351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:45.879258    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:46.279929 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:46.279894    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25"
Oct 31 19:52:46.280749 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:46.280723    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25"
Oct 31 19:52:46.282525 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:46.282505    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25"
Oct 31 19:52:46.659285 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:46.659207    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:47.470594 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:47.470560    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:47.975526 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:47.975489    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-c55cf1/auth-test-model-c55cf1-kserve-6bb6c9bd5c-qnh9f"]
Oct 31 19:52:47.980170 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:47.980136    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c55cf1/auth-test-model-c55cf1-kserve-6bb6c9bd5c-qnh9f"
Oct 31 19:52:47.989584 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:47.989542    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-c55cf1\"/\"default-dockercfg-dsnlr\""
Oct 31 19:52:47.989741 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:47.989629    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-c55cf1\"/\"auth-test-model-c55cf1-kserve-self-signed-certs\""
Oct 31 19:52:47.990566 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:47.990547    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-c55cf1\"/\"openshift-service-ca.crt\""
Oct 31 19:52:47.990746 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:47.990732    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-c55cf1\"/\"kube-root-ca.crt\""
Oct 31 19:52:47.997960 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:47.997935    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c55cf1/auth-test-model-c55cf1-kserve-6bb6c9bd5c-qnh9f"]
Oct 31 19:52:48.049785 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:48.049749    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:48.073602 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:48.073571    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ba2c7b6-54c7-4610-8051-8fc89f6f8731-tmp-dir\") pod \"auth-test-model-c55cf1-kserve-6bb6c9bd5c-qnh9f\" (UID: \"6ba2c7b6-54c7-4610-8051-8fc89f6f8731\") " pod="ai-tenant-e2e-shared-a-w3-c55cf1/auth-test-model-c55cf1-kserve-6bb6c9bd5c-qnh9f"
Oct 31 19:52:48.073731 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:48.073647    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ba2c7b6-54c7-4610-8051-8fc89f6f8731-home\") pod \"auth-test-model-c55cf1-kserve-6bb6c9bd5c-qnh9f\" (UID: \"6ba2c7b6-54c7-4610-8051-8fc89f6f8731\") " pod="ai-tenant-e2e-shared-a-w3-c55cf1/auth-test-model-c55cf1-kserve-6bb6c9bd5c-qnh9f"
Oct 31 19:52:48.073784 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:48.073749    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ba2c7b6-54c7-4610-8051-8fc89f6f8731-model-cache\") pod \"auth-test-model-c55cf1-kserve-6bb6c9bd5c-qnh9f\" (UID: \"6ba2c7b6-54c7-4610-8051-8fc89f6f8731\") " pod="ai-tenant-e2e-shared-a-w3-c55cf1/auth-test-model-c55cf1-kserve-6bb6c9bd5c-qnh9f"
Oct 31 19:52:48.073925 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:48.073903    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6kjl\" (UniqueName: \"kubernetes.io/projected/6ba2c7b6-54c7-4610-8051-8fc89f6f8731-kube-api-access-q6kjl\") pod \"auth-test-model-c55cf1-kserve-6bb6c9bd5c-qnh9f\" (UID: \"6ba2c7b6-54c7-4610-8051-8fc89f6f8731\") " pod="ai-tenant-e2e-shared-a-w3-c55cf1/auth-test-model-c55cf1-kserve-6bb6c9bd5c-qnh9f"
Oct 31 19:52:48.073978 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:48.073960    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ba2c7b6-54c7-4610-8051-8fc89f6f8731-tls-certs\") pod \"auth-test-model-c55cf1-kserve-6bb6c9bd5c-qnh9f\" (UID: \"6ba2c7b6-54c7-4610-8051-8fc89f6f8731\") " pod="ai-tenant-e2e-shared-a-w3-c55cf1/auth-test-model-c55cf1-kserve-6bb6c9bd5c-qnh9f"
Oct 31 19:52:48.074148 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:48.074132    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ba2c7b6-54c7-4610-8051-8fc89f6f8731-dshm\") pod \"auth-test-model-c55cf1-kserve-6bb6c9bd5c-qnh9f\" (UID: \"6ba2c7b6-54c7-4610-8051-8fc89f6f8731\") " pod="ai-tenant-e2e-shared-a-w3-c55cf1/auth-test-model-c55cf1-kserve-6bb6c9bd5c-qnh9f"
Oct 31 19:52:48.210394 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:48.191667    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6kjl\" (UniqueName: \"kubernetes.io/projected/6ba2c7b6-54c7-4610-8051-8fc89f6f8731-kube-api-access-q6kjl\") pod \"auth-test-model-c55cf1-kserve-6bb6c9bd5c-qnh9f\" (UID: \"6ba2c7b6-54c7-4610-8051-8fc89f6f8731\") " pod="ai-tenant-e2e-shared-a-w3-c55cf1/auth-test-model-c55cf1-kserve-6bb6c9bd5c-qnh9f"
Oct 31 19:52:48.210394 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:48.191751    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ba2c7b6-54c7-4610-8051-8fc89f6f8731-tls-certs\") pod \"auth-test-model-c55cf1-kserve-6bb6c9bd5c-qnh9f\" (UID: \"6ba2c7b6-54c7-4610-8051-8fc89f6f8731\") " pod="ai-tenant-e2e-shared-a-w3-c55cf1/auth-test-model-c55cf1-kserve-6bb6c9bd5c-qnh9f"
Oct 31 19:52:48.210394 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:48.191925    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ba2c7b6-54c7-4610-8051-8fc89f6f8731-dshm\") pod \"auth-test-model-c55cf1-kserve-6bb6c9bd5c-qnh9f\" (UID: \"6ba2c7b6-54c7-4610-8051-8fc89f6f8731\") " pod="ai-tenant-e2e-shared-a-w3-c55cf1/auth-test-model-c55cf1-kserve-6bb6c9bd5c-qnh9f"
Oct 31 19:52:48.210394 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:48.192056    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ba2c7b6-54c7-4610-8051-8fc89f6f8731-tmp-dir\") pod \"auth-test-model-c55cf1-kserve-6bb6c9bd5c-qnh9f\" (UID: \"6ba2c7b6-54c7-4610-8051-8fc89f6f8731\") " pod="ai-tenant-e2e-shared-a-w3-c55cf1/auth-test-model-c55cf1-kserve-6bb6c9bd5c-qnh9f"
Oct 31 19:52:48.210394 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:48.192132    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ba2c7b6-54c7-4610-8051-8fc89f6f8731-home\") pod \"auth-test-model-c55cf1-kserve-6bb6c9bd5c-qnh9f\" (UID: \"6ba2c7b6-54c7-4610-8051-8fc89f6f8731\") " pod="ai-tenant-e2e-shared-a-w3-c55cf1/auth-test-model-c55cf1-kserve-6bb6c9bd5c-qnh9f"
Oct 31 19:52:48.210394 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:48.192197    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ba2c7b6-54c7-4610-8051-8fc89f6f8731-model-cache\") pod \"auth-test-model-c55cf1-kserve-6bb6c9bd5c-qnh9f\" (UID: \"6ba2c7b6-54c7-4610-8051-8fc89f6f8731\") " pod="ai-tenant-e2e-shared-a-w3-c55cf1/auth-test-model-c55cf1-kserve-6bb6c9bd5c-qnh9f"
Oct 31 19:52:48.210394 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:48.192851    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ba2c7b6-54c7-4610-8051-8fc89f6f8731-model-cache\") pod \"auth-test-model-c55cf1-kserve-6bb6c9bd5c-qnh9f\" (UID: \"6ba2c7b6-54c7-4610-8051-8fc89f6f8731\") " pod="ai-tenant-e2e-shared-a-w3-c55cf1/auth-test-model-c55cf1-kserve-6bb6c9bd5c-qnh9f"
Oct 31 19:52:48.210394 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:48.200361    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ba2c7b6-54c7-4610-8051-8fc89f6f8731-tmp-dir\") pod \"auth-test-model-c55cf1-kserve-6bb6c9bd5c-qnh9f\" (UID: \"6ba2c7b6-54c7-4610-8051-8fc89f6f8731\") " pod="ai-tenant-e2e-shared-a-w3-c55cf1/auth-test-model-c55cf1-kserve-6bb6c9bd5c-qnh9f"
Oct 31 19:52:48.210394 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:48.200623    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ba2c7b6-54c7-4610-8051-8fc89f6f8731-home\") pod \"auth-test-model-c55cf1-kserve-6bb6c9bd5c-qnh9f\" (UID: \"6ba2c7b6-54c7-4610-8051-8fc89f6f8731\") " pod="ai-tenant-e2e-shared-a-w3-c55cf1/auth-test-model-c55cf1-kserve-6bb6c9bd5c-qnh9f"
Oct 31 19:52:48.224913 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:48.224875    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ba2c7b6-54c7-4610-8051-8fc89f6f8731-tls-certs\") pod \"auth-test-model-c55cf1-kserve-6bb6c9bd5c-qnh9f\" (UID: \"6ba2c7b6-54c7-4610-8051-8fc89f6f8731\") " pod="ai-tenant-e2e-shared-a-w3-c55cf1/auth-test-model-c55cf1-kserve-6bb6c9bd5c-qnh9f"
Oct 31 19:52:48.231734 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:48.231655    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6kjl\" (UniqueName: \"kubernetes.io/projected/6ba2c7b6-54c7-4610-8051-8fc89f6f8731-kube-api-access-q6kjl\") pod \"auth-test-model-c55cf1-kserve-6bb6c9bd5c-qnh9f\" (UID: \"6ba2c7b6-54c7-4610-8051-8fc89f6f8731\") " pod="ai-tenant-e2e-shared-a-w3-c55cf1/auth-test-model-c55cf1-kserve-6bb6c9bd5c-qnh9f"
Oct 31 19:52:48.244025 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:48.243676    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ba2c7b6-54c7-4610-8051-8fc89f6f8731-dshm\") pod \"auth-test-model-c55cf1-kserve-6bb6c9bd5c-qnh9f\" (UID: \"6ba2c7b6-54c7-4610-8051-8fc89f6f8731\") " pod="ai-tenant-e2e-shared-a-w3-c55cf1/auth-test-model-c55cf1-kserve-6bb6c9bd5c-qnh9f"
Oct 31 19:52:48.295825 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:48.295788    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c55cf1/auth-test-model-c55cf1-kserve-6bb6c9bd5c-qnh9f"
Oct 31 19:52:48.314644 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:48.314612    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:48.610674 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:48.610641    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c55cf1/auth-test-model-c55cf1-kserve-6bb6c9bd5c-qnh9f"]
Oct 31 19:52:48.611996 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:52:48.611964    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6ba2c7b6_54c7_4610_8051_8fc89f6f8731.slice/crio-3a493ca9dcc9b9111d50e08fa80c02770c9fa4702f882c3ba4b62492f76848be WatchSource:0}: Error finding container 3a493ca9dcc9b9111d50e08fa80c02770c9fa4702f882c3ba4b62492f76848be: Status 404 returned error can't find the container with id 3a493ca9dcc9b9111d50e08fa80c02770c9fa4702f882c3ba4b62492f76848be
Oct 31 19:52:49.022786 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:49.022752    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c55cf1/auth-test-model-c55cf1-kserve-6bb6c9bd5c-qnh9f" event={"ID":"6ba2c7b6-54c7-4610-8051-8fc89f6f8731","Type":"ContainerStarted","Data":"2ceeb3d7f440c730b397144ebfbf2d63313b7797cf0919686437ecf2507e674d"}
Oct 31 19:52:49.022954 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:49.022802    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c55cf1/auth-test-model-c55cf1-kserve-6bb6c9bd5c-qnh9f" event={"ID":"6ba2c7b6-54c7-4610-8051-8fc89f6f8731","Type":"ContainerStarted","Data":"3a493ca9dcc9b9111d50e08fa80c02770c9fa4702f882c3ba4b62492f76848be"}
Oct 31 19:52:49.042377 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:49.042283    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-c55cf1/auth-test-model-c55cf1-kserve-6bb6c9bd5c-qnh9f" podStartSLOduration=2.042264696 podStartE2EDuration="2.042264696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:52:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:52:49.040749433 +0000 UTC m=+1999.613983562" watchObservedRunningTime="2026-10-31 19:52:49.042264696 +0000 UTC m=+1999.615498923"
Oct 31 19:52:49.229036 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:49.229000    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:49.303424 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:49.296542    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-c55cf1/auth-test-model-c55cf1-kserve-6bb6c9bd5c-qnh9f"
Oct 31 19:52:49.993368 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:49.993306    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:50.211350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:50.208302    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:50.347464 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:50.343623    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:50.453683 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:50.453647    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:50.900527 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:50.900490    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:51.466452 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:51.466411    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:51.640915 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:51.640881    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:52.589378 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:52.589342    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:53.028474 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:53.027710    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-785d26-6467945f49-6xsr7" podUID="dfb5bd91-de17-4a3b-afd7-08cb9d538c50" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:52:53.772946 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:53.772897    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:54.035942 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:54.035864    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-785d26-589cf5bf5f-fmrnc" podUID="cdd7678b-849b-4a59-8cf5-227831aa2037" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:52:54.219957 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:54.219923    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w6-e1d060/auth-test-model-e1d060-kserve-589d89d7b9-2vkt5"
Oct 31 19:52:54.232757 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:54.232733    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w6-e1d060/auth-test-model-e1d060-kserve-589d89d7b9-2vkt5"
Oct 31 19:52:54.757951 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:54.757911    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:55.092893 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:55.092808    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w6-e1d060/auth-test-model-e1d060-kserve-589d89d7b9-2vkt5"
Oct 31 19:52:55.846633 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:55.846585    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:56.781011 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:56.780973    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:57.812887 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:57.812850    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:58.117955 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:58.117920    4645 generic.go:356] "Generic (PLEG): container finished" podID="dfb5bd91-de17-4a3b-afd7-08cb9d538c50" containerID="d64c55bd9ce7ec704d4cc217aed7dfa483df19e03ec11f1ff5ac94c22cf0099f" exitCode=0
Oct 31 19:52:58.118132 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:58.117999    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-785d26-6467945f49-6xsr7" event={"ID":"dfb5bd91-de17-4a3b-afd7-08cb9d538c50","Type":"ContainerDied","Data":"d64c55bd9ce7ec704d4cc217aed7dfa483df19e03ec11f1ff5ac94c22cf0099f"}
Oct 31 19:52:58.296880 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:58.296848    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-c55cf1/auth-test-model-c55cf1-kserve-6bb6c9bd5c-qnh9f"
Oct 31 19:52:58.301469 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:58.301445    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-785d26-6467945f49-6xsr7"
Oct 31 19:52:58.310426 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:58.310400    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-c55cf1/auth-test-model-c55cf1-kserve-6bb6c9bd5c-qnh9f"
Oct 31 19:52:58.347088 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:58.347058    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-785d26-589cf5bf5f-fmrnc"
Oct 31 19:52:58.388887 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:58.388850    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdd7678b-849b-4a59-8cf5-227831aa2037-kube-api-access-nbfl5\" (UniqueName: \"kubernetes.io/projected/cdd7678b-849b-4a59-8cf5-227831aa2037-kube-api-access-nbfl5\") pod \"cdd7678b-849b-4a59-8cf5-227831aa2037\" (UID: \"cdd7678b-849b-4a59-8cf5-227831aa2037\") "
Oct 31 19:52:58.389097 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:58.388939    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfb5bd91-de17-4a3b-afd7-08cb9d538c50-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfb5bd91-de17-4a3b-afd7-08cb9d538c50-plugins-config-volume\") pod \"dfb5bd91-de17-4a3b-afd7-08cb9d538c50\" (UID: \"dfb5bd91-de17-4a3b-afd7-08cb9d538c50\") "
Oct 31 19:52:58.389097 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:58.389065    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfb5bd91-de17-4a3b-afd7-08cb9d538c50-kube-api-access-n8rmh\" (UniqueName: \"kubernetes.io/projected/dfb5bd91-de17-4a3b-afd7-08cb9d538c50-kube-api-access-n8rmh\") pod \"dfb5bd91-de17-4a3b-afd7-08cb9d538c50\" (UID: \"dfb5bd91-de17-4a3b-afd7-08cb9d538c50\") "
Oct 31 19:52:58.389240 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:58.389107    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdd7678b-849b-4a59-8cf5-227831aa2037-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdd7678b-849b-4a59-8cf5-227831aa2037-plugins-config-volume\") pod \"cdd7678b-849b-4a59-8cf5-227831aa2037\" (UID: \"cdd7678b-849b-4a59-8cf5-227831aa2037\") "
Oct 31 19:52:58.390470 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:58.389692    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfb5bd91-de17-4a3b-afd7-08cb9d538c50-plugins-config-volume" pod "dfb5bd91-de17-4a3b-afd7-08cb9d538c50" (UID: "dfb5bd91-de17-4a3b-afd7-08cb9d538c50"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:52:58.390470 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:58.389728    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdd7678b-849b-4a59-8cf5-227831aa2037-plugins-config-volume" pod "cdd7678b-849b-4a59-8cf5-227831aa2037" (UID: "cdd7678b-849b-4a59-8cf5-227831aa2037"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:52:58.390470 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:58.390131    4645 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfb5bd91-de17-4a3b-afd7-08cb9d538c50-plugins-config-volume\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:52:58.390470 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:58.390155    4645 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cdd7678b-849b-4a59-8cf5-227831aa2037-plugins-config-volume\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:52:58.391722 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:58.391693    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdd7678b-849b-4a59-8cf5-227831aa2037-kube-api-access-nbfl5" pod "cdd7678b-849b-4a59-8cf5-227831aa2037" (UID: "cdd7678b-849b-4a59-8cf5-227831aa2037"). InnerVolumeSpecName "kube-api-access-nbfl5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:52:58.391833 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:58.391788    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfb5bd91-de17-4a3b-afd7-08cb9d538c50-kube-api-access-n8rmh" pod "dfb5bd91-de17-4a3b-afd7-08cb9d538c50" (UID: "dfb5bd91-de17-4a3b-afd7-08cb9d538c50"). InnerVolumeSpecName "kube-api-access-n8rmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:52:58.445648 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:58.445599    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:58.491452 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:58.491420    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbfl5\" (UniqueName: \"kubernetes.io/projected/cdd7678b-849b-4a59-8cf5-227831aa2037-kube-api-access-nbfl5\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:52:58.491452 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:58.491450    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8rmh\" (UniqueName: \"kubernetes.io/projected/dfb5bd91-de17-4a3b-afd7-08cb9d538c50-kube-api-access-n8rmh\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:52:58.591067 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:58.590981    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:58.743868 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:58.743829    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:58.925414 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:58.915229    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:52:59.127546 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:59.127507    4645 generic.go:356] "Generic (PLEG): container finished" podID="cdd7678b-849b-4a59-8cf5-227831aa2037" containerID="4a81ca6acd86ce80ca101ee87b197875a7b6133afca7c8d1c4e42e459e1275af" exitCode=0
Oct 31 19:52:59.127744 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:59.127554    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-785d26-589cf5bf5f-fmrnc"
Oct 31 19:52:59.127744 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:59.127627    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-785d26-589cf5bf5f-fmrnc" event={"ID":"cdd7678b-849b-4a59-8cf5-227831aa2037","Type":"ContainerDied","Data":"4a81ca6acd86ce80ca101ee87b197875a7b6133afca7c8d1c4e42e459e1275af"}
Oct 31 19:52:59.127744 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:59.127685    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-785d26-589cf5bf5f-fmrnc" event={"ID":"cdd7678b-849b-4a59-8cf5-227831aa2037","Type":"ContainerDied","Data":"061e4fe8dbd0fd77df88024ad36d50e06df20f8250204301188c6c975a9b1d98"}
Oct 31 19:52:59.127744 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:59.127711    4645 scope.go:122] "RemoveContainer" containerID="4a81ca6acd86ce80ca101ee87b197875a7b6133afca7c8d1c4e42e459e1275af"
Oct 31 19:52:59.131182 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:59.131161    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-785d26-6467945f49-6xsr7"
Oct 31 19:52:59.131516 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:59.131495    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-785d26-6467945f49-6xsr7" event={"ID":"dfb5bd91-de17-4a3b-afd7-08cb9d538c50","Type":"ContainerDied","Data":"ac85ea6f908322580eb05763371f92a079c762b66c4c418b927fe60d5c18f9fe"}
Oct 31 19:52:59.146518 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:59.146493    4645 scope.go:122] "RemoveContainer" containerID="4a81ca6acd86ce80ca101ee87b197875a7b6133afca7c8d1c4e42e459e1275af"
Oct 31 19:52:59.146945 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:52:59.146915    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a81ca6acd86ce80ca101ee87b197875a7b6133afca7c8d1c4e42e459e1275af\": container with ID starting with 4a81ca6acd86ce80ca101ee87b197875a7b6133afca7c8d1c4e42e459e1275af not found: ID does not exist" containerID="4a81ca6acd86ce80ca101ee87b197875a7b6133afca7c8d1c4e42e459e1275af"
Oct 31 19:52:59.147032 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:59.146957    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a81ca6acd86ce80ca101ee87b197875a7b6133afca7c8d1c4e42e459e1275af"} err="failed to get container status \"4a81ca6acd86ce80ca101ee87b197875a7b6133afca7c8d1c4e42e459e1275af\": rpc error: code = NotFound desc = could not find container \"4a81ca6acd86ce80ca101ee87b197875a7b6133afca7c8d1c4e42e459e1275af\": container with ID starting with 4a81ca6acd86ce80ca101ee87b197875a7b6133afca7c8d1c4e42e459e1275af not found: ID does not exist"
Oct 31 19:52:59.147032 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:59.146982    4645 scope.go:122] "RemoveContainer" containerID="d64c55bd9ce7ec704d4cc217aed7dfa483df19e03ec11f1ff5ac94c22cf0099f"
Oct 31 19:52:59.148388 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:59.147931    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-c55cf1/auth-test-model-c55cf1-kserve-6bb6c9bd5c-qnh9f"
Oct 31 19:52:59.168804 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:59.168775    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-785d26-589cf5bf5f-fmrnc"]
Oct 31 19:52:59.175210 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:59.175184    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-785d26-589cf5bf5f-fmrnc"]
Oct 31 19:52:59.200430 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:52:59.200355    4645 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdd7678b_849b_4a59_8cf5_227831aa2037.slice/crio-061e4fe8dbd0fd77df88024ad36d50e06df20f8250204301188c6c975a9b1d98\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfb5bd91_de17_4a3b_afd7_08cb9d538c50.slice/crio-ac85ea6f908322580eb05763371f92a079c762b66c4c418b927fe60d5c18f9fe\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfb5bd91_de17_4a3b_afd7_08cb9d538c50.slice\": RecentStats: unable to find data in memory cache]"
Oct 31 19:52:59.227838 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:59.227808    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-785d26-6467945f49-6xsr7"]
Oct 31 19:52:59.238658 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:59.238628    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-785d26-6467945f49-6xsr7"]
Oct 31 19:52:59.619582 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:59.619504    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdd7678b-849b-4a59-8cf5-227831aa2037" path="/var/lib/kubelet/pods/cdd7678b-849b-4a59-8cf5-227831aa2037/volumes"
Oct 31 19:52:59.619985 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:59.619970    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfb5bd91-de17-4a3b-afd7-08cb9d538c50" path="/var/lib/kubelet/pods/dfb5bd91-de17-4a3b-afd7-08cb9d538c50/volumes"
Oct 31 19:52:59.982472 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:52:59.982437    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:01.046354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:01.046296    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:01.780093 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:01.780059    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:02.673423 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:02.673380    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:03.297191 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:03.297150    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:03.449137 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:03.449102    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:03.579232 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:03.579130    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:04.362804 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:04.362767    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:05.225914 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:05.225878    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:05.436971 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:05.436937    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:06.247161 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:53:06.247122    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 19:53:06.351531 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:06.351497    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:07.163112 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:07.163071    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:08.081243 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:08.081209    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:08.867659 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:08.867625    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:09.470006 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:09.469973    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d8c398-69fd677f-s95lc"]
Oct 31 19:53:09.531436 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:09.531402    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:09.637312 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:09.637278    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:09.744298 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:09.744210    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:10.538668 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:10.538628    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:11.237217 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:11.237174    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:11.964896 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:11.964815    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:13.101730 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:13.101147    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:14.167617 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:14.167575    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:14.470708 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:14.470627    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d8c398-69fd677f-s95lc" podUID="475d73a5-f2c4-4cc8-b573-b42497e3e7a9" containerName="payload-pre-processing" containerID="cri-o://5eabd87c48f5fd10f070fa880b377084b28a4d3bd90e8104dc7ef49145100750" gracePeriod=25
Oct 31 19:53:15.259798 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:15.259760    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:16.162214 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:16.162175    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:16.784229 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:16.784180    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:17.137182 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:17.137092    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:17.296708 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:17.296664    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:18.235276 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:18.235235    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:18.367743 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:18.367708    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:19.044954 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:19.044920    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d8c398-56d6bc9d68-8ddtd"]
Oct 31 19:53:19.045142 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:19.045121    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d8c398-56d6bc9d68-8ddtd" podUID="25a436b2-50a2-404a-9e6d-3601d03edcc8" containerName="maas-api" containerID="cri-o://4006a6ccf8c79645c2bdf6e4a95df64af55c2b9e7982cd24d760ae6d92697c81" gracePeriod=30
Oct 31 19:53:19.103558 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:19.103529    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:19.466595 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:19.466291    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:19.527778 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:19.527743    4645 generic.go:356] "Generic (PLEG): container finished" podID="25a436b2-50a2-404a-9e6d-3601d03edcc8" containerID="4006a6ccf8c79645c2bdf6e4a95df64af55c2b9e7982cd24d760ae6d92697c81" exitCode=0
Oct 31 19:53:19.527940 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:19.527813    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d8c398-56d6bc9d68-8ddtd" event={"ID":"25a436b2-50a2-404a-9e6d-3601d03edcc8","Type":"ContainerDied","Data":"4006a6ccf8c79645c2bdf6e4a95df64af55c2b9e7982cd24d760ae6d92697c81"}
Oct 31 19:53:19.541638 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:19.541613    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d8c398-56d6bc9d68-8ddtd"
Oct 31 19:53:19.604433 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:19.604401    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25a436b2-50a2-404a-9e6d-3601d03edcc8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/25a436b2-50a2-404a-9e6d-3601d03edcc8-metrics-certs\") pod \"25a436b2-50a2-404a-9e6d-3601d03edcc8\" (UID: \"25a436b2-50a2-404a-9e6d-3601d03edcc8\") "
Oct 31 19:53:19.604611 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:19.604487    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25a436b2-50a2-404a-9e6d-3601d03edcc8-kube-api-access-khd6h\" (UniqueName: \"kubernetes.io/projected/25a436b2-50a2-404a-9e6d-3601d03edcc8-kube-api-access-khd6h\") pod \"25a436b2-50a2-404a-9e6d-3601d03edcc8\" (UID: \"25a436b2-50a2-404a-9e6d-3601d03edcc8\") "
Oct 31 19:53:19.604735 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:19.604720    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25a436b2-50a2-404a-9e6d-3601d03edcc8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25a436b2-50a2-404a-9e6d-3601d03edcc8-openshift-service-ca\") pod \"25a436b2-50a2-404a-9e6d-3601d03edcc8\" (UID: \"25a436b2-50a2-404a-9e6d-3601d03edcc8\") "
Oct 31 19:53:19.604810 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:19.604748    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25a436b2-50a2-404a-9e6d-3601d03edcc8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25a436b2-50a2-404a-9e6d-3601d03edcc8-maas-api-tls\") pod \"25a436b2-50a2-404a-9e6d-3601d03edcc8\" (UID: \"25a436b2-50a2-404a-9e6d-3601d03edcc8\") "
Oct 31 19:53:19.605128 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:19.605100    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25a436b2-50a2-404a-9e6d-3601d03edcc8-openshift-service-ca" pod "25a436b2-50a2-404a-9e6d-3601d03edcc8" (UID: "25a436b2-50a2-404a-9e6d-3601d03edcc8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:53:19.605407 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:19.605385    4645 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25a436b2-50a2-404a-9e6d-3601d03edcc8-openshift-service-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:53:19.606646 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:19.606619    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25a436b2-50a2-404a-9e6d-3601d03edcc8-metrics-certs" pod "25a436b2-50a2-404a-9e6d-3601d03edcc8" (UID: "25a436b2-50a2-404a-9e6d-3601d03edcc8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:53:19.606816 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:19.606794    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25a436b2-50a2-404a-9e6d-3601d03edcc8-kube-api-access-khd6h" pod "25a436b2-50a2-404a-9e6d-3601d03edcc8" (UID: "25a436b2-50a2-404a-9e6d-3601d03edcc8"). InnerVolumeSpecName "kube-api-access-khd6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:53:19.606980 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:19.606932    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25a436b2-50a2-404a-9e6d-3601d03edcc8-maas-api-tls" pod "25a436b2-50a2-404a-9e6d-3601d03edcc8" (UID: "25a436b2-50a2-404a-9e6d-3601d03edcc8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:53:19.673272 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:19.673233    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:19.717421 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:19.717257    4645 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25a436b2-50a2-404a-9e6d-3601d03edcc8-metrics-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:53:19.717421 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:19.717299    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khd6h\" (UniqueName: \"kubernetes.io/projected/25a436b2-50a2-404a-9e6d-3601d03edcc8-kube-api-access-khd6h\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:53:19.717421 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:19.717319    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25a436b2-50a2-404a-9e6d-3601d03edcc8-maas-api-tls\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:53:20.355105 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:20.355025    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:20.536891 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:20.536860    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d8c398-56d6bc9d68-8ddtd"
Oct 31 19:53:20.537393 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:20.536891    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-d8c398-56d6bc9d68-8ddtd" event={"ID":"25a436b2-50a2-404a-9e6d-3601d03edcc8","Type":"ContainerDied","Data":"61fdf19f5dff3c63ca8224472dfb9abc4eb88e558dcb0cddf4ebf1a89ac9967b"}
Oct 31 19:53:20.537393 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:20.536962    4645 scope.go:122] "RemoveContainer" containerID="4006a6ccf8c79645c2bdf6e4a95df64af55c2b9e7982cd24d760ae6d92697c81"
Oct 31 19:53:20.564265 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:20.564226    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d8c398-56d6bc9d68-8ddtd"]
Oct 31 19:53:20.570247 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:20.570223    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-d8c398-56d6bc9d68-8ddtd"]
Oct 31 19:53:21.042927 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:21.042892    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:21.629449 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:21.629416    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25a436b2-50a2-404a-9e6d-3601d03edcc8" path="/var/lib/kubelet/pods/25a436b2-50a2-404a-9e6d-3601d03edcc8/volumes"
Oct 31 19:53:21.863737 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:21.863699    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:22.664297 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:22.664257    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d8c398-69fd677f-s95lc" podUID="475d73a5-f2c4-4cc8-b573-b42497e3e7a9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:53:22.755120 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:22.755085    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:23.153407 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:23.153282    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-e618e1/auth-test-model-e618e1-kserve-765575658f-s9n47"]
Oct 31 19:53:23.154197 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:23.154171    4645 cpu_manager.go:414] "containerMap: removing container" podUID="cdd7678b-849b-4a59-8cf5-227831aa2037" containerName="payload-pre-processing"
Oct 31 19:53:23.154197 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:23.154197    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdd7678b-849b-4a59-8cf5-227831aa2037" containerName="payload-pre-processing"
Oct 31 19:53:23.154433 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:23.154225    4645 cpu_manager.go:414] "containerMap: removing container" podUID="25a436b2-50a2-404a-9e6d-3601d03edcc8" containerName="maas-api"
Oct 31 19:53:23.154433 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:23.154232    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25a436b2-50a2-404a-9e6d-3601d03edcc8" containerName="maas-api"
Oct 31 19:53:23.154433 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:23.154302    4645 cpu_manager.go:414] "containerMap: removing container" podUID="dfb5bd91-de17-4a3b-afd7-08cb9d538c50" containerName="payload-processing"
Oct 31 19:53:23.154433 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:23.154310    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfb5bd91-de17-4a3b-afd7-08cb9d538c50" containerName="payload-processing"
Oct 31 19:53:23.154643 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:23.154487    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfb5bd91-de17-4a3b-afd7-08cb9d538c50" containerName="payload-processing"
Oct 31 19:53:23.154643 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:23.154507    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdd7678b-849b-4a59-8cf5-227831aa2037" containerName="payload-pre-processing"
Oct 31 19:53:23.154643 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:23.154524    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="25a436b2-50a2-404a-9e6d-3601d03edcc8" containerName="maas-api"
Oct 31 19:53:23.160099 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:23.160075    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-e618e1/auth-test-model-e618e1-kserve-765575658f-s9n47"
Oct 31 19:53:23.172801 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:23.172777    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-e618e1/auth-test-model-e618e1-kserve-765575658f-s9n47"]
Oct 31 19:53:23.172917 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:23.172863    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-e618e1\"/\"kube-root-ca.crt\""
Oct 31 19:53:23.173062 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:23.173048    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-e618e1\"/\"openshift-service-ca.crt\""
Oct 31 19:53:23.173305 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:23.173286    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-e618e1\"/\"default-dockercfg-hf4r4\""
Oct 31 19:53:23.173444 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:23.173422    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-e618e1\"/\"auth-test-model-e618e1-kserve-self-signed-certs\""
Oct 31 19:53:23.240076 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:23.240051    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:23.266137 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:23.266112    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6588\" (UniqueName: \"kubernetes.io/projected/208c70bf-17d3-42fc-a106-9f30c2cc3c48-kube-api-access-k6588\") pod \"auth-test-model-e618e1-kserve-765575658f-s9n47\" (UID: \"208c70bf-17d3-42fc-a106-9f30c2cc3c48\") " pod="ai-tenant-e2e-shared-a-w4-e618e1/auth-test-model-e618e1-kserve-765575658f-s9n47"
Oct 31 19:53:23.266269 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:23.266148    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/208c70bf-17d3-42fc-a106-9f30c2cc3c48-tls-certs\") pod \"auth-test-model-e618e1-kserve-765575658f-s9n47\" (UID: \"208c70bf-17d3-42fc-a106-9f30c2cc3c48\") " pod="ai-tenant-e2e-shared-a-w4-e618e1/auth-test-model-e618e1-kserve-765575658f-s9n47"
Oct 31 19:53:23.266269 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:23.266233    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/208c70bf-17d3-42fc-a106-9f30c2cc3c48-tmp-dir\") pod \"auth-test-model-e618e1-kserve-765575658f-s9n47\" (UID: \"208c70bf-17d3-42fc-a106-9f30c2cc3c48\") " pod="ai-tenant-e2e-shared-a-w4-e618e1/auth-test-model-e618e1-kserve-765575658f-s9n47"
Oct 31 19:53:23.266404 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:23.266315    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/208c70bf-17d3-42fc-a106-9f30c2cc3c48-model-cache\") pod \"auth-test-model-e618e1-kserve-765575658f-s9n47\" (UID: \"208c70bf-17d3-42fc-a106-9f30c2cc3c48\") " pod="ai-tenant-e2e-shared-a-w4-e618e1/auth-test-model-e618e1-kserve-765575658f-s9n47"
Oct 31 19:53:23.266459 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:23.266446    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/208c70bf-17d3-42fc-a106-9f30c2cc3c48-dshm\") pod \"auth-test-model-e618e1-kserve-765575658f-s9n47\" (UID: \"208c70bf-17d3-42fc-a106-9f30c2cc3c48\") " pod="ai-tenant-e2e-shared-a-w4-e618e1/auth-test-model-e618e1-kserve-765575658f-s9n47"
Oct 31 19:53:23.266499 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:23.266480    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/208c70bf-17d3-42fc-a106-9f30c2cc3c48-home\") pod \"auth-test-model-e618e1-kserve-765575658f-s9n47\" (UID: \"208c70bf-17d3-42fc-a106-9f30c2cc3c48\") " pod="ai-tenant-e2e-shared-a-w4-e618e1/auth-test-model-e618e1-kserve-765575658f-s9n47"
Oct 31 19:53:23.368762 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:23.368711    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6588\" (UniqueName: \"kubernetes.io/projected/208c70bf-17d3-42fc-a106-9f30c2cc3c48-kube-api-access-k6588\") pod \"auth-test-model-e618e1-kserve-765575658f-s9n47\" (UID: \"208c70bf-17d3-42fc-a106-9f30c2cc3c48\") " pod="ai-tenant-e2e-shared-a-w4-e618e1/auth-test-model-e618e1-kserve-765575658f-s9n47"
Oct 31 19:53:23.369570 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:23.369541    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/208c70bf-17d3-42fc-a106-9f30c2cc3c48-tls-certs\") pod \"auth-test-model-e618e1-kserve-765575658f-s9n47\" (UID: \"208c70bf-17d3-42fc-a106-9f30c2cc3c48\") " pod="ai-tenant-e2e-shared-a-w4-e618e1/auth-test-model-e618e1-kserve-765575658f-s9n47"
Oct 31 19:53:23.372086 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:23.372062    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/208c70bf-17d3-42fc-a106-9f30c2cc3c48-tmp-dir\") pod \"auth-test-model-e618e1-kserve-765575658f-s9n47\" (UID: \"208c70bf-17d3-42fc-a106-9f30c2cc3c48\") " pod="ai-tenant-e2e-shared-a-w4-e618e1/auth-test-model-e618e1-kserve-765575658f-s9n47"
Oct 31 19:53:23.372759 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:23.372740    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/208c70bf-17d3-42fc-a106-9f30c2cc3c48-model-cache\") pod \"auth-test-model-e618e1-kserve-765575658f-s9n47\" (UID: \"208c70bf-17d3-42fc-a106-9f30c2cc3c48\") " pod="ai-tenant-e2e-shared-a-w4-e618e1/auth-test-model-e618e1-kserve-765575658f-s9n47"
Oct 31 19:53:23.373368 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:23.372636    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/208c70bf-17d3-42fc-a106-9f30c2cc3c48-tmp-dir\") pod \"auth-test-model-e618e1-kserve-765575658f-s9n47\" (UID: \"208c70bf-17d3-42fc-a106-9f30c2cc3c48\") " pod="ai-tenant-e2e-shared-a-w4-e618e1/auth-test-model-e618e1-kserve-765575658f-s9n47"
Oct 31 19:53:23.373819 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:23.373149    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/208c70bf-17d3-42fc-a106-9f30c2cc3c48-model-cache\") pod \"auth-test-model-e618e1-kserve-765575658f-s9n47\" (UID: \"208c70bf-17d3-42fc-a106-9f30c2cc3c48\") " pod="ai-tenant-e2e-shared-a-w4-e618e1/auth-test-model-e618e1-kserve-765575658f-s9n47"
Oct 31 19:53:23.373968 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:23.373496    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/208c70bf-17d3-42fc-a106-9f30c2cc3c48-dshm\") pod \"auth-test-model-e618e1-kserve-765575658f-s9n47\" (UID: \"208c70bf-17d3-42fc-a106-9f30c2cc3c48\") " pod="ai-tenant-e2e-shared-a-w4-e618e1/auth-test-model-e618e1-kserve-765575658f-s9n47"
Oct 31 19:53:23.374194 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:23.374177    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/208c70bf-17d3-42fc-a106-9f30c2cc3c48-home\") pod \"auth-test-model-e618e1-kserve-765575658f-s9n47\" (UID: \"208c70bf-17d3-42fc-a106-9f30c2cc3c48\") " pod="ai-tenant-e2e-shared-a-w4-e618e1/auth-test-model-e618e1-kserve-765575658f-s9n47"
Oct 31 19:53:23.374797 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:23.374572    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/208c70bf-17d3-42fc-a106-9f30c2cc3c48-home\") pod \"auth-test-model-e618e1-kserve-765575658f-s9n47\" (UID: \"208c70bf-17d3-42fc-a106-9f30c2cc3c48\") " pod="ai-tenant-e2e-shared-a-w4-e618e1/auth-test-model-e618e1-kserve-765575658f-s9n47"
Oct 31 19:53:23.379824 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:23.379801    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/208c70bf-17d3-42fc-a106-9f30c2cc3c48-dshm\") pod \"auth-test-model-e618e1-kserve-765575658f-s9n47\" (UID: \"208c70bf-17d3-42fc-a106-9f30c2cc3c48\") " pod="ai-tenant-e2e-shared-a-w4-e618e1/auth-test-model-e618e1-kserve-765575658f-s9n47"
Oct 31 19:53:23.379963 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:23.379916    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/208c70bf-17d3-42fc-a106-9f30c2cc3c48-tls-certs\") pod \"auth-test-model-e618e1-kserve-765575658f-s9n47\" (UID: \"208c70bf-17d3-42fc-a106-9f30c2cc3c48\") " pod="ai-tenant-e2e-shared-a-w4-e618e1/auth-test-model-e618e1-kserve-765575658f-s9n47"
Oct 31 19:53:23.414734 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:23.413970    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6588\" (UniqueName: \"kubernetes.io/projected/208c70bf-17d3-42fc-a106-9f30c2cc3c48-kube-api-access-k6588\") pod \"auth-test-model-e618e1-kserve-765575658f-s9n47\" (UID: \"208c70bf-17d3-42fc-a106-9f30c2cc3c48\") " pod="ai-tenant-e2e-shared-a-w4-e618e1/auth-test-model-e618e1-kserve-765575658f-s9n47"
Oct 31 19:53:23.477564 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:23.477529    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-e618e1/auth-test-model-e618e1-kserve-765575658f-s9n47"
Oct 31 19:53:23.831400 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:23.825011    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-e618e1/auth-test-model-e618e1-kserve-765575658f-s9n47"]
Oct 31 19:53:24.076160 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:24.076115    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:24.574882 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:24.574838    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-e618e1/auth-test-model-e618e1-kserve-765575658f-s9n47" event={"ID":"208c70bf-17d3-42fc-a106-9f30c2cc3c48","Type":"ContainerStarted","Data":"ac14057527c5b766dafcdf7cbdf87bef70afcd9dc728fd79e631d39ad6794649"}
Oct 31 19:53:24.575140 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:24.574893    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-e618e1/auth-test-model-e618e1-kserve-765575658f-s9n47" event={"ID":"208c70bf-17d3-42fc-a106-9f30c2cc3c48","Type":"ContainerStarted","Data":"b12f5af33c29c05fadd2e8b9485b04bbdde8179d5a4b297fb77221eb2c76b5af"}
Oct 31 19:53:24.595960 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:24.595900    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-e618e1/auth-test-model-e618e1-kserve-765575658f-s9n47" podStartSLOduration=1.595886024 podStartE2EDuration="1.595886024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:53:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:53:24.593829728 +0000 UTC m=+2035.167063857" watchObservedRunningTime="2026-10-31 19:53:24.595886024 +0000 UTC m=+2035.169120153"
Oct 31 19:53:25.266350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:25.259533    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:25.482351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:25.481442    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-e618e1/auth-test-model-e618e1-kserve-765575658f-s9n47"
Oct 31 19:53:26.482771 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:26.482736    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:27.455275 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:27.455238    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:28.584038 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:28.584003    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:29.135903 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:29.135847    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:29.277806 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:29.277769    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:29.427392 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:29.427355    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2"]
Oct 31 19:53:29.427618 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:29.427593    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2" podUID="90bf19b5-9c42-4481-873e-c55d96f717eb" containerName="istio-proxy" containerID="cri-o://b944ec729e9867c907806253346871b7687f9df0a792c3b7bcd79b72df4d7138" gracePeriod=30
Oct 31 19:53:29.488669 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:29.488633    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:29.615362 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:29.614991    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:30.805184 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:30.805134    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:31.682063 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:31.682023    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:32.561653 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:32.560476    4645 patch_prober.go:29] interesting pod/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 19:53:32.561653 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:32.560552    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2" podUID="90bf19b5-9c42-4481-873e-c55d96f717eb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 19:53:32.647154 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:32.647115    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:32.663050 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:32.663000    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d8c398-69fd677f-s95lc" podUID="475d73a5-f2c4-4cc8-b573-b42497e3e7a9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:53:33.452783 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:33.452746    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:33.478127 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:33.478092    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-e618e1/auth-test-model-e618e1-kserve-765575658f-s9n47"
Oct 31 19:53:33.507355 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:33.506565    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-e618e1/auth-test-model-e618e1-kserve-765575658f-s9n47"
Oct 31 19:53:33.961467 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:33.961432    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-e618e1/auth-test-model-e618e1-kserve-765575658f-s9n47"
Oct 31 19:53:34.319658 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:34.319570    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:34.844817 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:34.844776    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2"
Oct 31 19:53:34.883620 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:34.878391    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/90bf19b5-9c42-4481-873e-c55d96f717eb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90bf19b5-9c42-4481-873e-c55d96f717eb-istio-podinfo\") pod \"90bf19b5-9c42-4481-873e-c55d96f717eb\" (UID: \"90bf19b5-9c42-4481-873e-c55d96f717eb\") "
Oct 31 19:53:34.883620 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:34.878439    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90bf19b5-9c42-4481-873e-c55d96f717eb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90bf19b5-9c42-4481-873e-c55d96f717eb-istio-data\") pod \"90bf19b5-9c42-4481-873e-c55d96f717eb\" (UID: \"90bf19b5-9c42-4481-873e-c55d96f717eb\") "
Oct 31 19:53:34.883620 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:34.878484    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90bf19b5-9c42-4481-873e-c55d96f717eb-kube-api-access-z49st\" (UniqueName: \"kubernetes.io/projected/90bf19b5-9c42-4481-873e-c55d96f717eb-kube-api-access-z49st\") pod \"90bf19b5-9c42-4481-873e-c55d96f717eb\" (UID: \"90bf19b5-9c42-4481-873e-c55d96f717eb\") "
Oct 31 19:53:34.883620 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:34.878528    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90bf19b5-9c42-4481-873e-c55d96f717eb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90bf19b5-9c42-4481-873e-c55d96f717eb-credential-socket\") pod \"90bf19b5-9c42-4481-873e-c55d96f717eb\" (UID: \"90bf19b5-9c42-4481-873e-c55d96f717eb\") "
Oct 31 19:53:34.883620 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:34.878558    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90bf19b5-9c42-4481-873e-c55d96f717eb-istio-token\" (UniqueName: \"kubernetes.io/projected/90bf19b5-9c42-4481-873e-c55d96f717eb-istio-token\") pod \"90bf19b5-9c42-4481-873e-c55d96f717eb\" (UID: \"90bf19b5-9c42-4481-873e-c55d96f717eb\") "
Oct 31 19:53:34.883620 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:34.878674    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90bf19b5-9c42-4481-873e-c55d96f717eb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90bf19b5-9c42-4481-873e-c55d96f717eb-istio-envoy\") pod \"90bf19b5-9c42-4481-873e-c55d96f717eb\" (UID: \"90bf19b5-9c42-4481-873e-c55d96f717eb\") "
Oct 31 19:53:34.883620 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:34.878879    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90bf19b5-9c42-4481-873e-c55d96f717eb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90bf19b5-9c42-4481-873e-c55d96f717eb-workload-socket\") pod \"90bf19b5-9c42-4481-873e-c55d96f717eb\" (UID: \"90bf19b5-9c42-4481-873e-c55d96f717eb\") "
Oct 31 19:53:34.883620 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:34.878913    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90bf19b5-9c42-4481-873e-c55d96f717eb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90bf19b5-9c42-4481-873e-c55d96f717eb-workload-certs\") pod \"90bf19b5-9c42-4481-873e-c55d96f717eb\" (UID: \"90bf19b5-9c42-4481-873e-c55d96f717eb\") "
Oct 31 19:53:34.883620 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:34.879105    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90bf19b5-9c42-4481-873e-c55d96f717eb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90bf19b5-9c42-4481-873e-c55d96f717eb-istiod-ca-cert\") pod \"90bf19b5-9c42-4481-873e-c55d96f717eb\" (UID: \"90bf19b5-9c42-4481-873e-c55d96f717eb\") "
Oct 31 19:53:34.883620 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:34.879815    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90bf19b5-9c42-4481-873e-c55d96f717eb-workload-socket" pod "90bf19b5-9c42-4481-873e-c55d96f717eb" (UID: "90bf19b5-9c42-4481-873e-c55d96f717eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:53:34.883620 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:34.880050    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90bf19b5-9c42-4481-873e-c55d96f717eb-istio-data" pod "90bf19b5-9c42-4481-873e-c55d96f717eb" (UID: "90bf19b5-9c42-4481-873e-c55d96f717eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:53:34.883620 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:34.880130    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90bf19b5-9c42-4481-873e-c55d96f717eb-istiod-ca-cert" pod "90bf19b5-9c42-4481-873e-c55d96f717eb" (UID: "90bf19b5-9c42-4481-873e-c55d96f717eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:53:34.883620 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:34.880265    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90bf19b5-9c42-4481-873e-c55d96f717eb-workload-certs" pod "90bf19b5-9c42-4481-873e-c55d96f717eb" (UID: "90bf19b5-9c42-4481-873e-c55d96f717eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:53:34.883620 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:34.880318    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90bf19b5-9c42-4481-873e-c55d96f717eb-credential-socket" pod "90bf19b5-9c42-4481-873e-c55d96f717eb" (UID: "90bf19b5-9c42-4481-873e-c55d96f717eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:53:34.883620 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:34.882799    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90bf19b5-9c42-4481-873e-c55d96f717eb-kube-api-access-z49st" pod "90bf19b5-9c42-4481-873e-c55d96f717eb" (UID: "90bf19b5-9c42-4481-873e-c55d96f717eb"). InnerVolumeSpecName "kube-api-access-z49st". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:53:34.883620 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:34.883377    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90bf19b5-9c42-4481-873e-c55d96f717eb-istio-envoy" pod "90bf19b5-9c42-4481-873e-c55d96f717eb" (UID: "90bf19b5-9c42-4481-873e-c55d96f717eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:53:34.883620 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:34.883405    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/90bf19b5-9c42-4481-873e-c55d96f717eb-istio-podinfo" pod "90bf19b5-9c42-4481-873e-c55d96f717eb" (UID: "90bf19b5-9c42-4481-873e-c55d96f717eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:53:34.884466 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:34.884238    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90bf19b5-9c42-4481-873e-c55d96f717eb-istio-token" pod "90bf19b5-9c42-4481-873e-c55d96f717eb" (UID: "90bf19b5-9c42-4481-873e-c55d96f717eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:53:34.935213 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:34.935174    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d8c398-69fd677f-s95lc"
Oct 31 19:53:34.963270 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:34.963230    4645 generic.go:356] "Generic (PLEG): container finished" podID="90bf19b5-9c42-4481-873e-c55d96f717eb" containerID="b944ec729e9867c907806253346871b7687f9df0a792c3b7bcd79b72df4d7138" exitCode=0
Oct 31 19:53:34.963795 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:34.963463    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2"
Oct 31 19:53:34.965773 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:34.965733    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2" event={"ID":"90bf19b5-9c42-4481-873e-c55d96f717eb","Type":"ContainerDied","Data":"b944ec729e9867c907806253346871b7687f9df0a792c3b7bcd79b72df4d7138"}
Oct 31 19:53:34.965913 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:34.965794    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2" event={"ID":"90bf19b5-9c42-4481-873e-c55d96f717eb","Type":"ContainerDied","Data":"53650f939076865059f5e8a0782ceefe323d69f763d077b826fc656e218f159c"}
Oct 31 19:53:34.965913 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:34.965827    4645 scope.go:122] "RemoveContainer" containerID="b944ec729e9867c907806253346871b7687f9df0a792c3b7bcd79b72df4d7138"
Oct 31 19:53:34.970271 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:34.970206    4645 generic.go:356] "Generic (PLEG): container finished" podID="475d73a5-f2c4-4cc8-b573-b42497e3e7a9" containerID="5eabd87c48f5fd10f070fa880b377084b28a4d3bd90e8104dc7ef49145100750" exitCode=0
Oct 31 19:53:34.973023 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:34.971222    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d8c398-69fd677f-s95lc"
Oct 31 19:53:34.973023 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:34.971586    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d8c398-69fd677f-s95lc" event={"ID":"475d73a5-f2c4-4cc8-b573-b42497e3e7a9","Type":"ContainerDied","Data":"5eabd87c48f5fd10f070fa880b377084b28a4d3bd90e8104dc7ef49145100750"}
Oct 31 19:53:34.973023 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:34.971634    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d8c398-69fd677f-s95lc" event={"ID":"475d73a5-f2c4-4cc8-b573-b42497e3e7a9","Type":"ContainerDied","Data":"0f9e1644e9ff0c542349acd9e6be1c29c5c20163c0bf3c93103df9e6d1b90615"}
Oct 31 19:53:34.981855 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:34.980742    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/475d73a5-f2c4-4cc8-b573-b42497e3e7a9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/475d73a5-f2c4-4cc8-b573-b42497e3e7a9-plugins-config-volume\") pod \"475d73a5-f2c4-4cc8-b573-b42497e3e7a9\" (UID: \"475d73a5-f2c4-4cc8-b573-b42497e3e7a9\") "
Oct 31 19:53:34.981855 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:34.980814    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/475d73a5-f2c4-4cc8-b573-b42497e3e7a9-kube-api-access-kcztf\" (UniqueName: \"kubernetes.io/projected/475d73a5-f2c4-4cc8-b573-b42497e3e7a9-kube-api-access-kcztf\") pod \"475d73a5-f2c4-4cc8-b573-b42497e3e7a9\" (UID: \"475d73a5-f2c4-4cc8-b573-b42497e3e7a9\") "
Oct 31 19:53:34.981855 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:34.981798    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90bf19b5-9c42-4481-873e-c55d96f717eb-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:53:34.981855 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:34.981823    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90bf19b5-9c42-4481-873e-c55d96f717eb-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:53:34.981855 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:34.981823    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/475d73a5-f2c4-4cc8-b573-b42497e3e7a9-plugins-config-volume" pod "475d73a5-f2c4-4cc8-b573-b42497e3e7a9" (UID: "475d73a5-f2c4-4cc8-b573-b42497e3e7a9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:53:34.981855 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:34.981839    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90bf19b5-9c42-4481-873e-c55d96f717eb-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:53:34.981855 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:34.981853    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z49st\" (UniqueName: \"kubernetes.io/projected/90bf19b5-9c42-4481-873e-c55d96f717eb-kube-api-access-z49st\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:53:34.982223 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:34.981868    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90bf19b5-9c42-4481-873e-c55d96f717eb-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:53:34.982223 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:34.981884    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90bf19b5-9c42-4481-873e-c55d96f717eb-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:53:34.982223 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:34.981896    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90bf19b5-9c42-4481-873e-c55d96f717eb-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:53:34.982223 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:34.981910    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90bf19b5-9c42-4481-873e-c55d96f717eb-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:53:34.982223 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:34.981923    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90bf19b5-9c42-4481-873e-c55d96f717eb-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:53:34.984719 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:34.984697    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/475d73a5-f2c4-4cc8-b573-b42497e3e7a9-kube-api-access-kcztf" pod "475d73a5-f2c4-4cc8-b573-b42497e3e7a9" (UID: "475d73a5-f2c4-4cc8-b573-b42497e3e7a9"). InnerVolumeSpecName "kube-api-access-kcztf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:53:34.987004 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:34.986978    4645 scope.go:122] "RemoveContainer" containerID="b944ec729e9867c907806253346871b7687f9df0a792c3b7bcd79b72df4d7138"
Oct 31 19:53:34.987473 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:53:34.987448    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b944ec729e9867c907806253346871b7687f9df0a792c3b7bcd79b72df4d7138\": container with ID starting with b944ec729e9867c907806253346871b7687f9df0a792c3b7bcd79b72df4d7138 not found: ID does not exist" containerID="b944ec729e9867c907806253346871b7687f9df0a792c3b7bcd79b72df4d7138"
Oct 31 19:53:34.987558 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:34.987486    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b944ec729e9867c907806253346871b7687f9df0a792c3b7bcd79b72df4d7138"} err="failed to get container status \"b944ec729e9867c907806253346871b7687f9df0a792c3b7bcd79b72df4d7138\": rpc error: code = NotFound desc = could not find container \"b944ec729e9867c907806253346871b7687f9df0a792c3b7bcd79b72df4d7138\": container with ID starting with b944ec729e9867c907806253346871b7687f9df0a792c3b7bcd79b72df4d7138 not found: ID does not exist"
Oct 31 19:53:34.987558 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:34.987509    4645 scope.go:122] "RemoveContainer" containerID="5eabd87c48f5fd10f070fa880b377084b28a4d3bd90e8104dc7ef49145100750"
Oct 31 19:53:34.996380 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:34.994478    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2"]
Oct 31 19:53:35.000049 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:35.000023    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-d8c398-openshift-default-65cfb69fd7-l26k2"]
Oct 31 19:53:35.020670 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:35.020642    4645 scope.go:122] "RemoveContainer" containerID="5eabd87c48f5fd10f070fa880b377084b28a4d3bd90e8104dc7ef49145100750"
Oct 31 19:53:35.021031 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:53:35.021009    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5eabd87c48f5fd10f070fa880b377084b28a4d3bd90e8104dc7ef49145100750\": container with ID starting with 5eabd87c48f5fd10f070fa880b377084b28a4d3bd90e8104dc7ef49145100750 not found: ID does not exist" containerID="5eabd87c48f5fd10f070fa880b377084b28a4d3bd90e8104dc7ef49145100750"
Oct 31 19:53:35.021111 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:35.021041    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5eabd87c48f5fd10f070fa880b377084b28a4d3bd90e8104dc7ef49145100750"} err="failed to get container status \"5eabd87c48f5fd10f070fa880b377084b28a4d3bd90e8104dc7ef49145100750\": rpc error: code = NotFound desc = could not find container \"5eabd87c48f5fd10f070fa880b377084b28a4d3bd90e8104dc7ef49145100750\": container with ID starting with 5eabd87c48f5fd10f070fa880b377084b28a4d3bd90e8104dc7ef49145100750 not found: ID does not exist"
Oct 31 19:53:35.083541 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:35.083303    4645 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/475d73a5-f2c4-4cc8-b573-b42497e3e7a9-plugins-config-volume\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:53:35.083541 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:35.083378    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcztf\" (UniqueName: \"kubernetes.io/projected/475d73a5-f2c4-4cc8-b573-b42497e3e7a9-kube-api-access-kcztf\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:53:35.141489 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:35.141379    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:35.303748 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:35.303372    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d8c398-69fd677f-s95lc"]
Oct 31 19:53:35.310061 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:35.310033    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d8c398-69fd677f-s95lc"]
Oct 31 19:53:35.505512 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:35.505464    4645 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:53:35.505673 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:35.505541    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4" podUID="5c010664-b8c0-4a2c-91d7-3f66e006b4ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:53:35.621118 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:35.621076    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="475d73a5-f2c4-4cc8-b573-b42497e3e7a9" path="/var/lib/kubelet/pods/475d73a5-f2c4-4cc8-b573-b42497e3e7a9/volumes"
Oct 31 19:53:35.621670 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:35.621653    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90bf19b5-9c42-4481-873e-c55d96f717eb" path="/var/lib/kubelet/pods/90bf19b5-9c42-4481-873e-c55d96f717eb/volumes"
Oct 31 19:53:36.003797 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:36.003760    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:36.248386 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:53:36.248341    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 19:53:36.756497 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:36.756454    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:37.539260 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:37.539174    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:38.002106 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:38.002073    4645 generic.go:356] "Generic (PLEG): container finished" podID="5c010664-b8c0-4a2c-91d7-3f66e006b4ec" containerID="1e4c85c9543fa3181e0d806d168f058638109986d6fdb39ba16812d43c75a0a6" exitCode=0
Oct 31 19:53:38.002346 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:38.002144    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4" event={"ID":"5c010664-b8c0-4a2c-91d7-3f66e006b4ec","Type":"ContainerDied","Data":"1e4c85c9543fa3181e0d806d168f058638109986d6fdb39ba16812d43c75a0a6"}
Oct 31 19:53:38.002697 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:38.002677    4645 scope.go:122] "RemoveContainer" containerID="1e4c85c9543fa3181e0d806d168f058638109986d6fdb39ba16812d43c75a0a6"
Oct 31 19:53:38.004761 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:38.004730    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:53:38.004843 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:38.004812    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:53:38.004925 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:38.004850    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:53:38.462410 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:38.462366    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:38.545231 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:38.537157    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6"]
Oct 31 19:53:38.546600 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:38.546568    4645 cpu_manager.go:414] "containerMap: removing container" podUID="90bf19b5-9c42-4481-873e-c55d96f717eb" containerName="istio-proxy"
Oct 31 19:53:38.546600 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:38.546601    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90bf19b5-9c42-4481-873e-c55d96f717eb" containerName="istio-proxy"
Oct 31 19:53:38.546822 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:38.546638    4645 cpu_manager.go:414] "containerMap: removing container" podUID="475d73a5-f2c4-4cc8-b573-b42497e3e7a9" containerName="payload-pre-processing"
Oct 31 19:53:38.546822 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:38.546648    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="475d73a5-f2c4-4cc8-b573-b42497e3e7a9" containerName="payload-pre-processing"
Oct 31 19:53:38.546940 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:38.546925    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="90bf19b5-9c42-4481-873e-c55d96f717eb" containerName="istio-proxy"
Oct 31 19:53:38.546994 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:38.546947    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="475d73a5-f2c4-4cc8-b573-b42497e3e7a9" containerName="payload-pre-processing"
Oct 31 19:53:38.557782 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:38.557751    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6"
Oct 31 19:53:38.568871 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:38.568240    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6"]
Oct 31 19:53:38.650525 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:38.650490    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45ebe81b-1559-4c1b-860e-619f8e9e608e-istio-token\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6\" (UID: \"45ebe81b-1559-4c1b-860e-619f8e9e608e\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6"
Oct 31 19:53:38.650721 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:38.650555    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45ebe81b-1559-4c1b-860e-619f8e9e608e-istio-podinfo\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6\" (UID: \"45ebe81b-1559-4c1b-860e-619f8e9e608e\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6"
Oct 31 19:53:38.650721 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:38.650675    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45ebe81b-1559-4c1b-860e-619f8e9e608e-istio-envoy\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6\" (UID: \"45ebe81b-1559-4c1b-860e-619f8e9e608e\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6"
Oct 31 19:53:38.650721 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:38.650708    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45ebe81b-1559-4c1b-860e-619f8e9e608e-workload-socket\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6\" (UID: \"45ebe81b-1559-4c1b-860e-619f8e9e608e\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6"
Oct 31 19:53:38.650910 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:38.650730    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45ebe81b-1559-4c1b-860e-619f8e9e608e-credential-socket\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6\" (UID: \"45ebe81b-1559-4c1b-860e-619f8e9e608e\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6"
Oct 31 19:53:38.650968 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:38.650924    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45ebe81b-1559-4c1b-860e-619f8e9e608e-workload-certs\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6\" (UID: \"45ebe81b-1559-4c1b-860e-619f8e9e608e\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6"
Oct 31 19:53:38.651039 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:38.651022    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45ebe81b-1559-4c1b-860e-619f8e9e608e-istiod-ca-cert\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6\" (UID: \"45ebe81b-1559-4c1b-860e-619f8e9e608e\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6"
Oct 31 19:53:38.651169 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:38.651153    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw255\" (UniqueName: \"kubernetes.io/projected/45ebe81b-1559-4c1b-860e-619f8e9e608e-kube-api-access-fw255\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6\" (UID: \"45ebe81b-1559-4c1b-860e-619f8e9e608e\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6"
Oct 31 19:53:38.651233 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:38.651197    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45ebe81b-1559-4c1b-860e-619f8e9e608e-istio-data\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6\" (UID: \"45ebe81b-1559-4c1b-860e-619f8e9e608e\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6"
Oct 31 19:53:38.660852 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:38.660826    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-a891ce-openshift-default-dockercfg-t8vjd\""
Oct 31 19:53:38.752707 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:38.752611    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45ebe81b-1559-4c1b-860e-619f8e9e608e-istio-token\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6\" (UID: \"45ebe81b-1559-4c1b-860e-619f8e9e608e\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6"
Oct 31 19:53:38.752707 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:38.752685    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45ebe81b-1559-4c1b-860e-619f8e9e608e-istio-podinfo\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6\" (UID: \"45ebe81b-1559-4c1b-860e-619f8e9e608e\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6"
Oct 31 19:53:38.752961 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:38.752749    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45ebe81b-1559-4c1b-860e-619f8e9e608e-istio-envoy\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6\" (UID: \"45ebe81b-1559-4c1b-860e-619f8e9e608e\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6"
Oct 31 19:53:38.752961 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:38.752769    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45ebe81b-1559-4c1b-860e-619f8e9e608e-workload-socket\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6\" (UID: \"45ebe81b-1559-4c1b-860e-619f8e9e608e\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6"
Oct 31 19:53:38.752961 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:38.752784    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45ebe81b-1559-4c1b-860e-619f8e9e608e-credential-socket\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6\" (UID: \"45ebe81b-1559-4c1b-860e-619f8e9e608e\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6"
Oct 31 19:53:38.752961 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:38.752840    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45ebe81b-1559-4c1b-860e-619f8e9e608e-workload-certs\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6\" (UID: \"45ebe81b-1559-4c1b-860e-619f8e9e608e\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6"
Oct 31 19:53:38.752961 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:38.752885    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45ebe81b-1559-4c1b-860e-619f8e9e608e-istiod-ca-cert\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6\" (UID: \"45ebe81b-1559-4c1b-860e-619f8e9e608e\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6"
Oct 31 19:53:38.753271 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:38.752982    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw255\" (UniqueName: \"kubernetes.io/projected/45ebe81b-1559-4c1b-860e-619f8e9e608e-kube-api-access-fw255\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6\" (UID: \"45ebe81b-1559-4c1b-860e-619f8e9e608e\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6"
Oct 31 19:53:38.753271 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:38.753021    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45ebe81b-1559-4c1b-860e-619f8e9e608e-istio-data\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6\" (UID: \"45ebe81b-1559-4c1b-860e-619f8e9e608e\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6"
Oct 31 19:53:38.753793 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:38.753570    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45ebe81b-1559-4c1b-860e-619f8e9e608e-istio-data\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6\" (UID: \"45ebe81b-1559-4c1b-860e-619f8e9e608e\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6"
Oct 31 19:53:38.753793 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:38.753749    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45ebe81b-1559-4c1b-860e-619f8e9e608e-credential-socket\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6\" (UID: \"45ebe81b-1559-4c1b-860e-619f8e9e608e\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6"
Oct 31 19:53:38.753964 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:38.753822    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45ebe81b-1559-4c1b-860e-619f8e9e608e-workload-socket\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6\" (UID: \"45ebe81b-1559-4c1b-860e-619f8e9e608e\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6"
Oct 31 19:53:38.753964 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:38.753929    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45ebe81b-1559-4c1b-860e-619f8e9e608e-workload-certs\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6\" (UID: \"45ebe81b-1559-4c1b-860e-619f8e9e608e\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6"
Oct 31 19:53:38.754324 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:38.754295    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45ebe81b-1559-4c1b-860e-619f8e9e608e-istiod-ca-cert\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6\" (UID: \"45ebe81b-1559-4c1b-860e-619f8e9e608e\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6"
Oct 31 19:53:38.755760 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:38.755741    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45ebe81b-1559-4c1b-860e-619f8e9e608e-istio-envoy\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6\" (UID: \"45ebe81b-1559-4c1b-860e-619f8e9e608e\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6"
Oct 31 19:53:38.755907 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:38.755890    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45ebe81b-1559-4c1b-860e-619f8e9e608e-istio-podinfo\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6\" (UID: \"45ebe81b-1559-4c1b-860e-619f8e9e608e\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6"
Oct 31 19:53:38.764149 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:38.764126    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45ebe81b-1559-4c1b-860e-619f8e9e608e-istio-token\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6\" (UID: \"45ebe81b-1559-4c1b-860e-619f8e9e608e\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6"
Oct 31 19:53:38.764360 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:38.764306    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw255\" (UniqueName: \"kubernetes.io/projected/45ebe81b-1559-4c1b-860e-619f8e9e608e-kube-api-access-fw255\") pod \"e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6\" (UID: \"45ebe81b-1559-4c1b-860e-619f8e9e608e\") " pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6"
Oct 31 19:53:38.898381 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:38.898313    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6"
Oct 31 19:53:39.018695 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:39.018609    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4" event={"ID":"5c010664-b8c0-4a2c-91d7-3f66e006b4ec","Type":"ContainerStarted","Data":"262b687feec098c344273a668a9b3207ee0dafc35a0ef0dc7a36fc538dd8a3bd"}
Oct 31 19:53:39.206563 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:39.206529    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6"]
Oct 31 19:53:39.207015 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:53:39.206990    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45ebe81b_1559_4c1b_860e_619f8e9e608e.slice/crio-7a12fb767b5c65ff7f512838e1af96b776d8f8cc1f205b6c9b4f080e7fc733d0 WatchSource:0}: Error finding container 7a12fb767b5c65ff7f512838e1af96b776d8f8cc1f205b6c9b4f080e7fc733d0: Status 404 returned error can't find the container with id 7a12fb767b5c65ff7f512838e1af96b776d8f8cc1f205b6c9b4f080e7fc733d0
Oct 31 19:53:39.209918 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:39.209884    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:53:39.210017 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:39.209952    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:53:39.210017 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:39.209984    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:53:39.262382 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:39.259889    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4"
Oct 31 19:53:39.262382 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:39.260016    4645 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Oct 31 19:53:39.262382 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:39.260061    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4" podUID="5c010664-b8c0-4a2c-91d7-3f66e006b4ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Oct 31 19:53:39.372057 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:39.372016    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:39.450138 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:39.450099    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-4bac5a/rate-test-model-4bac5a-kserve-564dc4dfdf-vv676"]
Oct 31 19:53:39.500198 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:39.494118    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-4bac5a/rate-test-model-4bac5a-kserve-564dc4dfdf-vv676"
Oct 31 19:53:39.500198 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:39.497836    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:39.501154 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:39.501129    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-4bac5a\"/\"kube-root-ca.crt\""
Oct 31 19:53:39.502075 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:39.502057    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-4bac5a\"/\"openshift-service-ca.crt\""
Oct 31 19:53:39.502474 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:39.502445    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-4bac5a\"/\"default-dockercfg-9dvsj\""
Oct 31 19:53:39.505873 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:39.505831    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-4bac5a\"/\"rate-test-model-4bac5a-kserve-self-signed-certs\""
Oct 31 19:53:39.511741 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:39.511718    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-4bac5a/rate-test-model-4bac5a-kserve-564dc4dfdf-vv676"]
Oct 31 19:53:39.596546 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:39.596427    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9b4e54a-b150-415f-b266-caa45004d860-tmp-dir\") pod \"rate-test-model-4bac5a-kserve-564dc4dfdf-vv676\" (UID: \"f9b4e54a-b150-415f-b266-caa45004d860\") " pod="ai-tenant-e2e-shared-a-w2-4bac5a/rate-test-model-4bac5a-kserve-564dc4dfdf-vv676"
Oct 31 19:53:39.597142 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:39.597108    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9b4e54a-b150-415f-b266-caa45004d860-tls-certs\") pod \"rate-test-model-4bac5a-kserve-564dc4dfdf-vv676\" (UID: \"f9b4e54a-b150-415f-b266-caa45004d860\") " pod="ai-tenant-e2e-shared-a-w2-4bac5a/rate-test-model-4bac5a-kserve-564dc4dfdf-vv676"
Oct 31 19:53:39.597284 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:39.597165    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9b4e54a-b150-415f-b266-caa45004d860-home\") pod \"rate-test-model-4bac5a-kserve-564dc4dfdf-vv676\" (UID: \"f9b4e54a-b150-415f-b266-caa45004d860\") " pod="ai-tenant-e2e-shared-a-w2-4bac5a/rate-test-model-4bac5a-kserve-564dc4dfdf-vv676"
Oct 31 19:53:39.597393 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:39.597309    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9b4e54a-b150-415f-b266-caa45004d860-model-cache\") pod \"rate-test-model-4bac5a-kserve-564dc4dfdf-vv676\" (UID: \"f9b4e54a-b150-415f-b266-caa45004d860\") " pod="ai-tenant-e2e-shared-a-w2-4bac5a/rate-test-model-4bac5a-kserve-564dc4dfdf-vv676"
Oct 31 19:53:39.597527 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:39.597508    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hr84\" (UniqueName: \"kubernetes.io/projected/f9b4e54a-b150-415f-b266-caa45004d860-kube-api-access-5hr84\") pod \"rate-test-model-4bac5a-kserve-564dc4dfdf-vv676\" (UID: \"f9b4e54a-b150-415f-b266-caa45004d860\") " pod="ai-tenant-e2e-shared-a-w2-4bac5a/rate-test-model-4bac5a-kserve-564dc4dfdf-vv676"
Oct 31 19:53:39.597597 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:39.597576    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9b4e54a-b150-415f-b266-caa45004d860-dshm\") pod \"rate-test-model-4bac5a-kserve-564dc4dfdf-vv676\" (UID: \"f9b4e54a-b150-415f-b266-caa45004d860\") " pod="ai-tenant-e2e-shared-a-w2-4bac5a/rate-test-model-4bac5a-kserve-564dc4dfdf-vv676"
Oct 31 19:53:39.699916 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:39.699871    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hr84\" (UniqueName: \"kubernetes.io/projected/f9b4e54a-b150-415f-b266-caa45004d860-kube-api-access-5hr84\") pod \"rate-test-model-4bac5a-kserve-564dc4dfdf-vv676\" (UID: \"f9b4e54a-b150-415f-b266-caa45004d860\") " pod="ai-tenant-e2e-shared-a-w2-4bac5a/rate-test-model-4bac5a-kserve-564dc4dfdf-vv676"
Oct 31 19:53:39.700255 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:39.700228    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9b4e54a-b150-415f-b266-caa45004d860-dshm\") pod \"rate-test-model-4bac5a-kserve-564dc4dfdf-vv676\" (UID: \"f9b4e54a-b150-415f-b266-caa45004d860\") " pod="ai-tenant-e2e-shared-a-w2-4bac5a/rate-test-model-4bac5a-kserve-564dc4dfdf-vv676"
Oct 31 19:53:39.700532 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:39.700512    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9b4e54a-b150-415f-b266-caa45004d860-tmp-dir\") pod \"rate-test-model-4bac5a-kserve-564dc4dfdf-vv676\" (UID: \"f9b4e54a-b150-415f-b266-caa45004d860\") " pod="ai-tenant-e2e-shared-a-w2-4bac5a/rate-test-model-4bac5a-kserve-564dc4dfdf-vv676"
Oct 31 19:53:39.700753 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:39.700734    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9b4e54a-b150-415f-b266-caa45004d860-tls-certs\") pod \"rate-test-model-4bac5a-kserve-564dc4dfdf-vv676\" (UID: \"f9b4e54a-b150-415f-b266-caa45004d860\") " pod="ai-tenant-e2e-shared-a-w2-4bac5a/rate-test-model-4bac5a-kserve-564dc4dfdf-vv676"
Oct 31 19:53:39.721649 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:39.721612    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9b4e54a-b150-415f-b266-caa45004d860-home\") pod \"rate-test-model-4bac5a-kserve-564dc4dfdf-vv676\" (UID: \"f9b4e54a-b150-415f-b266-caa45004d860\") " pod="ai-tenant-e2e-shared-a-w2-4bac5a/rate-test-model-4bac5a-kserve-564dc4dfdf-vv676"
Oct 31 19:53:39.722252 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:39.702163    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9b4e54a-b150-415f-b266-caa45004d860-tmp-dir\") pod \"rate-test-model-4bac5a-kserve-564dc4dfdf-vv676\" (UID: \"f9b4e54a-b150-415f-b266-caa45004d860\") " pod="ai-tenant-e2e-shared-a-w2-4bac5a/rate-test-model-4bac5a-kserve-564dc4dfdf-vv676"
Oct 31 19:53:39.722516 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:39.722140    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9b4e54a-b150-415f-b266-caa45004d860-home\") pod \"rate-test-model-4bac5a-kserve-564dc4dfdf-vv676\" (UID: \"f9b4e54a-b150-415f-b266-caa45004d860\") " pod="ai-tenant-e2e-shared-a-w2-4bac5a/rate-test-model-4bac5a-kserve-564dc4dfdf-vv676"
Oct 31 19:53:39.722736 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:39.722718    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9b4e54a-b150-415f-b266-caa45004d860-model-cache\") pod \"rate-test-model-4bac5a-kserve-564dc4dfdf-vv676\" (UID: \"f9b4e54a-b150-415f-b266-caa45004d860\") " pod="ai-tenant-e2e-shared-a-w2-4bac5a/rate-test-model-4bac5a-kserve-564dc4dfdf-vv676"
Oct 31 19:53:39.723176 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:39.723073    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9b4e54a-b150-415f-b266-caa45004d860-model-cache\") pod \"rate-test-model-4bac5a-kserve-564dc4dfdf-vv676\" (UID: \"f9b4e54a-b150-415f-b266-caa45004d860\") " pod="ai-tenant-e2e-shared-a-w2-4bac5a/rate-test-model-4bac5a-kserve-564dc4dfdf-vv676"
Oct 31 19:53:39.740448 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:39.739844    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9b4e54a-b150-415f-b266-caa45004d860-dshm\") pod \"rate-test-model-4bac5a-kserve-564dc4dfdf-vv676\" (UID: \"f9b4e54a-b150-415f-b266-caa45004d860\") " pod="ai-tenant-e2e-shared-a-w2-4bac5a/rate-test-model-4bac5a-kserve-564dc4dfdf-vv676"
Oct 31 19:53:39.752351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:39.748992    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9b4e54a-b150-415f-b266-caa45004d860-tls-certs\") pod \"rate-test-model-4bac5a-kserve-564dc4dfdf-vv676\" (UID: \"f9b4e54a-b150-415f-b266-caa45004d860\") " pod="ai-tenant-e2e-shared-a-w2-4bac5a/rate-test-model-4bac5a-kserve-564dc4dfdf-vv676"
Oct 31 19:53:39.783355 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:39.767226    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hr84\" (UniqueName: \"kubernetes.io/projected/f9b4e54a-b150-415f-b266-caa45004d860-kube-api-access-5hr84\") pod \"rate-test-model-4bac5a-kserve-564dc4dfdf-vv676\" (UID: \"f9b4e54a-b150-415f-b266-caa45004d860\") " pod="ai-tenant-e2e-shared-a-w2-4bac5a/rate-test-model-4bac5a-kserve-564dc4dfdf-vv676"
Oct 31 19:53:39.818697 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:39.818661    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-4bac5a/rate-test-model-4bac5a-kserve-564dc4dfdf-vv676"
Oct 31 19:53:40.032522 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:40.032482    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6" event={"ID":"45ebe81b-1559-4c1b-860e-619f8e9e608e","Type":"ContainerStarted","Data":"b775495befc3cafaca74868b49ea438b27da6e77be98147c8111bd69d47fcfc5"}
Oct 31 19:53:40.032712 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:40.032556    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6" event={"ID":"45ebe81b-1559-4c1b-860e-619f8e9e608e","Type":"ContainerStarted","Data":"7a12fb767b5c65ff7f512838e1af96b776d8f8cc1f205b6c9b4f080e7fc733d0"}
Oct 31 19:53:40.141080 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:40.139553    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6" podStartSLOduration=2.139533695 podStartE2EDuration="2.139533695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:53:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:53:40.125048847 +0000 UTC m=+2050.698282978" watchObservedRunningTime="2026-10-31 19:53:40.139533695 +0000 UTC m=+2050.712767825"
Oct 31 19:53:40.183115 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:40.183082    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-4bac5a/rate-test-model-4bac5a-kserve-564dc4dfdf-vv676"]
Oct 31 19:53:40.185380 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:53:40.185351    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf9b4e54a_b150_415f_b266_caa45004d860.slice/crio-4768c937071a8140f6903bdc8373b5043059a0b580c8e26805bb54364960702c WatchSource:0}: Error finding container 4768c937071a8140f6903bdc8373b5043059a0b580c8e26805bb54364960702c: Status 404 returned error can't find the container with id 4768c937071a8140f6903bdc8373b5043059a0b580c8e26805bb54364960702c
Oct 31 19:53:40.282592 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:40.282553    4645 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Oct 31 19:53:40.282743 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:40.282630    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4" podUID="5c010664-b8c0-4a2c-91d7-3f66e006b4ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Oct 31 19:53:40.598408 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:40.598302    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:40.901408 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:40.900700    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6"
Oct 31 19:53:40.901408 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:40.900837    4645 patch_prober.go:29] interesting pod/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Oct 31 19:53:40.901408 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:40.900881    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6" podUID="45ebe81b-1559-4c1b-860e-619f8e9e608e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Oct 31 19:53:41.043453 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:41.043418    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-4bac5a/rate-test-model-4bac5a-kserve-564dc4dfdf-vv676" event={"ID":"f9b4e54a-b150-415f-b266-caa45004d860","Type":"ContainerStarted","Data":"58cadced6ea856f6d2d2ecd0660748c2cd605d10f7a4bd7c7504b26f000ad45d"}
Oct 31 19:53:41.043642 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:41.043466    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-4bac5a/rate-test-model-4bac5a-kserve-564dc4dfdf-vv676" event={"ID":"f9b4e54a-b150-415f-b266-caa45004d860","Type":"ContainerStarted","Data":"4768c937071a8140f6903bdc8373b5043059a0b580c8e26805bb54364960702c"}
Oct 31 19:53:41.273353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:41.263751    4645 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Oct 31 19:53:41.273353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:41.263824    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4" podUID="5c010664-b8c0-4a2c-91d7-3f66e006b4ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Oct 31 19:53:41.549246 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:41.549157    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:41.819807 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:41.819716    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-4bac5a/rate-test-model-4bac5a-kserve-564dc4dfdf-vv676"
Oct 31 19:53:41.900354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:41.900014    4645 patch_prober.go:29] interesting pod/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Oct 31 19:53:41.900354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:41.900077    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6" podUID="45ebe81b-1559-4c1b-860e-619f8e9e608e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Oct 31 19:53:42.281355 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:42.275763    4645 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Oct 31 19:53:42.281355 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:42.275846    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4" podUID="5c010664-b8c0-4a2c-91d7-3f66e006b4ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Oct 31 19:53:42.412648 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:42.412602    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:42.908349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:42.902775    4645 patch_prober.go:29] interesting pod/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Oct 31 19:53:42.908349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:42.902858    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6" podUID="45ebe81b-1559-4c1b-860e-619f8e9e608e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Oct 31 19:53:43.278361 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:43.276788    4645 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Oct 31 19:53:43.278361 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:43.276853    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4" podUID="5c010664-b8c0-4a2c-91d7-3f66e006b4ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Oct 31 19:53:43.352395 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:43.352356    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:43.900346 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:43.900276    4645 patch_prober.go:29] interesting pod/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Oct 31 19:53:43.900665 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:43.900631    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6" podUID="45ebe81b-1559-4c1b-860e-619f8e9e608e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Oct 31 19:53:44.254434 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:44.254393    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:44.267707 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:44.267667    4645 patch_prober.go:29] interesting pod/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Oct 31 19:53:44.267992 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:44.267959    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4" podUID="5c010664-b8c0-4a2c-91d7-3f66e006b4ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Oct 31 19:53:44.899443 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:44.899406    4645 patch_prober.go:29] interesting pod/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Oct 31 19:53:44.899610 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:44.899461    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6" podUID="45ebe81b-1559-4c1b-860e-619f8e9e608e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Oct 31 19:53:45.048191 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:45.048145    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:45.285988 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:45.285954    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4"
Oct 31 19:53:45.286536 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:45.286338    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4"
Oct 31 19:53:45.287158 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:45.287139    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4"
Oct 31 19:53:45.329538 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:45.329483    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-4bac5a/rate-test-model-4bac5a-kserve-564dc4dfdf-vv676" podStartSLOduration=6.329468354 podStartE2EDuration="6.329468354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:53:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:53:41.097941654 +0000 UTC m=+2051.671175784" watchObservedRunningTime="2026-10-31 19:53:45.329468354 +0000 UTC m=+2055.902702483"
Oct 31 19:53:45.898868 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:45.898831    4645 patch_prober.go:29] interesting pod/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Oct 31 19:53:45.899036 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:45.898890    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6" podUID="45ebe81b-1559-4c1b-860e-619f8e9e608e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Oct 31 19:53:45.982276 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:45.982228    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:46.675364 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:46.675297    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:46.903534 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:46.903503    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6"
Oct 31 19:53:46.904003 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:46.903989    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6"
Oct 31 19:53:46.904797 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:46.904772    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6"
Oct 31 19:53:47.633235 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:47.633199    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:48.434535 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:48.434496    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:49.119757 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:49.119727    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-d9c048/auth-test-model-d9c048-kserve-5d4b745b9-plqrp"]
Oct 31 19:53:49.123927 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:49.123907    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-d9c048/auth-test-model-d9c048-kserve-5d4b745b9-plqrp"
Oct 31 19:53:49.129027 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:49.128999    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-d9c048\"/\"kube-root-ca.crt\""
Oct 31 19:53:49.130086 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:49.129467    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-d9c048\"/\"default-dockercfg-xhl8s\""
Oct 31 19:53:49.130086 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:49.129558    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-d9c048\"/\"auth-test-model-d9c048-kserve-self-signed-certs\""
Oct 31 19:53:49.130086 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:49.129192    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-d9c048\"/\"openshift-service-ca.crt\""
Oct 31 19:53:49.136605 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:49.136575    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-d9c048/auth-test-model-d9c048-kserve-5d4b745b9-plqrp"]
Oct 31 19:53:49.182186 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:49.182131    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:49.217195 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:49.217157    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73cf0e08-4670-4dc1-b333-a1160099958f-model-cache\") pod \"auth-test-model-d9c048-kserve-5d4b745b9-plqrp\" (UID: \"73cf0e08-4670-4dc1-b333-a1160099958f\") " pod="ai-tenant-e2e-shared-b-w4-d9c048/auth-test-model-d9c048-kserve-5d4b745b9-plqrp"
Oct 31 19:53:49.217429 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:49.217205    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73cf0e08-4670-4dc1-b333-a1160099958f-home\") pod \"auth-test-model-d9c048-kserve-5d4b745b9-plqrp\" (UID: \"73cf0e08-4670-4dc1-b333-a1160099958f\") " pod="ai-tenant-e2e-shared-b-w4-d9c048/auth-test-model-d9c048-kserve-5d4b745b9-plqrp"
Oct 31 19:53:49.217429 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:49.217357    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73cf0e08-4670-4dc1-b333-a1160099958f-tls-certs\") pod \"auth-test-model-d9c048-kserve-5d4b745b9-plqrp\" (UID: \"73cf0e08-4670-4dc1-b333-a1160099958f\") " pod="ai-tenant-e2e-shared-b-w4-d9c048/auth-test-model-d9c048-kserve-5d4b745b9-plqrp"
Oct 31 19:53:49.217536 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:49.217514    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73cf0e08-4670-4dc1-b333-a1160099958f-tmp-dir\") pod \"auth-test-model-d9c048-kserve-5d4b745b9-plqrp\" (UID: \"73cf0e08-4670-4dc1-b333-a1160099958f\") " pod="ai-tenant-e2e-shared-b-w4-d9c048/auth-test-model-d9c048-kserve-5d4b745b9-plqrp"
Oct 31 19:53:49.217574 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:49.217549    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5j64\" (UniqueName: \"kubernetes.io/projected/73cf0e08-4670-4dc1-b333-a1160099958f-kube-api-access-p5j64\") pod \"auth-test-model-d9c048-kserve-5d4b745b9-plqrp\" (UID: \"73cf0e08-4670-4dc1-b333-a1160099958f\") " pod="ai-tenant-e2e-shared-b-w4-d9c048/auth-test-model-d9c048-kserve-5d4b745b9-plqrp"
Oct 31 19:53:49.217639 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:49.217627    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73cf0e08-4670-4dc1-b333-a1160099958f-dshm\") pod \"auth-test-model-d9c048-kserve-5d4b745b9-plqrp\" (UID: \"73cf0e08-4670-4dc1-b333-a1160099958f\") " pod="ai-tenant-e2e-shared-b-w4-d9c048/auth-test-model-d9c048-kserve-5d4b745b9-plqrp"
Oct 31 19:53:49.298970 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:49.298934    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:49.343520 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:49.343470    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73cf0e08-4670-4dc1-b333-a1160099958f-model-cache\") pod \"auth-test-model-d9c048-kserve-5d4b745b9-plqrp\" (UID: \"73cf0e08-4670-4dc1-b333-a1160099958f\") " pod="ai-tenant-e2e-shared-b-w4-d9c048/auth-test-model-d9c048-kserve-5d4b745b9-plqrp"
Oct 31 19:53:49.343745 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:49.343550    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73cf0e08-4670-4dc1-b333-a1160099958f-home\") pod \"auth-test-model-d9c048-kserve-5d4b745b9-plqrp\" (UID: \"73cf0e08-4670-4dc1-b333-a1160099958f\") " pod="ai-tenant-e2e-shared-b-w4-d9c048/auth-test-model-d9c048-kserve-5d4b745b9-plqrp"
Oct 31 19:53:49.343745 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:49.343614    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73cf0e08-4670-4dc1-b333-a1160099958f-tls-certs\") pod \"auth-test-model-d9c048-kserve-5d4b745b9-plqrp\" (UID: \"73cf0e08-4670-4dc1-b333-a1160099958f\") " pod="ai-tenant-e2e-shared-b-w4-d9c048/auth-test-model-d9c048-kserve-5d4b745b9-plqrp"
Oct 31 19:53:49.343880 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:49.343798    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73cf0e08-4670-4dc1-b333-a1160099958f-tmp-dir\") pod \"auth-test-model-d9c048-kserve-5d4b745b9-plqrp\" (UID: \"73cf0e08-4670-4dc1-b333-a1160099958f\") " pod="ai-tenant-e2e-shared-b-w4-d9c048/auth-test-model-d9c048-kserve-5d4b745b9-plqrp"
Oct 31 19:53:49.343880 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:49.343848    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5j64\" (UniqueName: \"kubernetes.io/projected/73cf0e08-4670-4dc1-b333-a1160099958f-kube-api-access-p5j64\") pod \"auth-test-model-d9c048-kserve-5d4b745b9-plqrp\" (UID: \"73cf0e08-4670-4dc1-b333-a1160099958f\") " pod="ai-tenant-e2e-shared-b-w4-d9c048/auth-test-model-d9c048-kserve-5d4b745b9-plqrp"
Oct 31 19:53:49.344006 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:49.343913    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73cf0e08-4670-4dc1-b333-a1160099958f-dshm\") pod \"auth-test-model-d9c048-kserve-5d4b745b9-plqrp\" (UID: \"73cf0e08-4670-4dc1-b333-a1160099958f\") " pod="ai-tenant-e2e-shared-b-w4-d9c048/auth-test-model-d9c048-kserve-5d4b745b9-plqrp"
Oct 31 19:53:49.358349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:49.346015    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73cf0e08-4670-4dc1-b333-a1160099958f-model-cache\") pod \"auth-test-model-d9c048-kserve-5d4b745b9-plqrp\" (UID: \"73cf0e08-4670-4dc1-b333-a1160099958f\") " pod="ai-tenant-e2e-shared-b-w4-d9c048/auth-test-model-d9c048-kserve-5d4b745b9-plqrp"
Oct 31 19:53:49.358774 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:49.348836    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73cf0e08-4670-4dc1-b333-a1160099958f-tmp-dir\") pod \"auth-test-model-d9c048-kserve-5d4b745b9-plqrp\" (UID: \"73cf0e08-4670-4dc1-b333-a1160099958f\") " pod="ai-tenant-e2e-shared-b-w4-d9c048/auth-test-model-d9c048-kserve-5d4b745b9-plqrp"
Oct 31 19:53:49.367888 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:49.349486    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73cf0e08-4670-4dc1-b333-a1160099958f-home\") pod \"auth-test-model-d9c048-kserve-5d4b745b9-plqrp\" (UID: \"73cf0e08-4670-4dc1-b333-a1160099958f\") " pod="ai-tenant-e2e-shared-b-w4-d9c048/auth-test-model-d9c048-kserve-5d4b745b9-plqrp"
Oct 31 19:53:49.376529 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:49.376449    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73cf0e08-4670-4dc1-b333-a1160099958f-dshm\") pod \"auth-test-model-d9c048-kserve-5d4b745b9-plqrp\" (UID: \"73cf0e08-4670-4dc1-b333-a1160099958f\") " pod="ai-tenant-e2e-shared-b-w4-d9c048/auth-test-model-d9c048-kserve-5d4b745b9-plqrp"
Oct 31 19:53:49.380315 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:49.380254    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5j64\" (UniqueName: \"kubernetes.io/projected/73cf0e08-4670-4dc1-b333-a1160099958f-kube-api-access-p5j64\") pod \"auth-test-model-d9c048-kserve-5d4b745b9-plqrp\" (UID: \"73cf0e08-4670-4dc1-b333-a1160099958f\") " pod="ai-tenant-e2e-shared-b-w4-d9c048/auth-test-model-d9c048-kserve-5d4b745b9-plqrp"
Oct 31 19:53:49.387356 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:49.381104    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73cf0e08-4670-4dc1-b333-a1160099958f-tls-certs\") pod \"auth-test-model-d9c048-kserve-5d4b745b9-plqrp\" (UID: \"73cf0e08-4670-4dc1-b333-a1160099958f\") " pod="ai-tenant-e2e-shared-b-w4-d9c048/auth-test-model-d9c048-kserve-5d4b745b9-plqrp"
Oct 31 19:53:49.448350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:49.444650    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-d9c048/auth-test-model-d9c048-kserve-5d4b745b9-plqrp"
Oct 31 19:53:49.742471 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:49.742439    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-d9c048/auth-test-model-d9c048-kserve-5d4b745b9-plqrp"]
Oct 31 19:53:49.742902 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:53:49.742864    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod73cf0e08_4670_4dc1_b333_a1160099958f.slice/crio-7c138e210f7e83a6703996d80da6c09cb12340611749269053c443c1a351ee34 WatchSource:0}: Error finding container 7c138e210f7e83a6703996d80da6c09cb12340611749269053c443c1a351ee34: Status 404 returned error can't find the container with id 7c138e210f7e83a6703996d80da6c09cb12340611749269053c443c1a351ee34
Oct 31 19:53:49.819496 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:49.819465    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-4bac5a/rate-test-model-4bac5a-kserve-564dc4dfdf-vv676"
Oct 31 19:53:49.833815 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:49.833793    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-4bac5a/rate-test-model-4bac5a-kserve-564dc4dfdf-vv676"
Oct 31 19:53:50.193253 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:50.193214    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-d9c048/auth-test-model-d9c048-kserve-5d4b745b9-plqrp" event={"ID":"73cf0e08-4670-4dc1-b333-a1160099958f","Type":"ContainerStarted","Data":"f46c7a11bf22bead2479903e4888efba0642b2682db21f5e236e05a515d9dee0"}
Oct 31 19:53:50.193484 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:50.193265    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-d9c048/auth-test-model-d9c048-kserve-5d4b745b9-plqrp" event={"ID":"73cf0e08-4670-4dc1-b333-a1160099958f","Type":"ContainerStarted","Data":"7c138e210f7e83a6703996d80da6c09cb12340611749269053c443c1a351ee34"}
Oct 31 19:53:50.204749 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:50.204713    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-4bac5a/rate-test-model-4bac5a-kserve-564dc4dfdf-vv676"
Oct 31 19:53:50.218129 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:50.218089    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-d9c048/auth-test-model-d9c048-kserve-5d4b745b9-plqrp" podStartSLOduration=1.218077087 podStartE2EDuration="1.218077087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:53:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:53:50.217584495 +0000 UTC m=+2060.790818624" watchObservedRunningTime="2026-10-31 19:53:50.218077087 +0000 UTC m=+2060.791311216"
Oct 31 19:53:50.365583 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:50.365543    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:50.449053 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:50.448938    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-d9c048/auth-test-model-d9c048-kserve-5d4b745b9-plqrp"
Oct 31 19:53:51.248608 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:51.248569    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:51.996283 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:51.996244    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:52.768747 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:52.768709    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:53.642011 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:53.641975    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:54.498089 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:54.498054    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:55.435071 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:55.435027    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:56.253655 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:56.253620    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:57.052849 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:57.052814    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:57.966296 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:57.966256    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:58.767842 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:58.767808    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:53:59.445944 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:59.445913    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-d9c048/auth-test-model-d9c048-kserve-5d4b745b9-plqrp"
Oct 31 19:53:59.458761 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:59.458732    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-d9c048/auth-test-model-d9c048-kserve-5d4b745b9-plqrp"
Oct 31 19:53:59.514517 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:53:59.514487    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:00.284283 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:00.284250    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-d9c048/auth-test-model-d9c048-kserve-5d4b745b9-plqrp"
Oct 31 19:54:00.452121 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:00.452088    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:01.281406 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:01.281368    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:02.069899 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:02.069867    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:02.536376 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:02.536341    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-ae2e60/rate-test-model-ae2e60-kserve-744475655c-xk2vp"]
Oct 31 19:54:02.542275 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:02.542248    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-ae2e60/rate-test-model-ae2e60-kserve-744475655c-xk2vp"
Oct 31 19:54:02.550951 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:02.550922    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-ae2e60\"/\"default-dockercfg-2kqbj\""
Oct 31 19:54:02.551120 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:02.551016    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-ae2e60\"/\"kube-root-ca.crt\""
Oct 31 19:54:02.551193 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:02.551145    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-ae2e60\"/\"rate-test-model-ae2e60-kserve-self-signed-certs\""
Oct 31 19:54:02.551434 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:02.551414    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-ae2e60\"/\"openshift-service-ca.crt\""
Oct 31 19:54:02.554399 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:02.554372    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-ae2e60/rate-test-model-ae2e60-kserve-744475655c-xk2vp"]
Oct 31 19:54:02.622718 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:02.622687    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:02.655239 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:02.655212    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gstzb\" (UniqueName: \"kubernetes.io/projected/44f86c77-4686-48fe-8cf3-cc0e688af7a4-kube-api-access-gstzb\") pod \"rate-test-model-ae2e60-kserve-744475655c-xk2vp\" (UID: \"44f86c77-4686-48fe-8cf3-cc0e688af7a4\") " pod="ai-tenant-e2e-shared-b-w2-ae2e60/rate-test-model-ae2e60-kserve-744475655c-xk2vp"
Oct 31 19:54:02.655391 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:02.655251    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44f86c77-4686-48fe-8cf3-cc0e688af7a4-model-cache\") pod \"rate-test-model-ae2e60-kserve-744475655c-xk2vp\" (UID: \"44f86c77-4686-48fe-8cf3-cc0e688af7a4\") " pod="ai-tenant-e2e-shared-b-w2-ae2e60/rate-test-model-ae2e60-kserve-744475655c-xk2vp"
Oct 31 19:54:02.655391 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:02.655376    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44f86c77-4686-48fe-8cf3-cc0e688af7a4-tmp-dir\") pod \"rate-test-model-ae2e60-kserve-744475655c-xk2vp\" (UID: \"44f86c77-4686-48fe-8cf3-cc0e688af7a4\") " pod="ai-tenant-e2e-shared-b-w2-ae2e60/rate-test-model-ae2e60-kserve-744475655c-xk2vp"
Oct 31 19:54:02.655479 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:02.655412    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44f86c77-4686-48fe-8cf3-cc0e688af7a4-dshm\") pod \"rate-test-model-ae2e60-kserve-744475655c-xk2vp\" (UID: \"44f86c77-4686-48fe-8cf3-cc0e688af7a4\") " pod="ai-tenant-e2e-shared-b-w2-ae2e60/rate-test-model-ae2e60-kserve-744475655c-xk2vp"
Oct 31 19:54:02.655516 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:02.655491    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44f86c77-4686-48fe-8cf3-cc0e688af7a4-home\") pod \"rate-test-model-ae2e60-kserve-744475655c-xk2vp\" (UID: \"44f86c77-4686-48fe-8cf3-cc0e688af7a4\") " pod="ai-tenant-e2e-shared-b-w2-ae2e60/rate-test-model-ae2e60-kserve-744475655c-xk2vp"
Oct 31 19:54:02.655690 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:02.655670    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44f86c77-4686-48fe-8cf3-cc0e688af7a4-tls-certs\") pod \"rate-test-model-ae2e60-kserve-744475655c-xk2vp\" (UID: \"44f86c77-4686-48fe-8cf3-cc0e688af7a4\") " pod="ai-tenant-e2e-shared-b-w2-ae2e60/rate-test-model-ae2e60-kserve-744475655c-xk2vp"
Oct 31 19:54:02.763159 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:02.762989    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44f86c77-4686-48fe-8cf3-cc0e688af7a4-tls-certs\") pod \"rate-test-model-ae2e60-kserve-744475655c-xk2vp\" (UID: \"44f86c77-4686-48fe-8cf3-cc0e688af7a4\") " pod="ai-tenant-e2e-shared-b-w2-ae2e60/rate-test-model-ae2e60-kserve-744475655c-xk2vp"
Oct 31 19:54:02.763159 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:02.763150    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gstzb\" (UniqueName: \"kubernetes.io/projected/44f86c77-4686-48fe-8cf3-cc0e688af7a4-kube-api-access-gstzb\") pod \"rate-test-model-ae2e60-kserve-744475655c-xk2vp\" (UID: \"44f86c77-4686-48fe-8cf3-cc0e688af7a4\") " pod="ai-tenant-e2e-shared-b-w2-ae2e60/rate-test-model-ae2e60-kserve-744475655c-xk2vp"
Oct 31 19:54:02.763456 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:02.763198    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44f86c77-4686-48fe-8cf3-cc0e688af7a4-model-cache\") pod \"rate-test-model-ae2e60-kserve-744475655c-xk2vp\" (UID: \"44f86c77-4686-48fe-8cf3-cc0e688af7a4\") " pod="ai-tenant-e2e-shared-b-w2-ae2e60/rate-test-model-ae2e60-kserve-744475655c-xk2vp"
Oct 31 19:54:02.763456 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:02.763306    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44f86c77-4686-48fe-8cf3-cc0e688af7a4-tmp-dir\") pod \"rate-test-model-ae2e60-kserve-744475655c-xk2vp\" (UID: \"44f86c77-4686-48fe-8cf3-cc0e688af7a4\") " pod="ai-tenant-e2e-shared-b-w2-ae2e60/rate-test-model-ae2e60-kserve-744475655c-xk2vp"
Oct 31 19:54:02.763456 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:02.763393    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44f86c77-4686-48fe-8cf3-cc0e688af7a4-dshm\") pod \"rate-test-model-ae2e60-kserve-744475655c-xk2vp\" (UID: \"44f86c77-4686-48fe-8cf3-cc0e688af7a4\") " pod="ai-tenant-e2e-shared-b-w2-ae2e60/rate-test-model-ae2e60-kserve-744475655c-xk2vp"
Oct 31 19:54:02.763609 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:02.763535    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44f86c77-4686-48fe-8cf3-cc0e688af7a4-home\") pod \"rate-test-model-ae2e60-kserve-744475655c-xk2vp\" (UID: \"44f86c77-4686-48fe-8cf3-cc0e688af7a4\") " pod="ai-tenant-e2e-shared-b-w2-ae2e60/rate-test-model-ae2e60-kserve-744475655c-xk2vp"
Oct 31 19:54:02.764132 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:02.764103    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44f86c77-4686-48fe-8cf3-cc0e688af7a4-home\") pod \"rate-test-model-ae2e60-kserve-744475655c-xk2vp\" (UID: \"44f86c77-4686-48fe-8cf3-cc0e688af7a4\") " pod="ai-tenant-e2e-shared-b-w2-ae2e60/rate-test-model-ae2e60-kserve-744475655c-xk2vp"
Oct 31 19:54:02.782922 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:02.771089    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44f86c77-4686-48fe-8cf3-cc0e688af7a4-model-cache\") pod \"rate-test-model-ae2e60-kserve-744475655c-xk2vp\" (UID: \"44f86c77-4686-48fe-8cf3-cc0e688af7a4\") " pod="ai-tenant-e2e-shared-b-w2-ae2e60/rate-test-model-ae2e60-kserve-744475655c-xk2vp"
Oct 31 19:54:02.782922 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:02.771365    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44f86c77-4686-48fe-8cf3-cc0e688af7a4-tmp-dir\") pod \"rate-test-model-ae2e60-kserve-744475655c-xk2vp\" (UID: \"44f86c77-4686-48fe-8cf3-cc0e688af7a4\") " pod="ai-tenant-e2e-shared-b-w2-ae2e60/rate-test-model-ae2e60-kserve-744475655c-xk2vp"
Oct 31 19:54:02.827250 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:02.827160    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gstzb\" (UniqueName: \"kubernetes.io/projected/44f86c77-4686-48fe-8cf3-cc0e688af7a4-kube-api-access-gstzb\") pod \"rate-test-model-ae2e60-kserve-744475655c-xk2vp\" (UID: \"44f86c77-4686-48fe-8cf3-cc0e688af7a4\") " pod="ai-tenant-e2e-shared-b-w2-ae2e60/rate-test-model-ae2e60-kserve-744475655c-xk2vp"
Oct 31 19:54:02.833747 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:02.833717    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44f86c77-4686-48fe-8cf3-cc0e688af7a4-dshm\") pod \"rate-test-model-ae2e60-kserve-744475655c-xk2vp\" (UID: \"44f86c77-4686-48fe-8cf3-cc0e688af7a4\") " pod="ai-tenant-e2e-shared-b-w2-ae2e60/rate-test-model-ae2e60-kserve-744475655c-xk2vp"
Oct 31 19:54:02.834021 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:02.833987    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44f86c77-4686-48fe-8cf3-cc0e688af7a4-tls-certs\") pod \"rate-test-model-ae2e60-kserve-744475655c-xk2vp\" (UID: \"44f86c77-4686-48fe-8cf3-cc0e688af7a4\") " pod="ai-tenant-e2e-shared-b-w2-ae2e60/rate-test-model-ae2e60-kserve-744475655c-xk2vp"
Oct 31 19:54:02.864365 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:02.862959    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-ae2e60/rate-test-model-ae2e60-kserve-744475655c-xk2vp"
Oct 31 19:54:03.168764 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:54:03.168734    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod44f86c77_4686_48fe_8cf3_cc0e688af7a4.slice/crio-9c4602ce6a49471da8257b4e3472e0c0932e552af560c466d513b57478d8d924 WatchSource:0}: Error finding container 9c4602ce6a49471da8257b4e3472e0c0932e552af560c466d513b57478d8d924: Status 404 returned error can't find the container with id 9c4602ce6a49471da8257b4e3472e0c0932e552af560c466d513b57478d8d924
Oct 31 19:54:03.169111 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:03.168954    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-ae2e60/rate-test-model-ae2e60-kserve-744475655c-xk2vp"]
Oct 31 19:54:03.295282 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:03.295252    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:03.299019 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:03.298983    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-ae2e60/rate-test-model-ae2e60-kserve-744475655c-xk2vp" event={"ID":"44f86c77-4686-48fe-8cf3-cc0e688af7a4","Type":"ContainerStarted","Data":"84b053c3f262360af868dd7a162b1266fc3c7d2d4f764dcd68807583fbe1ab35"}
Oct 31 19:54:03.299152 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:03.299025    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-ae2e60/rate-test-model-ae2e60-kserve-744475655c-xk2vp" event={"ID":"44f86c77-4686-48fe-8cf3-cc0e688af7a4","Type":"ContainerStarted","Data":"9c4602ce6a49471da8257b4e3472e0c0932e552af560c466d513b57478d8d924"}
Oct 31 19:54:03.320941 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:03.320901    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-ae2e60/rate-test-model-ae2e60-kserve-744475655c-xk2vp" podStartSLOduration=1.320888205 podStartE2EDuration="1.320888205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:54:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:54:03.316768322 +0000 UTC m=+2073.890002452" watchObservedRunningTime="2026-10-31 19:54:03.320888205 +0000 UTC m=+2073.894122360"
Oct 31 19:54:03.865591 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:03.865559    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-ae2e60/rate-test-model-ae2e60-kserve-744475655c-xk2vp"
Oct 31 19:54:04.161541 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:04.161508    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:04.980376 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:04.977797    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:05.771415 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:05.771371    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:06.247246 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:54:06.247210    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 19:54:06.659492 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:06.659395    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:07.453807 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:07.453773    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:08.178763 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:08.178707    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:09.630183 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:09.630150    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:09.903601 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:09.903516    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:10.702125 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:10.702082    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:11.458709 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:11.458672    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:12.378430 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:12.378392    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:12.865656 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:12.865621    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-ae2e60/rate-test-model-ae2e60-kserve-744475655c-xk2vp"
Oct 31 19:54:12.878750 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:12.878723    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-ae2e60/rate-test-model-ae2e60-kserve-744475655c-xk2vp"
Oct 31 19:54:13.058568 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:13.058524    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:13.391449 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:13.391420    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-ae2e60/rate-test-model-ae2e60-kserve-744475655c-xk2vp"
Oct 31 19:54:13.848292 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:13.848254    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:14.656815 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:14.656780    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:15.069430 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:15.069394    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ceb3f51e-hwggz"]
Oct 31 19:54:15.070982 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:15.070960    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ceb3f51e-hwggz"]
Oct 31 19:54:15.579729 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:15.579693    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:15.642375 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:15.641478    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6eea5ad-a82b-47ba-a674-a3adecfacb68" path="/var/lib/kubelet/pods/d6eea5ad-a82b-47ba-a674-a3adecfacb68/volumes"
Oct 31 19:54:16.597450 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:16.597406    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:17.174496 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:17.174457    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-e618e1/auth-test-model-e618e1-kserve-765575658f-s9n47"]
Oct 31 19:54:17.258261 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:17.258161    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:17.479975 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:17.479881    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:17.680988 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:17.680942    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:18.294781 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:18.294719    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:19.157745 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:19.157705    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:20.507724 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:20.507681    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:21.735405 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:21.735372    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:23.358182 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:23.358100    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:25.028352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:25.025996    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:26.446351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:26.446290    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:27.775226 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:27.775193    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:28.896660 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:28.896618    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:29.912399 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:29.909663    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:31.019452 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:31.019406    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:32.075271 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:32.075231    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:32.195521 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:32.195487    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-e618e1/auth-test-model-e618e1-kserve-765575658f-s9n47" podUID="208c70bf-17d3-42fc-a106-9f30c2cc3c48" containerName="main" containerID="cri-o://ac14057527c5b766dafcdf7cbdf87bef70afcd9dc728fd79e631d39ad6794649" gracePeriod=45
Oct 31 19:54:32.489924 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:32.489897    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-e618e1/auth-test-model-e618e1-kserve-765575658f-s9n47"
Oct 31 19:54:32.523907 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:32.523876    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/208c70bf-17d3-42fc-a106-9f30c2cc3c48-dshm\" (UniqueName: \"kubernetes.io/empty-dir/208c70bf-17d3-42fc-a106-9f30c2cc3c48-dshm\") pod \"208c70bf-17d3-42fc-a106-9f30c2cc3c48\" (UID: \"208c70bf-17d3-42fc-a106-9f30c2cc3c48\") "
Oct 31 19:54:32.524097 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:32.523965    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/208c70bf-17d3-42fc-a106-9f30c2cc3c48-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/208c70bf-17d3-42fc-a106-9f30c2cc3c48-model-cache\") pod \"208c70bf-17d3-42fc-a106-9f30c2cc3c48\" (UID: \"208c70bf-17d3-42fc-a106-9f30c2cc3c48\") "
Oct 31 19:54:32.524097 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:32.524020    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/208c70bf-17d3-42fc-a106-9f30c2cc3c48-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/208c70bf-17d3-42fc-a106-9f30c2cc3c48-tmp-dir\") pod \"208c70bf-17d3-42fc-a106-9f30c2cc3c48\" (UID: \"208c70bf-17d3-42fc-a106-9f30c2cc3c48\") "
Oct 31 19:54:32.524316 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:32.524144    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/208c70bf-17d3-42fc-a106-9f30c2cc3c48-tls-certs\" (UniqueName: \"kubernetes.io/secret/208c70bf-17d3-42fc-a106-9f30c2cc3c48-tls-certs\") pod \"208c70bf-17d3-42fc-a106-9f30c2cc3c48\" (UID: \"208c70bf-17d3-42fc-a106-9f30c2cc3c48\") "
Oct 31 19:54:32.524316 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:32.524209    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/208c70bf-17d3-42fc-a106-9f30c2cc3c48-home\" (UniqueName: \"kubernetes.io/empty-dir/208c70bf-17d3-42fc-a106-9f30c2cc3c48-home\") pod \"208c70bf-17d3-42fc-a106-9f30c2cc3c48\" (UID: \"208c70bf-17d3-42fc-a106-9f30c2cc3c48\") "
Oct 31 19:54:32.524492 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:32.524360    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/208c70bf-17d3-42fc-a106-9f30c2cc3c48-tmp-dir" pod "208c70bf-17d3-42fc-a106-9f30c2cc3c48" (UID: "208c70bf-17d3-42fc-a106-9f30c2cc3c48"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:54:32.524492 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:32.524364    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/208c70bf-17d3-42fc-a106-9f30c2cc3c48-kube-api-access-k6588\" (UniqueName: \"kubernetes.io/projected/208c70bf-17d3-42fc-a106-9f30c2cc3c48-kube-api-access-k6588\") pod \"208c70bf-17d3-42fc-a106-9f30c2cc3c48\" (UID: \"208c70bf-17d3-42fc-a106-9f30c2cc3c48\") "
Oct 31 19:54:32.524492 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:32.524421    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/208c70bf-17d3-42fc-a106-9f30c2cc3c48-model-cache" pod "208c70bf-17d3-42fc-a106-9f30c2cc3c48" (UID: "208c70bf-17d3-42fc-a106-9f30c2cc3c48"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:54:32.524746 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:32.524720    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/208c70bf-17d3-42fc-a106-9f30c2cc3c48-home" pod "208c70bf-17d3-42fc-a106-9f30c2cc3c48" (UID: "208c70bf-17d3-42fc-a106-9f30c2cc3c48"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:54:32.525168 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:32.525149    4645 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/208c70bf-17d3-42fc-a106-9f30c2cc3c48-home\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:54:32.525244 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:32.525171    4645 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/208c70bf-17d3-42fc-a106-9f30c2cc3c48-model-cache\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:54:32.525244 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:32.525180    4645 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/208c70bf-17d3-42fc-a106-9f30c2cc3c48-tmp-dir\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:54:32.526939 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:32.526910    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/208c70bf-17d3-42fc-a106-9f30c2cc3c48-tls-certs" pod "208c70bf-17d3-42fc-a106-9f30c2cc3c48" (UID: "208c70bf-17d3-42fc-a106-9f30c2cc3c48"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:54:32.527053 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:32.526951    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/208c70bf-17d3-42fc-a106-9f30c2cc3c48-kube-api-access-k6588" pod "208c70bf-17d3-42fc-a106-9f30c2cc3c48" (UID: "208c70bf-17d3-42fc-a106-9f30c2cc3c48"). InnerVolumeSpecName "kube-api-access-k6588". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:54:32.527295 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:32.527275    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/208c70bf-17d3-42fc-a106-9f30c2cc3c48-dshm" pod "208c70bf-17d3-42fc-a106-9f30c2cc3c48" (UID: "208c70bf-17d3-42fc-a106-9f30c2cc3c48"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:54:32.545691 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:32.545655    4645 generic.go:356] "Generic (PLEG): container finished" podID="208c70bf-17d3-42fc-a106-9f30c2cc3c48" containerID="ac14057527c5b766dafcdf7cbdf87bef70afcd9dc728fd79e631d39ad6794649" exitCode=0
Oct 31 19:54:32.545871 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:32.545792    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-e618e1/auth-test-model-e618e1-kserve-765575658f-s9n47" event={"ID":"208c70bf-17d3-42fc-a106-9f30c2cc3c48","Type":"ContainerDied","Data":"ac14057527c5b766dafcdf7cbdf87bef70afcd9dc728fd79e631d39ad6794649"}
Oct 31 19:54:32.545871 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:32.545831    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-e618e1/auth-test-model-e618e1-kserve-765575658f-s9n47" event={"ID":"208c70bf-17d3-42fc-a106-9f30c2cc3c48","Type":"ContainerDied","Data":"b12f5af33c29c05fadd2e8b9485b04bbdde8179d5a4b297fb77221eb2c76b5af"}
Oct 31 19:54:32.545871 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:32.545855    4645 scope.go:122] "RemoveContainer" containerID="ac14057527c5b766dafcdf7cbdf87bef70afcd9dc728fd79e631d39ad6794649"
Oct 31 19:54:32.546105 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:32.546079    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-e618e1/auth-test-model-e618e1-kserve-765575658f-s9n47"
Oct 31 19:54:32.564376 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:32.564045    4645 scope.go:122] "RemoveContainer" containerID="ac14057527c5b766dafcdf7cbdf87bef70afcd9dc728fd79e631d39ad6794649"
Oct 31 19:54:32.564629 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:54:32.564599    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac14057527c5b766dafcdf7cbdf87bef70afcd9dc728fd79e631d39ad6794649\": container with ID starting with ac14057527c5b766dafcdf7cbdf87bef70afcd9dc728fd79e631d39ad6794649 not found: ID does not exist" containerID="ac14057527c5b766dafcdf7cbdf87bef70afcd9dc728fd79e631d39ad6794649"
Oct 31 19:54:32.564788 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:32.564637    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac14057527c5b766dafcdf7cbdf87bef70afcd9dc728fd79e631d39ad6794649"} err="failed to get container status \"ac14057527c5b766dafcdf7cbdf87bef70afcd9dc728fd79e631d39ad6794649\": rpc error: code = NotFound desc = could not find container \"ac14057527c5b766dafcdf7cbdf87bef70afcd9dc728fd79e631d39ad6794649\": container with ID starting with ac14057527c5b766dafcdf7cbdf87bef70afcd9dc728fd79e631d39ad6794649 not found: ID does not exist"
Oct 31 19:54:32.587294 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:32.587265    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-e618e1/auth-test-model-e618e1-kserve-765575658f-s9n47"]
Oct 31 19:54:32.593785 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:32.593756    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-e618e1/auth-test-model-e618e1-kserve-765575658f-s9n47"]
Oct 31 19:54:32.626664 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:32.626589    4645 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/208c70bf-17d3-42fc-a106-9f30c2cc3c48-dshm\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:54:32.626664 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:32.626619    4645 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/208c70bf-17d3-42fc-a106-9f30c2cc3c48-tls-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:54:32.626664 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:32.626629    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6588\" (UniqueName: \"kubernetes.io/projected/208c70bf-17d3-42fc-a106-9f30c2cc3c48-kube-api-access-k6588\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:54:33.101079 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:33.101039    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5e1b35a-cnbfq"]
Oct 31 19:54:33.108139 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:33.108106    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5e1b35a-cnbfq"]
Oct 31 19:54:33.251411 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:33.251369    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:33.619855 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:33.619818    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="208c70bf-17d3-42fc-a106-9f30c2cc3c48" path="/var/lib/kubelet/pods/208c70bf-17d3-42fc-a106-9f30c2cc3c48/volumes"
Oct 31 19:54:33.620361 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:33.620344    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd48fc95-b72d-427d-ac18-b86487770086" path="/var/lib/kubelet/pods/cd48fc95-b72d-427d-ac18-b86487770086/volumes"
Oct 31 19:54:34.445044 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:34.444997    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:35.326412 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:35.326317    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:36.247895 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:54:36.247860    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 19:54:36.374308 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:36.374270    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:37.789219 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:37.789174    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:38.252827 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:38.252797    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-d9c048/auth-test-model-d9c048-kserve-5d4b745b9-plqrp"]
Oct 31 19:54:38.308601 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:38.308566    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:38.883172 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:38.883136    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:39.970820 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:39.970786    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:40.798953 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:40.798917    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:41.764202 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:41.764153    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:42.587762 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:42.587722    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:43.445357 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:43.443121    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:44.397941 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:44.397904    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:45.256170 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:45.256124    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:45.965671 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:45.965637    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:46.771483 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:46.771451    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:47.784186 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:47.784143    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:48.833080 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:48.833042    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:49.811873 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:49.808383    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:50.866170 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:50.866138    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:51.694813 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:51.694775    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:52.625303 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:52.625260    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:53.274407 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:53.274322    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-d9c048/auth-test-model-d9c048-kserve-5d4b745b9-plqrp" podUID="73cf0e08-4670-4dc1-b333-a1160099958f" containerName="main" containerID="cri-o://f46c7a11bf22bead2479903e4888efba0642b2682db21f5e236e05a515d9dee0" gracePeriod=45
Oct 31 19:54:53.278725 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:53.278690    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-d9c048/auth-test-model-d9c048-kserve-5d4b745b9-plqrp" podUID="73cf0e08-4670-4dc1-b333-a1160099958f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.215:8000/ready\": EOF"
Oct 31 19:54:53.596511 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:53.596473    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99"]
Oct 31 19:54:53.597252 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:53.597226    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-d9c048/auth-test-model-d9c048-kserve-5d4b745b9-plqrp"
Oct 31 19:54:53.597414 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:53.597255    4645 cpu_manager.go:414] "containerMap: removing container" podUID="208c70bf-17d3-42fc-a106-9f30c2cc3c48" containerName="main"
Oct 31 19:54:53.597414 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:53.597274    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="208c70bf-17d3-42fc-a106-9f30c2cc3c48" containerName="main"
Oct 31 19:54:53.597494 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:53.597424    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="73cf0e08-4670-4dc1-b333-a1160099958f" containerName="main"
Oct 31 19:54:53.597494 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:53.597437    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="208c70bf-17d3-42fc-a106-9f30c2cc3c48" containerName="main"
Oct 31 19:54:53.600975 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:53.600953    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99"
Oct 31 19:54:53.641253 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:53.641213    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-4bac5a-openshift-default-dockercfg-p2t4w\""
Oct 31 19:54:53.666349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:53.660297    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:53.683509 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:53.683475    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99"]
Oct 31 19:54:53.683677 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:53.683460    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73cf0e08-4670-4dc1-b333-a1160099958f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73cf0e08-4670-4dc1-b333-a1160099958f-tmp-dir\") pod \"73cf0e08-4670-4dc1-b333-a1160099958f\" (UID: \"73cf0e08-4670-4dc1-b333-a1160099958f\") "
Oct 31 19:54:53.683749 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:53.683699    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73cf0e08-4670-4dc1-b333-a1160099958f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73cf0e08-4670-4dc1-b333-a1160099958f-model-cache\") pod \"73cf0e08-4670-4dc1-b333-a1160099958f\" (UID: \"73cf0e08-4670-4dc1-b333-a1160099958f\") "
Oct 31 19:54:53.683851 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:53.683752    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73cf0e08-4670-4dc1-b333-a1160099958f-tmp-dir" pod "73cf0e08-4670-4dc1-b333-a1160099958f" (UID: "73cf0e08-4670-4dc1-b333-a1160099958f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:54:53.683851 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:53.683824    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73cf0e08-4670-4dc1-b333-a1160099958f-kube-api-access-p5j64\" (UniqueName: \"kubernetes.io/projected/73cf0e08-4670-4dc1-b333-a1160099958f-kube-api-access-p5j64\") pod \"73cf0e08-4670-4dc1-b333-a1160099958f\" (UID: \"73cf0e08-4670-4dc1-b333-a1160099958f\") "
Oct 31 19:54:53.683983 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:53.683862    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73cf0e08-4670-4dc1-b333-a1160099958f-home\" (UniqueName: \"kubernetes.io/empty-dir/73cf0e08-4670-4dc1-b333-a1160099958f-home\") pod \"73cf0e08-4670-4dc1-b333-a1160099958f\" (UID: \"73cf0e08-4670-4dc1-b333-a1160099958f\") "
Oct 31 19:54:53.683983 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:53.683907    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73cf0e08-4670-4dc1-b333-a1160099958f-tls-certs\" (UniqueName: \"kubernetes.io/secret/73cf0e08-4670-4dc1-b333-a1160099958f-tls-certs\") pod \"73cf0e08-4670-4dc1-b333-a1160099958f\" (UID: \"73cf0e08-4670-4dc1-b333-a1160099958f\") "
Oct 31 19:54:53.684059 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:53.684011    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73cf0e08-4670-4dc1-b333-a1160099958f-model-cache" pod "73cf0e08-4670-4dc1-b333-a1160099958f" (UID: "73cf0e08-4670-4dc1-b333-a1160099958f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:54:53.684103 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:53.684071    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73cf0e08-4670-4dc1-b333-a1160099958f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/73cf0e08-4670-4dc1-b333-a1160099958f-dshm\") pod \"73cf0e08-4670-4dc1-b333-a1160099958f\" (UID: \"73cf0e08-4670-4dc1-b333-a1160099958f\") "
Oct 31 19:54:53.684565 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:53.684420    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/726d2f42-5772-4a85-ae19-2fcf878567d5-istio-token\") pod \"e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99\" (UID: \"726d2f42-5772-4a85-ae19-2fcf878567d5\") " pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99"
Oct 31 19:54:53.684565 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:53.684469    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/726d2f42-5772-4a85-ae19-2fcf878567d5-workload-certs\") pod \"e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99\" (UID: \"726d2f42-5772-4a85-ae19-2fcf878567d5\") " pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99"
Oct 31 19:54:53.684565 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:53.684505    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/726d2f42-5772-4a85-ae19-2fcf878567d5-credential-socket\") pod \"e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99\" (UID: \"726d2f42-5772-4a85-ae19-2fcf878567d5\") " pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99"
Oct 31 19:54:53.686236 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:53.684642    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/726d2f42-5772-4a85-ae19-2fcf878567d5-istio-envoy\") pod \"e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99\" (UID: \"726d2f42-5772-4a85-ae19-2fcf878567d5\") " pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99"
Oct 31 19:54:53.686236 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:53.684671    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/726d2f42-5772-4a85-ae19-2fcf878567d5-istio-podinfo\") pod \"e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99\" (UID: \"726d2f42-5772-4a85-ae19-2fcf878567d5\") " pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99"
Oct 31 19:54:53.686236 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:53.684807    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpq9n\" (UniqueName: \"kubernetes.io/projected/726d2f42-5772-4a85-ae19-2fcf878567d5-kube-api-access-bpq9n\") pod \"e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99\" (UID: \"726d2f42-5772-4a85-ae19-2fcf878567d5\") " pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99"
Oct 31 19:54:53.686236 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:53.684846    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/726d2f42-5772-4a85-ae19-2fcf878567d5-istiod-ca-cert\") pod \"e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99\" (UID: \"726d2f42-5772-4a85-ae19-2fcf878567d5\") " pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99"
Oct 31 19:54:53.686236 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:53.684910    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/726d2f42-5772-4a85-ae19-2fcf878567d5-istio-data\") pod \"e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99\" (UID: \"726d2f42-5772-4a85-ae19-2fcf878567d5\") " pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99"
Oct 31 19:54:53.686236 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:53.685016    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/726d2f42-5772-4a85-ae19-2fcf878567d5-workload-socket\") pod \"e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99\" (UID: \"726d2f42-5772-4a85-ae19-2fcf878567d5\") " pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99"
Oct 31 19:54:53.686236 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:53.685263    4645 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73cf0e08-4670-4dc1-b333-a1160099958f-tmp-dir\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:54:53.686236 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:53.685280    4645 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73cf0e08-4670-4dc1-b333-a1160099958f-model-cache\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:54:53.686236 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:53.685943    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73cf0e08-4670-4dc1-b333-a1160099958f-home" pod "73cf0e08-4670-4dc1-b333-a1160099958f" (UID: "73cf0e08-4670-4dc1-b333-a1160099958f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:54:53.707109 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:53.695745    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73cf0e08-4670-4dc1-b333-a1160099958f-tls-certs" pod "73cf0e08-4670-4dc1-b333-a1160099958f" (UID: "73cf0e08-4670-4dc1-b333-a1160099958f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:54:53.707109 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:53.700484    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73cf0e08-4670-4dc1-b333-a1160099958f-dshm" pod "73cf0e08-4670-4dc1-b333-a1160099958f" (UID: "73cf0e08-4670-4dc1-b333-a1160099958f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:54:53.721664 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:53.721577    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73cf0e08-4670-4dc1-b333-a1160099958f-kube-api-access-p5j64" pod "73cf0e08-4670-4dc1-b333-a1160099958f" (UID: "73cf0e08-4670-4dc1-b333-a1160099958f"). InnerVolumeSpecName "kube-api-access-p5j64". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:54:53.736930 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:53.736891    4645 generic.go:356] "Generic (PLEG): container finished" podID="73cf0e08-4670-4dc1-b333-a1160099958f" containerID="f46c7a11bf22bead2479903e4888efba0642b2682db21f5e236e05a515d9dee0" exitCode=0
Oct 31 19:54:53.737289 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:53.736981    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-d9c048/auth-test-model-d9c048-kserve-5d4b745b9-plqrp"
Oct 31 19:54:53.737564 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:53.737004    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-d9c048/auth-test-model-d9c048-kserve-5d4b745b9-plqrp" event={"ID":"73cf0e08-4670-4dc1-b333-a1160099958f","Type":"ContainerDied","Data":"f46c7a11bf22bead2479903e4888efba0642b2682db21f5e236e05a515d9dee0"}
Oct 31 19:54:53.737700 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:53.737681    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-d9c048/auth-test-model-d9c048-kserve-5d4b745b9-plqrp" event={"ID":"73cf0e08-4670-4dc1-b333-a1160099958f","Type":"ContainerDied","Data":"7c138e210f7e83a6703996d80da6c09cb12340611749269053c443c1a351ee34"}
Oct 31 19:54:53.737804 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:53.737793    4645 scope.go:122] "RemoveContainer" containerID="f46c7a11bf22bead2479903e4888efba0642b2682db21f5e236e05a515d9dee0"
Oct 31 19:54:53.750910 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:53.750888    4645 scope.go:122] "RemoveContainer" containerID="f46c7a11bf22bead2479903e4888efba0642b2682db21f5e236e05a515d9dee0"
Oct 31 19:54:53.751246 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:54:53.751219    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f46c7a11bf22bead2479903e4888efba0642b2682db21f5e236e05a515d9dee0\": container with ID starting with f46c7a11bf22bead2479903e4888efba0642b2682db21f5e236e05a515d9dee0 not found: ID does not exist" containerID="f46c7a11bf22bead2479903e4888efba0642b2682db21f5e236e05a515d9dee0"
Oct 31 19:54:53.751312 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:53.751259    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f46c7a11bf22bead2479903e4888efba0642b2682db21f5e236e05a515d9dee0"} err="failed to get container status \"f46c7a11bf22bead2479903e4888efba0642b2682db21f5e236e05a515d9dee0\": rpc error: code = NotFound desc = could not find container \"f46c7a11bf22bead2479903e4888efba0642b2682db21f5e236e05a515d9dee0\": container with ID starting with f46c7a11bf22bead2479903e4888efba0642b2682db21f5e236e05a515d9dee0 not found: ID does not exist"
Oct 31 19:54:53.766652 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:53.766614    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-d9c048/auth-test-model-d9c048-kserve-5d4b745b9-plqrp"]
Oct 31 19:54:53.771249 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:53.771212    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-d9c048/auth-test-model-d9c048-kserve-5d4b745b9-plqrp"]
Oct 31 19:54:53.786925 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:53.786879    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/726d2f42-5772-4a85-ae19-2fcf878567d5-istio-envoy\") pod \"e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99\" (UID: \"726d2f42-5772-4a85-ae19-2fcf878567d5\") " pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99"
Oct 31 19:54:53.787075 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:53.786928    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/726d2f42-5772-4a85-ae19-2fcf878567d5-istio-podinfo\") pod \"e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99\" (UID: \"726d2f42-5772-4a85-ae19-2fcf878567d5\") " pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99"
Oct 31 19:54:53.787075 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:53.787035    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpq9n\" (UniqueName: \"kubernetes.io/projected/726d2f42-5772-4a85-ae19-2fcf878567d5-kube-api-access-bpq9n\") pod \"e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99\" (UID: \"726d2f42-5772-4a85-ae19-2fcf878567d5\") " pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99"
Oct 31 19:54:53.787196 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:53.787079    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/726d2f42-5772-4a85-ae19-2fcf878567d5-istiod-ca-cert\") pod \"e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99\" (UID: \"726d2f42-5772-4a85-ae19-2fcf878567d5\") " pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99"
Oct 31 19:54:53.787196 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:53.787133    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/726d2f42-5772-4a85-ae19-2fcf878567d5-istio-data\") pod \"e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99\" (UID: \"726d2f42-5772-4a85-ae19-2fcf878567d5\") " pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99"
Oct 31 19:54:53.787297 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:53.787224    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/726d2f42-5772-4a85-ae19-2fcf878567d5-workload-socket\") pod \"e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99\" (UID: \"726d2f42-5772-4a85-ae19-2fcf878567d5\") " pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99"
Oct 31 19:54:53.787517 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:53.787492    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/726d2f42-5772-4a85-ae19-2fcf878567d5-istio-token\") pod \"e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99\" (UID: \"726d2f42-5772-4a85-ae19-2fcf878567d5\") " pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99"
Oct 31 19:54:53.787650 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:53.787539    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/726d2f42-5772-4a85-ae19-2fcf878567d5-workload-certs\") pod \"e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99\" (UID: \"726d2f42-5772-4a85-ae19-2fcf878567d5\") " pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99"
Oct 31 19:54:53.787650 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:53.787548    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/726d2f42-5772-4a85-ae19-2fcf878567d5-istio-data\") pod \"e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99\" (UID: \"726d2f42-5772-4a85-ae19-2fcf878567d5\") " pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99"
Oct 31 19:54:53.787650 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:53.787571    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/726d2f42-5772-4a85-ae19-2fcf878567d5-credential-socket\") pod \"e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99\" (UID: \"726d2f42-5772-4a85-ae19-2fcf878567d5\") " pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99"
Oct 31 19:54:53.787846 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:53.787697    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/726d2f42-5772-4a85-ae19-2fcf878567d5-workload-socket\") pod \"e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99\" (UID: \"726d2f42-5772-4a85-ae19-2fcf878567d5\") " pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99"
Oct 31 19:54:53.787918 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:53.787852    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/726d2f42-5772-4a85-ae19-2fcf878567d5-credential-socket\") pod \"e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99\" (UID: \"726d2f42-5772-4a85-ae19-2fcf878567d5\") " pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99"
Oct 31 19:54:53.787918 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:53.787862    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5j64\" (UniqueName: \"kubernetes.io/projected/73cf0e08-4670-4dc1-b333-a1160099958f-kube-api-access-p5j64\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:54:53.787918 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:53.787887    4645 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73cf0e08-4670-4dc1-b333-a1160099958f-home\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:54:53.787918 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:53.787903    4645 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73cf0e08-4670-4dc1-b333-a1160099958f-tls-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:54:53.787918 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:53.787915    4645 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73cf0e08-4670-4dc1-b333-a1160099958f-dshm\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:54:53.788105 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:53.787970    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/726d2f42-5772-4a85-ae19-2fcf878567d5-workload-certs\") pod \"e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99\" (UID: \"726d2f42-5772-4a85-ae19-2fcf878567d5\") " pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99"
Oct 31 19:54:53.788105 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:53.787975    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/726d2f42-5772-4a85-ae19-2fcf878567d5-istiod-ca-cert\") pod \"e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99\" (UID: \"726d2f42-5772-4a85-ae19-2fcf878567d5\") " pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99"
Oct 31 19:54:53.789556 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:53.789533    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/726d2f42-5772-4a85-ae19-2fcf878567d5-istio-envoy\") pod \"e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99\" (UID: \"726d2f42-5772-4a85-ae19-2fcf878567d5\") " pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99"
Oct 31 19:54:53.789784 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:53.789765    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/726d2f42-5772-4a85-ae19-2fcf878567d5-istio-podinfo\") pod \"e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99\" (UID: \"726d2f42-5772-4a85-ae19-2fcf878567d5\") " pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99"
Oct 31 19:54:53.800759 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:53.800721    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpq9n\" (UniqueName: \"kubernetes.io/projected/726d2f42-5772-4a85-ae19-2fcf878567d5-kube-api-access-bpq9n\") pod \"e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99\" (UID: \"726d2f42-5772-4a85-ae19-2fcf878567d5\") " pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99"
Oct 31 19:54:53.803273 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:53.803245    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/726d2f42-5772-4a85-ae19-2fcf878567d5-istio-token\") pod \"e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99\" (UID: \"726d2f42-5772-4a85-ae19-2fcf878567d5\") " pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99"
Oct 31 19:54:53.943527 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:53.943479    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99"
Oct 31 19:54:54.250478 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:54.250435    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99"]
Oct 31 19:54:54.251254 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:54:54.251218    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod726d2f42_5772_4a85_ae19_2fcf878567d5.slice/crio-8306699aa9e18cfa1c7c45c8a2cac5816574fe7c6cd0bdc0bc280609526fefb1 WatchSource:0}: Error finding container 8306699aa9e18cfa1c7c45c8a2cac5816574fe7c6cd0bdc0bc280609526fefb1: Status 404 returned error can't find the container with id 8306699aa9e18cfa1c7c45c8a2cac5816574fe7c6cd0bdc0bc280609526fefb1
Oct 31 19:54:54.254416 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:54.254375    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:54:54.254515 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:54.254458    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:54:54.254515 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:54.254491    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:54:54.483456 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:54.483402    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:54.757755 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:54.757710    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99" event={"ID":"726d2f42-5772-4a85-ae19-2fcf878567d5","Type":"ContainerStarted","Data":"b1c4b5407b6eac7792ec915125e7714b46ffeec197e61582ec1682b4120690b5"}
Oct 31 19:54:54.758300 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:54.757788    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99" event={"ID":"726d2f42-5772-4a85-ae19-2fcf878567d5","Type":"ContainerStarted","Data":"8306699aa9e18cfa1c7c45c8a2cac5816574fe7c6cd0bdc0bc280609526fefb1"}
Oct 31 19:54:54.786745 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:54.786705    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-e618e1/sub-test-model-e618e1-kserve-584c8d9c65-42m96"]
Oct 31 19:54:54.787435 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:54.787418    4645 cpu_manager.go:414] "containerMap: removing container" podUID="73cf0e08-4670-4dc1-b333-a1160099958f" containerName="main"
Oct 31 19:54:54.787522 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:54.787437    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73cf0e08-4670-4dc1-b333-a1160099958f" containerName="main"
Oct 31 19:54:54.791669 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:54.791634    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-e618e1/sub-test-model-e618e1-kserve-584c8d9c65-42m96"
Oct 31 19:54:54.796751 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:54.796654    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99" podStartSLOduration=1.7966286120000001 podStartE2EDuration="1.796628612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:54:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:54:54.792809705 +0000 UTC m=+2125.366043835" watchObservedRunningTime="2026-10-31 19:54:54.796628612 +0000 UTC m=+2125.369862742"
Oct 31 19:54:54.799070 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:54.799038    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-e618e1\"/\"default-dockercfg-hf4r4\""
Oct 31 19:54:54.803542 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:54.803485    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-e618e1\"/\"sub-test-model-e618e1-kserve-self-signed-certs\""
Oct 31 19:54:54.806716 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:54.806681    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-e618e1\"/\"openshift-service-ca.crt\""
Oct 31 19:54:54.807073 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:54.807007    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-e618e1\"/\"kube-root-ca.crt\""
Oct 31 19:54:54.808410 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:54.808376    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-e618e1/sub-test-model-e618e1-kserve-584c8d9c65-42m96"]
Oct 31 19:54:54.824451 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:54.824405    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:54.915983 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:54.915946    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f876b87-1980-4834-89d7-47b4db24a2c1-home\") pod \"sub-test-model-e618e1-kserve-584c8d9c65-42m96\" (UID: \"4f876b87-1980-4834-89d7-47b4db24a2c1\") " pod="ai-tenant-e2e-shared-a-w4-e618e1/sub-test-model-e618e1-kserve-584c8d9c65-42m96"
Oct 31 19:54:54.916198 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:54.916071    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f876b87-1980-4834-89d7-47b4db24a2c1-dshm\") pod \"sub-test-model-e618e1-kserve-584c8d9c65-42m96\" (UID: \"4f876b87-1980-4834-89d7-47b4db24a2c1\") " pod="ai-tenant-e2e-shared-a-w4-e618e1/sub-test-model-e618e1-kserve-584c8d9c65-42m96"
Oct 31 19:54:54.916246 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:54.916224    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkvlq\" (UniqueName: \"kubernetes.io/projected/4f876b87-1980-4834-89d7-47b4db24a2c1-kube-api-access-dkvlq\") pod \"sub-test-model-e618e1-kserve-584c8d9c65-42m96\" (UID: \"4f876b87-1980-4834-89d7-47b4db24a2c1\") " pod="ai-tenant-e2e-shared-a-w4-e618e1/sub-test-model-e618e1-kserve-584c8d9c65-42m96"
Oct 31 19:54:54.916290 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:54.916256    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f876b87-1980-4834-89d7-47b4db24a2c1-model-cache\") pod \"sub-test-model-e618e1-kserve-584c8d9c65-42m96\" (UID: \"4f876b87-1980-4834-89d7-47b4db24a2c1\") " pod="ai-tenant-e2e-shared-a-w4-e618e1/sub-test-model-e618e1-kserve-584c8d9c65-42m96"
Oct 31 19:54:54.916290 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:54.916282    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f876b87-1980-4834-89d7-47b4db24a2c1-tls-certs\") pod \"sub-test-model-e618e1-kserve-584c8d9c65-42m96\" (UID: \"4f876b87-1980-4834-89d7-47b4db24a2c1\") " pod="ai-tenant-e2e-shared-a-w4-e618e1/sub-test-model-e618e1-kserve-584c8d9c65-42m96"
Oct 31 19:54:54.916423 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:54.916406    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f876b87-1980-4834-89d7-47b4db24a2c1-tmp-dir\") pod \"sub-test-model-e618e1-kserve-584c8d9c65-42m96\" (UID: \"4f876b87-1980-4834-89d7-47b4db24a2c1\") " pod="ai-tenant-e2e-shared-a-w4-e618e1/sub-test-model-e618e1-kserve-584c8d9c65-42m96"
Oct 31 19:54:54.957353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:54.950437    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99"
Oct 31 19:54:55.020408 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:55.019542    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f876b87-1980-4834-89d7-47b4db24a2c1-home\") pod \"sub-test-model-e618e1-kserve-584c8d9c65-42m96\" (UID: \"4f876b87-1980-4834-89d7-47b4db24a2c1\") " pod="ai-tenant-e2e-shared-a-w4-e618e1/sub-test-model-e618e1-kserve-584c8d9c65-42m96"
Oct 31 19:54:55.020408 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:55.019646    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f876b87-1980-4834-89d7-47b4db24a2c1-dshm\") pod \"sub-test-model-e618e1-kserve-584c8d9c65-42m96\" (UID: \"4f876b87-1980-4834-89d7-47b4db24a2c1\") " pod="ai-tenant-e2e-shared-a-w4-e618e1/sub-test-model-e618e1-kserve-584c8d9c65-42m96"
Oct 31 19:54:55.020408 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:55.020241    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f876b87-1980-4834-89d7-47b4db24a2c1-home\") pod \"sub-test-model-e618e1-kserve-584c8d9c65-42m96\" (UID: \"4f876b87-1980-4834-89d7-47b4db24a2c1\") " pod="ai-tenant-e2e-shared-a-w4-e618e1/sub-test-model-e618e1-kserve-584c8d9c65-42m96"
Oct 31 19:54:55.020794 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:55.020760    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkvlq\" (UniqueName: \"kubernetes.io/projected/4f876b87-1980-4834-89d7-47b4db24a2c1-kube-api-access-dkvlq\") pod \"sub-test-model-e618e1-kserve-584c8d9c65-42m96\" (UID: \"4f876b87-1980-4834-89d7-47b4db24a2c1\") " pod="ai-tenant-e2e-shared-a-w4-e618e1/sub-test-model-e618e1-kserve-584c8d9c65-42m96"
Oct 31 19:54:55.020854 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:55.020829    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f876b87-1980-4834-89d7-47b4db24a2c1-model-cache\") pod \"sub-test-model-e618e1-kserve-584c8d9c65-42m96\" (UID: \"4f876b87-1980-4834-89d7-47b4db24a2c1\") " pod="ai-tenant-e2e-shared-a-w4-e618e1/sub-test-model-e618e1-kserve-584c8d9c65-42m96"
Oct 31 19:54:55.020911 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:55.020858    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f876b87-1980-4834-89d7-47b4db24a2c1-tls-certs\") pod \"sub-test-model-e618e1-kserve-584c8d9c65-42m96\" (UID: \"4f876b87-1980-4834-89d7-47b4db24a2c1\") " pod="ai-tenant-e2e-shared-a-w4-e618e1/sub-test-model-e618e1-kserve-584c8d9c65-42m96"
Oct 31 19:54:55.020971 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:55.020945    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f876b87-1980-4834-89d7-47b4db24a2c1-tmp-dir\") pod \"sub-test-model-e618e1-kserve-584c8d9c65-42m96\" (UID: \"4f876b87-1980-4834-89d7-47b4db24a2c1\") " pod="ai-tenant-e2e-shared-a-w4-e618e1/sub-test-model-e618e1-kserve-584c8d9c65-42m96"
Oct 31 19:54:55.022056 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:55.022024    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f876b87-1980-4834-89d7-47b4db24a2c1-tmp-dir\") pod \"sub-test-model-e618e1-kserve-584c8d9c65-42m96\" (UID: \"4f876b87-1980-4834-89d7-47b4db24a2c1\") " pod="ai-tenant-e2e-shared-a-w4-e618e1/sub-test-model-e618e1-kserve-584c8d9c65-42m96"
Oct 31 19:54:55.022270 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:55.022247    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f876b87-1980-4834-89d7-47b4db24a2c1-model-cache\") pod \"sub-test-model-e618e1-kserve-584c8d9c65-42m96\" (UID: \"4f876b87-1980-4834-89d7-47b4db24a2c1\") " pod="ai-tenant-e2e-shared-a-w4-e618e1/sub-test-model-e618e1-kserve-584c8d9c65-42m96"
Oct 31 19:54:55.040352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:55.040239    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f876b87-1980-4834-89d7-47b4db24a2c1-tls-certs\") pod \"sub-test-model-e618e1-kserve-584c8d9c65-42m96\" (UID: \"4f876b87-1980-4834-89d7-47b4db24a2c1\") " pod="ai-tenant-e2e-shared-a-w4-e618e1/sub-test-model-e618e1-kserve-584c8d9c65-42m96"
Oct 31 19:54:55.041354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:55.040571    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f876b87-1980-4834-89d7-47b4db24a2c1-dshm\") pod \"sub-test-model-e618e1-kserve-584c8d9c65-42m96\" (UID: \"4f876b87-1980-4834-89d7-47b4db24a2c1\") " pod="ai-tenant-e2e-shared-a-w4-e618e1/sub-test-model-e618e1-kserve-584c8d9c65-42m96"
Oct 31 19:54:55.051281 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:55.049957    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkvlq\" (UniqueName: \"kubernetes.io/projected/4f876b87-1980-4834-89d7-47b4db24a2c1-kube-api-access-dkvlq\") pod \"sub-test-model-e618e1-kserve-584c8d9c65-42m96\" (UID: \"4f876b87-1980-4834-89d7-47b4db24a2c1\") " pod="ai-tenant-e2e-shared-a-w4-e618e1/sub-test-model-e618e1-kserve-584c8d9c65-42m96"
Oct 31 19:54:55.131628 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:55.131548    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-e618e1/sub-test-model-e618e1-kserve-584c8d9c65-42m96"
Oct 31 19:54:55.433603 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:54:55.433568    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4f876b87_1980_4834_89d7_47b4db24a2c1.slice/crio-ccc4bda146ca580969e2a7841362360c68c022df65d86b8fc839da86cbf6f28c WatchSource:0}: Error finding container ccc4bda146ca580969e2a7841362360c68c022df65d86b8fc839da86cbf6f28c: Status 404 returned error can't find the container with id ccc4bda146ca580969e2a7841362360c68c022df65d86b8fc839da86cbf6f28c
Oct 31 19:54:55.434126 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:55.434093    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-e618e1/sub-test-model-e618e1-kserve-584c8d9c65-42m96"]
Oct 31 19:54:55.588612 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:55.588562    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:55.691431 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:55.691290    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73cf0e08-4670-4dc1-b333-a1160099958f" path="/var/lib/kubelet/pods/73cf0e08-4670-4dc1-b333-a1160099958f/volumes"
Oct 31 19:54:55.788413 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:55.788371    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-e618e1/sub-test-model-e618e1-kserve-584c8d9c65-42m96" event={"ID":"4f876b87-1980-4834-89d7-47b4db24a2c1","Type":"ContainerStarted","Data":"43a30004ad65415df37e1ec6f8998895fabb74123a46f998c67fb358c9687ffc"}
Oct 31 19:54:55.788875 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:55.788426    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-e618e1/sub-test-model-e618e1-kserve-584c8d9c65-42m96" event={"ID":"4f876b87-1980-4834-89d7-47b4db24a2c1","Type":"ContainerStarted","Data":"ccc4bda146ca580969e2a7841362360c68c022df65d86b8fc839da86cbf6f28c"}
Oct 31 19:54:55.810131 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:55.810071    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-e618e1/sub-test-model-e618e1-kserve-584c8d9c65-42m96" podStartSLOduration=1.810053276 podStartE2EDuration="1.810053276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:54:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:54:55.80627736 +0000 UTC m=+2126.379511489" watchObservedRunningTime="2026-10-31 19:54:55.810053276 +0000 UTC m=+2126.383287404"
Oct 31 19:54:55.950907 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:55.950483    4645 patch_prober.go:29] interesting pod/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body=
Oct 31 19:54:55.950907 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:55.950555    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99" podUID="726d2f42-5772-4a85-ae19-2fcf878567d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused"
Oct 31 19:54:56.132575 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:56.132522    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-e618e1/sub-test-model-e618e1-kserve-584c8d9c65-42m96"
Oct 31 19:54:56.496968 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:56.496926    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:56.944778 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:56.944735    4645 patch_prober.go:29] interesting pod/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body=
Oct 31 19:54:56.945157 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:56.944786    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99" podUID="726d2f42-5772-4a85-ae19-2fcf878567d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused"
Oct 31 19:54:57.481477 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:57.481438    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:57.943906 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:57.943869    4645 patch_prober.go:29] interesting pod/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body=
Oct 31 19:54:57.944086 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:57.943922    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99" podUID="726d2f42-5772-4a85-ae19-2fcf878567d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused"
Oct 31 19:54:58.281628 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:58.281541    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:58.944145 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:58.944109    4645 patch_prober.go:29] interesting pod/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body=
Oct 31 19:54:58.944353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:58.944172    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99" podUID="726d2f42-5772-4a85-ae19-2fcf878567d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused"
Oct 31 19:54:59.043930 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:59.043895    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d448ce9-mm5sz"]
Oct 31 19:54:59.048295 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:59.048269    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7b1e78a-qg2kl"]
Oct 31 19:54:59.051721 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:59.051699    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d448ce9-mm5sz"]
Oct 31 19:54:59.074419 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:59.074385    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2683f624-g7tlm"]
Oct 31 19:54:59.086606 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:59.086576    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7b1e78a-qg2kl"]
Oct 31 19:54:59.092438 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:59.092411    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2683f624-g7tlm"]
Oct 31 19:54:59.143353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:59.132030    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:54:59.623418 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:59.623378    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3003da29-b66f-4a97-a303-decfacf3250a" path="/var/lib/kubelet/pods/3003da29-b66f-4a97-a303-decfacf3250a/volumes"
Oct 31 19:54:59.624222 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:59.624199    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa03581c-60ab-4f65-a8ea-e5ba6b47a16d" path="/var/lib/kubelet/pods/aa03581c-60ab-4f65-a8ea-e5ba6b47a16d/volumes"
Oct 31 19:54:59.624886 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:59.624866    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2f889ae-016e-472f-ab38-346ceb2cd980" path="/var/lib/kubelet/pods/d2f889ae-016e-472f-ab38-346ceb2cd980/volumes"
Oct 31 19:54:59.946459 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:59.944849    4645 patch_prober.go:29] interesting pod/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body=
Oct 31 19:54:59.946459 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:54:59.944927    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99" podUID="726d2f42-5772-4a85-ae19-2fcf878567d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused"
Oct 31 19:55:00.349385 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:00.349288    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:55:00.945634 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:00.945597    4645 patch_prober.go:29] interesting pod/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body=
Oct 31 19:55:00.958183 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:00.958140    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99" podUID="726d2f42-5772-4a85-ae19-2fcf878567d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused"
Oct 31 19:55:01.345815 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:01.345704    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:55:01.948310 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:01.948280    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99"
Oct 31 19:55:01.948789 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:01.948770    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99"
Oct 31 19:55:01.949401 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:01.949380    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99"
Oct 31 19:55:02.309310 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:02.299085    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:55:03.192400 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:03.192367    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:55:03.931935 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:03.931900    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:55:04.669740 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:04.669705    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:55:05.132376 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:05.132272    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-e618e1/sub-test-model-e618e1-kserve-584c8d9c65-42m96"
Oct 31 19:55:05.145291 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:05.145263    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-e618e1/sub-test-model-e618e1-kserve-584c8d9c65-42m96"
Oct 31 19:55:05.446005 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:05.445969    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:55:05.932506 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:05.932477    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-e618e1/sub-test-model-e618e1-kserve-584c8d9c65-42m96"
Oct 31 19:55:06.219024 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:06.218930    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:55:06.246696 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:55:06.246664    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 19:55:07.139564 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:07.139531    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:55:07.865364 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:07.865311    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:55:08.637752 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:08.637720    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:55:09.350175 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:09.350141    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:55:10.120439 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:10.120405    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d914121-zfkpr"]
Oct 31 19:55:10.127368 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:10.127344    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e160cf2-wqzvm"]
Oct 31 19:55:10.143793 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:10.143769    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d914121-zfkpr"]
Oct 31 19:55:10.149058 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:10.149022    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e160cf2-wqzvm"]
Oct 31 19:55:10.283769 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:10.283744    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:55:11.147968 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:11.147936    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:55:11.187483 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:11.187447    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-d9c048/sub-test-model-d9c048-kserve-6fff8b86d5-2mb99"]
Oct 31 19:55:11.193861 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:11.193833    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-d9c048/sub-test-model-d9c048-kserve-6fff8b86d5-2mb99"
Oct 31 19:55:11.199523 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:11.199499    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-d9c048\"/\"kube-root-ca.crt\""
Oct 31 19:55:11.199914 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:11.199777    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-d9c048\"/\"sub-test-model-d9c048-kserve-self-signed-certs\""
Oct 31 19:55:11.200058 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:11.199862    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-d9c048\"/\"default-dockercfg-xhl8s\""
Oct 31 19:55:11.201105 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:11.201087    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-d9c048\"/\"openshift-service-ca.crt\""
Oct 31 19:55:11.203644 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:11.203621    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-d9c048/sub-test-model-d9c048-kserve-6fff8b86d5-2mb99"]
Oct 31 19:55:11.260278 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:11.260253    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:55:11.296954 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:11.296928    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2665b4f7-db95-40bf-8027-63b27e28afa3-model-cache\") pod \"sub-test-model-d9c048-kserve-6fff8b86d5-2mb99\" (UID: \"2665b4f7-db95-40bf-8027-63b27e28afa3\") " pod="ai-tenant-e2e-shared-b-w4-d9c048/sub-test-model-d9c048-kserve-6fff8b86d5-2mb99"
Oct 31 19:55:11.297087 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:11.297049    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2665b4f7-db95-40bf-8027-63b27e28afa3-dshm\") pod \"sub-test-model-d9c048-kserve-6fff8b86d5-2mb99\" (UID: \"2665b4f7-db95-40bf-8027-63b27e28afa3\") " pod="ai-tenant-e2e-shared-b-w4-d9c048/sub-test-model-d9c048-kserve-6fff8b86d5-2mb99"
Oct 31 19:55:11.297139 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:11.297108    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mpn4\" (UniqueName: \"kubernetes.io/projected/2665b4f7-db95-40bf-8027-63b27e28afa3-kube-api-access-6mpn4\") pod \"sub-test-model-d9c048-kserve-6fff8b86d5-2mb99\" (UID: \"2665b4f7-db95-40bf-8027-63b27e28afa3\") " pod="ai-tenant-e2e-shared-b-w4-d9c048/sub-test-model-d9c048-kserve-6fff8b86d5-2mb99"
Oct 31 19:55:11.297218 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:11.297204    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2665b4f7-db95-40bf-8027-63b27e28afa3-tmp-dir\") pod \"sub-test-model-d9c048-kserve-6fff8b86d5-2mb99\" (UID: \"2665b4f7-db95-40bf-8027-63b27e28afa3\") " pod="ai-tenant-e2e-shared-b-w4-d9c048/sub-test-model-d9c048-kserve-6fff8b86d5-2mb99"
Oct 31 19:55:11.297268 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:11.297251    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2665b4f7-db95-40bf-8027-63b27e28afa3-home\") pod \"sub-test-model-d9c048-kserve-6fff8b86d5-2mb99\" (UID: \"2665b4f7-db95-40bf-8027-63b27e28afa3\") " pod="ai-tenant-e2e-shared-b-w4-d9c048/sub-test-model-d9c048-kserve-6fff8b86d5-2mb99"
Oct 31 19:55:11.297360 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:11.297323    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2665b4f7-db95-40bf-8027-63b27e28afa3-tls-certs\") pod \"sub-test-model-d9c048-kserve-6fff8b86d5-2mb99\" (UID: \"2665b4f7-db95-40bf-8027-63b27e28afa3\") " pod="ai-tenant-e2e-shared-b-w4-d9c048/sub-test-model-d9c048-kserve-6fff8b86d5-2mb99"
Oct 31 19:55:11.399174 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:11.399076    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2665b4f7-db95-40bf-8027-63b27e28afa3-tmp-dir\") pod \"sub-test-model-d9c048-kserve-6fff8b86d5-2mb99\" (UID: \"2665b4f7-db95-40bf-8027-63b27e28afa3\") " pod="ai-tenant-e2e-shared-b-w4-d9c048/sub-test-model-d9c048-kserve-6fff8b86d5-2mb99"
Oct 31 19:55:11.399500 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:11.399474    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2665b4f7-db95-40bf-8027-63b27e28afa3-home\") pod \"sub-test-model-d9c048-kserve-6fff8b86d5-2mb99\" (UID: \"2665b4f7-db95-40bf-8027-63b27e28afa3\") " pod="ai-tenant-e2e-shared-b-w4-d9c048/sub-test-model-d9c048-kserve-6fff8b86d5-2mb99"
Oct 31 19:55:11.399746 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:11.399728    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2665b4f7-db95-40bf-8027-63b27e28afa3-tls-certs\") pod \"sub-test-model-d9c048-kserve-6fff8b86d5-2mb99\" (UID: \"2665b4f7-db95-40bf-8027-63b27e28afa3\") " pod="ai-tenant-e2e-shared-b-w4-d9c048/sub-test-model-d9c048-kserve-6fff8b86d5-2mb99"
Oct 31 19:55:11.399876 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:11.399863    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2665b4f7-db95-40bf-8027-63b27e28afa3-model-cache\") pod \"sub-test-model-d9c048-kserve-6fff8b86d5-2mb99\" (UID: \"2665b4f7-db95-40bf-8027-63b27e28afa3\") " pod="ai-tenant-e2e-shared-b-w4-d9c048/sub-test-model-d9c048-kserve-6fff8b86d5-2mb99"
Oct 31 19:55:11.400104 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:11.400086    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2665b4f7-db95-40bf-8027-63b27e28afa3-dshm\") pod \"sub-test-model-d9c048-kserve-6fff8b86d5-2mb99\" (UID: \"2665b4f7-db95-40bf-8027-63b27e28afa3\") " pod="ai-tenant-e2e-shared-b-w4-d9c048/sub-test-model-d9c048-kserve-6fff8b86d5-2mb99"
Oct 31 19:55:11.400246 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:11.400224    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mpn4\" (UniqueName: \"kubernetes.io/projected/2665b4f7-db95-40bf-8027-63b27e28afa3-kube-api-access-6mpn4\") pod \"sub-test-model-d9c048-kserve-6fff8b86d5-2mb99\" (UID: \"2665b4f7-db95-40bf-8027-63b27e28afa3\") " pod="ai-tenant-e2e-shared-b-w4-d9c048/sub-test-model-d9c048-kserve-6fff8b86d5-2mb99"
Oct 31 19:55:11.410357 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:11.401212    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2665b4f7-db95-40bf-8027-63b27e28afa3-tmp-dir\") pod \"sub-test-model-d9c048-kserve-6fff8b86d5-2mb99\" (UID: \"2665b4f7-db95-40bf-8027-63b27e28afa3\") " pod="ai-tenant-e2e-shared-b-w4-d9c048/sub-test-model-d9c048-kserve-6fff8b86d5-2mb99"
Oct 31 19:55:11.411073 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:11.408038    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2665b4f7-db95-40bf-8027-63b27e28afa3-home\") pod \"sub-test-model-d9c048-kserve-6fff8b86d5-2mb99\" (UID: \"2665b4f7-db95-40bf-8027-63b27e28afa3\") " pod="ai-tenant-e2e-shared-b-w4-d9c048/sub-test-model-d9c048-kserve-6fff8b86d5-2mb99"
Oct 31 19:55:11.425421 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:11.409501    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2665b4f7-db95-40bf-8027-63b27e28afa3-model-cache\") pod \"sub-test-model-d9c048-kserve-6fff8b86d5-2mb99\" (UID: \"2665b4f7-db95-40bf-8027-63b27e28afa3\") " pod="ai-tenant-e2e-shared-b-w4-d9c048/sub-test-model-d9c048-kserve-6fff8b86d5-2mb99"
Oct 31 19:55:11.444346 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:11.440693    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2665b4f7-db95-40bf-8027-63b27e28afa3-dshm\") pod \"sub-test-model-d9c048-kserve-6fff8b86d5-2mb99\" (UID: \"2665b4f7-db95-40bf-8027-63b27e28afa3\") " pod="ai-tenant-e2e-shared-b-w4-d9c048/sub-test-model-d9c048-kserve-6fff8b86d5-2mb99"
Oct 31 19:55:11.458847 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:11.458815    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2665b4f7-db95-40bf-8027-63b27e28afa3-tls-certs\") pod \"sub-test-model-d9c048-kserve-6fff8b86d5-2mb99\" (UID: \"2665b4f7-db95-40bf-8027-63b27e28afa3\") " pod="ai-tenant-e2e-shared-b-w4-d9c048/sub-test-model-d9c048-kserve-6fff8b86d5-2mb99"
Oct 31 19:55:11.462866 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:11.459660    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mpn4\" (UniqueName: \"kubernetes.io/projected/2665b4f7-db95-40bf-8027-63b27e28afa3-kube-api-access-6mpn4\") pod \"sub-test-model-d9c048-kserve-6fff8b86d5-2mb99\" (UID: \"2665b4f7-db95-40bf-8027-63b27e28afa3\") " pod="ai-tenant-e2e-shared-b-w4-d9c048/sub-test-model-d9c048-kserve-6fff8b86d5-2mb99"
Oct 31 19:55:11.519911 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:11.519620    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-d9c048/sub-test-model-d9c048-kserve-6fff8b86d5-2mb99"
Oct 31 19:55:11.626148 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:11.625813    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bda9b5d-489e-4834-9676-1ed463ed81e0" path="/var/lib/kubelet/pods/4bda9b5d-489e-4834-9676-1ed463ed81e0/volumes"
Oct 31 19:55:11.626907 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:11.626883    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c13db3a-c224-4a77-b079-5d491d2592a0" path="/var/lib/kubelet/pods/5c13db3a-c224-4a77-b079-5d491d2592a0/volumes"
Oct 31 19:55:11.827775 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:11.827745    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-d9c048/sub-test-model-d9c048-kserve-6fff8b86d5-2mb99"]
Oct 31 19:55:11.828377 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:55:11.828320    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2665b4f7_db95_40bf_8027_63b27e28afa3.slice/crio-cffc36185c0f78c5fe6fda6bb360d1d7e09f846e4ee567e947a64b332b7a6cd7 WatchSource:0}: Error finding container cffc36185c0f78c5fe6fda6bb360d1d7e09f846e4ee567e947a64b332b7a6cd7: Status 404 returned error can't find the container with id cffc36185c0f78c5fe6fda6bb360d1d7e09f846e4ee567e947a64b332b7a6cd7
Oct 31 19:55:11.980697 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:11.980663    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-d9c048/sub-test-model-d9c048-kserve-6fff8b86d5-2mb99" event={"ID":"2665b4f7-db95-40bf-8027-63b27e28afa3","Type":"ContainerStarted","Data":"05237464e141ec8e599962cfdb22f91f992cc45c86421c1f0fae5b386a60debc"}
Oct 31 19:55:11.980859 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:11.980708    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-d9c048/sub-test-model-d9c048-kserve-6fff8b86d5-2mb99" event={"ID":"2665b4f7-db95-40bf-8027-63b27e28afa3","Type":"ContainerStarted","Data":"cffc36185c0f78c5fe6fda6bb360d1d7e09f846e4ee567e947a64b332b7a6cd7"}
Oct 31 19:55:12.000239 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:12.000174    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-d9c048/sub-test-model-d9c048-kserve-6fff8b86d5-2mb99" podStartSLOduration=1.000153295 podStartE2EDuration="1.000153295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:55:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:55:11.999239406 +0000 UTC m=+2142.572473535" watchObservedRunningTime="2026-10-31 19:55:12.000153295 +0000 UTC m=+2142.573387441"
Oct 31 19:55:12.031105 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:12.031063    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:55:12.521432 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:12.521403    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-d9c048/sub-test-model-d9c048-kserve-6fff8b86d5-2mb99"
Oct 31 19:55:12.748190 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:12.748144    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:55:13.563403 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:13.563367    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:55:14.339011 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:14.338971    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:55:15.151677 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:15.151627    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:55:15.869704 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:15.869664    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:55:16.731649 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:16.731601    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:55:17.180899 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:17.180863    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:55:17.564430 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:17.564345    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:55:18.261666 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:18.261626    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:55:18.440809 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:18.440768    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:55:19.363688 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:19.363653    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:55:20.342596 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:20.342562    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:55:21.520899 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:21.520865    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-d9c048/sub-test-model-d9c048-kserve-6fff8b86d5-2mb99"
Oct 31 19:55:21.534010 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:21.533983    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-d9c048/sub-test-model-d9c048-kserve-6fff8b86d5-2mb99"
Oct 31 19:55:22.066739 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:22.066705    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-d9c048/sub-test-model-d9c048-kserve-6fff8b86d5-2mb99"
Oct 31 19:55:31.119748 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:31.119716    4645 scope.go:122] "RemoveContainer" containerID="6733c2051b46505dc74e54ebd20a03f1993f062851339d2c76632b64dbc42f1d"
Oct 31 19:55:31.130579 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:31.130540    4645 scope.go:122] "RemoveContainer" containerID="7b99f12652fa8c4c40f711674d62c4f47524f1511395b30f98bd2c72bc2c60ca"
Oct 31 19:55:34.074669 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:34.074635    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk"]
Oct 31 19:55:34.080673 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:34.080647    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk"
Oct 31 19:55:34.094099 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:34.094076    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-ae2e60-openshift-default-dockercfg-ts6pq\""
Oct 31 19:55:34.119342 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:34.119291    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22d8af28-f4e5-47ea-a25a-9076b9afbc11-workload-certs\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk\" (UID: \"22d8af28-f4e5-47ea-a25a-9076b9afbc11\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk"
Oct 31 19:55:34.119650 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:34.119629    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22d8af28-f4e5-47ea-a25a-9076b9afbc11-istio-token\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk\" (UID: \"22d8af28-f4e5-47ea-a25a-9076b9afbc11\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk"
Oct 31 19:55:34.119770 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:34.119756    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22d8af28-f4e5-47ea-a25a-9076b9afbc11-istio-podinfo\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk\" (UID: \"22d8af28-f4e5-47ea-a25a-9076b9afbc11\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk"
Oct 31 19:55:34.119906 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:34.119884    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h62pb\" (UniqueName: \"kubernetes.io/projected/22d8af28-f4e5-47ea-a25a-9076b9afbc11-kube-api-access-h62pb\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk\" (UID: \"22d8af28-f4e5-47ea-a25a-9076b9afbc11\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk"
Oct 31 19:55:34.120047 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:34.119926    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22d8af28-f4e5-47ea-a25a-9076b9afbc11-workload-socket\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk\" (UID: \"22d8af28-f4e5-47ea-a25a-9076b9afbc11\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk"
Oct 31 19:55:34.120120 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:34.120019    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22d8af28-f4e5-47ea-a25a-9076b9afbc11-credential-socket\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk\" (UID: \"22d8af28-f4e5-47ea-a25a-9076b9afbc11\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk"
Oct 31 19:55:34.120268 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:34.120251    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22d8af28-f4e5-47ea-a25a-9076b9afbc11-istio-data\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk\" (UID: \"22d8af28-f4e5-47ea-a25a-9076b9afbc11\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk"
Oct 31 19:55:34.120359 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:34.120299    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22d8af28-f4e5-47ea-a25a-9076b9afbc11-istio-envoy\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk\" (UID: \"22d8af28-f4e5-47ea-a25a-9076b9afbc11\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk"
Oct 31 19:55:34.120359 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:34.120344    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22d8af28-f4e5-47ea-a25a-9076b9afbc11-istiod-ca-cert\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk\" (UID: \"22d8af28-f4e5-47ea-a25a-9076b9afbc11\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk"
Oct 31 19:55:34.123601 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:34.123578    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk"]
Oct 31 19:55:34.221751 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:34.221721    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22d8af28-f4e5-47ea-a25a-9076b9afbc11-workload-socket\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk\" (UID: \"22d8af28-f4e5-47ea-a25a-9076b9afbc11\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk"
Oct 31 19:55:34.221912 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:34.221797    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22d8af28-f4e5-47ea-a25a-9076b9afbc11-credential-socket\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk\" (UID: \"22d8af28-f4e5-47ea-a25a-9076b9afbc11\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk"
Oct 31 19:55:34.221962 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:34.221912    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22d8af28-f4e5-47ea-a25a-9076b9afbc11-istio-data\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk\" (UID: \"22d8af28-f4e5-47ea-a25a-9076b9afbc11\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk"
Oct 31 19:55:34.221962 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:34.221943    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22d8af28-f4e5-47ea-a25a-9076b9afbc11-istio-envoy\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk\" (UID: \"22d8af28-f4e5-47ea-a25a-9076b9afbc11\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk"
Oct 31 19:55:34.222132 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:34.221961    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22d8af28-f4e5-47ea-a25a-9076b9afbc11-istiod-ca-cert\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk\" (UID: \"22d8af28-f4e5-47ea-a25a-9076b9afbc11\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk"
Oct 31 19:55:34.222132 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:34.222033    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22d8af28-f4e5-47ea-a25a-9076b9afbc11-workload-certs\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk\" (UID: \"22d8af28-f4e5-47ea-a25a-9076b9afbc11\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk"
Oct 31 19:55:34.222250 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:34.222148    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22d8af28-f4e5-47ea-a25a-9076b9afbc11-workload-socket\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk\" (UID: \"22d8af28-f4e5-47ea-a25a-9076b9afbc11\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk"
Oct 31 19:55:34.222250 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:34.222154    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22d8af28-f4e5-47ea-a25a-9076b9afbc11-istio-token\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk\" (UID: \"22d8af28-f4e5-47ea-a25a-9076b9afbc11\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk"
Oct 31 19:55:34.222250 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:34.222207    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22d8af28-f4e5-47ea-a25a-9076b9afbc11-istio-podinfo\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk\" (UID: \"22d8af28-f4e5-47ea-a25a-9076b9afbc11\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk"
Oct 31 19:55:34.222473 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:34.222251    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h62pb\" (UniqueName: \"kubernetes.io/projected/22d8af28-f4e5-47ea-a25a-9076b9afbc11-kube-api-access-h62pb\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk\" (UID: \"22d8af28-f4e5-47ea-a25a-9076b9afbc11\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk"
Oct 31 19:55:34.222473 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:34.222269    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22d8af28-f4e5-47ea-a25a-9076b9afbc11-istio-data\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk\" (UID: \"22d8af28-f4e5-47ea-a25a-9076b9afbc11\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk"
Oct 31 19:55:34.222635 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:34.222609    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22d8af28-f4e5-47ea-a25a-9076b9afbc11-workload-certs\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk\" (UID: \"22d8af28-f4e5-47ea-a25a-9076b9afbc11\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk"
Oct 31 19:55:34.222765 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:34.222742    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22d8af28-f4e5-47ea-a25a-9076b9afbc11-istiod-ca-cert\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk\" (UID: \"22d8af28-f4e5-47ea-a25a-9076b9afbc11\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk"
Oct 31 19:55:34.222864 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:34.222804    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22d8af28-f4e5-47ea-a25a-9076b9afbc11-credential-socket\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk\" (UID: \"22d8af28-f4e5-47ea-a25a-9076b9afbc11\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk"
Oct 31 19:55:34.224415 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:34.224393    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22d8af28-f4e5-47ea-a25a-9076b9afbc11-istio-envoy\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk\" (UID: \"22d8af28-f4e5-47ea-a25a-9076b9afbc11\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk"
Oct 31 19:55:34.224943 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:34.224923    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22d8af28-f4e5-47ea-a25a-9076b9afbc11-istio-podinfo\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk\" (UID: \"22d8af28-f4e5-47ea-a25a-9076b9afbc11\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk"
Oct 31 19:55:34.236920 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:34.236898    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h62pb\" (UniqueName: \"kubernetes.io/projected/22d8af28-f4e5-47ea-a25a-9076b9afbc11-kube-api-access-h62pb\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk\" (UID: \"22d8af28-f4e5-47ea-a25a-9076b9afbc11\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk"
Oct 31 19:55:34.239170 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:34.239150    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22d8af28-f4e5-47ea-a25a-9076b9afbc11-istio-token\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk\" (UID: \"22d8af28-f4e5-47ea-a25a-9076b9afbc11\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk"
Oct 31 19:55:34.395522 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:34.395473    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk"
Oct 31 19:55:34.691365 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:55:34.691298    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22d8af28_f4e5_47ea_a25a_9076b9afbc11.slice/crio-3d54ceb5ab03e959f991cc3213f4c901ef9d688116a2c8f4626537e59cbfbbcc WatchSource:0}: Error finding container 3d54ceb5ab03e959f991cc3213f4c901ef9d688116a2c8f4626537e59cbfbbcc: Status 404 returned error can't find the container with id 3d54ceb5ab03e959f991cc3213f4c901ef9d688116a2c8f4626537e59cbfbbcc
Oct 31 19:55:34.692368 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:34.692324    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk"]
Oct 31 19:55:34.694152 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:34.694115    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:55:34.694261 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:34.694187    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:55:34.694261 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:34.694224    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:55:35.172838 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:35.172801    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk" event={"ID":"22d8af28-f4e5-47ea-a25a-9076b9afbc11","Type":"ContainerStarted","Data":"9490917c1e492122036f9de619e640b0911012219d2c82f4dcf059fb9e3808f0"}
Oct 31 19:55:35.173244 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:35.172855    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk" event={"ID":"22d8af28-f4e5-47ea-a25a-9076b9afbc11","Type":"ContainerStarted","Data":"3d54ceb5ab03e959f991cc3213f4c901ef9d688116a2c8f4626537e59cbfbbcc"}
Oct 31 19:55:35.198476 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:35.198430    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk" podStartSLOduration=1.198415622 podStartE2EDuration="1.198415622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:55:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:55:35.196394901 +0000 UTC m=+2165.769629024" watchObservedRunningTime="2026-10-31 19:55:35.198415622 +0000 UTC m=+2165.771649751"
Oct 31 19:55:35.395633 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:35.395595    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk"
Oct 31 19:55:35.397604 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:35.397578    4645 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body=
Oct 31 19:55:35.397743 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:35.397631    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk" podUID="22d8af28-f4e5-47ea-a25a-9076b9afbc11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused"
Oct 31 19:55:36.023502 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:36.023466    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-754e0a/test-model-754e0a-kserve-767c88f75f-8xwns"]
Oct 31 19:55:36.137358 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:36.131214    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:55:36.305410 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:55:36.303851    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 19:55:36.396614 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:36.396576    4645 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body=
Oct 31 19:55:36.396839 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:36.396641    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk" podUID="22d8af28-f4e5-47ea-a25a-9076b9afbc11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused"
Oct 31 19:55:37.208619 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:37.208584    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:55:37.395814 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:37.395785    4645 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body=
Oct 31 19:55:37.396217 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:37.395832    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk" podUID="22d8af28-f4e5-47ea-a25a-9076b9afbc11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused"
Oct 31 19:55:37.799799 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:37.799750    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:55:38.396557 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:38.396522    4645 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body=
Oct 31 19:55:38.396982 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:38.396590    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk" podUID="22d8af28-f4e5-47ea-a25a-9076b9afbc11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused"
Oct 31 19:55:39.395762 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:39.395722    4645 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body=
Oct 31 19:55:39.395979 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:39.395776    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk" podUID="22d8af28-f4e5-47ea-a25a-9076b9afbc11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused"
Oct 31 19:55:40.035671 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:40.035627    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:55:40.396735 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:40.396649    4645 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body=
Oct 31 19:55:40.396735 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:40.396711    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk" podUID="22d8af28-f4e5-47ea-a25a-9076b9afbc11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused"
Oct 31 19:55:41.395810 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:41.395757    4645 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused" start-of-body=
Oct 31 19:55:41.396219 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:41.395822    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk" podUID="22d8af28-f4e5-47ea-a25a-9076b9afbc11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.220:15021/healthz/ready\": dial tcp 10.133.0.220:15021: connect: connection refused"
Oct 31 19:55:42.400609 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:42.400575    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk"
Oct 31 19:55:42.400981 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:42.400939    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk"
Oct 31 19:55:42.401520 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:42.401503    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk"
Oct 31 19:55:45.540928 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:45.540890    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:55:48.499279 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:48.499238    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-7c892a/test-model-7c892a-kserve-57875c9b5f-r4j8f"]
Oct 31 19:55:48.867075 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:48.866975    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:55:48.987823 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:48.987783    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:55:49.106883 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:49.106852    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz"]
Oct 31 19:55:49.111456 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:49.111437    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz"
Oct 31 19:55:49.134940 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:49.134911    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz"]
Oct 31 19:55:49.180950 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:49.180922    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l"]
Oct 31 19:55:49.185365 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:49.185344    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l"
Oct 31 19:55:49.197158 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:49.197132    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l"]
Oct 31 19:55:49.285372 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:49.285312    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cc42e0b-07f3-49f4-bb0b-8324751ea2e8-istio-podinfo\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l\" (UID: \"5cc42e0b-07f3-49f4-bb0b-8324751ea2e8\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l"
Oct 31 19:55:49.285539 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:49.285439    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0404b7e2-fd55-484a-bbec-3dd056db3e6b-istio-token\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz\" (UID: \"0404b7e2-fd55-484a-bbec-3dd056db3e6b\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz"
Oct 31 19:55:49.285539 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:49.285471    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0404b7e2-fd55-484a-bbec-3dd056db3e6b-credential-socket\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz\" (UID: \"0404b7e2-fd55-484a-bbec-3dd056db3e6b\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz"
Oct 31 19:55:49.285664 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:49.285544    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0404b7e2-fd55-484a-bbec-3dd056db3e6b-workload-socket\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz\" (UID: \"0404b7e2-fd55-484a-bbec-3dd056db3e6b\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz"
Oct 31 19:55:49.285664 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:49.285592    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cc42e0b-07f3-49f4-bb0b-8324751ea2e8-istio-envoy\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l\" (UID: \"5cc42e0b-07f3-49f4-bb0b-8324751ea2e8\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l"
Oct 31 19:55:49.285664 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:49.285646    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0404b7e2-fd55-484a-bbec-3dd056db3e6b-istio-envoy\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz\" (UID: \"0404b7e2-fd55-484a-bbec-3dd056db3e6b\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz"
Oct 31 19:55:49.285775 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:49.285682    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0404b7e2-fd55-484a-bbec-3dd056db3e6b-workload-certs\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz\" (UID: \"0404b7e2-fd55-484a-bbec-3dd056db3e6b\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz"
Oct 31 19:55:49.285775 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:49.285709    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cc42e0b-07f3-49f4-bb0b-8324751ea2e8-credential-socket\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l\" (UID: \"5cc42e0b-07f3-49f4-bb0b-8324751ea2e8\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l"
Oct 31 19:55:49.285775 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:49.285732    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0404b7e2-fd55-484a-bbec-3dd056db3e6b-istio-podinfo\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz\" (UID: \"0404b7e2-fd55-484a-bbec-3dd056db3e6b\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz"
Oct 31 19:55:49.285894 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:49.285793    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0404b7e2-fd55-484a-bbec-3dd056db3e6b-istiod-ca-cert\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz\" (UID: \"0404b7e2-fd55-484a-bbec-3dd056db3e6b\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz"
Oct 31 19:55:49.285894 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:49.285867    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cc42e0b-07f3-49f4-bb0b-8324751ea2e8-istiod-ca-cert\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l\" (UID: \"5cc42e0b-07f3-49f4-bb0b-8324751ea2e8\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l"
Oct 31 19:55:49.285994 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:49.285893    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0404b7e2-fd55-484a-bbec-3dd056db3e6b-istio-data\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz\" (UID: \"0404b7e2-fd55-484a-bbec-3dd056db3e6b\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz"
Oct 31 19:55:49.285994 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:49.285920    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cc42e0b-07f3-49f4-bb0b-8324751ea2e8-workload-certs\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l\" (UID: \"5cc42e0b-07f3-49f4-bb0b-8324751ea2e8\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l"
Oct 31 19:55:49.286076 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:49.286038    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cc42e0b-07f3-49f4-bb0b-8324751ea2e8-workload-socket\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l\" (UID: \"5cc42e0b-07f3-49f4-bb0b-8324751ea2e8\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l"
Oct 31 19:55:49.286126 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:49.286090    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s4qf\" (UniqueName: \"kubernetes.io/projected/5cc42e0b-07f3-49f4-bb0b-8324751ea2e8-kube-api-access-9s4qf\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l\" (UID: \"5cc42e0b-07f3-49f4-bb0b-8324751ea2e8\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l"
Oct 31 19:55:49.286216 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:49.286203    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cc42e0b-07f3-49f4-bb0b-8324751ea2e8-istio-data\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l\" (UID: \"5cc42e0b-07f3-49f4-bb0b-8324751ea2e8\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l"
Oct 31 19:55:49.286260 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:49.286238    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxgcw\" (UniqueName: \"kubernetes.io/projected/0404b7e2-fd55-484a-bbec-3dd056db3e6b-kube-api-access-sxgcw\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz\" (UID: \"0404b7e2-fd55-484a-bbec-3dd056db3e6b\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz"
Oct 31 19:55:49.286305 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:49.286260    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cc42e0b-07f3-49f4-bb0b-8324751ea2e8-istio-token\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l\" (UID: \"5cc42e0b-07f3-49f4-bb0b-8324751ea2e8\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l"
Oct 31 19:55:49.387312 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:49.387230    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cc42e0b-07f3-49f4-bb0b-8324751ea2e8-istio-data\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l\" (UID: \"5cc42e0b-07f3-49f4-bb0b-8324751ea2e8\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l"
Oct 31 19:55:49.387312 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:49.387277    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxgcw\" (UniqueName: \"kubernetes.io/projected/0404b7e2-fd55-484a-bbec-3dd056db3e6b-kube-api-access-sxgcw\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz\" (UID: \"0404b7e2-fd55-484a-bbec-3dd056db3e6b\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz"
Oct 31 19:55:49.387312 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:49.387299    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cc42e0b-07f3-49f4-bb0b-8324751ea2e8-istio-token\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l\" (UID: \"5cc42e0b-07f3-49f4-bb0b-8324751ea2e8\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l"
Oct 31 19:55:49.387639 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:49.387370    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cc42e0b-07f3-49f4-bb0b-8324751ea2e8-istio-podinfo\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l\" (UID: \"5cc42e0b-07f3-49f4-bb0b-8324751ea2e8\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l"
Oct 31 19:55:49.387639 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:49.387421    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0404b7e2-fd55-484a-bbec-3dd056db3e6b-istio-token\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz\" (UID: \"0404b7e2-fd55-484a-bbec-3dd056db3e6b\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz"
Oct 31 19:55:49.387639 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:49.387601    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0404b7e2-fd55-484a-bbec-3dd056db3e6b-credential-socket\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz\" (UID: \"0404b7e2-fd55-484a-bbec-3dd056db3e6b\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz"
Oct 31 19:55:49.387797 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:49.387646    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cc42e0b-07f3-49f4-bb0b-8324751ea2e8-istio-data\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l\" (UID: \"5cc42e0b-07f3-49f4-bb0b-8324751ea2e8\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l"
Oct 31 19:55:49.387797 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:49.387651    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0404b7e2-fd55-484a-bbec-3dd056db3e6b-workload-socket\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz\" (UID: \"0404b7e2-fd55-484a-bbec-3dd056db3e6b\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz"
Oct 31 19:55:49.387797 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:49.387706    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cc42e0b-07f3-49f4-bb0b-8324751ea2e8-istio-envoy\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l\" (UID: \"5cc42e0b-07f3-49f4-bb0b-8324751ea2e8\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l"
Oct 31 19:55:49.387797 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:49.387724    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0404b7e2-fd55-484a-bbec-3dd056db3e6b-istio-envoy\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz\" (UID: \"0404b7e2-fd55-484a-bbec-3dd056db3e6b\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz"
Oct 31 19:55:49.387797 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:49.387760    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0404b7e2-fd55-484a-bbec-3dd056db3e6b-workload-certs\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz\" (UID: \"0404b7e2-fd55-484a-bbec-3dd056db3e6b\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz"
Oct 31 19:55:49.388064 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:49.387815    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cc42e0b-07f3-49f4-bb0b-8324751ea2e8-credential-socket\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l\" (UID: \"5cc42e0b-07f3-49f4-bb0b-8324751ea2e8\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l"
Oct 31 19:55:49.388064 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:49.387851    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0404b7e2-fd55-484a-bbec-3dd056db3e6b-istio-podinfo\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz\" (UID: \"0404b7e2-fd55-484a-bbec-3dd056db3e6b\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz"
Oct 31 19:55:49.388064 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:49.387917    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0404b7e2-fd55-484a-bbec-3dd056db3e6b-istiod-ca-cert\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz\" (UID: \"0404b7e2-fd55-484a-bbec-3dd056db3e6b\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz"
Oct 31 19:55:49.388064 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:49.388016    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cc42e0b-07f3-49f4-bb0b-8324751ea2e8-istiod-ca-cert\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l\" (UID: \"5cc42e0b-07f3-49f4-bb0b-8324751ea2e8\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l"
Oct 31 19:55:49.388280 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:49.388062    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0404b7e2-fd55-484a-bbec-3dd056db3e6b-istio-data\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz\" (UID: \"0404b7e2-fd55-484a-bbec-3dd056db3e6b\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz"
Oct 31 19:55:49.388280 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:49.388091    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0404b7e2-fd55-484a-bbec-3dd056db3e6b-credential-socket\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz\" (UID: \"0404b7e2-fd55-484a-bbec-3dd056db3e6b\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz"
Oct 31 19:55:49.388280 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:49.388106    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cc42e0b-07f3-49f4-bb0b-8324751ea2e8-workload-certs\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l\" (UID: \"5cc42e0b-07f3-49f4-bb0b-8324751ea2e8\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l"
Oct 31 19:55:49.388280 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:49.388229    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cc42e0b-07f3-49f4-bb0b-8324751ea2e8-workload-socket\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l\" (UID: \"5cc42e0b-07f3-49f4-bb0b-8324751ea2e8\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l"
Oct 31 19:55:49.388520 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:49.388321    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s4qf\" (UniqueName: \"kubernetes.io/projected/5cc42e0b-07f3-49f4-bb0b-8324751ea2e8-kube-api-access-9s4qf\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l\" (UID: \"5cc42e0b-07f3-49f4-bb0b-8324751ea2e8\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l"
Oct 31 19:55:49.388520 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:49.388412    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0404b7e2-fd55-484a-bbec-3dd056db3e6b-workload-certs\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz\" (UID: \"0404b7e2-fd55-484a-bbec-3dd056db3e6b\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz"
Oct 31 19:55:49.388520 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:49.388094    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0404b7e2-fd55-484a-bbec-3dd056db3e6b-workload-socket\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz\" (UID: \"0404b7e2-fd55-484a-bbec-3dd056db3e6b\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz"
Oct 31 19:55:49.388831 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:49.388810    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cc42e0b-07f3-49f4-bb0b-8324751ea2e8-workload-certs\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l\" (UID: \"5cc42e0b-07f3-49f4-bb0b-8324751ea2e8\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l"
Oct 31 19:55:49.388935 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:49.388912    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0404b7e2-fd55-484a-bbec-3dd056db3e6b-istiod-ca-cert\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz\" (UID: \"0404b7e2-fd55-484a-bbec-3dd056db3e6b\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz"
Oct 31 19:55:49.389062 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:49.389039    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cc42e0b-07f3-49f4-bb0b-8324751ea2e8-workload-socket\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l\" (UID: \"5cc42e0b-07f3-49f4-bb0b-8324751ea2e8\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l"
Oct 31 19:55:49.389123 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:49.389069    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0404b7e2-fd55-484a-bbec-3dd056db3e6b-istio-data\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz\" (UID: \"0404b7e2-fd55-484a-bbec-3dd056db3e6b\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz"
Oct 31 19:55:49.389201 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:49.389179    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cc42e0b-07f3-49f4-bb0b-8324751ea2e8-credential-socket\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l\" (UID: \"5cc42e0b-07f3-49f4-bb0b-8324751ea2e8\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l"
Oct 31 19:55:49.389755 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:49.389730    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cc42e0b-07f3-49f4-bb0b-8324751ea2e8-istiod-ca-cert\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l\" (UID: \"5cc42e0b-07f3-49f4-bb0b-8324751ea2e8\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l"
Oct 31 19:55:49.390227 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:49.390198    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cc42e0b-07f3-49f4-bb0b-8324751ea2e8-istio-envoy\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l\" (UID: \"5cc42e0b-07f3-49f4-bb0b-8324751ea2e8\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l"
Oct 31 19:55:49.390680 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:49.390660    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cc42e0b-07f3-49f4-bb0b-8324751ea2e8-istio-podinfo\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l\" (UID: \"5cc42e0b-07f3-49f4-bb0b-8324751ea2e8\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l"
Oct 31 19:55:49.390854 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:49.390831    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0404b7e2-fd55-484a-bbec-3dd056db3e6b-istio-podinfo\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz\" (UID: \"0404b7e2-fd55-484a-bbec-3dd056db3e6b\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz"
Oct 31 19:55:49.391092 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:49.391071    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0404b7e2-fd55-484a-bbec-3dd056db3e6b-istio-envoy\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz\" (UID: \"0404b7e2-fd55-484a-bbec-3dd056db3e6b\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz"
Oct 31 19:55:49.400824 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:49.400804    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cc42e0b-07f3-49f4-bb0b-8324751ea2e8-istio-token\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l\" (UID: \"5cc42e0b-07f3-49f4-bb0b-8324751ea2e8\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l"
Oct 31 19:55:49.403985 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:49.403963    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0404b7e2-fd55-484a-bbec-3dd056db3e6b-istio-token\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz\" (UID: \"0404b7e2-fd55-484a-bbec-3dd056db3e6b\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz"
Oct 31 19:55:49.404135 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:49.404116    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s4qf\" (UniqueName: \"kubernetes.io/projected/5cc42e0b-07f3-49f4-bb0b-8324751ea2e8-kube-api-access-9s4qf\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l\" (UID: \"5cc42e0b-07f3-49f4-bb0b-8324751ea2e8\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l"
Oct 31 19:55:49.404365 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:49.404323    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxgcw\" (UniqueName: \"kubernetes.io/projected/0404b7e2-fd55-484a-bbec-3dd056db3e6b-kube-api-access-sxgcw\") pod \"e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz\" (UID: \"0404b7e2-fd55-484a-bbec-3dd056db3e6b\") " pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz"
Oct 31 19:55:49.426487 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:49.426464    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz"
Oct 31 19:55:49.509379 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:49.508817    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l"
Oct 31 19:55:49.717531 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:55:49.717488    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0404b7e2_fd55_484a_bbec_3dd056db3e6b.slice/crio-f79ddf9dcee6156078a69efc81842aeed1dcf53b9b86dc98576e67a20138ab54 WatchSource:0}: Error finding container f79ddf9dcee6156078a69efc81842aeed1dcf53b9b86dc98576e67a20138ab54: Status 404 returned error can't find the container with id f79ddf9dcee6156078a69efc81842aeed1dcf53b9b86dc98576e67a20138ab54
Oct 31 19:55:49.717638 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:49.717558    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz"]
Oct 31 19:55:49.721130 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:49.721086    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:55:49.721241 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:49.721173    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:55:49.721311 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:49.721251    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:55:49.827491 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:49.827462    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l"]
Oct 31 19:55:49.828440 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:55:49.828409    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cc42e0b_07f3_49f4_bb0b_8324751ea2e8.slice/crio-6bb672ef664c1120f0014e25204e8d28c146cacdd873deb091174fb52a7979b4 WatchSource:0}: Error finding container 6bb672ef664c1120f0014e25204e8d28c146cacdd873deb091174fb52a7979b4: Status 404 returned error can't find the container with id 6bb672ef664c1120f0014e25204e8d28c146cacdd873deb091174fb52a7979b4
Oct 31 19:55:49.832506 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:49.832460    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:55:49.832611 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:49.832554    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:55:49.832611 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:49.832596    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:55:50.332061 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:50.332020    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l" event={"ID":"5cc42e0b-07f3-49f4-bb0b-8324751ea2e8","Type":"ContainerStarted","Data":"940af39b2e9946a3254be3fdd68594126537b9dc89696440c2a32857414513ef"}
Oct 31 19:55:50.332241 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:50.332081    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l" event={"ID":"5cc42e0b-07f3-49f4-bb0b-8324751ea2e8","Type":"ContainerStarted","Data":"6bb672ef664c1120f0014e25204e8d28c146cacdd873deb091174fb52a7979b4"}
Oct 31 19:55:50.341354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:50.341161    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz" event={"ID":"0404b7e2-fd55-484a-bbec-3dd056db3e6b","Type":"ContainerStarted","Data":"aebbcc9192682a9b18b6a94a4f8f5ec8f3f244bdbb2bfb2a5db987db8094deb5"}
Oct 31 19:55:50.341354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:50.341269    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz" event={"ID":"0404b7e2-fd55-484a-bbec-3dd056db3e6b","Type":"ContainerStarted","Data":"f79ddf9dcee6156078a69efc81842aeed1dcf53b9b86dc98576e67a20138ab54"}
Oct 31 19:55:50.364227 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:50.364176    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l" podStartSLOduration=1.364155237 podStartE2EDuration="1.364155237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:55:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:55:50.357154196 +0000 UTC m=+2180.930388325" watchObservedRunningTime="2026-10-31 19:55:50.364155237 +0000 UTC m=+2180.937389366"
Oct 31 19:55:50.384293 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:50.384249    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz" podStartSLOduration=1.384236745 podStartE2EDuration="1.384236745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:55:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:55:50.380476017 +0000 UTC m=+2180.953710146" watchObservedRunningTime="2026-10-31 19:55:50.384236745 +0000 UTC m=+2180.957470873"
Oct 31 19:55:50.427053 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:50.427029    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz"
Oct 31 19:55:50.428239 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:50.428218    4645 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body=
Oct 31 19:55:50.428300 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:50.428261    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz" podUID="0404b7e2-fd55-484a-bbec-3dd056db3e6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused"
Oct 31 19:55:50.509779 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:50.509756    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l"
Oct 31 19:55:50.511276 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:50.511252    4645 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Oct 31 19:55:50.511404 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:50.511310    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l" podUID="5cc42e0b-07f3-49f4-bb0b-8324751ea2e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Oct 31 19:55:51.044682 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:51.044628    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-754e0a/test-model-754e0a-kserve-767c88f75f-8xwns" podUID="55751f79-d0e3-474e-9623-5e37f8d1f396" containerName="main" containerID="cri-o://82940fe1be15510f34d139309ad98a64aeb93293ab80eaf0b63709b8f1c7b6be" gracePeriod=45
Oct 31 19:55:51.428492 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:51.427816    4645 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body=
Oct 31 19:55:51.428492 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:51.427884    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz" podUID="0404b7e2-fd55-484a-bbec-3dd056db3e6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused"
Oct 31 19:55:51.433929 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:51.433895    4645 generic.go:356] "Generic (PLEG): container finished" podID="55751f79-d0e3-474e-9623-5e37f8d1f396" containerID="82940fe1be15510f34d139309ad98a64aeb93293ab80eaf0b63709b8f1c7b6be" exitCode=0
Oct 31 19:55:51.435187 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:51.435167    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-754e0a/test-model-754e0a-kserve-767c88f75f-8xwns" event={"ID":"55751f79-d0e3-474e-9623-5e37f8d1f396","Type":"ContainerDied","Data":"82940fe1be15510f34d139309ad98a64aeb93293ab80eaf0b63709b8f1c7b6be"}
Oct 31 19:55:51.511385 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:51.510837    4645 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Oct 31 19:55:51.511385 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:51.510901    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l" podUID="5cc42e0b-07f3-49f4-bb0b-8324751ea2e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Oct 31 19:55:51.548072 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:51.548046    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-754e0a/test-model-754e0a-kserve-767c88f75f-8xwns"
Oct 31 19:55:51.734901 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:51.734866    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55751f79-d0e3-474e-9623-5e37f8d1f396-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55751f79-d0e3-474e-9623-5e37f8d1f396-model-cache\") pod \"55751f79-d0e3-474e-9623-5e37f8d1f396\" (UID: \"55751f79-d0e3-474e-9623-5e37f8d1f396\") "
Oct 31 19:55:51.735124 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:51.734914    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55751f79-d0e3-474e-9623-5e37f8d1f396-tls-certs\" (UniqueName: \"kubernetes.io/secret/55751f79-d0e3-474e-9623-5e37f8d1f396-tls-certs\") pod \"55751f79-d0e3-474e-9623-5e37f8d1f396\" (UID: \"55751f79-d0e3-474e-9623-5e37f8d1f396\") "
Oct 31 19:55:51.735124 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:51.734968    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55751f79-d0e3-474e-9623-5e37f8d1f396-home\" (UniqueName: \"kubernetes.io/empty-dir/55751f79-d0e3-474e-9623-5e37f8d1f396-home\") pod \"55751f79-d0e3-474e-9623-5e37f8d1f396\" (UID: \"55751f79-d0e3-474e-9623-5e37f8d1f396\") "
Oct 31 19:55:51.735250 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:51.735181    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55751f79-d0e3-474e-9623-5e37f8d1f396-model-cache" pod "55751f79-d0e3-474e-9623-5e37f8d1f396" (UID: "55751f79-d0e3-474e-9623-5e37f8d1f396"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:55:51.735377 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:51.735323    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55751f79-d0e3-474e-9623-5e37f8d1f396-dshm\" (UniqueName: \"kubernetes.io/empty-dir/55751f79-d0e3-474e-9623-5e37f8d1f396-dshm\") pod \"55751f79-d0e3-474e-9623-5e37f8d1f396\" (UID: \"55751f79-d0e3-474e-9623-5e37f8d1f396\") "
Oct 31 19:55:51.735461 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:51.735375    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55751f79-d0e3-474e-9623-5e37f8d1f396-home" pod "55751f79-d0e3-474e-9623-5e37f8d1f396" (UID: "55751f79-d0e3-474e-9623-5e37f8d1f396"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:55:51.735461 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:51.735403    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55751f79-d0e3-474e-9623-5e37f8d1f396-kube-api-access-hkzhb\" (UniqueName: \"kubernetes.io/projected/55751f79-d0e3-474e-9623-5e37f8d1f396-kube-api-access-hkzhb\") pod \"55751f79-d0e3-474e-9623-5e37f8d1f396\" (UID: \"55751f79-d0e3-474e-9623-5e37f8d1f396\") "
Oct 31 19:55:51.735577 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:51.735505    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55751f79-d0e3-474e-9623-5e37f8d1f396-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55751f79-d0e3-474e-9623-5e37f8d1f396-tmp-dir\") pod \"55751f79-d0e3-474e-9623-5e37f8d1f396\" (UID: \"55751f79-d0e3-474e-9623-5e37f8d1f396\") "
Oct 31 19:55:51.735808 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:51.735781    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55751f79-d0e3-474e-9623-5e37f8d1f396-tmp-dir" pod "55751f79-d0e3-474e-9623-5e37f8d1f396" (UID: "55751f79-d0e3-474e-9623-5e37f8d1f396"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:55:51.736210 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:51.736192    4645 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55751f79-d0e3-474e-9623-5e37f8d1f396-tmp-dir\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:55:51.736263 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:51.736212    4645 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55751f79-d0e3-474e-9623-5e37f8d1f396-model-cache\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:55:51.736263 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:51.736221    4645 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55751f79-d0e3-474e-9623-5e37f8d1f396-home\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:55:51.737871 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:51.737848    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55751f79-d0e3-474e-9623-5e37f8d1f396-tls-certs" pod "55751f79-d0e3-474e-9623-5e37f8d1f396" (UID: "55751f79-d0e3-474e-9623-5e37f8d1f396"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:55:51.737962 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:51.737850    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55751f79-d0e3-474e-9623-5e37f8d1f396-kube-api-access-hkzhb" pod "55751f79-d0e3-474e-9623-5e37f8d1f396" (UID: "55751f79-d0e3-474e-9623-5e37f8d1f396"). InnerVolumeSpecName "kube-api-access-hkzhb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:55:51.738091 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:51.738067    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55751f79-d0e3-474e-9623-5e37f8d1f396-dshm" pod "55751f79-d0e3-474e-9623-5e37f8d1f396" (UID: "55751f79-d0e3-474e-9623-5e37f8d1f396"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:55:51.867300 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:51.867257    4645 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55751f79-d0e3-474e-9623-5e37f8d1f396-dshm\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:55:51.879471 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:51.879436    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkzhb\" (UniqueName: \"kubernetes.io/projected/55751f79-d0e3-474e-9623-5e37f8d1f396-kube-api-access-hkzhb\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:55:51.879633 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:51.879616    4645 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55751f79-d0e3-474e-9623-5e37f8d1f396-tls-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:55:52.426801 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:52.426766    4645 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body=
Oct 31 19:55:52.427006 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:52.426817    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz" podUID="0404b7e2-fd55-484a-bbec-3dd056db3e6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused"
Oct 31 19:55:52.454912 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:52.454881    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-754e0a/test-model-754e0a-kserve-767c88f75f-8xwns"
Oct 31 19:55:52.455054 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:52.454929    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-754e0a/test-model-754e0a-kserve-767c88f75f-8xwns" event={"ID":"55751f79-d0e3-474e-9623-5e37f8d1f396","Type":"ContainerDied","Data":"0fea81aa49f4f5b108823c3136a47aa2843f9106a008a275935c37743bcc8469"}
Oct 31 19:55:52.455054 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:52.454972    4645 scope.go:122] "RemoveContainer" containerID="82940fe1be15510f34d139309ad98a64aeb93293ab80eaf0b63709b8f1c7b6be"
Oct 31 19:55:52.509887 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:52.509850    4645 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Oct 31 19:55:52.510184 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:52.510154    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l" podUID="5cc42e0b-07f3-49f4-bb0b-8324751ea2e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Oct 31 19:55:52.543404 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:52.543375    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-754e0a/test-model-754e0a-kserve-767c88f75f-8xwns"]
Oct 31 19:55:52.551408 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:52.551385    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-754e0a/test-model-754e0a-kserve-767c88f75f-8xwns"]
Oct 31 19:55:52.616082 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:55:52.616046    4645 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod55751f79_d0e3_474e_9623_5e37f8d1f396.slice/crio-0fea81aa49f4f5b108823c3136a47aa2843f9106a008a275935c37743bcc8469\": RecentStats: unable to find data in memory cache]"
Oct 31 19:55:52.906597 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:52.903843    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:55:53.029189 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:53.029149    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:55:53.251384 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:53.251319    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:55:53.427051 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:53.427014    4645 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body=
Oct 31 19:55:53.427255 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:53.427070    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz" podUID="0404b7e2-fd55-484a-bbec-3dd056db3e6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused"
Oct 31 19:55:53.509745 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:53.509654    4645 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Oct 31 19:55:53.509745 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:53.509724    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l" podUID="5cc42e0b-07f3-49f4-bb0b-8324751ea2e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Oct 31 19:55:53.622854 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:53.622817    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55751f79-d0e3-474e-9623-5e37f8d1f396" path="/var/lib/kubelet/pods/55751f79-d0e3-474e-9623-5e37f8d1f396/volumes"
Oct 31 19:55:53.993352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:53.991284    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:55:54.427951 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:54.427906    4645 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body=
Oct 31 19:55:54.428173 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:54.427974    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz" podUID="0404b7e2-fd55-484a-bbec-3dd056db3e6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused"
Oct 31 19:55:54.511489 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:54.510230    4645 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Oct 31 19:55:54.511489 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:54.510301    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l" podUID="5cc42e0b-07f3-49f4-bb0b-8324751ea2e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Oct 31 19:55:55.122348 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:55.119018    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:55:55.427859 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:55.427819    4645 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body=
Oct 31 19:55:55.428039 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:55.427891    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz" podUID="0404b7e2-fd55-484a-bbec-3dd056db3e6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused"
Oct 31 19:55:55.514571 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:55.514530    4645 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Oct 31 19:55:55.514792 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:55.514605    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l" podUID="5cc42e0b-07f3-49f4-bb0b-8324751ea2e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Oct 31 19:55:56.441353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:56.440066    4645 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body=
Oct 31 19:55:56.441353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:56.440138    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz" podUID="0404b7e2-fd55-484a-bbec-3dd056db3e6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused"
Oct 31 19:55:56.513303 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:56.513266    4645 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Oct 31 19:55:56.513618 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:56.513587    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l" podUID="5cc42e0b-07f3-49f4-bb0b-8324751ea2e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Oct 31 19:55:57.260823 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:57.260770    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:55:57.387547 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:57.387474    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-ae2e60/rate-test-model-ae2e60-kserve-744475655c-xk2vp" podUID="44f86c77-4686-48fe-8cf3-cc0e688af7a4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.216:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:55:57.408754 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:57.408695    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-435728/auth-test-model-435728-kserve-76df7cb4d9-t5bn2" podUID="fa08927f-11e4-458e-beac-c96cfa4de425" containerName="main" probeResult="failure" output="Get \"https://10.133.0.207:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:55:57.431071 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:57.431022    4645 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused" start-of-body=
Oct 31 19:55:57.431391 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:57.431360    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz" podUID="0404b7e2-fd55-484a-bbec-3dd056db3e6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.221:15021/healthz/ready\": dial tcp 10.133.0.221:15021: connect: connection refused"
Oct 31 19:55:57.454897 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:57.454851    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4" podUID="6032a997-b795-4561-805c-110cf67a9d5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:55:57.495488 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:57.495409    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669" podUID="9285bee0-bf7c-4fd4-9da7-a1db021cae05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:55:57.499838 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:57.499793    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-cd9729/auth-test-model-cd9729-kserve-65f895f85c-pcprs" podUID="96edc0b2-28d0-4058-8aff-c0e179fe5f2e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.202:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:55:57.515503 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:57.514671    4645 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused" start-of-body=
Oct 31 19:55:57.515503 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:57.514721    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l" podUID="5cc42e0b-07f3-49f4-bb0b-8324751ea2e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.222:15021/healthz/ready\": dial tcp 10.133.0.222:15021: connect: connection refused"
Oct 31 19:55:57.515503 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:57.514786    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw" podUID="d504b2d3-c26d-4dcf-a7eb-efe20732ae4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:55:57.591363 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:57.585269    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n" podUID="dbb97168-db93-4757-9a1b-f9c5eb704fc5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:55:57.591363 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:57.586179    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x" podUID="a774e286-dcb8-4a2f-8541-26bf231eb08d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 19:55:57.641575 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:57.641539    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-f5c1ab/auth-test-model-f5c1ab-kserve-77f7d9ccc-4ljhr" podUID="d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.204:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:55:58.433735 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:58.433698    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz"
Oct 31 19:55:58.434163 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:58.434136    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz"
Oct 31 19:55:58.435188 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:58.435165    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz"
Oct 31 19:55:58.521583 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:58.521553    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l"
Oct 31 19:55:58.523499 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:58.523475    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l"
Oct 31 19:55:58.526381 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:58.526288    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l"
Oct 31 19:55:58.714429 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:58.704636    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:55:59.505305 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:59.505271    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:55:59.932199 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:55:59.932168    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:00.506070 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:00.506032    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:01.534750 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:01.534717    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:02.390514 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:02.390478    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:03.168100 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:03.168063    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:03.521278 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:03.521200    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-7c892a/test-model-7c892a-kserve-57875c9b5f-r4j8f" podUID="7be66b1c-b038-45f8-8e28-cb3b3cd3c526" containerName="main" containerID="cri-o://9295a364f0bea976491737426de95a27251994ea30dbe97442874ec97cb985e7" gracePeriod=45
Oct 31 19:56:03.756876 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:03.756840    4645 generic.go:356] "Generic (PLEG): container finished" podID="7be66b1c-b038-45f8-8e28-cb3b3cd3c526" containerID="9295a364f0bea976491737426de95a27251994ea30dbe97442874ec97cb985e7" exitCode=0
Oct 31 19:56:03.757046 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:03.756912    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7c892a/test-model-7c892a-kserve-57875c9b5f-r4j8f" event={"ID":"7be66b1c-b038-45f8-8e28-cb3b3cd3c526","Type":"ContainerDied","Data":"9295a364f0bea976491737426de95a27251994ea30dbe97442874ec97cb985e7"}
Oct 31 19:56:03.809478 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:03.809453    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7c892a/test-model-7c892a-kserve-57875c9b5f-r4j8f"
Oct 31 19:56:03.846221 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:03.846194    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7be66b1c-b038-45f8-8e28-cb3b3cd3c526-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7be66b1c-b038-45f8-8e28-cb3b3cd3c526-tmp-dir\") pod \"7be66b1c-b038-45f8-8e28-cb3b3cd3c526\" (UID: \"7be66b1c-b038-45f8-8e28-cb3b3cd3c526\") "
Oct 31 19:56:03.846402 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:03.846250    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7be66b1c-b038-45f8-8e28-cb3b3cd3c526-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7be66b1c-b038-45f8-8e28-cb3b3cd3c526-model-cache\") pod \"7be66b1c-b038-45f8-8e28-cb3b3cd3c526\" (UID: \"7be66b1c-b038-45f8-8e28-cb3b3cd3c526\") "
Oct 31 19:56:03.846402 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:03.846369    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7be66b1c-b038-45f8-8e28-cb3b3cd3c526-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7be66b1c-b038-45f8-8e28-cb3b3cd3c526-dshm\") pod \"7be66b1c-b038-45f8-8e28-cb3b3cd3c526\" (UID: \"7be66b1c-b038-45f8-8e28-cb3b3cd3c526\") "
Oct 31 19:56:03.846535 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:03.846419    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7be66b1c-b038-45f8-8e28-cb3b3cd3c526-home\" (UniqueName: \"kubernetes.io/empty-dir/7be66b1c-b038-45f8-8e28-cb3b3cd3c526-home\") pod \"7be66b1c-b038-45f8-8e28-cb3b3cd3c526\" (UID: \"7be66b1c-b038-45f8-8e28-cb3b3cd3c526\") "
Oct 31 19:56:03.846535 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:03.846491    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7be66b1c-b038-45f8-8e28-cb3b3cd3c526-tmp-dir" pod "7be66b1c-b038-45f8-8e28-cb3b3cd3c526" (UID: "7be66b1c-b038-45f8-8e28-cb3b3cd3c526"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:56:03.846640 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:03.846534    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7be66b1c-b038-45f8-8e28-cb3b3cd3c526-kube-api-access-cnwdz\" (UniqueName: \"kubernetes.io/projected/7be66b1c-b038-45f8-8e28-cb3b3cd3c526-kube-api-access-cnwdz\") pod \"7be66b1c-b038-45f8-8e28-cb3b3cd3c526\" (UID: \"7be66b1c-b038-45f8-8e28-cb3b3cd3c526\") "
Oct 31 19:56:03.846700 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:03.846686    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7be66b1c-b038-45f8-8e28-cb3b3cd3c526-tls-certs\" (UniqueName: \"kubernetes.io/secret/7be66b1c-b038-45f8-8e28-cb3b3cd3c526-tls-certs\") pod \"7be66b1c-b038-45f8-8e28-cb3b3cd3c526\" (UID: \"7be66b1c-b038-45f8-8e28-cb3b3cd3c526\") "
Oct 31 19:56:03.846754 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:03.846728    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7be66b1c-b038-45f8-8e28-cb3b3cd3c526-home" pod "7be66b1c-b038-45f8-8e28-cb3b3cd3c526" (UID: "7be66b1c-b038-45f8-8e28-cb3b3cd3c526"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:56:03.847225 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:03.847181    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7be66b1c-b038-45f8-8e28-cb3b3cd3c526-model-cache" pod "7be66b1c-b038-45f8-8e28-cb3b3cd3c526" (UID: "7be66b1c-b038-45f8-8e28-cb3b3cd3c526"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:56:03.847481 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:03.847455    4645 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7be66b1c-b038-45f8-8e28-cb3b3cd3c526-tmp-dir\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:03.847604 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:03.847485    4645 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7be66b1c-b038-45f8-8e28-cb3b3cd3c526-model-cache\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:03.847604 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:03.847501    4645 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7be66b1c-b038-45f8-8e28-cb3b3cd3c526-home\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:03.849651 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:03.849629    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7be66b1c-b038-45f8-8e28-cb3b3cd3c526-dshm" pod "7be66b1c-b038-45f8-8e28-cb3b3cd3c526" (UID: "7be66b1c-b038-45f8-8e28-cb3b3cd3c526"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:56:03.849877 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:03.849841    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7be66b1c-b038-45f8-8e28-cb3b3cd3c526-tls-certs" pod "7be66b1c-b038-45f8-8e28-cb3b3cd3c526" (UID: "7be66b1c-b038-45f8-8e28-cb3b3cd3c526"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:56:03.850093 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:03.850062    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7be66b1c-b038-45f8-8e28-cb3b3cd3c526-kube-api-access-cnwdz" pod "7be66b1c-b038-45f8-8e28-cb3b3cd3c526" (UID: "7be66b1c-b038-45f8-8e28-cb3b3cd3c526"). InnerVolumeSpecName "kube-api-access-cnwdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:56:03.949360 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:03.949322    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnwdz\" (UniqueName: \"kubernetes.io/projected/7be66b1c-b038-45f8-8e28-cb3b3cd3c526-kube-api-access-cnwdz\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:03.949360 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:03.949358    4645 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7be66b1c-b038-45f8-8e28-cb3b3cd3c526-tls-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:03.949523 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:03.949368    4645 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7be66b1c-b038-45f8-8e28-cb3b3cd3c526-dshm\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:04.034785 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:04.034530    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:04.765184 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:04.765154    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7c892a/test-model-7c892a-kserve-57875c9b5f-r4j8f"
Oct 31 19:56:04.765621 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:04.765189    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7c892a/test-model-7c892a-kserve-57875c9b5f-r4j8f" event={"ID":"7be66b1c-b038-45f8-8e28-cb3b3cd3c526","Type":"ContainerDied","Data":"bdf2b43aca451df0cdde4f192b9beda33588babd00c5037fca57fc602fbb777e"}
Oct 31 19:56:04.765621 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:04.765239    4645 scope.go:122] "RemoveContainer" containerID="9295a364f0bea976491737426de95a27251994ea30dbe97442874ec97cb985e7"
Oct 31 19:56:04.789933 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:04.789907    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-7c892a/test-model-7c892a-kserve-57875c9b5f-r4j8f"]
Oct 31 19:56:04.797151 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:04.797130    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-7c892a/test-model-7c892a-kserve-57875c9b5f-r4j8f"]
Oct 31 19:56:04.885340 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:04.885308    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:05.619754 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:05.619713    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7be66b1c-b038-45f8-8e28-cb3b3cd3c526" path="/var/lib/kubelet/pods/7be66b1c-b038-45f8-8e28-cb3b3cd3c526/volumes"
Oct 31 19:56:05.708842 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:05.708403    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:06.247441 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:56:06.247405    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 19:56:06.715650 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:06.715606    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:07.576975 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:07.576944    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:08.393661 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:08.393621    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:09.264138 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:09.264107    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:10.142922 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:10.142887    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:10.964463 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:10.964430    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:11.602752 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:11.602699    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:11.708267 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:11.708220    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:11.842379 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:11.842351    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:12.593215 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:12.593175    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:13.332150 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:13.332123    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:13.886480 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:13.886437    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:14.002411 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:14.002379    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:14.160792 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:14.160758    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:14.546272 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:14.546187    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:15.058877 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:15.058844    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:16.021538 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:16.021501    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:16.934373 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:16.933832    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:17.305169 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:17.305074    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:17.795708 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:17.795677    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:18.314558 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:18.314519    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:18.577631 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:18.577563    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:19.575488 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:19.575449    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:20.611405 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:20.611369    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:21.601904 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:21.601860    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:22.793846 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:22.793811    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:23.764961 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:23.764923    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:23.916742 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:23.916523    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:24.192645 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:24.192612    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679"]
Oct 31 19:56:24.192850 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:24.192823    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679" podUID="8824b6ff-8d47-4df9-a907-8306aa909993" containerName="istio-proxy" containerID="cri-o://4b7f911128454c22b30bd5bee1def6d15118dd4942154af22635dd3ad1d9caa5" gracePeriod=30
Oct 31 19:56:24.228169 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:24.228131    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:24.230553 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:24.230532    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9"]
Oct 31 19:56:24.230752 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:24.230728    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9" podUID="d734b787-3a19-4da6-8b9b-93aa92eaa6a5" containerName="istio-proxy" containerID="cri-o://999b4f4360f74f671fabd6a034749d4ae588137a56eb551083bde28a51f099f2" gracePeriod=30
Oct 31 19:56:24.235476 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:24.235451    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g"]
Oct 31 19:56:24.235624 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:24.235607    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g" podUID="f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b" containerName="istio-proxy" containerID="cri-o://8c66143ccc602e9d179ba7b3f611d5f0ef71cc1f87c4bf05a0ff1829a9637efa" gracePeriod=30
Oct 31 19:56:25.250819 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:25.250779    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:26.194636 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:26.194599    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:27.107305 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:27.107272    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk"]
Oct 31 19:56:27.107965 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:27.107948    4645 cpu_manager.go:414] "containerMap: removing container" podUID="7be66b1c-b038-45f8-8e28-cb3b3cd3c526" containerName="main"
Oct 31 19:56:27.108012 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:27.107966    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7be66b1c-b038-45f8-8e28-cb3b3cd3c526" containerName="main"
Oct 31 19:56:27.108012 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:27.107983    4645 cpu_manager.go:414] "containerMap: removing container" podUID="55751f79-d0e3-474e-9623-5e37f8d1f396" containerName="main"
Oct 31 19:56:27.108012 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:27.107988    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55751f79-d0e3-474e-9623-5e37f8d1f396" containerName="main"
Oct 31 19:56:27.108115 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:27.108108    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="55751f79-d0e3-474e-9623-5e37f8d1f396" containerName="main"
Oct 31 19:56:27.108154 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:27.108127    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="7be66b1c-b038-45f8-8e28-cb3b3cd3c526" containerName="main"
Oct 31 19:56:27.111620 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:27.111603    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk"
Oct 31 19:56:27.116647 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:27.116625    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-d9c048-openshift-default-dockercfg-fk2wl\""
Oct 31 19:56:27.127715 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:27.127690    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk"]
Oct 31 19:56:27.172868 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:27.172838    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:27.249060 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:27.249019    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd191bcb-3c5e-49ff-accd-8a706eef5bd9-workload-certs\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk\" (UID: \"bd191bcb-3c5e-49ff-accd-8a706eef5bd9\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk"
Oct 31 19:56:27.249251 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:27.249093    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd191bcb-3c5e-49ff-accd-8a706eef5bd9-istio-token\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk\" (UID: \"bd191bcb-3c5e-49ff-accd-8a706eef5bd9\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk"
Oct 31 19:56:27.249317 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:27.249290    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd191bcb-3c5e-49ff-accd-8a706eef5bd9-credential-socket\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk\" (UID: \"bd191bcb-3c5e-49ff-accd-8a706eef5bd9\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk"
Oct 31 19:56:27.249414 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:27.249393    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd191bcb-3c5e-49ff-accd-8a706eef5bd9-istio-data\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk\" (UID: \"bd191bcb-3c5e-49ff-accd-8a706eef5bd9\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk"
Oct 31 19:56:27.249507 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:27.249430    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd191bcb-3c5e-49ff-accd-8a706eef5bd9-istio-podinfo\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk\" (UID: \"bd191bcb-3c5e-49ff-accd-8a706eef5bd9\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk"
Oct 31 19:56:27.249554 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:27.249508    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgzn7\" (UniqueName: \"kubernetes.io/projected/bd191bcb-3c5e-49ff-accd-8a706eef5bd9-kube-api-access-hgzn7\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk\" (UID: \"bd191bcb-3c5e-49ff-accd-8a706eef5bd9\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk"
Oct 31 19:56:27.249600 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:27.249569    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd191bcb-3c5e-49ff-accd-8a706eef5bd9-istiod-ca-cert\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk\" (UID: \"bd191bcb-3c5e-49ff-accd-8a706eef5bd9\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk"
Oct 31 19:56:27.249638 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:27.249606    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd191bcb-3c5e-49ff-accd-8a706eef5bd9-istio-envoy\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk\" (UID: \"bd191bcb-3c5e-49ff-accd-8a706eef5bd9\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk"
Oct 31 19:56:27.249690 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:27.249645    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd191bcb-3c5e-49ff-accd-8a706eef5bd9-workload-socket\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk\" (UID: \"bd191bcb-3c5e-49ff-accd-8a706eef5bd9\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk"
Oct 31 19:56:27.351212 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:27.351180    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd191bcb-3c5e-49ff-accd-8a706eef5bd9-credential-socket\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk\" (UID: \"bd191bcb-3c5e-49ff-accd-8a706eef5bd9\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk"
Oct 31 19:56:27.351403 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:27.351235    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd191bcb-3c5e-49ff-accd-8a706eef5bd9-istio-data\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk\" (UID: \"bd191bcb-3c5e-49ff-accd-8a706eef5bd9\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk"
Oct 31 19:56:27.351403 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:27.351374    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd191bcb-3c5e-49ff-accd-8a706eef5bd9-istio-podinfo\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk\" (UID: \"bd191bcb-3c5e-49ff-accd-8a706eef5bd9\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk"
Oct 31 19:56:27.351501 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:27.351468    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgzn7\" (UniqueName: \"kubernetes.io/projected/bd191bcb-3c5e-49ff-accd-8a706eef5bd9-kube-api-access-hgzn7\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk\" (UID: \"bd191bcb-3c5e-49ff-accd-8a706eef5bd9\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk"
Oct 31 19:56:27.351562 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:27.351517    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd191bcb-3c5e-49ff-accd-8a706eef5bd9-istiod-ca-cert\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk\" (UID: \"bd191bcb-3c5e-49ff-accd-8a706eef5bd9\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk"
Oct 31 19:56:27.351625 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:27.351565    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd191bcb-3c5e-49ff-accd-8a706eef5bd9-istio-envoy\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk\" (UID: \"bd191bcb-3c5e-49ff-accd-8a706eef5bd9\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk"
Oct 31 19:56:27.351625 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:27.351580    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd191bcb-3c5e-49ff-accd-8a706eef5bd9-credential-socket\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk\" (UID: \"bd191bcb-3c5e-49ff-accd-8a706eef5bd9\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk"
Oct 31 19:56:27.351727 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:27.351617    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd191bcb-3c5e-49ff-accd-8a706eef5bd9-istio-data\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk\" (UID: \"bd191bcb-3c5e-49ff-accd-8a706eef5bd9\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk"
Oct 31 19:56:27.351727 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:27.351641    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd191bcb-3c5e-49ff-accd-8a706eef5bd9-workload-socket\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk\" (UID: \"bd191bcb-3c5e-49ff-accd-8a706eef5bd9\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk"
Oct 31 19:56:27.351838 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:27.351734    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd191bcb-3c5e-49ff-accd-8a706eef5bd9-workload-certs\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk\" (UID: \"bd191bcb-3c5e-49ff-accd-8a706eef5bd9\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk"
Oct 31 19:56:27.351838 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:27.351780    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd191bcb-3c5e-49ff-accd-8a706eef5bd9-istio-token\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk\" (UID: \"bd191bcb-3c5e-49ff-accd-8a706eef5bd9\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk"
Oct 31 19:56:27.351987 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:27.351963    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd191bcb-3c5e-49ff-accd-8a706eef5bd9-workload-socket\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk\" (UID: \"bd191bcb-3c5e-49ff-accd-8a706eef5bd9\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk"
Oct 31 19:56:27.352132 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:27.352107    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd191bcb-3c5e-49ff-accd-8a706eef5bd9-workload-certs\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk\" (UID: \"bd191bcb-3c5e-49ff-accd-8a706eef5bd9\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk"
Oct 31 19:56:27.352242 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:27.352224    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd191bcb-3c5e-49ff-accd-8a706eef5bd9-istiod-ca-cert\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk\" (UID: \"bd191bcb-3c5e-49ff-accd-8a706eef5bd9\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk"
Oct 31 19:56:27.353957 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:27.353928    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd191bcb-3c5e-49ff-accd-8a706eef5bd9-istio-envoy\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk\" (UID: \"bd191bcb-3c5e-49ff-accd-8a706eef5bd9\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk"
Oct 31 19:56:27.354173 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:27.354153    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd191bcb-3c5e-49ff-accd-8a706eef5bd9-istio-podinfo\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk\" (UID: \"bd191bcb-3c5e-49ff-accd-8a706eef5bd9\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk"
Oct 31 19:56:27.364270 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:27.364222    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd191bcb-3c5e-49ff-accd-8a706eef5bd9-istio-token\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk\" (UID: \"bd191bcb-3c5e-49ff-accd-8a706eef5bd9\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk"
Oct 31 19:56:27.364576 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:27.364552    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgzn7\" (UniqueName: \"kubernetes.io/projected/bd191bcb-3c5e-49ff-accd-8a706eef5bd9-kube-api-access-hgzn7\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk\" (UID: \"bd191bcb-3c5e-49ff-accd-8a706eef5bd9\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk"
Oct 31 19:56:27.426149 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:27.426123    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk"
Oct 31 19:56:27.726863 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:27.726828    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk"]
Oct 31 19:56:27.728986 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:27.728951    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:56:27.729101 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:27.729018    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:56:27.729101 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:27.729048    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:56:27.964419 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:27.964375    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk" event={"ID":"bd191bcb-3c5e-49ff-accd-8a706eef5bd9","Type":"ContainerStarted","Data":"b7800bf245a0dd9aeec96bdbcdbcf45b1b443b61f10a69633b95126ed0f10543"}
Oct 31 19:56:27.964628 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:27.964454    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk" event={"ID":"bd191bcb-3c5e-49ff-accd-8a706eef5bd9","Type":"ContainerStarted","Data":"98e41f1339d8b077eb29842725638af1c4cb3ddfc261af970d8df04e52afa18a"}
Oct 31 19:56:28.003117 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:28.003025    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk" podStartSLOduration=1.003011985 podStartE2EDuration="1.003011985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:56:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:56:28.000666105 +0000 UTC m=+2218.573900254" watchObservedRunningTime="2026-10-31 19:56:28.003011985 +0000 UTC m=+2218.576246174"
Oct 31 19:56:28.144039 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:28.144000    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:28.426441 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:28.426402    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk"
Oct 31 19:56:28.428270 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:28.428240    4645 patch_prober.go:29] interesting pod/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body=
Oct 31 19:56:28.428466 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:28.428296    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk" podUID="bd191bcb-3c5e-49ff-accd-8a706eef5bd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused"
Oct 31 19:56:29.119396 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.118485    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:29.427578 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.427544    4645 patch_prober.go:29] interesting pod/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body=
Oct 31 19:56:29.427950 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.427598    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk" podUID="bd191bcb-3c5e-49ff-accd-8a706eef5bd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused"
Oct 31 19:56:29.600218 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.599885    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679"
Oct 31 19:56:29.738408 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.732706    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9"
Oct 31 19:56:29.740151 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.740115    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d734b787-3a19-4da6-8b9b-93aa92eaa6a5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d734b787-3a19-4da6-8b9b-93aa92eaa6a5-workload-socket\") pod \"d734b787-3a19-4da6-8b9b-93aa92eaa6a5\" (UID: \"d734b787-3a19-4da6-8b9b-93aa92eaa6a5\") "
Oct 31 19:56:29.740318 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.740259    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8824b6ff-8d47-4df9-a907-8306aa909993-kube-api-access-fpw72\" (UniqueName: \"kubernetes.io/projected/8824b6ff-8d47-4df9-a907-8306aa909993-kube-api-access-fpw72\") pod \"8824b6ff-8d47-4df9-a907-8306aa909993\" (UID: \"8824b6ff-8d47-4df9-a907-8306aa909993\") "
Oct 31 19:56:29.740318 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.740312    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d734b787-3a19-4da6-8b9b-93aa92eaa6a5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d734b787-3a19-4da6-8b9b-93aa92eaa6a5-istio-podinfo\") pod \"d734b787-3a19-4da6-8b9b-93aa92eaa6a5\" (UID: \"d734b787-3a19-4da6-8b9b-93aa92eaa6a5\") "
Oct 31 19:56:29.740462 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.740377    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8824b6ff-8d47-4df9-a907-8306aa909993-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8824b6ff-8d47-4df9-a907-8306aa909993-workload-socket\") pod \"8824b6ff-8d47-4df9-a907-8306aa909993\" (UID: \"8824b6ff-8d47-4df9-a907-8306aa909993\") "
Oct 31 19:56:29.740462 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.740422    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8824b6ff-8d47-4df9-a907-8306aa909993-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8824b6ff-8d47-4df9-a907-8306aa909993-istio-data\") pod \"8824b6ff-8d47-4df9-a907-8306aa909993\" (UID: \"8824b6ff-8d47-4df9-a907-8306aa909993\") "
Oct 31 19:56:29.740462 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.740451    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8824b6ff-8d47-4df9-a907-8306aa909993-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8824b6ff-8d47-4df9-a907-8306aa909993-istiod-ca-cert\") pod \"8824b6ff-8d47-4df9-a907-8306aa909993\" (UID: \"8824b6ff-8d47-4df9-a907-8306aa909993\") "
Oct 31 19:56:29.740611 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.740506    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8824b6ff-8d47-4df9-a907-8306aa909993-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8824b6ff-8d47-4df9-a907-8306aa909993-credential-socket\") pod \"8824b6ff-8d47-4df9-a907-8306aa909993\" (UID: \"8824b6ff-8d47-4df9-a907-8306aa909993\") "
Oct 31 19:56:29.740611 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.740537    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8824b6ff-8d47-4df9-a907-8306aa909993-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8824b6ff-8d47-4df9-a907-8306aa909993-istio-envoy\") pod \"8824b6ff-8d47-4df9-a907-8306aa909993\" (UID: \"8824b6ff-8d47-4df9-a907-8306aa909993\") "
Oct 31 19:56:29.740611 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.740582    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d734b787-3a19-4da6-8b9b-93aa92eaa6a5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d734b787-3a19-4da6-8b9b-93aa92eaa6a5-credential-socket\") pod \"d734b787-3a19-4da6-8b9b-93aa92eaa6a5\" (UID: \"d734b787-3a19-4da6-8b9b-93aa92eaa6a5\") "
Oct 31 19:56:29.740768 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.740623    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d734b787-3a19-4da6-8b9b-93aa92eaa6a5-kube-api-access-x5nth\" (UniqueName: \"kubernetes.io/projected/d734b787-3a19-4da6-8b9b-93aa92eaa6a5-kube-api-access-x5nth\") pod \"d734b787-3a19-4da6-8b9b-93aa92eaa6a5\" (UID: \"d734b787-3a19-4da6-8b9b-93aa92eaa6a5\") "
Oct 31 19:56:29.740768 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.740664    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8824b6ff-8d47-4df9-a907-8306aa909993-istio-token\" (UniqueName: \"kubernetes.io/projected/8824b6ff-8d47-4df9-a907-8306aa909993-istio-token\") pod \"8824b6ff-8d47-4df9-a907-8306aa909993\" (UID: \"8824b6ff-8d47-4df9-a907-8306aa909993\") "
Oct 31 19:56:29.740768 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.740749    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d734b787-3a19-4da6-8b9b-93aa92eaa6a5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d734b787-3a19-4da6-8b9b-93aa92eaa6a5-istiod-ca-cert\") pod \"d734b787-3a19-4da6-8b9b-93aa92eaa6a5\" (UID: \"d734b787-3a19-4da6-8b9b-93aa92eaa6a5\") "
Oct 31 19:56:29.740926 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.740813    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d734b787-3a19-4da6-8b9b-93aa92eaa6a5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d734b787-3a19-4da6-8b9b-93aa92eaa6a5-istio-data\") pod \"d734b787-3a19-4da6-8b9b-93aa92eaa6a5\" (UID: \"d734b787-3a19-4da6-8b9b-93aa92eaa6a5\") "
Oct 31 19:56:29.740926 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.740845    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d734b787-3a19-4da6-8b9b-93aa92eaa6a5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d734b787-3a19-4da6-8b9b-93aa92eaa6a5-workload-certs\") pod \"d734b787-3a19-4da6-8b9b-93aa92eaa6a5\" (UID: \"d734b787-3a19-4da6-8b9b-93aa92eaa6a5\") "
Oct 31 19:56:29.740926 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.740906    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d734b787-3a19-4da6-8b9b-93aa92eaa6a5-istio-token\" (UniqueName: \"kubernetes.io/projected/d734b787-3a19-4da6-8b9b-93aa92eaa6a5-istio-token\") pod \"d734b787-3a19-4da6-8b9b-93aa92eaa6a5\" (UID: \"d734b787-3a19-4da6-8b9b-93aa92eaa6a5\") "
Oct 31 19:56:29.741080 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.740936    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8824b6ff-8d47-4df9-a907-8306aa909993-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8824b6ff-8d47-4df9-a907-8306aa909993-istio-podinfo\") pod \"8824b6ff-8d47-4df9-a907-8306aa909993\" (UID: \"8824b6ff-8d47-4df9-a907-8306aa909993\") "
Oct 31 19:56:29.741080 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.740991    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d734b787-3a19-4da6-8b9b-93aa92eaa6a5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d734b787-3a19-4da6-8b9b-93aa92eaa6a5-istio-envoy\") pod \"d734b787-3a19-4da6-8b9b-93aa92eaa6a5\" (UID: \"d734b787-3a19-4da6-8b9b-93aa92eaa6a5\") "
Oct 31 19:56:29.741080 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.741053    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8824b6ff-8d47-4df9-a907-8306aa909993-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8824b6ff-8d47-4df9-a907-8306aa909993-workload-certs\") pod \"8824b6ff-8d47-4df9-a907-8306aa909993\" (UID: \"8824b6ff-8d47-4df9-a907-8306aa909993\") "
Oct 31 19:56:29.742526 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.742026    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d734b787-3a19-4da6-8b9b-93aa92eaa6a5-istio-data" pod "d734b787-3a19-4da6-8b9b-93aa92eaa6a5" (UID: "d734b787-3a19-4da6-8b9b-93aa92eaa6a5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:56:29.742526 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.742285    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8824b6ff-8d47-4df9-a907-8306aa909993-workload-certs" pod "8824b6ff-8d47-4df9-a907-8306aa909993" (UID: "8824b6ff-8d47-4df9-a907-8306aa909993"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:56:29.742526 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.742294    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d734b787-3a19-4da6-8b9b-93aa92eaa6a5-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:29.742790 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.742558    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d734b787-3a19-4da6-8b9b-93aa92eaa6a5-workload-certs" pod "d734b787-3a19-4da6-8b9b-93aa92eaa6a5" (UID: "d734b787-3a19-4da6-8b9b-93aa92eaa6a5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:56:29.742855 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.742825    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d734b787-3a19-4da6-8b9b-93aa92eaa6a5-credential-socket" pod "d734b787-3a19-4da6-8b9b-93aa92eaa6a5" (UID: "d734b787-3a19-4da6-8b9b-93aa92eaa6a5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:56:29.743050 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.743019    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d734b787-3a19-4da6-8b9b-93aa92eaa6a5-workload-socket" pod "d734b787-3a19-4da6-8b9b-93aa92eaa6a5" (UID: "d734b787-3a19-4da6-8b9b-93aa92eaa6a5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:56:29.743151 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.743129    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8824b6ff-8d47-4df9-a907-8306aa909993-istio-data" pod "8824b6ff-8d47-4df9-a907-8306aa909993" (UID: "8824b6ff-8d47-4df9-a907-8306aa909993"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:56:29.743423 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.743394    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8824b6ff-8d47-4df9-a907-8306aa909993-istiod-ca-cert" pod "8824b6ff-8d47-4df9-a907-8306aa909993" (UID: "8824b6ff-8d47-4df9-a907-8306aa909993"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:56:29.743683 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.743663    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8824b6ff-8d47-4df9-a907-8306aa909993-credential-socket" pod "8824b6ff-8d47-4df9-a907-8306aa909993" (UID: "8824b6ff-8d47-4df9-a907-8306aa909993"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:56:29.744402 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.744376    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d734b787-3a19-4da6-8b9b-93aa92eaa6a5-istiod-ca-cert" pod "d734b787-3a19-4da6-8b9b-93aa92eaa6a5" (UID: "d734b787-3a19-4da6-8b9b-93aa92eaa6a5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:56:29.746265 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.746225    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d734b787-3a19-4da6-8b9b-93aa92eaa6a5-istio-token" pod "d734b787-3a19-4da6-8b9b-93aa92eaa6a5" (UID: "d734b787-3a19-4da6-8b9b-93aa92eaa6a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:56:29.746452 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.746240    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8824b6ff-8d47-4df9-a907-8306aa909993-workload-socket" pod "8824b6ff-8d47-4df9-a907-8306aa909993" (UID: "8824b6ff-8d47-4df9-a907-8306aa909993"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:56:29.747953 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.747927    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8824b6ff-8d47-4df9-a907-8306aa909993-istio-podinfo" pod "8824b6ff-8d47-4df9-a907-8306aa909993" (UID: "8824b6ff-8d47-4df9-a907-8306aa909993"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:56:29.748536 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.748292    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d734b787-3a19-4da6-8b9b-93aa92eaa6a5-istio-podinfo" pod "d734b787-3a19-4da6-8b9b-93aa92eaa6a5" (UID: "d734b787-3a19-4da6-8b9b-93aa92eaa6a5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:56:29.750677 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.750652    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8824b6ff-8d47-4df9-a907-8306aa909993-kube-api-access-fpw72" pod "8824b6ff-8d47-4df9-a907-8306aa909993" (UID: "8824b6ff-8d47-4df9-a907-8306aa909993"). InnerVolumeSpecName "kube-api-access-fpw72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:56:29.750799 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.750695    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8824b6ff-8d47-4df9-a907-8306aa909993-istio-token" pod "8824b6ff-8d47-4df9-a907-8306aa909993" (UID: "8824b6ff-8d47-4df9-a907-8306aa909993"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:56:29.774403 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.753436    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d734b787-3a19-4da6-8b9b-93aa92eaa6a5-kube-api-access-x5nth" pod "d734b787-3a19-4da6-8b9b-93aa92eaa6a5" (UID: "d734b787-3a19-4da6-8b9b-93aa92eaa6a5"). InnerVolumeSpecName "kube-api-access-x5nth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:56:29.793370 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.793267    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8824b6ff-8d47-4df9-a907-8306aa909993-istio-envoy" pod "8824b6ff-8d47-4df9-a907-8306aa909993" (UID: "8824b6ff-8d47-4df9-a907-8306aa909993"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:56:29.793912 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.793838    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d734b787-3a19-4da6-8b9b-93aa92eaa6a5-istio-envoy" pod "d734b787-3a19-4da6-8b9b-93aa92eaa6a5" (UID: "d734b787-3a19-4da6-8b9b-93aa92eaa6a5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:56:29.802786 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.802679    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g"
Oct 31 19:56:29.843589 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.843562    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b-istio-envoy\") pod \"f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b\" (UID: \"f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b\") "
Oct 31 19:56:29.843735 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.843675    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b-istio-data\") pod \"f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b\" (UID: \"f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b\") "
Oct 31 19:56:29.843735 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.843721    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b-workload-certs\") pod \"f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b\" (UID: \"f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b\") "
Oct 31 19:56:29.843858 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.843752    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b-istiod-ca-cert\") pod \"f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b\" (UID: \"f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b\") "
Oct 31 19:56:29.843858 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.843795    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b-istio-token\" (UniqueName: \"kubernetes.io/projected/f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b-istio-token\") pod \"f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b\" (UID: \"f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b\") "
Oct 31 19:56:29.843858 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.843835    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b-credential-socket\") pod \"f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b\" (UID: \"f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b\") "
Oct 31 19:56:29.844019 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.843889    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b-kube-api-access-v8ltp\" (UniqueName: \"kubernetes.io/projected/f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b-kube-api-access-v8ltp\") pod \"f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b\" (UID: \"f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b\") "
Oct 31 19:56:29.844090 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.844074    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b-workload-socket\") pod \"f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b\" (UID: \"f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b\") "
Oct 31 19:56:29.844401 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.844259    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b-istio-podinfo\") pod \"f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b\" (UID: \"f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b\") "
Oct 31 19:56:29.844401 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.844295    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b-workload-certs" pod "f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b" (UID: "f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:56:29.844844 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.844823    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:29.844940 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.844850    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpw72\" (UniqueName: \"kubernetes.io/projected/8824b6ff-8d47-4df9-a907-8306aa909993-kube-api-access-fpw72\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:29.844940 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.844865    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d734b787-3a19-4da6-8b9b-93aa92eaa6a5-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:29.844940 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.844878    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8824b6ff-8d47-4df9-a907-8306aa909993-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:29.844940 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.844893    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8824b6ff-8d47-4df9-a907-8306aa909993-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:29.844940 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.844907    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8824b6ff-8d47-4df9-a907-8306aa909993-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:29.845213 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.844943    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8824b6ff-8d47-4df9-a907-8306aa909993-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:29.845213 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.844958    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8824b6ff-8d47-4df9-a907-8306aa909993-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:29.845213 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.844976    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d734b787-3a19-4da6-8b9b-93aa92eaa6a5-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:29.845213 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.844991    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5nth\" (UniqueName: \"kubernetes.io/projected/d734b787-3a19-4da6-8b9b-93aa92eaa6a5-kube-api-access-x5nth\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:29.845213 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.845005    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8824b6ff-8d47-4df9-a907-8306aa909993-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:29.845213 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.845019    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d734b787-3a19-4da6-8b9b-93aa92eaa6a5-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:29.845213 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.845033    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d734b787-3a19-4da6-8b9b-93aa92eaa6a5-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:29.845213 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.845046    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d734b787-3a19-4da6-8b9b-93aa92eaa6a5-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:29.845213 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.845060    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8824b6ff-8d47-4df9-a907-8306aa909993-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:29.845213 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.845088    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d734b787-3a19-4da6-8b9b-93aa92eaa6a5-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:29.845213 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.845104    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8824b6ff-8d47-4df9-a907-8306aa909993-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:29.845213 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.845117    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d734b787-3a19-4da6-8b9b-93aa92eaa6a5-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:29.846002 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.845560    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b-istiod-ca-cert" pod "f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b" (UID: "f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:56:29.846002 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.845919    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b-workload-socket" pod "f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b" (UID: "f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:56:29.846113 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.846073    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b-credential-socket" pod "f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b" (UID: "f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:56:29.846186 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.846109    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b-istio-data" pod "f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b" (UID: "f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:56:29.846584 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.846556    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b-istio-envoy" pod "f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b" (UID: "f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:56:29.847832 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.847810    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b-istio-podinfo" pod "f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b" (UID: "f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:56:29.848068 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.848044    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b-kube-api-access-v8ltp" pod "f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b" (UID: "f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b"). InnerVolumeSpecName "kube-api-access-v8ltp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:56:29.848725 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.848700    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b-istio-token" pod "f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b" (UID: "f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:56:29.937851 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.937702    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:29.947406 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.947248    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:29.947406 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.947279    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:29.947406 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.947293    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:29.947406 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.947308    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:29.947406 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.947322    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:29.947406 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.947355    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:29.947406 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.947368    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:29.947406 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:29.947383    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8ltp\" (UniqueName: \"kubernetes.io/projected/f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b-kube-api-access-v8ltp\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:30.035015 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:30.033976    4645 generic.go:356] "Generic (PLEG): container finished" podID="f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b" containerID="8c66143ccc602e9d179ba7b3f611d5f0ef71cc1f87c4bf05a0ff1829a9637efa" exitCode=0
Oct 31 19:56:30.035015 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:30.034512    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g"
Oct 31 19:56:30.035535 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:30.035502    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g" event={"ID":"f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b","Type":"ContainerDied","Data":"8c66143ccc602e9d179ba7b3f611d5f0ef71cc1f87c4bf05a0ff1829a9637efa"}
Oct 31 19:56:30.036012 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:30.035827    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g" event={"ID":"f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b","Type":"ContainerDied","Data":"873fce5daf2b2d684e79a31ff1a3b3430d94f11ccf65ab31a5dcfec22fdc93d0"}
Oct 31 19:56:30.036012 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:30.035874    4645 scope.go:122] "RemoveContainer" containerID="8c66143ccc602e9d179ba7b3f611d5f0ef71cc1f87c4bf05a0ff1829a9637efa"
Oct 31 19:56:30.047366 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:30.040909    4645 generic.go:356] "Generic (PLEG): container finished" podID="8824b6ff-8d47-4df9-a907-8306aa909993" containerID="4b7f911128454c22b30bd5bee1def6d15118dd4942154af22635dd3ad1d9caa5" exitCode=0
Oct 31 19:56:30.047366 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:30.040999    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679" event={"ID":"8824b6ff-8d47-4df9-a907-8306aa909993","Type":"ContainerDied","Data":"4b7f911128454c22b30bd5bee1def6d15118dd4942154af22635dd3ad1d9caa5"}
Oct 31 19:56:30.047366 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:30.041042    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679" event={"ID":"8824b6ff-8d47-4df9-a907-8306aa909993","Type":"ContainerDied","Data":"857fdd596bb5e97c022bd228c4aca9f7ab2f37d5fe195c59955034a44ef4dfb5"}
Oct 31 19:56:30.047366 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:30.041132    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679"
Oct 31 19:56:30.081647 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:30.080614    4645 generic.go:356] "Generic (PLEG): container finished" podID="d734b787-3a19-4da6-8b9b-93aa92eaa6a5" containerID="999b4f4360f74f671fabd6a034749d4ae588137a56eb551083bde28a51f099f2" exitCode=0
Oct 31 19:56:30.081647 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:30.081419    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9" event={"ID":"d734b787-3a19-4da6-8b9b-93aa92eaa6a5","Type":"ContainerDied","Data":"999b4f4360f74f671fabd6a034749d4ae588137a56eb551083bde28a51f099f2"}
Oct 31 19:56:30.081647 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:30.081456    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9"
Oct 31 19:56:30.081647 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:30.081473    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9" event={"ID":"d734b787-3a19-4da6-8b9b-93aa92eaa6a5","Type":"ContainerDied","Data":"fd20c7d607dbd9b3ceaf78d3911197f6dcdf293e42ae083b14225a77ca84c891"}
Oct 31 19:56:30.103215 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:30.102226    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g"]
Oct 31 19:56:30.103215 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:30.102592    4645 scope.go:122] "RemoveContainer" containerID="8c66143ccc602e9d179ba7b3f611d5f0ef71cc1f87c4bf05a0ff1829a9637efa"
Oct 31 19:56:30.103215 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:56:30.102904    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c66143ccc602e9d179ba7b3f611d5f0ef71cc1f87c4bf05a0ff1829a9637efa\": container with ID starting with 8c66143ccc602e9d179ba7b3f611d5f0ef71cc1f87c4bf05a0ff1829a9637efa not found: ID does not exist" containerID="8c66143ccc602e9d179ba7b3f611d5f0ef71cc1f87c4bf05a0ff1829a9637efa"
Oct 31 19:56:30.103215 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:30.102938    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c66143ccc602e9d179ba7b3f611d5f0ef71cc1f87c4bf05a0ff1829a9637efa"} err="failed to get container status \"8c66143ccc602e9d179ba7b3f611d5f0ef71cc1f87c4bf05a0ff1829a9637efa\": rpc error: code = NotFound desc = could not find container \"8c66143ccc602e9d179ba7b3f611d5f0ef71cc1f87c4bf05a0ff1829a9637efa\": container with ID starting with 8c66143ccc602e9d179ba7b3f611d5f0ef71cc1f87c4bf05a0ff1829a9637efa not found: ID does not exist"
Oct 31 19:56:30.103215 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:30.102963    4645 scope.go:122] "RemoveContainer" containerID="4b7f911128454c22b30bd5bee1def6d15118dd4942154af22635dd3ad1d9caa5"
Oct 31 19:56:30.107887 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:30.107864    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dxl4g"]
Oct 31 19:56:30.115027 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:30.115008    4645 scope.go:122] "RemoveContainer" containerID="4b7f911128454c22b30bd5bee1def6d15118dd4942154af22635dd3ad1d9caa5"
Oct 31 19:56:30.115575 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:56:30.115549    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b7f911128454c22b30bd5bee1def6d15118dd4942154af22635dd3ad1d9caa5\": container with ID starting with 4b7f911128454c22b30bd5bee1def6d15118dd4942154af22635dd3ad1d9caa5 not found: ID does not exist" containerID="4b7f911128454c22b30bd5bee1def6d15118dd4942154af22635dd3ad1d9caa5"
Oct 31 19:56:30.115656 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:30.115585    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b7f911128454c22b30bd5bee1def6d15118dd4942154af22635dd3ad1d9caa5"} err="failed to get container status \"4b7f911128454c22b30bd5bee1def6d15118dd4942154af22635dd3ad1d9caa5\": rpc error: code = NotFound desc = could not find container \"4b7f911128454c22b30bd5bee1def6d15118dd4942154af22635dd3ad1d9caa5\": container with ID starting with 4b7f911128454c22b30bd5bee1def6d15118dd4942154af22635dd3ad1d9caa5 not found: ID does not exist"
Oct 31 19:56:30.115656 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:30.115607    4645 scope.go:122] "RemoveContainer" containerID="999b4f4360f74f671fabd6a034749d4ae588137a56eb551083bde28a51f099f2"
Oct 31 19:56:30.115747 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:30.115716    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679"]
Oct 31 19:56:30.125145 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:30.125125    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-lv679"]
Oct 31 19:56:30.128897 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:30.128874    4645 scope.go:122] "RemoveContainer" containerID="c710846a10f377410aaa0b72fde1570797cc359b3750714bf0cd93cbd7686fcf"
Oct 31 19:56:30.142176 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:30.142128    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9"]
Oct 31 19:56:30.144038 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:30.144018    4645 scope.go:122] "RemoveContainer" containerID="999b4f4360f74f671fabd6a034749d4ae588137a56eb551083bde28a51f099f2"
Oct 31 19:56:30.144456 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:56:30.144323    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"999b4f4360f74f671fabd6a034749d4ae588137a56eb551083bde28a51f099f2\": container with ID starting with 999b4f4360f74f671fabd6a034749d4ae588137a56eb551083bde28a51f099f2 not found: ID does not exist" containerID="999b4f4360f74f671fabd6a034749d4ae588137a56eb551083bde28a51f099f2"
Oct 31 19:56:30.144456 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:30.144375    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"999b4f4360f74f671fabd6a034749d4ae588137a56eb551083bde28a51f099f2"} err="failed to get container status \"999b4f4360f74f671fabd6a034749d4ae588137a56eb551083bde28a51f099f2\": rpc error: code = NotFound desc = could not find container \"999b4f4360f74f671fabd6a034749d4ae588137a56eb551083bde28a51f099f2\": container with ID starting with 999b4f4360f74f671fabd6a034749d4ae588137a56eb551083bde28a51f099f2 not found: ID does not exist"
Oct 31 19:56:30.144456 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:30.144395    4645 scope.go:122] "RemoveContainer" containerID="c710846a10f377410aaa0b72fde1570797cc359b3750714bf0cd93cbd7686fcf"
Oct 31 19:56:30.144711 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:56:30.144693    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c710846a10f377410aaa0b72fde1570797cc359b3750714bf0cd93cbd7686fcf\": container with ID starting with c710846a10f377410aaa0b72fde1570797cc359b3750714bf0cd93cbd7686fcf not found: ID does not exist" containerID="c710846a10f377410aaa0b72fde1570797cc359b3750714bf0cd93cbd7686fcf"
Oct 31 19:56:30.144755 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:30.144717    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c710846a10f377410aaa0b72fde1570797cc359b3750714bf0cd93cbd7686fcf"} err="failed to get container status \"c710846a10f377410aaa0b72fde1570797cc359b3750714bf0cd93cbd7686fcf\": rpc error: code = NotFound desc = could not find container \"c710846a10f377410aaa0b72fde1570797cc359b3750714bf0cd93cbd7686fcf\": container with ID starting with c710846a10f377410aaa0b72fde1570797cc359b3750714bf0cd93cbd7686fcf not found: ID does not exist"
Oct 31 19:56:30.147712 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:30.147691    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-754e0a-openshift-default-7c7cdd7487-dl2c9"]
Oct 31 19:56:30.426412 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:30.426379    4645 patch_prober.go:29] interesting pod/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body=
Oct 31 19:56:30.426607 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:30.426443    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk" podUID="bd191bcb-3c5e-49ff-accd-8a706eef5bd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused"
Oct 31 19:56:30.842483 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:30.842391    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:31.182644 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:31.182613    4645 scope.go:122] "RemoveContainer" containerID="37e487f42e91062eeeb1a3a9b1fad11b450b86067e96d8912ddc748ceb865121"
Oct 31 19:56:31.193305 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:31.193262    4645 scope.go:122] "RemoveContainer" containerID="7f49c83612a8ab8d67a97adac1df12964bb1b4949eacc3260b89a87fcbffce6c"
Oct 31 19:56:31.203354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:31.203298    4645 scope.go:122] "RemoveContainer" containerID="896d2fcb2bbf70573710480e8f8493ab08d97a4e109aa4fad5bf4f55af4dbd43"
Oct 31 19:56:31.213461 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:31.213442    4645 scope.go:122] "RemoveContainer" containerID="9fa564799d0d32b7e29caae6df8363518d394ba3ccb8069669d4fcf60e2b6a06"
Oct 31 19:56:31.222601 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:31.222580    4645 scope.go:122] "RemoveContainer" containerID="5ace4d970d9936d0fba97d5352a10d2453d8caa8a9205e732b43f50b79744b3e"
Oct 31 19:56:31.231587 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:31.231568    4645 scope.go:122] "RemoveContainer" containerID="e597ec4feffc1554329c2f52f5e4185887c38ea0e679c1d77f726e6d98f66729"
Oct 31 19:56:31.240296 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:31.240279    4645 scope.go:122] "RemoveContainer" containerID="1524c1647e87702dc653abc736a896cf5f0e26cefcf35c787e4ae5828515fe4a"
Oct 31 19:56:31.248829 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:31.248811    4645 scope.go:122] "RemoveContainer" containerID="0bccce786d3a1f1d55d96a02b4ad8e53aea0f0ecc0c826b785512076d4551e1e"
Oct 31 19:56:31.426662 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:31.426635    4645 patch_prober.go:29] interesting pod/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body=
Oct 31 19:56:31.426778 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:31.426686    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk" podUID="bd191bcb-3c5e-49ff-accd-8a706eef5bd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused"
Oct 31 19:56:31.619464 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:31.619390    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8824b6ff-8d47-4df9-a907-8306aa909993" path="/var/lib/kubelet/pods/8824b6ff-8d47-4df9-a907-8306aa909993/volumes"
Oct 31 19:56:31.619912 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:31.619898    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d734b787-3a19-4da6-8b9b-93aa92eaa6a5" path="/var/lib/kubelet/pods/d734b787-3a19-4da6-8b9b-93aa92eaa6a5/volumes"
Oct 31 19:56:31.620423 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:31.620408    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b" path="/var/lib/kubelet/pods/f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b/volumes"
Oct 31 19:56:31.744150 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:31.744121    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:32.427324 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:32.427295    4645 patch_prober.go:29] interesting pod/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body=
Oct 31 19:56:32.427733 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:32.427371    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk" podUID="bd191bcb-3c5e-49ff-accd-8a706eef5bd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused"
Oct 31 19:56:32.480521 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:32.480486    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:33.185149 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:33.178346    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:33.427013 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:33.426980    4645 patch_prober.go:29] interesting pod/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body=
Oct 31 19:56:33.427170 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:33.427033    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk" podUID="bd191bcb-3c5e-49ff-accd-8a706eef5bd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused"
Oct 31 19:56:33.959674 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:33.959627    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:34.432071 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:34.432038    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk"
Oct 31 19:56:34.432537 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:34.432520    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk"
Oct 31 19:56:34.433181 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:34.433162    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk"
Oct 31 19:56:34.742001 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:34.741911    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:35.009087 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:35.009002    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7c892a-98d9d78c9-qrnvx"]
Oct 31 19:56:35.057111 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:35.057080    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:35.159825 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:35.159769    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:35.640247 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:35.640211    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:36.247850 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:56:36.247814    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 19:56:36.454587 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:36.454556    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:37.251145 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:37.251100    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:37.955671 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:37.955639    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:38.504352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:38.502614    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7c892a-6f48cdf487-v46nr"]
Oct 31 19:56:38.504352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:38.502848    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7c892a-6f48cdf487-v46nr" podUID="4d444a98-d697-4fe3-ba5c-ed7158998ab7" containerName="maas-api" containerID="cri-o://ab24419c9537a6219a4369c7bf63ee457dc3085cab2df6b5c8fa7e7b95e0c2c9" gracePeriod=30
Oct 31 19:56:38.554664 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:38.554557    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:38.796604 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:38.790074    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:38.928021 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:38.927914    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:39.039199 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:39.039171    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7c892a-6f48cdf487-v46nr"
Oct 31 19:56:39.061704 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:39.061629    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:39.132178 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:39.132132    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d444a98-d697-4fe3-ba5c-ed7158998ab7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d444a98-d697-4fe3-ba5c-ed7158998ab7-maas-api-tls\") pod \"4d444a98-d697-4fe3-ba5c-ed7158998ab7\" (UID: \"4d444a98-d697-4fe3-ba5c-ed7158998ab7\") "
Oct 31 19:56:39.132361 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:39.132274    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d444a98-d697-4fe3-ba5c-ed7158998ab7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d444a98-d697-4fe3-ba5c-ed7158998ab7-metrics-certs\") pod \"4d444a98-d697-4fe3-ba5c-ed7158998ab7\" (UID: \"4d444a98-d697-4fe3-ba5c-ed7158998ab7\") "
Oct 31 19:56:39.132361 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:39.132307    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d444a98-d697-4fe3-ba5c-ed7158998ab7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d444a98-d697-4fe3-ba5c-ed7158998ab7-openshift-service-ca\") pod \"4d444a98-d697-4fe3-ba5c-ed7158998ab7\" (UID: \"4d444a98-d697-4fe3-ba5c-ed7158998ab7\") "
Oct 31 19:56:39.132555 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:39.132533    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d444a98-d697-4fe3-ba5c-ed7158998ab7-kube-api-access-88gsc\" (UniqueName: \"kubernetes.io/projected/4d444a98-d697-4fe3-ba5c-ed7158998ab7-kube-api-access-88gsc\") pod \"4d444a98-d697-4fe3-ba5c-ed7158998ab7\" (UID: \"4d444a98-d697-4fe3-ba5c-ed7158998ab7\") "
Oct 31 19:56:39.132782 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:39.132753    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d444a98-d697-4fe3-ba5c-ed7158998ab7-openshift-service-ca" pod "4d444a98-d697-4fe3-ba5c-ed7158998ab7" (UID: "4d444a98-d697-4fe3-ba5c-ed7158998ab7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:56:39.133663 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:39.133628    4645 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d444a98-d697-4fe3-ba5c-ed7158998ab7-openshift-service-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:39.135185 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:39.135015    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d444a98-d697-4fe3-ba5c-ed7158998ab7-maas-api-tls" pod "4d444a98-d697-4fe3-ba5c-ed7158998ab7" (UID: "4d444a98-d697-4fe3-ba5c-ed7158998ab7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:56:39.135185 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:39.135059    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d444a98-d697-4fe3-ba5c-ed7158998ab7-kube-api-access-88gsc" pod "4d444a98-d697-4fe3-ba5c-ed7158998ab7" (UID: "4d444a98-d697-4fe3-ba5c-ed7158998ab7"). InnerVolumeSpecName "kube-api-access-88gsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:56:39.135185 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:39.135142    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d444a98-d697-4fe3-ba5c-ed7158998ab7-metrics-certs" pod "4d444a98-d697-4fe3-ba5c-ed7158998ab7" (UID: "4d444a98-d697-4fe3-ba5c-ed7158998ab7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:56:39.158886 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:39.158859    4645 generic.go:356] "Generic (PLEG): container finished" podID="4d444a98-d697-4fe3-ba5c-ed7158998ab7" containerID="ab24419c9537a6219a4369c7bf63ee457dc3085cab2df6b5c8fa7e7b95e0c2c9" exitCode=0
Oct 31 19:56:39.158976 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:39.158914    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7c892a-6f48cdf487-v46nr"
Oct 31 19:56:39.158976 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:39.158939    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7c892a-6f48cdf487-v46nr" event={"ID":"4d444a98-d697-4fe3-ba5c-ed7158998ab7","Type":"ContainerDied","Data":"ab24419c9537a6219a4369c7bf63ee457dc3085cab2df6b5c8fa7e7b95e0c2c9"}
Oct 31 19:56:39.159056 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:39.158998    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7c892a-6f48cdf487-v46nr" event={"ID":"4d444a98-d697-4fe3-ba5c-ed7158998ab7","Type":"ContainerDied","Data":"7dc8d8e22c0a10802100d121b9ea2df0d68aa227ee16c007f5623126eafceb11"}
Oct 31 19:56:39.159056 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:39.159028    4645 scope.go:122] "RemoveContainer" containerID="ab24419c9537a6219a4369c7bf63ee457dc3085cab2df6b5c8fa7e7b95e0c2c9"
Oct 31 19:56:39.169978 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:39.169961    4645 scope.go:122] "RemoveContainer" containerID="ab24419c9537a6219a4369c7bf63ee457dc3085cab2df6b5c8fa7e7b95e0c2c9"
Oct 31 19:56:39.170259 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:56:39.170237    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab24419c9537a6219a4369c7bf63ee457dc3085cab2df6b5c8fa7e7b95e0c2c9\": container with ID starting with ab24419c9537a6219a4369c7bf63ee457dc3085cab2df6b5c8fa7e7b95e0c2c9 not found: ID does not exist" containerID="ab24419c9537a6219a4369c7bf63ee457dc3085cab2df6b5c8fa7e7b95e0c2c9"
Oct 31 19:56:39.170390 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:39.170264    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab24419c9537a6219a4369c7bf63ee457dc3085cab2df6b5c8fa7e7b95e0c2c9"} err="failed to get container status \"ab24419c9537a6219a4369c7bf63ee457dc3085cab2df6b5c8fa7e7b95e0c2c9\": rpc error: code = NotFound desc = could not find container \"ab24419c9537a6219a4369c7bf63ee457dc3085cab2df6b5c8fa7e7b95e0c2c9\": container with ID starting with ab24419c9537a6219a4369c7bf63ee457dc3085cab2df6b5c8fa7e7b95e0c2c9 not found: ID does not exist"
Oct 31 19:56:39.220138 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:39.206935    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7c892a-6f48cdf487-v46nr"]
Oct 31 19:56:39.239837 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:39.238210    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7c892a-6f48cdf487-v46nr"]
Oct 31 19:56:39.241603 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:39.241570    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d444a98-d697-4fe3-ba5c-ed7158998ab7-maas-api-tls\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:39.241767 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:39.241604    4645 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d444a98-d697-4fe3-ba5c-ed7158998ab7-metrics-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:39.241767 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:39.241622    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88gsc\" (UniqueName: \"kubernetes.io/projected/4d444a98-d697-4fe3-ba5c-ed7158998ab7-kube-api-access-88gsc\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:39.460717 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:39.460691    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:39.620771 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:39.620744    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d444a98-d697-4fe3-ba5c-ed7158998ab7" path="/var/lib/kubelet/pods/4d444a98-d697-4fe3-ba5c-ed7158998ab7/volumes"
Oct 31 19:56:39.947020 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:39.946989    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:40.015727 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:40.015427    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7c892a-98d9d78c9-qrnvx" podUID="dad64038-a18c-4944-80c8-b3f052a98dbb" containerName="payload-pre-processing" containerID="cri-o://a3ed5340dd2ca8710d32df3a1e01e31447636ccd988cad30a18044d290cbe69f" gracePeriod=25
Oct 31 19:56:40.688314 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:40.688269    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:41.402873 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:41.402830    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:42.242344 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:42.242302    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:42.962692 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:42.962653    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:43.747883 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:43.747818    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:44.548805 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:44.548762    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:45.277535 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:45.277501    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:46.160835 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:46.160799    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:46.814558 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:46.814527    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:46.922628 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:46.922597    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:47.024514 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:47.024477    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:47.255613 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:47.255571    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9"]
Oct 31 19:56:47.255943 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:47.255891    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9" podUID="aceb3312-c698-4f69-9b44-be9bd3ef0e6b" containerName="istio-proxy" containerID="cri-o://ebb0038c362dad3b8d3f2bbbed721b338c570108d2d175d8e3474999a978b949" gracePeriod=30
Oct 31 19:56:47.300353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:47.300312    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:47.940552 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:47.940516    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:48.194218 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:48.194125    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:48.629905 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:48.629830    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs"]
Oct 31 19:56:48.630594 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:48.630568    4645 cpu_manager.go:414] "containerMap: removing container" podUID="8824b6ff-8d47-4df9-a907-8306aa909993" containerName="istio-proxy"
Oct 31 19:56:48.630594 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:48.630589    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8824b6ff-8d47-4df9-a907-8306aa909993" containerName="istio-proxy"
Oct 31 19:56:48.630688 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:48.630607    4645 cpu_manager.go:414] "containerMap: removing container" podUID="f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b" containerName="istio-proxy"
Oct 31 19:56:48.630688 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:48.630612    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b" containerName="istio-proxy"
Oct 31 19:56:48.630688 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:48.630635    4645 cpu_manager.go:414] "containerMap: removing container" podUID="d734b787-3a19-4da6-8b9b-93aa92eaa6a5" containerName="istio-proxy"
Oct 31 19:56:48.630688 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:48.630642    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d734b787-3a19-4da6-8b9b-93aa92eaa6a5" containerName="istio-proxy"
Oct 31 19:56:48.630688 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:48.630650    4645 cpu_manager.go:414] "containerMap: removing container" podUID="4d444a98-d697-4fe3-ba5c-ed7158998ab7" containerName="maas-api"
Oct 31 19:56:48.630688 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:48.630655    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d444a98-d697-4fe3-ba5c-ed7158998ab7" containerName="maas-api"
Oct 31 19:56:48.630688 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:48.630670    4645 cpu_manager.go:414] "containerMap: removing container" podUID="d734b787-3a19-4da6-8b9b-93aa92eaa6a5" containerName="istio-proxy"
Oct 31 19:56:48.630688 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:48.630675    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d734b787-3a19-4da6-8b9b-93aa92eaa6a5" containerName="istio-proxy"
Oct 31 19:56:48.630918 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:48.630783    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="8824b6ff-8d47-4df9-a907-8306aa909993" containerName="istio-proxy"
Oct 31 19:56:48.630918 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:48.630791    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="f540b229-b1ea-4fe2-b7ac-a3e3bc140a1b" containerName="istio-proxy"
Oct 31 19:56:48.630918 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:48.630801    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="d734b787-3a19-4da6-8b9b-93aa92eaa6a5" containerName="istio-proxy"
Oct 31 19:56:48.630918 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:48.630808    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d444a98-d697-4fe3-ba5c-ed7158998ab7" containerName="maas-api"
Oct 31 19:56:48.630918 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:48.630817    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="d734b787-3a19-4da6-8b9b-93aa92eaa6a5" containerName="istio-proxy"
Oct 31 19:56:48.636082 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:48.636063    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs"
Oct 31 19:56:48.639777 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:48.639754    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-1f2781-openshift-default-dockercfg-jkblx\""
Oct 31 19:56:48.661710 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:48.661685    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs"]
Oct 31 19:56:48.663511 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:48.663479    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:48.822355 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:48.794489    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48ab68ad-2abf-44c8-98dc-c3ec72d36362-istio-token\") pod \"e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs\" (UID: \"48ab68ad-2abf-44c8-98dc-c3ec72d36362\") " pod="openshift-ingress/e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs"
Oct 31 19:56:48.822781 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:48.822748    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48ab68ad-2abf-44c8-98dc-c3ec72d36362-istio-data\") pod \"e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs\" (UID: \"48ab68ad-2abf-44c8-98dc-c3ec72d36362\") " pod="openshift-ingress/e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs"
Oct 31 19:56:48.822972 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:48.822950    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48ab68ad-2abf-44c8-98dc-c3ec72d36362-credential-socket\") pod \"e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs\" (UID: \"48ab68ad-2abf-44c8-98dc-c3ec72d36362\") " pod="openshift-ingress/e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs"
Oct 31 19:56:48.823093 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:48.823078    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48ab68ad-2abf-44c8-98dc-c3ec72d36362-istio-podinfo\") pod \"e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs\" (UID: \"48ab68ad-2abf-44c8-98dc-c3ec72d36362\") " pod="openshift-ingress/e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs"
Oct 31 19:56:48.823273 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:48.823257    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48ab68ad-2abf-44c8-98dc-c3ec72d36362-istiod-ca-cert\") pod \"e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs\" (UID: \"48ab68ad-2abf-44c8-98dc-c3ec72d36362\") " pod="openshift-ingress/e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs"
Oct 31 19:56:48.823483 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:48.823463    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtkxc\" (UniqueName: \"kubernetes.io/projected/48ab68ad-2abf-44c8-98dc-c3ec72d36362-kube-api-access-xtkxc\") pod \"e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs\" (UID: \"48ab68ad-2abf-44c8-98dc-c3ec72d36362\") " pod="openshift-ingress/e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs"
Oct 31 19:56:48.823689 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:48.823670    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48ab68ad-2abf-44c8-98dc-c3ec72d36362-istio-envoy\") pod \"e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs\" (UID: \"48ab68ad-2abf-44c8-98dc-c3ec72d36362\") " pod="openshift-ingress/e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs"
Oct 31 19:56:48.823811 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:48.823795    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48ab68ad-2abf-44c8-98dc-c3ec72d36362-workload-socket\") pod \"e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs\" (UID: \"48ab68ad-2abf-44c8-98dc-c3ec72d36362\") " pod="openshift-ingress/e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs"
Oct 31 19:56:48.824058 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:48.824039    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48ab68ad-2abf-44c8-98dc-c3ec72d36362-workload-certs\") pod \"e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs\" (UID: \"48ab68ad-2abf-44c8-98dc-c3ec72d36362\") " pod="openshift-ingress/e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs"
Oct 31 19:56:48.828349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:48.817571    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:48.963195 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:48.963085    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48ab68ad-2abf-44c8-98dc-c3ec72d36362-istio-token\") pod \"e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs\" (UID: \"48ab68ad-2abf-44c8-98dc-c3ec72d36362\") " pod="openshift-ingress/e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs"
Oct 31 19:56:48.963907 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:48.963880    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48ab68ad-2abf-44c8-98dc-c3ec72d36362-istio-data\") pod \"e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs\" (UID: \"48ab68ad-2abf-44c8-98dc-c3ec72d36362\") " pod="openshift-ingress/e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs"
Oct 31 19:56:48.964064 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:48.964045    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48ab68ad-2abf-44c8-98dc-c3ec72d36362-credential-socket\") pod \"e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs\" (UID: \"48ab68ad-2abf-44c8-98dc-c3ec72d36362\") " pod="openshift-ingress/e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs"
Oct 31 19:56:48.964193 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:48.964176    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48ab68ad-2abf-44c8-98dc-c3ec72d36362-istio-podinfo\") pod \"e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs\" (UID: \"48ab68ad-2abf-44c8-98dc-c3ec72d36362\") " pod="openshift-ingress/e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs"
Oct 31 19:56:48.964318 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:48.964296    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:48.965949 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:48.964454    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48ab68ad-2abf-44c8-98dc-c3ec72d36362-istiod-ca-cert\") pod \"e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs\" (UID: \"48ab68ad-2abf-44c8-98dc-c3ec72d36362\") " pod="openshift-ingress/e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs"
Oct 31 19:56:48.966170 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:48.966148    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtkxc\" (UniqueName: \"kubernetes.io/projected/48ab68ad-2abf-44c8-98dc-c3ec72d36362-kube-api-access-xtkxc\") pod \"e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs\" (UID: \"48ab68ad-2abf-44c8-98dc-c3ec72d36362\") " pod="openshift-ingress/e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs"
Oct 31 19:56:48.966386 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:48.966366    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48ab68ad-2abf-44c8-98dc-c3ec72d36362-istio-envoy\") pod \"e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs\" (UID: \"48ab68ad-2abf-44c8-98dc-c3ec72d36362\") " pod="openshift-ingress/e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs"
Oct 31 19:56:48.966516 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:48.966498    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48ab68ad-2abf-44c8-98dc-c3ec72d36362-workload-socket\") pod \"e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs\" (UID: \"48ab68ad-2abf-44c8-98dc-c3ec72d36362\") " pod="openshift-ingress/e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs"
Oct 31 19:56:48.966616 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:48.965397    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48ab68ad-2abf-44c8-98dc-c3ec72d36362-credential-socket\") pod \"e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs\" (UID: \"48ab68ad-2abf-44c8-98dc-c3ec72d36362\") " pod="openshift-ingress/e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs"
Oct 31 19:56:48.966680 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:48.964816    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48ab68ad-2abf-44c8-98dc-c3ec72d36362-istio-data\") pod \"e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs\" (UID: \"48ab68ad-2abf-44c8-98dc-c3ec72d36362\") " pod="openshift-ingress/e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs"
Oct 31 19:56:48.967086 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:48.967063    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48ab68ad-2abf-44c8-98dc-c3ec72d36362-workload-socket\") pod \"e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs\" (UID: \"48ab68ad-2abf-44c8-98dc-c3ec72d36362\") " pod="openshift-ingress/e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs"
Oct 31 19:56:48.967417 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:48.966529    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48ab68ad-2abf-44c8-98dc-c3ec72d36362-istiod-ca-cert\") pod \"e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs\" (UID: \"48ab68ad-2abf-44c8-98dc-c3ec72d36362\") " pod="openshift-ingress/e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs"
Oct 31 19:56:48.967607 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:48.967587    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48ab68ad-2abf-44c8-98dc-c3ec72d36362-workload-certs\") pod \"e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs\" (UID: \"48ab68ad-2abf-44c8-98dc-c3ec72d36362\") " pod="openshift-ingress/e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs"
Oct 31 19:56:48.968178 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:48.968145    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48ab68ad-2abf-44c8-98dc-c3ec72d36362-workload-certs\") pod \"e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs\" (UID: \"48ab68ad-2abf-44c8-98dc-c3ec72d36362\") " pod="openshift-ingress/e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs"
Oct 31 19:56:48.985816 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:48.985781    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48ab68ad-2abf-44c8-98dc-c3ec72d36362-istio-token\") pod \"e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs\" (UID: \"48ab68ad-2abf-44c8-98dc-c3ec72d36362\") " pod="openshift-ingress/e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs"
Oct 31 19:56:48.999810 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:48.999645    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtkxc\" (UniqueName: \"kubernetes.io/projected/48ab68ad-2abf-44c8-98dc-c3ec72d36362-kube-api-access-xtkxc\") pod \"e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs\" (UID: \"48ab68ad-2abf-44c8-98dc-c3ec72d36362\") " pod="openshift-ingress/e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs"
Oct 31 19:56:49.013804 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:49.006758    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48ab68ad-2abf-44c8-98dc-c3ec72d36362-istio-envoy\") pod \"e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs\" (UID: \"48ab68ad-2abf-44c8-98dc-c3ec72d36362\") " pod="openshift-ingress/e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs"
Oct 31 19:56:49.021716 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:49.016135    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48ab68ad-2abf-44c8-98dc-c3ec72d36362-istio-podinfo\") pod \"e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs\" (UID: \"48ab68ad-2abf-44c8-98dc-c3ec72d36362\") " pod="openshift-ingress/e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs"
Oct 31 19:56:49.024016 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:49.023966    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7c892a-98d9d78c9-qrnvx" podUID="dad64038-a18c-4944-80c8-b3f052a98dbb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:56:49.250556 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:49.250479    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs"
Oct 31 19:56:49.542807 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:49.542772    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs"]
Oct 31 19:56:49.542958 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:56:49.542940    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48ab68ad_2abf_44c8_98dc_c3ec72d36362.slice/crio-72ebcb4a54b2abdc3415279aa576d28255fbbe5533649092e67ac8e42e37e339 WatchSource:0}: Error finding container 72ebcb4a54b2abdc3415279aa576d28255fbbe5533649092e67ac8e42e37e339: Status 404 returned error can't find the container with id 72ebcb4a54b2abdc3415279aa576d28255fbbe5533649092e67ac8e42e37e339
Oct 31 19:56:49.546186 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:49.546146    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:56:49.546298 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:49.546228    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:56:49.546298 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:49.546271    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:56:49.896451 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:49.896357    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:50.279040 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:50.279009    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs" event={"ID":"48ab68ad-2abf-44c8-98dc-c3ec72d36362","Type":"ContainerStarted","Data":"1d7aeedb1b68c93d708d6174b24a7c7cdd3b72f8761a9c34d5da7b479223a126"}
Oct 31 19:56:50.279447 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:50.279064    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs" event={"ID":"48ab68ad-2abf-44c8-98dc-c3ec72d36362","Type":"ContainerStarted","Data":"72ebcb4a54b2abdc3415279aa576d28255fbbe5533649092e67ac8e42e37e339"}
Oct 31 19:56:50.303379 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:50.303313    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs" podStartSLOduration=2.303299221 podStartE2EDuration="2.303299221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:56:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:56:50.300651332 +0000 UTC m=+2240.873885461" watchObservedRunningTime="2026-10-31 19:56:50.303299221 +0000 UTC m=+2240.876533348"
Oct 31 19:56:50.660557 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:50.660518    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:51.251202 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:51.251174    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs"
Oct 31 19:56:51.256798 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:51.256771    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs"
Oct 31 19:56:51.286776 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:51.286745    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs"
Oct 31 19:56:51.287556 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:51.287531    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs"
Oct 31 19:56:51.448134 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:51.448101    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:51.886370 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:51.885155    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn"]
Oct 31 19:56:51.886370 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:51.885438    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn" podUID="15ee340d-0b42-4985-a3bd-fb0ba596bc9d" containerName="istio-proxy" containerID="cri-o://bc1463068f4e9e25a981a20fa1ddf217cc71add26b92cfd2e7e417f52a531629" gracePeriod=30
Oct 31 19:56:52.428817 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:52.428783    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:52.619459 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:52.619432    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9"
Oct 31 19:56:52.632052 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:52.632027    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aceb3312-c698-4f69-9b44-be9bd3ef0e6b-istio-token\" (UniqueName: \"kubernetes.io/projected/aceb3312-c698-4f69-9b44-be9bd3ef0e6b-istio-token\") pod \"aceb3312-c698-4f69-9b44-be9bd3ef0e6b\" (UID: \"aceb3312-c698-4f69-9b44-be9bd3ef0e6b\") "
Oct 31 19:56:52.632261 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:52.632235    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aceb3312-c698-4f69-9b44-be9bd3ef0e6b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aceb3312-c698-4f69-9b44-be9bd3ef0e6b-credential-socket\") pod \"aceb3312-c698-4f69-9b44-be9bd3ef0e6b\" (UID: \"aceb3312-c698-4f69-9b44-be9bd3ef0e6b\") "
Oct 31 19:56:52.632416 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:52.632299    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aceb3312-c698-4f69-9b44-be9bd3ef0e6b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aceb3312-c698-4f69-9b44-be9bd3ef0e6b-istio-podinfo\") pod \"aceb3312-c698-4f69-9b44-be9bd3ef0e6b\" (UID: \"aceb3312-c698-4f69-9b44-be9bd3ef0e6b\") "
Oct 31 19:56:52.632416 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:52.632318    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aceb3312-c698-4f69-9b44-be9bd3ef0e6b-kube-api-access-2vbwr\" (UniqueName: \"kubernetes.io/projected/aceb3312-c698-4f69-9b44-be9bd3ef0e6b-kube-api-access-2vbwr\") pod \"aceb3312-c698-4f69-9b44-be9bd3ef0e6b\" (UID: \"aceb3312-c698-4f69-9b44-be9bd3ef0e6b\") "
Oct 31 19:56:52.632534 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:52.632412    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aceb3312-c698-4f69-9b44-be9bd3ef0e6b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aceb3312-c698-4f69-9b44-be9bd3ef0e6b-workload-certs\") pod \"aceb3312-c698-4f69-9b44-be9bd3ef0e6b\" (UID: \"aceb3312-c698-4f69-9b44-be9bd3ef0e6b\") "
Oct 31 19:56:52.632601 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:52.632531    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aceb3312-c698-4f69-9b44-be9bd3ef0e6b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aceb3312-c698-4f69-9b44-be9bd3ef0e6b-istio-envoy\") pod \"aceb3312-c698-4f69-9b44-be9bd3ef0e6b\" (UID: \"aceb3312-c698-4f69-9b44-be9bd3ef0e6b\") "
Oct 31 19:56:52.632601 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:52.632554    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aceb3312-c698-4f69-9b44-be9bd3ef0e6b-credential-socket" pod "aceb3312-c698-4f69-9b44-be9bd3ef0e6b" (UID: "aceb3312-c698-4f69-9b44-be9bd3ef0e6b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:56:52.632601 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:52.632572    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aceb3312-c698-4f69-9b44-be9bd3ef0e6b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aceb3312-c698-4f69-9b44-be9bd3ef0e6b-workload-socket\") pod \"aceb3312-c698-4f69-9b44-be9bd3ef0e6b\" (UID: \"aceb3312-c698-4f69-9b44-be9bd3ef0e6b\") "
Oct 31 19:56:52.632757 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:52.632615    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aceb3312-c698-4f69-9b44-be9bd3ef0e6b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aceb3312-c698-4f69-9b44-be9bd3ef0e6b-istio-data\") pod \"aceb3312-c698-4f69-9b44-be9bd3ef0e6b\" (UID: \"aceb3312-c698-4f69-9b44-be9bd3ef0e6b\") "
Oct 31 19:56:52.632757 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:52.632716    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aceb3312-c698-4f69-9b44-be9bd3ef0e6b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aceb3312-c698-4f69-9b44-be9bd3ef0e6b-istiod-ca-cert\") pod \"aceb3312-c698-4f69-9b44-be9bd3ef0e6b\" (UID: \"aceb3312-c698-4f69-9b44-be9bd3ef0e6b\") "
Oct 31 19:56:52.632955 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:52.632712    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aceb3312-c698-4f69-9b44-be9bd3ef0e6b-workload-certs" pod "aceb3312-c698-4f69-9b44-be9bd3ef0e6b" (UID: "aceb3312-c698-4f69-9b44-be9bd3ef0e6b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:56:52.633075 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:52.632844    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aceb3312-c698-4f69-9b44-be9bd3ef0e6b-workload-socket" pod "aceb3312-c698-4f69-9b44-be9bd3ef0e6b" (UID: "aceb3312-c698-4f69-9b44-be9bd3ef0e6b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:56:52.633075 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:52.632904    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aceb3312-c698-4f69-9b44-be9bd3ef0e6b-istio-data" pod "aceb3312-c698-4f69-9b44-be9bd3ef0e6b" (UID: "aceb3312-c698-4f69-9b44-be9bd3ef0e6b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:56:52.633075 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:52.633055    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aceb3312-c698-4f69-9b44-be9bd3ef0e6b-istiod-ca-cert" pod "aceb3312-c698-4f69-9b44-be9bd3ef0e6b" (UID: "aceb3312-c698-4f69-9b44-be9bd3ef0e6b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:56:52.633701 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:52.633680    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aceb3312-c698-4f69-9b44-be9bd3ef0e6b-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:52.633775 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:52.633709    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aceb3312-c698-4f69-9b44-be9bd3ef0e6b-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:52.633775 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:52.633723    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aceb3312-c698-4f69-9b44-be9bd3ef0e6b-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:52.633775 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:52.633737    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aceb3312-c698-4f69-9b44-be9bd3ef0e6b-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:52.633775 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:52.633751    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aceb3312-c698-4f69-9b44-be9bd3ef0e6b-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:52.635273 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:52.635253    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aceb3312-c698-4f69-9b44-be9bd3ef0e6b-kube-api-access-2vbwr" pod "aceb3312-c698-4f69-9b44-be9bd3ef0e6b" (UID: "aceb3312-c698-4f69-9b44-be9bd3ef0e6b"). InnerVolumeSpecName "kube-api-access-2vbwr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:56:52.635781 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:52.635491    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aceb3312-c698-4f69-9b44-be9bd3ef0e6b-istio-podinfo" pod "aceb3312-c698-4f69-9b44-be9bd3ef0e6b" (UID: "aceb3312-c698-4f69-9b44-be9bd3ef0e6b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:56:52.635781 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:52.635560    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aceb3312-c698-4f69-9b44-be9bd3ef0e6b-istio-envoy" pod "aceb3312-c698-4f69-9b44-be9bd3ef0e6b" (UID: "aceb3312-c698-4f69-9b44-be9bd3ef0e6b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:56:52.636250 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:52.636217    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aceb3312-c698-4f69-9b44-be9bd3ef0e6b-istio-token" pod "aceb3312-c698-4f69-9b44-be9bd3ef0e6b" (UID: "aceb3312-c698-4f69-9b44-be9bd3ef0e6b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:56:52.667218 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:52.667161    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:52.735598 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:52.735569    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aceb3312-c698-4f69-9b44-be9bd3ef0e6b-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:52.735598 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:52.735599    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aceb3312-c698-4f69-9b44-be9bd3ef0e6b-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:52.735814 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:52.735616    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vbwr\" (UniqueName: \"kubernetes.io/projected/aceb3312-c698-4f69-9b44-be9bd3ef0e6b-kube-api-access-2vbwr\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:52.735814 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:52.735629    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aceb3312-c698-4f69-9b44-be9bd3ef0e6b-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:52.796364 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:52.796300    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:53.303500 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:53.303463    4645 generic.go:356] "Generic (PLEG): container finished" podID="aceb3312-c698-4f69-9b44-be9bd3ef0e6b" containerID="ebb0038c362dad3b8d3f2bbbed721b338c570108d2d175d8e3474999a978b949" exitCode=0
Oct 31 19:56:53.303736 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:53.303524    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9"
Oct 31 19:56:53.303736 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:53.303542    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9" event={"ID":"aceb3312-c698-4f69-9b44-be9bd3ef0e6b","Type":"ContainerDied","Data":"ebb0038c362dad3b8d3f2bbbed721b338c570108d2d175d8e3474999a978b949"}
Oct 31 19:56:53.303736 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:53.303589    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9" event={"ID":"aceb3312-c698-4f69-9b44-be9bd3ef0e6b","Type":"ContainerDied","Data":"43e44c03a13c066202b1bab74cafb0f8f407b0921ddd569f881b1afecee972d8"}
Oct 31 19:56:53.303736 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:53.303612    4645 scope.go:122] "RemoveContainer" containerID="ebb0038c362dad3b8d3f2bbbed721b338c570108d2d175d8e3474999a978b949"
Oct 31 19:56:53.316159 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:53.316136    4645 scope.go:122] "RemoveContainer" containerID="ebb0038c362dad3b8d3f2bbbed721b338c570108d2d175d8e3474999a978b949"
Oct 31 19:56:53.316509 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:56:53.316491    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebb0038c362dad3b8d3f2bbbed721b338c570108d2d175d8e3474999a978b949\": container with ID starting with ebb0038c362dad3b8d3f2bbbed721b338c570108d2d175d8e3474999a978b949 not found: ID does not exist" containerID="ebb0038c362dad3b8d3f2bbbed721b338c570108d2d175d8e3474999a978b949"
Oct 31 19:56:53.316591 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:53.316516    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebb0038c362dad3b8d3f2bbbed721b338c570108d2d175d8e3474999a978b949"} err="failed to get container status \"ebb0038c362dad3b8d3f2bbbed721b338c570108d2d175d8e3474999a978b949\": rpc error: code = NotFound desc = could not find container \"ebb0038c362dad3b8d3f2bbbed721b338c570108d2d175d8e3474999a978b949\": container with ID starting with ebb0038c362dad3b8d3f2bbbed721b338c570108d2d175d8e3474999a978b949 not found: ID does not exist"
Oct 31 19:56:53.334817 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:53.334790    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9"]
Oct 31 19:56:53.340409 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:53.340380    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-7c892a-openshift-default-c8946d869-qv6n9"]
Oct 31 19:56:53.456622 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:53.456585    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:53.620434 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:53.620359    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aceb3312-c698-4f69-9b44-be9bd3ef0e6b" path="/var/lib/kubelet/pods/aceb3312-c698-4f69-9b44-be9bd3ef0e6b/volumes"
Oct 31 19:56:53.908123 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:53.908045    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:54.049277 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:54.049239    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:54.744040 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:54.744002    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:55.758963 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:55.758922    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:56.386367 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:56.386314    4645 patch_prober.go:29] interesting pod/e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 19:56:56.386652 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:56.386392    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn" podUID="15ee340d-0b42-4985-a3bd-fb0ba596bc9d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 19:56:56.635873 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:56.635832    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:57.218808 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:57.218784    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn"
Oct 31 19:56:57.353112 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:57.353079    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15ee340d-0b42-4985-a3bd-fb0ba596bc9d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15ee340d-0b42-4985-a3bd-fb0ba596bc9d-istio-data\") pod \"15ee340d-0b42-4985-a3bd-fb0ba596bc9d\" (UID: \"15ee340d-0b42-4985-a3bd-fb0ba596bc9d\") "
Oct 31 19:56:57.353304 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:57.353153    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15ee340d-0b42-4985-a3bd-fb0ba596bc9d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15ee340d-0b42-4985-a3bd-fb0ba596bc9d-istiod-ca-cert\") pod \"15ee340d-0b42-4985-a3bd-fb0ba596bc9d\" (UID: \"15ee340d-0b42-4985-a3bd-fb0ba596bc9d\") "
Oct 31 19:56:57.353304 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:57.353190    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15ee340d-0b42-4985-a3bd-fb0ba596bc9d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15ee340d-0b42-4985-a3bd-fb0ba596bc9d-istio-envoy\") pod \"15ee340d-0b42-4985-a3bd-fb0ba596bc9d\" (UID: \"15ee340d-0b42-4985-a3bd-fb0ba596bc9d\") "
Oct 31 19:56:57.353304 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:57.353226    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15ee340d-0b42-4985-a3bd-fb0ba596bc9d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15ee340d-0b42-4985-a3bd-fb0ba596bc9d-workload-certs\") pod \"15ee340d-0b42-4985-a3bd-fb0ba596bc9d\" (UID: \"15ee340d-0b42-4985-a3bd-fb0ba596bc9d\") "
Oct 31 19:56:57.353304 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:57.353298    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15ee340d-0b42-4985-a3bd-fb0ba596bc9d-kube-api-access-mgwz2\" (UniqueName: \"kubernetes.io/projected/15ee340d-0b42-4985-a3bd-fb0ba596bc9d-kube-api-access-mgwz2\") pod \"15ee340d-0b42-4985-a3bd-fb0ba596bc9d\" (UID: \"15ee340d-0b42-4985-a3bd-fb0ba596bc9d\") "
Oct 31 19:56:57.353598 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:57.353349    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15ee340d-0b42-4985-a3bd-fb0ba596bc9d-istio-token\" (UniqueName: \"kubernetes.io/projected/15ee340d-0b42-4985-a3bd-fb0ba596bc9d-istio-token\") pod \"15ee340d-0b42-4985-a3bd-fb0ba596bc9d\" (UID: \"15ee340d-0b42-4985-a3bd-fb0ba596bc9d\") "
Oct 31 19:56:57.353598 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:57.353436    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15ee340d-0b42-4985-a3bd-fb0ba596bc9d-istio-data" pod "15ee340d-0b42-4985-a3bd-fb0ba596bc9d" (UID: "15ee340d-0b42-4985-a3bd-fb0ba596bc9d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:56:57.353598 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:57.353449    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15ee340d-0b42-4985-a3bd-fb0ba596bc9d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15ee340d-0b42-4985-a3bd-fb0ba596bc9d-credential-socket\") pod \"15ee340d-0b42-4985-a3bd-fb0ba596bc9d\" (UID: \"15ee340d-0b42-4985-a3bd-fb0ba596bc9d\") "
Oct 31 19:56:57.353598 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:57.353578    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15ee340d-0b42-4985-a3bd-fb0ba596bc9d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15ee340d-0b42-4985-a3bd-fb0ba596bc9d-workload-socket\") pod \"15ee340d-0b42-4985-a3bd-fb0ba596bc9d\" (UID: \"15ee340d-0b42-4985-a3bd-fb0ba596bc9d\") "
Oct 31 19:56:57.353829 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:57.353626    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/15ee340d-0b42-4985-a3bd-fb0ba596bc9d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15ee340d-0b42-4985-a3bd-fb0ba596bc9d-istio-podinfo\") pod \"15ee340d-0b42-4985-a3bd-fb0ba596bc9d\" (UID: \"15ee340d-0b42-4985-a3bd-fb0ba596bc9d\") "
Oct 31 19:56:57.353829 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:57.353669    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15ee340d-0b42-4985-a3bd-fb0ba596bc9d-credential-socket" pod "15ee340d-0b42-4985-a3bd-fb0ba596bc9d" (UID: "15ee340d-0b42-4985-a3bd-fb0ba596bc9d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:56:57.353829 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:57.353690    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15ee340d-0b42-4985-a3bd-fb0ba596bc9d-istiod-ca-cert" pod "15ee340d-0b42-4985-a3bd-fb0ba596bc9d" (UID: "15ee340d-0b42-4985-a3bd-fb0ba596bc9d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:56:57.353984 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:57.353905    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15ee340d-0b42-4985-a3bd-fb0ba596bc9d-workload-socket" pod "15ee340d-0b42-4985-a3bd-fb0ba596bc9d" (UID: "15ee340d-0b42-4985-a3bd-fb0ba596bc9d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:56:57.353984 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:57.353958    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15ee340d-0b42-4985-a3bd-fb0ba596bc9d-workload-certs" pod "15ee340d-0b42-4985-a3bd-fb0ba596bc9d" (UID: "15ee340d-0b42-4985-a3bd-fb0ba596bc9d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:56:57.354524 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:57.354496    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15ee340d-0b42-4985-a3bd-fb0ba596bc9d-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:57.354672 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:57.354528    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15ee340d-0b42-4985-a3bd-fb0ba596bc9d-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:57.354672 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:57.354545    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15ee340d-0b42-4985-a3bd-fb0ba596bc9d-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:57.354672 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:57.354561    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15ee340d-0b42-4985-a3bd-fb0ba596bc9d-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:57.354672 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:57.354576    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15ee340d-0b42-4985-a3bd-fb0ba596bc9d-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:57.355966 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:57.355938    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15ee340d-0b42-4985-a3bd-fb0ba596bc9d-istio-token" pod "15ee340d-0b42-4985-a3bd-fb0ba596bc9d" (UID: "15ee340d-0b42-4985-a3bd-fb0ba596bc9d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:56:57.356095 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:57.355966    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15ee340d-0b42-4985-a3bd-fb0ba596bc9d-istio-envoy" pod "15ee340d-0b42-4985-a3bd-fb0ba596bc9d" (UID: "15ee340d-0b42-4985-a3bd-fb0ba596bc9d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:56:57.356095 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:57.355974    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15ee340d-0b42-4985-a3bd-fb0ba596bc9d-kube-api-access-mgwz2" pod "15ee340d-0b42-4985-a3bd-fb0ba596bc9d" (UID: "15ee340d-0b42-4985-a3bd-fb0ba596bc9d"). InnerVolumeSpecName "kube-api-access-mgwz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:56:57.356095 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:57.356018    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/15ee340d-0b42-4985-a3bd-fb0ba596bc9d-istio-podinfo" pod "15ee340d-0b42-4985-a3bd-fb0ba596bc9d" (UID: "15ee340d-0b42-4985-a3bd-fb0ba596bc9d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:56:57.362069 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:57.362046    4645 generic.go:356] "Generic (PLEG): container finished" podID="15ee340d-0b42-4985-a3bd-fb0ba596bc9d" containerID="bc1463068f4e9e25a981a20fa1ddf217cc71add26b92cfd2e7e417f52a531629" exitCode=0
Oct 31 19:56:57.362171 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:57.362095    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn"
Oct 31 19:56:57.362171 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:57.362126    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn" event={"ID":"15ee340d-0b42-4985-a3bd-fb0ba596bc9d","Type":"ContainerDied","Data":"bc1463068f4e9e25a981a20fa1ddf217cc71add26b92cfd2e7e417f52a531629"}
Oct 31 19:56:57.362295 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:57.362187    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn" event={"ID":"15ee340d-0b42-4985-a3bd-fb0ba596bc9d","Type":"ContainerDied","Data":"f91073323a171a4678273685be7c7fe2e8db1e6e248988f3e8421922885ecce0"}
Oct 31 19:56:57.362295 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:57.362210    4645 scope.go:122] "RemoveContainer" containerID="bc1463068f4e9e25a981a20fa1ddf217cc71add26b92cfd2e7e417f52a531629"
Oct 31 19:56:57.393316 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:57.393237    4645 scope.go:122] "RemoveContainer" containerID="bc1463068f4e9e25a981a20fa1ddf217cc71add26b92cfd2e7e417f52a531629"
Oct 31 19:56:57.393693 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:56:57.393668    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc1463068f4e9e25a981a20fa1ddf217cc71add26b92cfd2e7e417f52a531629\": container with ID starting with bc1463068f4e9e25a981a20fa1ddf217cc71add26b92cfd2e7e417f52a531629 not found: ID does not exist" containerID="bc1463068f4e9e25a981a20fa1ddf217cc71add26b92cfd2e7e417f52a531629"
Oct 31 19:56:57.393792 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:57.393697    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc1463068f4e9e25a981a20fa1ddf217cc71add26b92cfd2e7e417f52a531629"} err="failed to get container status \"bc1463068f4e9e25a981a20fa1ddf217cc71add26b92cfd2e7e417f52a531629\": rpc error: code = NotFound desc = could not find container \"bc1463068f4e9e25a981a20fa1ddf217cc71add26b92cfd2e7e417f52a531629\": container with ID starting with bc1463068f4e9e25a981a20fa1ddf217cc71add26b92cfd2e7e417f52a531629 not found: ID does not exist"
Oct 31 19:56:57.407387 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:57.407361    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn"]
Oct 31 19:56:57.412951 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:57.412930    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-996c32-openshift-default-58dc84fb9-ghgbn"]
Oct 31 19:56:57.439658 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:57.439606    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:57.456643 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:57.456622    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15ee340d-0b42-4985-a3bd-fb0ba596bc9d-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:57.456643 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:57.456643    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15ee340d-0b42-4985-a3bd-fb0ba596bc9d-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:57.456763 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:57.456654    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgwz2\" (UniqueName: \"kubernetes.io/projected/15ee340d-0b42-4985-a3bd-fb0ba596bc9d-kube-api-access-mgwz2\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:57.456763 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:57.456664    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15ee340d-0b42-4985-a3bd-fb0ba596bc9d-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:56:57.631602 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:57.631559    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15ee340d-0b42-4985-a3bd-fb0ba596bc9d" path="/var/lib/kubelet/pods/15ee340d-0b42-4985-a3bd-fb0ba596bc9d/volumes"
Oct 31 19:56:58.198345 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:58.198295    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:58.950440 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:58.950405    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:56:59.022353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:59.021647    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7c892a-98d9d78c9-qrnvx" podUID="dad64038-a18c-4944-80c8-b3f052a98dbb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 19:56:59.746878 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:56:59.746842    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:00.320148 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:00.320122    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7c892a-98d9d78c9-qrnvx"
Oct 31 19:57:00.391682 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:00.391648    4645 generic.go:356] "Generic (PLEG): container finished" podID="dad64038-a18c-4944-80c8-b3f052a98dbb" containerID="a3ed5340dd2ca8710d32df3a1e01e31447636ccd988cad30a18044d290cbe69f" exitCode=0
Oct 31 19:57:00.391852 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:00.391692    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7c892a-98d9d78c9-qrnvx"
Oct 31 19:57:00.391852 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:00.391732    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7c892a-98d9d78c9-qrnvx" event={"ID":"dad64038-a18c-4944-80c8-b3f052a98dbb","Type":"ContainerDied","Data":"a3ed5340dd2ca8710d32df3a1e01e31447636ccd988cad30a18044d290cbe69f"}
Oct 31 19:57:00.391852 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:00.391798    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7c892a-98d9d78c9-qrnvx" event={"ID":"dad64038-a18c-4944-80c8-b3f052a98dbb","Type":"ContainerDied","Data":"f8a56bee5ec8381858d7d72c43bd85e82a70ac7a4f61859beacec99835cf1fd1"}
Oct 31 19:57:00.391852 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:00.391830    4645 scope.go:122] "RemoveContainer" containerID="a3ed5340dd2ca8710d32df3a1e01e31447636ccd988cad30a18044d290cbe69f"
Oct 31 19:57:00.403234 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:00.403218    4645 scope.go:122] "RemoveContainer" containerID="a3ed5340dd2ca8710d32df3a1e01e31447636ccd988cad30a18044d290cbe69f"
Oct 31 19:57:00.403548 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:57:00.403521    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3ed5340dd2ca8710d32df3a1e01e31447636ccd988cad30a18044d290cbe69f\": container with ID starting with a3ed5340dd2ca8710d32df3a1e01e31447636ccd988cad30a18044d290cbe69f not found: ID does not exist" containerID="a3ed5340dd2ca8710d32df3a1e01e31447636ccd988cad30a18044d290cbe69f"
Oct 31 19:57:00.403659 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:00.403551    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3ed5340dd2ca8710d32df3a1e01e31447636ccd988cad30a18044d290cbe69f"} err="failed to get container status \"a3ed5340dd2ca8710d32df3a1e01e31447636ccd988cad30a18044d290cbe69f\": rpc error: code = NotFound desc = could not find container \"a3ed5340dd2ca8710d32df3a1e01e31447636ccd988cad30a18044d290cbe69f\": container with ID starting with a3ed5340dd2ca8710d32df3a1e01e31447636ccd988cad30a18044d290cbe69f not found: ID does not exist"
Oct 31 19:57:00.418875 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:00.418855    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dad64038-a18c-4944-80c8-b3f052a98dbb-kube-api-access-f946j\" (UniqueName: \"kubernetes.io/projected/dad64038-a18c-4944-80c8-b3f052a98dbb-kube-api-access-f946j\") pod \"dad64038-a18c-4944-80c8-b3f052a98dbb\" (UID: \"dad64038-a18c-4944-80c8-b3f052a98dbb\") "
Oct 31 19:57:00.419021 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:00.419004    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dad64038-a18c-4944-80c8-b3f052a98dbb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dad64038-a18c-4944-80c8-b3f052a98dbb-plugins-config-volume\") pod \"dad64038-a18c-4944-80c8-b3f052a98dbb\" (UID: \"dad64038-a18c-4944-80c8-b3f052a98dbb\") "
Oct 31 19:57:00.419343 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:00.419311    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dad64038-a18c-4944-80c8-b3f052a98dbb-plugins-config-volume" pod "dad64038-a18c-4944-80c8-b3f052a98dbb" (UID: "dad64038-a18c-4944-80c8-b3f052a98dbb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:57:00.419746 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:00.419727    4645 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dad64038-a18c-4944-80c8-b3f052a98dbb-plugins-config-volume\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:57:00.420863 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:00.420846    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dad64038-a18c-4944-80c8-b3f052a98dbb-kube-api-access-f946j" pod "dad64038-a18c-4944-80c8-b3f052a98dbb" (UID: "dad64038-a18c-4944-80c8-b3f052a98dbb"). InnerVolumeSpecName "kube-api-access-f946j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:57:00.522166 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:00.521036    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f946j\" (UniqueName: \"kubernetes.io/projected/dad64038-a18c-4944-80c8-b3f052a98dbb-kube-api-access-f946j\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:57:00.633955 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:00.633895    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:00.737196 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:00.737153    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7c892a-98d9d78c9-qrnvx"]
Oct 31 19:57:00.742233 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:00.742200    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7c892a-98d9d78c9-qrnvx"]
Oct 31 19:57:01.420408 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:01.420370    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:01.619052 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:01.619020    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dad64038-a18c-4944-80c8-b3f052a98dbb" path="/var/lib/kubelet/pods/dad64038-a18c-4944-80c8-b3f052a98dbb/volumes"
Oct 31 19:57:02.237821 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:02.237785    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:02.939872 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:02.939835    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:03.729991 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:03.729938    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:04.547223 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:04.547189    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:05.501517 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:05.501485    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:06.246018 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:57:06.245977    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 19:57:06.253373 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:06.253304    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:07.073886 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:07.073839    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:07.910413 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:07.910378    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:08.660418 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:08.660378    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:09.461615 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:09.461580    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:09.586347 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:09.573350    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:09.696738 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:09.696705    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:10.578474 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:10.569922    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:11.361761 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:11.361720    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:12.149180 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:12.149145    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:12.870978 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:12.870948    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:13.663024 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:13.662996    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:14.435663 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:14.435634    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:15.271129 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:15.271092    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:16.213179 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:16.213147    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:17.108927 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:17.093320    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:17.300891 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:17.300856    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:17.854149 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:17.854112    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:18.351590 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:18.351560    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:18.647245 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:18.647174    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:19.441603 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:19.441561    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:20.151263 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:20.151227    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:20.401102 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:20.401061    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:21.072514 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:21.072475    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:21.842039 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:21.842009    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:22.778796 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:22.778763    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:23.648812 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:23.648776    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:24.399019 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:24.398979    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:25.176486 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:25.176450    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:25.960014 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:25.959980    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:26.738470 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:26.738436    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:27.710987 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:27.710944    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:28.287779 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:28.287743    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:28.387999 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:28.387969    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:28.542928 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:28.542836    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:28.666907 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:28.666867    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:29.453102 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:29.453062    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:30.162270 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:30.162238    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:31.007840 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:31.007751    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:31.834244 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:31.834212    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:32.565688 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:32.565653    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:33.324338 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:33.324278    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:34.043648 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:34.043613    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:34.823205 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:34.823173    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:35.562421 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:35.562382    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:36.246449 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:57:36.246413    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 19:57:36.364520 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:36.364487    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:37.132975 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:37.132940    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:37.867053 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:37.867019    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:38.661534 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:38.661502    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:39.590795 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:39.590763    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:40.341107 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:40.341071    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:41.070922 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:41.070884    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:42.051018 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:42.050932    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:43.014666 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:43.014627    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:43.869361 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:43.867766    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:44.604005 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:44.603947    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:44.723432 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:44.723400    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:45.338006 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:45.337968    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:46.142272 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:46.142235    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:46.940857 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:46.940819    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:47.880403 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:47.880286    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:48.744836 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:48.744801    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:49.546378 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:49.546316    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:50.348571 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:50.348536    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:51.231300 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:51.231270    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:51.689815 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:51.689786    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s"]
Oct 31 19:57:51.690287 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:51.689971    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s" podUID="426c8fc3-1152-4a03-81e6-ada81e4d0007" containerName="istio-proxy" containerID="cri-o://9e97adecf885e756917ec05986dd7dbf6db69ab86c5cf75fcfea13ebdf5264ca" gracePeriod=30
Oct 31 19:57:52.069304 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:52.069226    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:52.335669 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:52.335562    4645 patch_prober.go:29] interesting pod/e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.224:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 19:57:52.335931 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:52.335897    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs" podUID="48ab68ad-2abf-44c8-98dc-c3ec72d36362" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.224:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:57:52.833401 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:52.833359    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:52.967717 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:52.967684    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9th7f"]
Oct 31 19:57:52.967896 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:52.967878    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9th7f" podUID="76bc1f3a-74b3-49b3-bde1-cc3b305260d6" containerName="istio-proxy" containerID="cri-o://2f0c74238d8b1117dda0cd0bb0aada4ee7f7ac6057576ed3a76b0fc2162f754e" gracePeriod=30
Oct 31 19:57:53.718724 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:53.711176    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:54.537937 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:54.537907    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:55.172670 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:55.172631    4645 generic.go:356] "Generic (PLEG): container finished" podID="48ab68ad-2abf-44c8-98dc-c3ec72d36362" containerID="1d7aeedb1b68c93d708d6174b24a7c7cdd3b72f8761a9c34d5da7b479223a126" exitCode=0
Oct 31 19:57:55.172847 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:55.172698    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs" event={"ID":"48ab68ad-2abf-44c8-98dc-c3ec72d36362","Type":"ContainerDied","Data":"1d7aeedb1b68c93d708d6174b24a7c7cdd3b72f8761a9c34d5da7b479223a126"}
Oct 31 19:57:55.173050 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:55.173035    4645 scope.go:122] "RemoveContainer" containerID="1d7aeedb1b68c93d708d6174b24a7c7cdd3b72f8761a9c34d5da7b479223a126"
Oct 31 19:57:55.174630 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:55.174603    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:57:55.174722 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:55.174668    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:57:55.174722 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:55.174699    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:57:55.357836 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:55.357802    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:56.188373 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:56.188300    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs" event={"ID":"48ab68ad-2abf-44c8-98dc-c3ec72d36362","Type":"ContainerStarted","Data":"bcbcd6e75fa56be2a3f5a818fd449b59252d483240db33b92cf8d944e34b716e"}
Oct 31 19:57:56.250974 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:56.250945    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs"
Oct 31 19:57:56.251185 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:56.251165    4645 patch_prober.go:29] interesting pod/e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" start-of-body=
Oct 31 19:57:56.251241 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:56.251219    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs" podUID="48ab68ad-2abf-44c8-98dc-c3ec72d36362" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused"
Oct 31 19:57:56.477117 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:56.477015    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:57.029859 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:57.029835    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s"
Oct 31 19:57:57.200002 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:57.199962    4645 generic.go:356] "Generic (PLEG): container finished" podID="426c8fc3-1152-4a03-81e6-ada81e4d0007" containerID="9e97adecf885e756917ec05986dd7dbf6db69ab86c5cf75fcfea13ebdf5264ca" exitCode=0
Oct 31 19:57:57.200497 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:57.200028    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s"
Oct 31 19:57:57.200497 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:57.200039    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s" event={"ID":"426c8fc3-1152-4a03-81e6-ada81e4d0007","Type":"ContainerDied","Data":"9e97adecf885e756917ec05986dd7dbf6db69ab86c5cf75fcfea13ebdf5264ca"}
Oct 31 19:57:57.200497 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:57.200094    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s" event={"ID":"426c8fc3-1152-4a03-81e6-ada81e4d0007","Type":"ContainerDied","Data":"7c5692b7c4524597651ee600c9a5373c784a9b63775a53f991bdb169e453b2ae"}
Oct 31 19:57:57.200497 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:57.200122    4645 scope.go:122] "RemoveContainer" containerID="9e97adecf885e756917ec05986dd7dbf6db69ab86c5cf75fcfea13ebdf5264ca"
Oct 31 19:57:57.205279 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:57.205253    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/426c8fc3-1152-4a03-81e6-ada81e4d0007-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/426c8fc3-1152-4a03-81e6-ada81e4d0007-credential-socket\") pod \"426c8fc3-1152-4a03-81e6-ada81e4d0007\" (UID: \"426c8fc3-1152-4a03-81e6-ada81e4d0007\") "
Oct 31 19:57:57.205416 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:57.205388    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/426c8fc3-1152-4a03-81e6-ada81e4d0007-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/426c8fc3-1152-4a03-81e6-ada81e4d0007-istiod-ca-cert\") pod \"426c8fc3-1152-4a03-81e6-ada81e4d0007\" (UID: \"426c8fc3-1152-4a03-81e6-ada81e4d0007\") "
Oct 31 19:57:57.205535 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:57.205519    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/426c8fc3-1152-4a03-81e6-ada81e4d0007-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/426c8fc3-1152-4a03-81e6-ada81e4d0007-istio-data\") pod \"426c8fc3-1152-4a03-81e6-ada81e4d0007\" (UID: \"426c8fc3-1152-4a03-81e6-ada81e4d0007\") "
Oct 31 19:57:57.205572 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:57.205552    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/426c8fc3-1152-4a03-81e6-ada81e4d0007-credential-socket" pod "426c8fc3-1152-4a03-81e6-ada81e4d0007" (UID: "426c8fc3-1152-4a03-81e6-ada81e4d0007"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:57:57.205612 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:57.205578    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/426c8fc3-1152-4a03-81e6-ada81e4d0007-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/426c8fc3-1152-4a03-81e6-ada81e4d0007-istio-podinfo\") pod \"426c8fc3-1152-4a03-81e6-ada81e4d0007\" (UID: \"426c8fc3-1152-4a03-81e6-ada81e4d0007\") "
Oct 31 19:57:57.205657 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:57.205614    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/426c8fc3-1152-4a03-81e6-ada81e4d0007-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/426c8fc3-1152-4a03-81e6-ada81e4d0007-workload-certs\") pod \"426c8fc3-1152-4a03-81e6-ada81e4d0007\" (UID: \"426c8fc3-1152-4a03-81e6-ada81e4d0007\") "
Oct 31 19:57:57.205710 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:57.205653    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/426c8fc3-1152-4a03-81e6-ada81e4d0007-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/426c8fc3-1152-4a03-81e6-ada81e4d0007-workload-socket\") pod \"426c8fc3-1152-4a03-81e6-ada81e4d0007\" (UID: \"426c8fc3-1152-4a03-81e6-ada81e4d0007\") "
Oct 31 19:57:57.205751 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:57.205733    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/426c8fc3-1152-4a03-81e6-ada81e4d0007-istio-data" pod "426c8fc3-1152-4a03-81e6-ada81e4d0007" (UID: "426c8fc3-1152-4a03-81e6-ada81e4d0007"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:57:57.205789 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:57.205751    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/426c8fc3-1152-4a03-81e6-ada81e4d0007-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/426c8fc3-1152-4a03-81e6-ada81e4d0007-istio-envoy\") pod \"426c8fc3-1152-4a03-81e6-ada81e4d0007\" (UID: \"426c8fc3-1152-4a03-81e6-ada81e4d0007\") "
Oct 31 19:57:57.205789 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:57.205784    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/426c8fc3-1152-4a03-81e6-ada81e4d0007-istio-token\" (UniqueName: \"kubernetes.io/projected/426c8fc3-1152-4a03-81e6-ada81e4d0007-istio-token\") pod \"426c8fc3-1152-4a03-81e6-ada81e4d0007\" (UID: \"426c8fc3-1152-4a03-81e6-ada81e4d0007\") "
Oct 31 19:57:57.205856 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:57.205797    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/426c8fc3-1152-4a03-81e6-ada81e4d0007-istiod-ca-cert" pod "426c8fc3-1152-4a03-81e6-ada81e4d0007" (UID: "426c8fc3-1152-4a03-81e6-ada81e4d0007"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:57:57.205856 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:57.205851    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/426c8fc3-1152-4a03-81e6-ada81e4d0007-kube-api-access-26rf9\" (UniqueName: \"kubernetes.io/projected/426c8fc3-1152-4a03-81e6-ada81e4d0007-kube-api-access-26rf9\") pod \"426c8fc3-1152-4a03-81e6-ada81e4d0007\" (UID: \"426c8fc3-1152-4a03-81e6-ada81e4d0007\") "
Oct 31 19:57:57.205923 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:57.205907    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/426c8fc3-1152-4a03-81e6-ada81e4d0007-workload-certs" pod "426c8fc3-1152-4a03-81e6-ada81e4d0007" (UID: "426c8fc3-1152-4a03-81e6-ada81e4d0007"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:57:57.206252 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:57.206233    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/426c8fc3-1152-4a03-81e6-ada81e4d0007-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:57:57.206349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:57.206254    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/426c8fc3-1152-4a03-81e6-ada81e4d0007-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:57:57.206349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:57.206269    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/426c8fc3-1152-4a03-81e6-ada81e4d0007-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:57:57.206349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:57.206281    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/426c8fc3-1152-4a03-81e6-ada81e4d0007-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:57:57.206349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:57.206308    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/426c8fc3-1152-4a03-81e6-ada81e4d0007-workload-socket" pod "426c8fc3-1152-4a03-81e6-ada81e4d0007" (UID: "426c8fc3-1152-4a03-81e6-ada81e4d0007"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:57:57.208382 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:57.208291    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/426c8fc3-1152-4a03-81e6-ada81e4d0007-istio-podinfo" pod "426c8fc3-1152-4a03-81e6-ada81e4d0007" (UID: "426c8fc3-1152-4a03-81e6-ada81e4d0007"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:57:57.208382 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:57.208364    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/426c8fc3-1152-4a03-81e6-ada81e4d0007-istio-envoy" pod "426c8fc3-1152-4a03-81e6-ada81e4d0007" (UID: "426c8fc3-1152-4a03-81e6-ada81e4d0007"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:57:57.208604 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:57.208584    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/426c8fc3-1152-4a03-81e6-ada81e4d0007-kube-api-access-26rf9" pod "426c8fc3-1152-4a03-81e6-ada81e4d0007" (UID: "426c8fc3-1152-4a03-81e6-ada81e4d0007"). InnerVolumeSpecName "kube-api-access-26rf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:57:57.210263 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:57.210239    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/426c8fc3-1152-4a03-81e6-ada81e4d0007-istio-token" pod "426c8fc3-1152-4a03-81e6-ada81e4d0007" (UID: "426c8fc3-1152-4a03-81e6-ada81e4d0007"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:57:57.213307 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:57.213287    4645 scope.go:122] "RemoveContainer" containerID="f87c1ec8f52beaaa4f9d63a1ec96e4d1889c7e71e71aa74a742ef3fa1f7ed58f"
Oct 31 19:57:57.238788 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:57.238758    4645 scope.go:122] "RemoveContainer" containerID="9e97adecf885e756917ec05986dd7dbf6db69ab86c5cf75fcfea13ebdf5264ca"
Oct 31 19:57:57.239071 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:57:57.239044    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e97adecf885e756917ec05986dd7dbf6db69ab86c5cf75fcfea13ebdf5264ca\": container with ID starting with 9e97adecf885e756917ec05986dd7dbf6db69ab86c5cf75fcfea13ebdf5264ca not found: ID does not exist" containerID="9e97adecf885e756917ec05986dd7dbf6db69ab86c5cf75fcfea13ebdf5264ca"
Oct 31 19:57:57.239141 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:57.239082    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e97adecf885e756917ec05986dd7dbf6db69ab86c5cf75fcfea13ebdf5264ca"} err="failed to get container status \"9e97adecf885e756917ec05986dd7dbf6db69ab86c5cf75fcfea13ebdf5264ca\": rpc error: code = NotFound desc = could not find container \"9e97adecf885e756917ec05986dd7dbf6db69ab86c5cf75fcfea13ebdf5264ca\": container with ID starting with 9e97adecf885e756917ec05986dd7dbf6db69ab86c5cf75fcfea13ebdf5264ca not found: ID does not exist"
Oct 31 19:57:57.239141 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:57.239100    4645 scope.go:122] "RemoveContainer" containerID="f87c1ec8f52beaaa4f9d63a1ec96e4d1889c7e71e71aa74a742ef3fa1f7ed58f"
Oct 31 19:57:57.239384 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:57:57.239364    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f87c1ec8f52beaaa4f9d63a1ec96e4d1889c7e71e71aa74a742ef3fa1f7ed58f\": container with ID starting with f87c1ec8f52beaaa4f9d63a1ec96e4d1889c7e71e71aa74a742ef3fa1f7ed58f not found: ID does not exist" containerID="f87c1ec8f52beaaa4f9d63a1ec96e4d1889c7e71e71aa74a742ef3fa1f7ed58f"
Oct 31 19:57:57.239440 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:57.239389    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f87c1ec8f52beaaa4f9d63a1ec96e4d1889c7e71e71aa74a742ef3fa1f7ed58f"} err="failed to get container status \"f87c1ec8f52beaaa4f9d63a1ec96e4d1889c7e71e71aa74a742ef3fa1f7ed58f\": rpc error: code = NotFound desc = could not find container \"f87c1ec8f52beaaa4f9d63a1ec96e4d1889c7e71e71aa74a742ef3fa1f7ed58f\": container with ID starting with f87c1ec8f52beaaa4f9d63a1ec96e4d1889c7e71e71aa74a742ef3fa1f7ed58f not found: ID does not exist"
Oct 31 19:57:57.251771 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:57.251722    4645 patch_prober.go:29] interesting pod/e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" start-of-body=
Oct 31 19:57:57.251893 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:57.251770    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs" podUID="48ab68ad-2abf-44c8-98dc-c3ec72d36362" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused"
Oct 31 19:57:57.287140 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:57.287116    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:57.308021 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:57.307997    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/426c8fc3-1152-4a03-81e6-ada81e4d0007-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:57:57.308118 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:57.308024    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/426c8fc3-1152-4a03-81e6-ada81e4d0007-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:57:57.308118 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:57.308038    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/426c8fc3-1152-4a03-81e6-ada81e4d0007-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:57:57.308118 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:57.308051    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26rf9\" (UniqueName: \"kubernetes.io/projected/426c8fc3-1152-4a03-81e6-ada81e4d0007-kube-api-access-26rf9\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:57:57.308118 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:57.308064    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/426c8fc3-1152-4a03-81e6-ada81e4d0007-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:57:57.530153 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:57.530122    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s"]
Oct 31 19:57:57.534694 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:57.534669    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-5bbf75-openshift-default-7cf9cc5c65-p2x8s"]
Oct 31 19:57:57.626347 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:57.621825    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="426c8fc3-1152-4a03-81e6-ada81e4d0007" path="/var/lib/kubelet/pods/426c8fc3-1152-4a03-81e6-ada81e4d0007/volumes"
Oct 31 19:57:58.081402 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:58.079370    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:58.210732 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:58.210702    4645 generic.go:356] "Generic (PLEG): container finished" podID="76bc1f3a-74b3-49b3-bde1-cc3b305260d6" containerID="2f0c74238d8b1117dda0cd0bb0aada4ee7f7ac6057576ed3a76b0fc2162f754e" exitCode=0
Oct 31 19:57:58.211082 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:58.210797    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9th7f" event={"ID":"76bc1f3a-74b3-49b3-bde1-cc3b305260d6","Type":"ContainerDied","Data":"2f0c74238d8b1117dda0cd0bb0aada4ee7f7ac6057576ed3a76b0fc2162f754e"}
Oct 31 19:57:58.251354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:58.251306    4645 patch_prober.go:29] interesting pod/e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" start-of-body=
Oct 31 19:57:58.251522 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:58.251395    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs" podUID="48ab68ad-2abf-44c8-98dc-c3ec72d36362" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused"
Oct 31 19:57:58.334544 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:58.334518    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9th7f"
Oct 31 19:57:58.432829 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:58.432798    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76bc1f3a-74b3-49b3-bde1-cc3b305260d6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76bc1f3a-74b3-49b3-bde1-cc3b305260d6-credential-socket\") pod \"76bc1f3a-74b3-49b3-bde1-cc3b305260d6\" (UID: \"76bc1f3a-74b3-49b3-bde1-cc3b305260d6\") "
Oct 31 19:57:58.432829 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:58.432832    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76bc1f3a-74b3-49b3-bde1-cc3b305260d6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76bc1f3a-74b3-49b3-bde1-cc3b305260d6-workload-certs\") pod \"76bc1f3a-74b3-49b3-bde1-cc3b305260d6\" (UID: \"76bc1f3a-74b3-49b3-bde1-cc3b305260d6\") "
Oct 31 19:57:58.433059 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:58.432895    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/76bc1f3a-74b3-49b3-bde1-cc3b305260d6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76bc1f3a-74b3-49b3-bde1-cc3b305260d6-istio-podinfo\") pod \"76bc1f3a-74b3-49b3-bde1-cc3b305260d6\" (UID: \"76bc1f3a-74b3-49b3-bde1-cc3b305260d6\") "
Oct 31 19:57:58.433059 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:58.432931    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76bc1f3a-74b3-49b3-bde1-cc3b305260d6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76bc1f3a-74b3-49b3-bde1-cc3b305260d6-istiod-ca-cert\") pod \"76bc1f3a-74b3-49b3-bde1-cc3b305260d6\" (UID: \"76bc1f3a-74b3-49b3-bde1-cc3b305260d6\") "
Oct 31 19:57:58.433059 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:58.432970    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76bc1f3a-74b3-49b3-bde1-cc3b305260d6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76bc1f3a-74b3-49b3-bde1-cc3b305260d6-workload-socket\") pod \"76bc1f3a-74b3-49b3-bde1-cc3b305260d6\" (UID: \"76bc1f3a-74b3-49b3-bde1-cc3b305260d6\") "
Oct 31 19:57:58.433059 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:58.433016    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76bc1f3a-74b3-49b3-bde1-cc3b305260d6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76bc1f3a-74b3-49b3-bde1-cc3b305260d6-istio-envoy\") pod \"76bc1f3a-74b3-49b3-bde1-cc3b305260d6\" (UID: \"76bc1f3a-74b3-49b3-bde1-cc3b305260d6\") "
Oct 31 19:57:58.433059 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:58.433042    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76bc1f3a-74b3-49b3-bde1-cc3b305260d6-kube-api-access-v574w\" (UniqueName: \"kubernetes.io/projected/76bc1f3a-74b3-49b3-bde1-cc3b305260d6-kube-api-access-v574w\") pod \"76bc1f3a-74b3-49b3-bde1-cc3b305260d6\" (UID: \"76bc1f3a-74b3-49b3-bde1-cc3b305260d6\") "
Oct 31 19:57:58.433388 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:58.433091    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76bc1f3a-74b3-49b3-bde1-cc3b305260d6-credential-socket" pod "76bc1f3a-74b3-49b3-bde1-cc3b305260d6" (UID: "76bc1f3a-74b3-49b3-bde1-cc3b305260d6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:57:58.433388 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:58.433109    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76bc1f3a-74b3-49b3-bde1-cc3b305260d6-istio-token\" (UniqueName: \"kubernetes.io/projected/76bc1f3a-74b3-49b3-bde1-cc3b305260d6-istio-token\") pod \"76bc1f3a-74b3-49b3-bde1-cc3b305260d6\" (UID: \"76bc1f3a-74b3-49b3-bde1-cc3b305260d6\") "
Oct 31 19:57:58.433388 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:58.433150    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76bc1f3a-74b3-49b3-bde1-cc3b305260d6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76bc1f3a-74b3-49b3-bde1-cc3b305260d6-istio-data\") pod \"76bc1f3a-74b3-49b3-bde1-cc3b305260d6\" (UID: \"76bc1f3a-74b3-49b3-bde1-cc3b305260d6\") "
Oct 31 19:57:58.433388 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:58.433209    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76bc1f3a-74b3-49b3-bde1-cc3b305260d6-workload-certs" pod "76bc1f3a-74b3-49b3-bde1-cc3b305260d6" (UID: "76bc1f3a-74b3-49b3-bde1-cc3b305260d6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:57:58.433388 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:58.433380    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76bc1f3a-74b3-49b3-bde1-cc3b305260d6-workload-socket" pod "76bc1f3a-74b3-49b3-bde1-cc3b305260d6" (UID: "76bc1f3a-74b3-49b3-bde1-cc3b305260d6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:57:58.433742 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:58.433491    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76bc1f3a-74b3-49b3-bde1-cc3b305260d6-istio-data" pod "76bc1f3a-74b3-49b3-bde1-cc3b305260d6" (UID: "76bc1f3a-74b3-49b3-bde1-cc3b305260d6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:57:58.433742 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:58.433505    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76bc1f3a-74b3-49b3-bde1-cc3b305260d6-istiod-ca-cert" pod "76bc1f3a-74b3-49b3-bde1-cc3b305260d6" (UID: "76bc1f3a-74b3-49b3-bde1-cc3b305260d6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:57:58.434923 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:58.433907    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76bc1f3a-74b3-49b3-bde1-cc3b305260d6-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:57:58.434923 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:58.433933    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76bc1f3a-74b3-49b3-bde1-cc3b305260d6-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:57:58.434923 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:58.433947    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76bc1f3a-74b3-49b3-bde1-cc3b305260d6-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:57:58.434923 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:58.433959    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76bc1f3a-74b3-49b3-bde1-cc3b305260d6-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:57:58.434923 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:58.433972    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76bc1f3a-74b3-49b3-bde1-cc3b305260d6-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:57:58.435494 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:58.435473    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76bc1f3a-74b3-49b3-bde1-cc3b305260d6-istio-envoy" pod "76bc1f3a-74b3-49b3-bde1-cc3b305260d6" (UID: "76bc1f3a-74b3-49b3-bde1-cc3b305260d6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:57:58.435633 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:58.435530    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76bc1f3a-74b3-49b3-bde1-cc3b305260d6-kube-api-access-v574w" pod "76bc1f3a-74b3-49b3-bde1-cc3b305260d6" (UID: "76bc1f3a-74b3-49b3-bde1-cc3b305260d6"). InnerVolumeSpecName "kube-api-access-v574w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:57:58.435837 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:58.435811    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76bc1f3a-74b3-49b3-bde1-cc3b305260d6-istio-token" pod "76bc1f3a-74b3-49b3-bde1-cc3b305260d6" (UID: "76bc1f3a-74b3-49b3-bde1-cc3b305260d6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:57:58.435954 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:58.435887    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/76bc1f3a-74b3-49b3-bde1-cc3b305260d6-istio-podinfo" pod "76bc1f3a-74b3-49b3-bde1-cc3b305260d6" (UID: "76bc1f3a-74b3-49b3-bde1-cc3b305260d6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 19:57:58.535533 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:58.535502    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76bc1f3a-74b3-49b3-bde1-cc3b305260d6-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:57:58.535533 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:58.535531    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76bc1f3a-74b3-49b3-bde1-cc3b305260d6-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:57:58.535753 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:58.535540    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v574w\" (UniqueName: \"kubernetes.io/projected/76bc1f3a-74b3-49b3-bde1-cc3b305260d6-kube-api-access-v574w\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:57:58.535753 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:58.535551    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76bc1f3a-74b3-49b3-bde1-cc3b305260d6-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:57:58.854295 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:58.854256    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:59.218711 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:59.218684    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9th7f"
Oct 31 19:57:59.219231 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:59.218724    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9th7f" event={"ID":"76bc1f3a-74b3-49b3-bde1-cc3b305260d6","Type":"ContainerDied","Data":"74f01100ae9d6e4cd9d6ca79a35d7def9de1fc822c2afcaa12e4d3f15865572a"}
Oct 31 19:57:59.219231 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:59.218790    4645 scope.go:122] "RemoveContainer" containerID="2f0c74238d8b1117dda0cd0bb0aada4ee7f7ac6057576ed3a76b0fc2162f754e"
Oct 31 19:57:59.243772 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:59.243727    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9th7f"]
Oct 31 19:57:59.247731 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:59.247711    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-9th7f"]
Oct 31 19:57:59.250935 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:59.250916    4645 patch_prober.go:29] interesting pod/e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" start-of-body=
Oct 31 19:57:59.251001 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:59.250956    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs" podUID="48ab68ad-2abf-44c8-98dc-c3ec72d36362" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused"
Oct 31 19:57:59.599307 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:59.599214    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:57:59.621420 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:59.621373    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76bc1f3a-74b3-49b3-bde1-cc3b305260d6" path="/var/lib/kubelet/pods/76bc1f3a-74b3-49b3-bde1-cc3b305260d6/volumes"
Oct 31 19:57:59.712350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:57:59.710442    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:00.251349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:00.250754    4645 patch_prober.go:29] interesting pod/e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused" start-of-body=
Oct 31 19:58:00.251349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:00.250811    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs" podUID="48ab68ad-2abf-44c8-98dc-c3ec72d36362" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.224:15021/healthz/ready\": dial tcp 10.133.0.224:15021: connect: connection refused"
Oct 31 19:58:00.389305 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:00.389268    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:01.105369 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:01.105308    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:01.330191 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:01.330153    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs"
Oct 31 19:58:01.331299 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:01.331187    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs"
Oct 31 19:58:01.332489 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:01.332461    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs"
Oct 31 19:58:01.880066 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:01.880014    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:02.946720 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:02.946691    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:03.700545 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:03.700514    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:04.469071 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:04.469040    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:05.275196 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:05.275165    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:05.617082 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:05.616987    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:06.041279 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:06.041232    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n"]
Oct 31 19:58:06.042197 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:06.042170    4645 cpu_manager.go:414] "containerMap: removing container" podUID="426c8fc3-1152-4a03-81e6-ada81e4d0007" containerName="istio-proxy"
Oct 31 19:58:06.042197 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:06.042196    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="426c8fc3-1152-4a03-81e6-ada81e4d0007" containerName="istio-proxy"
Oct 31 19:58:06.042439 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:06.042235    4645 cpu_manager.go:414] "containerMap: removing container" podUID="aceb3312-c698-4f69-9b44-be9bd3ef0e6b" containerName="istio-proxy"
Oct 31 19:58:06.042439 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:06.042246    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aceb3312-c698-4f69-9b44-be9bd3ef0e6b" containerName="istio-proxy"
Oct 31 19:58:06.042439 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:06.042269    4645 cpu_manager.go:414] "containerMap: removing container" podUID="15ee340d-0b42-4985-a3bd-fb0ba596bc9d" containerName="istio-proxy"
Oct 31 19:58:06.042439 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:06.042277    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15ee340d-0b42-4985-a3bd-fb0ba596bc9d" containerName="istio-proxy"
Oct 31 19:58:06.042439 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:06.042308    4645 cpu_manager.go:414] "containerMap: removing container" podUID="76bc1f3a-74b3-49b3-bde1-cc3b305260d6" containerName="istio-proxy"
Oct 31 19:58:06.042439 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:06.042314    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76bc1f3a-74b3-49b3-bde1-cc3b305260d6" containerName="istio-proxy"
Oct 31 19:58:06.042439 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:06.042372    4645 cpu_manager.go:414] "containerMap: removing container" podUID="dad64038-a18c-4944-80c8-b3f052a98dbb" containerName="payload-pre-processing"
Oct 31 19:58:06.042439 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:06.042381    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dad64038-a18c-4944-80c8-b3f052a98dbb" containerName="payload-pre-processing"
Oct 31 19:58:06.042872 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:06.042564    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="426c8fc3-1152-4a03-81e6-ada81e4d0007" containerName="istio-proxy"
Oct 31 19:58:06.042872 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:06.042583    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="aceb3312-c698-4f69-9b44-be9bd3ef0e6b" containerName="istio-proxy"
Oct 31 19:58:06.042872 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:06.042597    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="15ee340d-0b42-4985-a3bd-fb0ba596bc9d" containerName="istio-proxy"
Oct 31 19:58:06.042872 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:06.042610    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="426c8fc3-1152-4a03-81e6-ada81e4d0007" containerName="istio-proxy"
Oct 31 19:58:06.042872 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:06.042620    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="76bc1f3a-74b3-49b3-bde1-cc3b305260d6" containerName="istio-proxy"
Oct 31 19:58:06.042872 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:06.042631    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="dad64038-a18c-4944-80c8-b3f052a98dbb" containerName="payload-pre-processing"
Oct 31 19:58:06.048400 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:06.048376    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n"
Oct 31 19:58:06.057892 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:06.057864    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-01afb5-openshift-default-dockercfg-czt6t\""
Oct 31 19:58:06.065047 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:06.065021    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n"]
Oct 31 19:58:06.083376 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:06.083350    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:06.092376 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:06.092323    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f54f6ce5-712d-430a-be69-8ec4a29e44d0-istiod-ca-cert\") pod \"e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n\" (UID: \"f54f6ce5-712d-430a-be69-8ec4a29e44d0\") " pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n"
Oct 31 19:58:06.092499 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:06.092476    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f54f6ce5-712d-430a-be69-8ec4a29e44d0-workload-certs\") pod \"e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n\" (UID: \"f54f6ce5-712d-430a-be69-8ec4a29e44d0\") " pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n"
Oct 31 19:58:06.092564 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:06.092530    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f54f6ce5-712d-430a-be69-8ec4a29e44d0-credential-socket\") pod \"e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n\" (UID: \"f54f6ce5-712d-430a-be69-8ec4a29e44d0\") " pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n"
Oct 31 19:58:06.092627 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:06.092582    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f54f6ce5-712d-430a-be69-8ec4a29e44d0-istio-envoy\") pod \"e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n\" (UID: \"f54f6ce5-712d-430a-be69-8ec4a29e44d0\") " pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n"
Oct 31 19:58:06.092627 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:06.092617    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f54f6ce5-712d-430a-be69-8ec4a29e44d0-workload-socket\") pod \"e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n\" (UID: \"f54f6ce5-712d-430a-be69-8ec4a29e44d0\") " pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n"
Oct 31 19:58:06.092741 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:06.092656    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f54f6ce5-712d-430a-be69-8ec4a29e44d0-istio-podinfo\") pod \"e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n\" (UID: \"f54f6ce5-712d-430a-be69-8ec4a29e44d0\") " pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n"
Oct 31 19:58:06.092838 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:06.092818    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfwzg\" (UniqueName: \"kubernetes.io/projected/f54f6ce5-712d-430a-be69-8ec4a29e44d0-kube-api-access-hfwzg\") pod \"e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n\" (UID: \"f54f6ce5-712d-430a-be69-8ec4a29e44d0\") " pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n"
Oct 31 19:58:06.092896 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:06.092875    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f54f6ce5-712d-430a-be69-8ec4a29e44d0-istio-data\") pod \"e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n\" (UID: \"f54f6ce5-712d-430a-be69-8ec4a29e44d0\") " pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n"
Oct 31 19:58:06.092955 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:06.092937    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f54f6ce5-712d-430a-be69-8ec4a29e44d0-istio-token\") pod \"e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n\" (UID: \"f54f6ce5-712d-430a-be69-8ec4a29e44d0\") " pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n"
Oct 31 19:58:06.197354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:06.197233    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f54f6ce5-712d-430a-be69-8ec4a29e44d0-workload-certs\") pod \"e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n\" (UID: \"f54f6ce5-712d-430a-be69-8ec4a29e44d0\") " pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n"
Oct 31 19:58:06.197354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:06.197342    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f54f6ce5-712d-430a-be69-8ec4a29e44d0-credential-socket\") pod \"e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n\" (UID: \"f54f6ce5-712d-430a-be69-8ec4a29e44d0\") " pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n"
Oct 31 19:58:06.197650 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:06.197423    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f54f6ce5-712d-430a-be69-8ec4a29e44d0-istio-envoy\") pod \"e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n\" (UID: \"f54f6ce5-712d-430a-be69-8ec4a29e44d0\") " pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n"
Oct 31 19:58:06.197650 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:06.197493    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f54f6ce5-712d-430a-be69-8ec4a29e44d0-workload-socket\") pod \"e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n\" (UID: \"f54f6ce5-712d-430a-be69-8ec4a29e44d0\") " pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n"
Oct 31 19:58:06.197650 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:06.197554    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f54f6ce5-712d-430a-be69-8ec4a29e44d0-istio-podinfo\") pod \"e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n\" (UID: \"f54f6ce5-712d-430a-be69-8ec4a29e44d0\") " pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n"
Oct 31 19:58:06.197650 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:06.197624    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfwzg\" (UniqueName: \"kubernetes.io/projected/f54f6ce5-712d-430a-be69-8ec4a29e44d0-kube-api-access-hfwzg\") pod \"e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n\" (UID: \"f54f6ce5-712d-430a-be69-8ec4a29e44d0\") " pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n"
Oct 31 19:58:06.197884 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:06.197686    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f54f6ce5-712d-430a-be69-8ec4a29e44d0-istio-data\") pod \"e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n\" (UID: \"f54f6ce5-712d-430a-be69-8ec4a29e44d0\") " pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n"
Oct 31 19:58:06.197884 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:06.197741    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f54f6ce5-712d-430a-be69-8ec4a29e44d0-istio-token\") pod \"e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n\" (UID: \"f54f6ce5-712d-430a-be69-8ec4a29e44d0\") " pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n"
Oct 31 19:58:06.220350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:06.210391    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f54f6ce5-712d-430a-be69-8ec4a29e44d0-istiod-ca-cert\") pod \"e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n\" (UID: \"f54f6ce5-712d-430a-be69-8ec4a29e44d0\") " pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n"
Oct 31 19:58:06.220350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:06.211249    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f54f6ce5-712d-430a-be69-8ec4a29e44d0-istiod-ca-cert\") pod \"e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n\" (UID: \"f54f6ce5-712d-430a-be69-8ec4a29e44d0\") " pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n"
Oct 31 19:58:06.220350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:06.211538    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f54f6ce5-712d-430a-be69-8ec4a29e44d0-workload-certs\") pod \"e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n\" (UID: \"f54f6ce5-712d-430a-be69-8ec4a29e44d0\") " pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n"
Oct 31 19:58:06.220350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:06.211747    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f54f6ce5-712d-430a-be69-8ec4a29e44d0-credential-socket\") pod \"e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n\" (UID: \"f54f6ce5-712d-430a-be69-8ec4a29e44d0\") " pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n"
Oct 31 19:58:06.220350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:06.212608    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f54f6ce5-712d-430a-be69-8ec4a29e44d0-workload-socket\") pod \"e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n\" (UID: \"f54f6ce5-712d-430a-be69-8ec4a29e44d0\") " pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n"
Oct 31 19:58:06.227345 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:06.225599    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f54f6ce5-712d-430a-be69-8ec4a29e44d0-istio-data\") pod \"e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n\" (UID: \"f54f6ce5-712d-430a-be69-8ec4a29e44d0\") " pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n"
Oct 31 19:58:06.241427 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:06.229880    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:06.249700 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:06.249668    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfwzg\" (UniqueName: \"kubernetes.io/projected/f54f6ce5-712d-430a-be69-8ec4a29e44d0-kube-api-access-hfwzg\") pod \"e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n\" (UID: \"f54f6ce5-712d-430a-be69-8ec4a29e44d0\") " pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n"
Oct 31 19:58:06.252084 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:06.252063    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f54f6ce5-712d-430a-be69-8ec4a29e44d0-istio-token\") pod \"e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n\" (UID: \"f54f6ce5-712d-430a-be69-8ec4a29e44d0\") " pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n"
Oct 31 19:58:06.261689 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:06.261635    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f54f6ce5-712d-430a-be69-8ec4a29e44d0-istio-envoy\") pod \"e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n\" (UID: \"f54f6ce5-712d-430a-be69-8ec4a29e44d0\") " pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n"
Oct 31 19:58:06.264362 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:06.264274    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f54f6ce5-712d-430a-be69-8ec4a29e44d0-istio-podinfo\") pod \"e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n\" (UID: \"f54f6ce5-712d-430a-be69-8ec4a29e44d0\") " pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n"
Oct 31 19:58:06.338610 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:58:06.338514    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 19:58:06.379921 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:06.379656    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n"
Oct 31 19:58:06.671643 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:06.671614    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:06.685826 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:58:06.685799    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf54f6ce5_712d_430a_be69_8ec4a29e44d0.slice/crio-a6be4e00341227506d05c169f9b766396d20c99da89adf05141a4ecee7aaf4d1 WatchSource:0}: Error finding container a6be4e00341227506d05c169f9b766396d20c99da89adf05141a4ecee7aaf4d1: Status 404 returned error can't find the container with id a6be4e00341227506d05c169f9b766396d20c99da89adf05141a4ecee7aaf4d1
Oct 31 19:58:06.686564 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:06.686543    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n"]
Oct 31 19:58:06.689031 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:06.688995    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:58:06.689151 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:06.689084    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:58:06.689151 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:06.689131    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:58:07.081162 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:07.081033    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:07.340042 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:07.339963    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n" event={"ID":"f54f6ce5-712d-430a-be69-8ec4a29e44d0","Type":"ContainerStarted","Data":"f7a48a62a3970b16f21af40bb7b30de5f7d063f23813c6ccd68033d4c6e884c9"}
Oct 31 19:58:07.340042 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:07.340024    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n" event={"ID":"f54f6ce5-712d-430a-be69-8ec4a29e44d0","Type":"ContainerStarted","Data":"a6be4e00341227506d05c169f9b766396d20c99da89adf05141a4ecee7aaf4d1"}
Oct 31 19:58:07.366165 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:07.366112    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n" podStartSLOduration=1.36609719 podStartE2EDuration="1.36609719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:58:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:58:07.363945004 +0000 UTC m=+2317.937179133" watchObservedRunningTime="2026-10-31 19:58:07.36609719 +0000 UTC m=+2317.939331312"
Oct 31 19:58:07.380384 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:07.380358    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n"
Oct 31 19:58:07.385918 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:07.385898    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n"
Oct 31 19:58:08.002993 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:08.002952    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:08.347305 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:08.347232    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n"
Oct 31 19:58:08.347969 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:08.347949    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n"
Oct 31 19:58:09.057563 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:09.057534    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:09.673570 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:09.673540    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:09.818511 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:09.818474    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:10.275856 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:10.275817    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:10.463985 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:10.463950    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-fd1c91/auth-test-model-fd1c91-kserve-55d79cc5b5-svsm4"]
Oct 31 19:58:10.504342 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:10.504296    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:11.398413 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:11.398368    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:11.518390 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:11.518356    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-01afb5-7956f95fdf-4sgpr"]
Oct 31 19:58:11.519286 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:11.519270    4645 cpu_manager.go:414] "containerMap: removing container" podUID="426c8fc3-1152-4a03-81e6-ada81e4d0007" containerName="istio-proxy"
Oct 31 19:58:11.519365 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:11.519291    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="426c8fc3-1152-4a03-81e6-ada81e4d0007" containerName="istio-proxy"
Oct 31 19:58:11.522689 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:11.522659    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:11.524918 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:11.524900    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-01afb5-7956f95fdf-4sgpr"
Oct 31 19:58:11.537213 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:11.537140    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-01afb5-7956f95fdf-4sgpr"]
Oct 31 19:58:11.615354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:11.608909    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-01afb5-dockercfg-9jzvm\""
Oct 31 19:58:11.615354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:11.609024    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-01afb5\""
Oct 31 19:58:11.758396 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:11.758317    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64qnt\" (UniqueName: \"kubernetes.io/projected/d3b92f72-3871-446f-bd45-f279f2e5d712-kube-api-access-64qnt\") pod \"payload-processing-e2e-ipp-b-01afb5-7956f95fdf-4sgpr\" (UID: \"d3b92f72-3871-446f-bd45-f279f2e5d712\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-01afb5-7956f95fdf-4sgpr"
Oct 31 19:58:11.759181 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:11.759086    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3b92f72-3871-446f-bd45-f279f2e5d712-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-01afb5-7956f95fdf-4sgpr\" (UID: \"d3b92f72-3871-446f-bd45-f279f2e5d712\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-01afb5-7956f95fdf-4sgpr"
Oct 31 19:58:11.860804 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:11.860766    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3b92f72-3871-446f-bd45-f279f2e5d712-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-01afb5-7956f95fdf-4sgpr\" (UID: \"d3b92f72-3871-446f-bd45-f279f2e5d712\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-01afb5-7956f95fdf-4sgpr"
Oct 31 19:58:11.861017 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:11.860979    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64qnt\" (UniqueName: \"kubernetes.io/projected/d3b92f72-3871-446f-bd45-f279f2e5d712-kube-api-access-64qnt\") pod \"payload-processing-e2e-ipp-b-01afb5-7956f95fdf-4sgpr\" (UID: \"d3b92f72-3871-446f-bd45-f279f2e5d712\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-01afb5-7956f95fdf-4sgpr"
Oct 31 19:58:11.861935 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:11.861908    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3b92f72-3871-446f-bd45-f279f2e5d712-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-01afb5-7956f95fdf-4sgpr\" (UID: \"d3b92f72-3871-446f-bd45-f279f2e5d712\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-01afb5-7956f95fdf-4sgpr"
Oct 31 19:58:11.873783 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:11.873754    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64qnt\" (UniqueName: \"kubernetes.io/projected/d3b92f72-3871-446f-bd45-f279f2e5d712-kube-api-access-64qnt\") pod \"payload-processing-e2e-ipp-b-01afb5-7956f95fdf-4sgpr\" (UID: \"d3b92f72-3871-446f-bd45-f279f2e5d712\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-01afb5-7956f95fdf-4sgpr"
Oct 31 19:58:12.150550 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:12.150467    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-01afb5-7956f95fdf-4sgpr"
Oct 31 19:58:12.352823 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:12.352795    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:12.449205 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:12.449174    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-01afb5-7956f95fdf-4sgpr"]
Oct 31 19:58:12.450594 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:58:12.450566    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3b92f72_3871_446f_bd45_f279f2e5d712.slice/crio-f325e82dfde38427e4787da6aef33c3e4cf5c914d234aee1945c72f32ebdad82 WatchSource:0}: Error finding container f325e82dfde38427e4787da6aef33c3e4cf5c914d234aee1945c72f32ebdad82: Status 404 returned error can't find the container with id f325e82dfde38427e4787da6aef33c3e4cf5c914d234aee1945c72f32ebdad82
Oct 31 19:58:13.393946 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:13.393909    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-01afb5-7956f95fdf-4sgpr" event={"ID":"d3b92f72-3871-446f-bd45-f279f2e5d712","Type":"ContainerStarted","Data":"4a59b465a65eb1ac3a42110ec801f2f8863183505ff3058a521a80efb5c6a11f"}
Oct 31 19:58:13.394132 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:13.394003    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-01afb5-7956f95fdf-4sgpr" event={"ID":"d3b92f72-3871-446f-bd45-f279f2e5d712","Type":"ContainerStarted","Data":"f325e82dfde38427e4787da6aef33c3e4cf5c914d234aee1945c72f32ebdad82"}
Oct 31 19:58:13.394132 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:13.394035    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-01afb5-7956f95fdf-4sgpr"
Oct 31 19:58:13.404318 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:13.404130    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:13.414424 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:13.414378    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-01afb5-7956f95fdf-4sgpr" podStartSLOduration=2.414361877 podStartE2EDuration="2.414361877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:58:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:58:13.41089451 +0000 UTC m=+2323.984128640" watchObservedRunningTime="2026-10-31 19:58:13.414361877 +0000 UTC m=+2323.987596007"
Oct 31 19:58:14.345515 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:14.345479    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:15.271526 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:15.271481    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:16.061292 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:16.061256    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:16.762819 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:16.762783    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:17.352389 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:17.352346    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:17.657823 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:17.657727    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:18.402975 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:18.402942    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:18.518964 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:18.518911    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:19.330653 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:19.330619    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:20.145994 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:20.145959    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:20.899502 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:20.899468    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-f5c1ab/auth-test-model-f5c1ab-kserve-77f7d9ccc-4ljhr"]
Oct 31 19:58:20.938208 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:20.938178    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:21.055354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:21.048300    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:21.838895 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:21.838862    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:21.953273 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:21.953240    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:22.691601 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:22.691565    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:22.901473 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:22.901432    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:23.071312 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:23.071231    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:24.151510 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:24.151470    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:24.426085 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:24.426057    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-01afb5-7956f95fdf-4sgpr"
Oct 31 19:58:24.936068 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:24.936031    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:25.486094 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:25.486037    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-fd1c91/auth-test-model-fd1c91-kserve-55d79cc5b5-svsm4" podUID="74a345a2-e822-4ba3-8fd9-0e8a41df4ffb" containerName="main" containerID="cri-o://ac4d000a63a29a7672c418ca586f440a375202969940eba9ae4d3cf6c8ba6bb3" gracePeriod=45
Oct 31 19:58:25.762173 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:25.762120    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:25.777921 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:25.777896    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-fd1c91/auth-test-model-fd1c91-kserve-55d79cc5b5-svsm4"
Oct 31 19:58:25.954899 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:25.954868    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74a345a2-e822-4ba3-8fd9-0e8a41df4ffb-kube-api-access-zkg86\" (UniqueName: \"kubernetes.io/projected/74a345a2-e822-4ba3-8fd9-0e8a41df4ffb-kube-api-access-zkg86\") pod \"74a345a2-e822-4ba3-8fd9-0e8a41df4ffb\" (UID: \"74a345a2-e822-4ba3-8fd9-0e8a41df4ffb\") "
Oct 31 19:58:25.955107 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:25.955078    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74a345a2-e822-4ba3-8fd9-0e8a41df4ffb-home\" (UniqueName: \"kubernetes.io/empty-dir/74a345a2-e822-4ba3-8fd9-0e8a41df4ffb-home\") pod \"74a345a2-e822-4ba3-8fd9-0e8a41df4ffb\" (UID: \"74a345a2-e822-4ba3-8fd9-0e8a41df4ffb\") "
Oct 31 19:58:25.955202 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:25.955170    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74a345a2-e822-4ba3-8fd9-0e8a41df4ffb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/74a345a2-e822-4ba3-8fd9-0e8a41df4ffb-dshm\") pod \"74a345a2-e822-4ba3-8fd9-0e8a41df4ffb\" (UID: \"74a345a2-e822-4ba3-8fd9-0e8a41df4ffb\") "
Oct 31 19:58:25.955267 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:25.955217    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74a345a2-e822-4ba3-8fd9-0e8a41df4ffb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74a345a2-e822-4ba3-8fd9-0e8a41df4ffb-model-cache\") pod \"74a345a2-e822-4ba3-8fd9-0e8a41df4ffb\" (UID: \"74a345a2-e822-4ba3-8fd9-0e8a41df4ffb\") "
Oct 31 19:58:25.955377 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:25.955360    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74a345a2-e822-4ba3-8fd9-0e8a41df4ffb-tls-certs\" (UniqueName: \"kubernetes.io/secret/74a345a2-e822-4ba3-8fd9-0e8a41df4ffb-tls-certs\") pod \"74a345a2-e822-4ba3-8fd9-0e8a41df4ffb\" (UID: \"74a345a2-e822-4ba3-8fd9-0e8a41df4ffb\") "
Oct 31 19:58:25.955462 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:25.955446    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74a345a2-e822-4ba3-8fd9-0e8a41df4ffb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74a345a2-e822-4ba3-8fd9-0e8a41df4ffb-tmp-dir\") pod \"74a345a2-e822-4ba3-8fd9-0e8a41df4ffb\" (UID: \"74a345a2-e822-4ba3-8fd9-0e8a41df4ffb\") "
Oct 31 19:58:25.955462 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:25.955442    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74a345a2-e822-4ba3-8fd9-0e8a41df4ffb-home" pod "74a345a2-e822-4ba3-8fd9-0e8a41df4ffb" (UID: "74a345a2-e822-4ba3-8fd9-0e8a41df4ffb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:58:25.955574 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:25.955547    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74a345a2-e822-4ba3-8fd9-0e8a41df4ffb-model-cache" pod "74a345a2-e822-4ba3-8fd9-0e8a41df4ffb" (UID: "74a345a2-e822-4ba3-8fd9-0e8a41df4ffb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:58:25.955792 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:25.955767    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74a345a2-e822-4ba3-8fd9-0e8a41df4ffb-tmp-dir" pod "74a345a2-e822-4ba3-8fd9-0e8a41df4ffb" (UID: "74a345a2-e822-4ba3-8fd9-0e8a41df4ffb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:58:25.956251 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:25.956211    4645 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74a345a2-e822-4ba3-8fd9-0e8a41df4ffb-tmp-dir\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:58:25.956251 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:25.956237    4645 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74a345a2-e822-4ba3-8fd9-0e8a41df4ffb-home\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:58:25.956251 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:25.956250    4645 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74a345a2-e822-4ba3-8fd9-0e8a41df4ffb-model-cache\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:58:25.957550 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:25.957524    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74a345a2-e822-4ba3-8fd9-0e8a41df4ffb-tls-certs" pod "74a345a2-e822-4ba3-8fd9-0e8a41df4ffb" (UID: "74a345a2-e822-4ba3-8fd9-0e8a41df4ffb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:58:25.957662 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:25.957618    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74a345a2-e822-4ba3-8fd9-0e8a41df4ffb-dshm" pod "74a345a2-e822-4ba3-8fd9-0e8a41df4ffb" (UID: "74a345a2-e822-4ba3-8fd9-0e8a41df4ffb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:58:25.957662 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:25.957636    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74a345a2-e822-4ba3-8fd9-0e8a41df4ffb-kube-api-access-zkg86" pod "74a345a2-e822-4ba3-8fd9-0e8a41df4ffb" (UID: "74a345a2-e822-4ba3-8fd9-0e8a41df4ffb"). InnerVolumeSpecName "kube-api-access-zkg86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:58:26.033227 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:26.033136    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:26.057564 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:26.057525    4645 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74a345a2-e822-4ba3-8fd9-0e8a41df4ffb-dshm\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:58:26.057564 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:26.057563    4645 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74a345a2-e822-4ba3-8fd9-0e8a41df4ffb-tls-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:58:26.057769 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:26.057579    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkg86\" (UniqueName: \"kubernetes.io/projected/74a345a2-e822-4ba3-8fd9-0e8a41df4ffb-kube-api-access-zkg86\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:58:26.155759 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:26.155723    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:26.593261 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:26.593225    4645 generic.go:356] "Generic (PLEG): container finished" podID="74a345a2-e822-4ba3-8fd9-0e8a41df4ffb" containerID="ac4d000a63a29a7672c418ca586f440a375202969940eba9ae4d3cf6c8ba6bb3" exitCode=0
Oct 31 19:58:26.593698 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:26.593268    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-fd1c91/auth-test-model-fd1c91-kserve-55d79cc5b5-svsm4"
Oct 31 19:58:26.593698 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:26.593304    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-fd1c91/auth-test-model-fd1c91-kserve-55d79cc5b5-svsm4" event={"ID":"74a345a2-e822-4ba3-8fd9-0e8a41df4ffb","Type":"ContainerDied","Data":"ac4d000a63a29a7672c418ca586f440a375202969940eba9ae4d3cf6c8ba6bb3"}
Oct 31 19:58:26.593698 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:26.593361    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-fd1c91/auth-test-model-fd1c91-kserve-55d79cc5b5-svsm4" event={"ID":"74a345a2-e822-4ba3-8fd9-0e8a41df4ffb","Type":"ContainerDied","Data":"298d829391f260a4221c2e879f887d662e0893fa41751e23cfa58b6214d71ae3"}
Oct 31 19:58:26.593698 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:26.593382    4645 scope.go:122] "RemoveContainer" containerID="ac4d000a63a29a7672c418ca586f440a375202969940eba9ae4d3cf6c8ba6bb3"
Oct 31 19:58:26.604747 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:26.604731    4645 scope.go:122] "RemoveContainer" containerID="ac4d000a63a29a7672c418ca586f440a375202969940eba9ae4d3cf6c8ba6bb3"
Oct 31 19:58:26.605003 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:58:26.604982    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac4d000a63a29a7672c418ca586f440a375202969940eba9ae4d3cf6c8ba6bb3\": container with ID starting with ac4d000a63a29a7672c418ca586f440a375202969940eba9ae4d3cf6c8ba6bb3 not found: ID does not exist" containerID="ac4d000a63a29a7672c418ca586f440a375202969940eba9ae4d3cf6c8ba6bb3"
Oct 31 19:58:26.605088 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:26.605015    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac4d000a63a29a7672c418ca586f440a375202969940eba9ae4d3cf6c8ba6bb3"} err="failed to get container status \"ac4d000a63a29a7672c418ca586f440a375202969940eba9ae4d3cf6c8ba6bb3\": rpc error: code = NotFound desc = could not find container \"ac4d000a63a29a7672c418ca586f440a375202969940eba9ae4d3cf6c8ba6bb3\": container with ID starting with ac4d000a63a29a7672c418ca586f440a375202969940eba9ae4d3cf6c8ba6bb3 not found: ID does not exist"
Oct 31 19:58:26.617047 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:26.617024    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-fd1c91/auth-test-model-fd1c91-kserve-55d79cc5b5-svsm4"]
Oct 31 19:58:26.627097 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:26.627077    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-fd1c91/auth-test-model-fd1c91-kserve-55d79cc5b5-svsm4"]
Oct 31 19:58:26.859977 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:26.859891    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:27.310710 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:27.310670    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:27.376503 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:27.376469    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-26cd59-554cf8bc5f-hfrgd"]
Oct 31 19:58:27.377161 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:27.377147    4645 cpu_manager.go:414] "containerMap: removing container" podUID="74a345a2-e822-4ba3-8fd9-0e8a41df4ffb" containerName="main"
Oct 31 19:58:27.377225 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:27.377163    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74a345a2-e822-4ba3-8fd9-0e8a41df4ffb" containerName="main"
Oct 31 19:58:27.377304 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:27.377294    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="74a345a2-e822-4ba3-8fd9-0e8a41df4ffb" containerName="main"
Oct 31 19:58:27.382089 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:27.382066    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-26cd59-554cf8bc5f-hfrgd"
Oct 31 19:58:27.391692 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:27.391669    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-26cd59\""
Oct 31 19:58:27.392029 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:27.392009    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-26cd59-dockercfg-md5zh\""
Oct 31 19:58:27.403396 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:27.400810    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-26cd59-554cf8bc5f-hfrgd"]
Oct 31 19:58:27.432882 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:27.432852    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-26cd59-79c7d4d9c9-j7z8j"]
Oct 31 19:58:27.438275 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:27.438252    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-26cd59-79c7d4d9c9-j7z8j"
Oct 31 19:58:27.448219 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:27.448089    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-26cd59-79c7d4d9c9-j7z8j"]
Oct 31 19:58:27.481429 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:27.481382    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7tkq\" (UniqueName: \"kubernetes.io/projected/28db49c5-ce35-4752-8e9e-11fd054e4884-kube-api-access-j7tkq\") pod \"payload-pre-processing-e2e-ipp-a-26cd59-554cf8bc5f-hfrgd\" (UID: \"28db49c5-ce35-4752-8e9e-11fd054e4884\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-26cd59-554cf8bc5f-hfrgd"
Oct 31 19:58:27.481648 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:27.481574    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dcnm\" (UniqueName: \"kubernetes.io/projected/29c8935f-7368-41b3-9b04-febe7848fd41-kube-api-access-7dcnm\") pod \"payload-processing-e2e-ipp-a-26cd59-79c7d4d9c9-j7z8j\" (UID: \"29c8935f-7368-41b3-9b04-febe7848fd41\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-26cd59-79c7d4d9c9-j7z8j"
Oct 31 19:58:27.481756 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:27.481731    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29c8935f-7368-41b3-9b04-febe7848fd41-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-26cd59-79c7d4d9c9-j7z8j\" (UID: \"29c8935f-7368-41b3-9b04-febe7848fd41\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-26cd59-79c7d4d9c9-j7z8j"
Oct 31 19:58:27.481883 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:27.481859    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28db49c5-ce35-4752-8e9e-11fd054e4884-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-26cd59-554cf8bc5f-hfrgd\" (UID: \"28db49c5-ce35-4752-8e9e-11fd054e4884\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-26cd59-554cf8bc5f-hfrgd"
Oct 31 19:58:27.490178 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:27.490145    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:27.601408 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:27.588431    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7tkq\" (UniqueName: \"kubernetes.io/projected/28db49c5-ce35-4752-8e9e-11fd054e4884-kube-api-access-j7tkq\") pod \"payload-pre-processing-e2e-ipp-a-26cd59-554cf8bc5f-hfrgd\" (UID: \"28db49c5-ce35-4752-8e9e-11fd054e4884\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-26cd59-554cf8bc5f-hfrgd"
Oct 31 19:58:27.601408 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:27.588566    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dcnm\" (UniqueName: \"kubernetes.io/projected/29c8935f-7368-41b3-9b04-febe7848fd41-kube-api-access-7dcnm\") pod \"payload-processing-e2e-ipp-a-26cd59-79c7d4d9c9-j7z8j\" (UID: \"29c8935f-7368-41b3-9b04-febe7848fd41\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-26cd59-79c7d4d9c9-j7z8j"
Oct 31 19:58:27.601408 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:27.588688    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29c8935f-7368-41b3-9b04-febe7848fd41-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-26cd59-79c7d4d9c9-j7z8j\" (UID: \"29c8935f-7368-41b3-9b04-febe7848fd41\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-26cd59-79c7d4d9c9-j7z8j"
Oct 31 19:58:27.601408 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:27.588777    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28db49c5-ce35-4752-8e9e-11fd054e4884-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-26cd59-554cf8bc5f-hfrgd\" (UID: \"28db49c5-ce35-4752-8e9e-11fd054e4884\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-26cd59-554cf8bc5f-hfrgd"
Oct 31 19:58:27.601408 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:27.589708    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28db49c5-ce35-4752-8e9e-11fd054e4884-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-26cd59-554cf8bc5f-hfrgd\" (UID: \"28db49c5-ce35-4752-8e9e-11fd054e4884\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-26cd59-554cf8bc5f-hfrgd"
Oct 31 19:58:27.601408 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:27.590875    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29c8935f-7368-41b3-9b04-febe7848fd41-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-26cd59-79c7d4d9c9-j7z8j\" (UID: \"29c8935f-7368-41b3-9b04-febe7848fd41\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-26cd59-79c7d4d9c9-j7z8j"
Oct 31 19:58:27.618348 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:27.615042    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dcnm\" (UniqueName: \"kubernetes.io/projected/29c8935f-7368-41b3-9b04-febe7848fd41-kube-api-access-7dcnm\") pod \"payload-processing-e2e-ipp-a-26cd59-79c7d4d9c9-j7z8j\" (UID: \"29c8935f-7368-41b3-9b04-febe7848fd41\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-26cd59-79c7d4d9c9-j7z8j"
Oct 31 19:58:27.653356 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:27.650050    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7tkq\" (UniqueName: \"kubernetes.io/projected/28db49c5-ce35-4752-8e9e-11fd054e4884-kube-api-access-j7tkq\") pod \"payload-pre-processing-e2e-ipp-a-26cd59-554cf8bc5f-hfrgd\" (UID: \"28db49c5-ce35-4752-8e9e-11fd054e4884\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-26cd59-554cf8bc5f-hfrgd"
Oct 31 19:58:27.707469 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:27.707436    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-26cd59-554cf8bc5f-hfrgd"
Oct 31 19:58:27.749609 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:27.749453    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74a345a2-e822-4ba3-8fd9-0e8a41df4ffb" path="/var/lib/kubelet/pods/74a345a2-e822-4ba3-8fd9-0e8a41df4ffb/volumes"
Oct 31 19:58:27.758718 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:27.758637    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-26cd59-79c7d4d9c9-j7z8j"
Oct 31 19:58:27.937113 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:27.937075    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:28.067236 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:28.067201    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-26cd59-554cf8bc5f-hfrgd"]
Oct 31 19:58:28.069608 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:58:28.069454    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28db49c5_ce35_4752_8e9e_11fd054e4884.slice/crio-adc2ee8480beba2726414b7cb1689a87c3c880b25ea235e516f6454dfea1048e WatchSource:0}: Error finding container adc2ee8480beba2726414b7cb1689a87c3c880b25ea235e516f6454dfea1048e: Status 404 returned error can't find the container with id adc2ee8480beba2726414b7cb1689a87c3c880b25ea235e516f6454dfea1048e
Oct 31 19:58:28.071452 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:58:28.071425    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29c8935f_7368_41b3_9b04_febe7848fd41.slice/crio-ce5a8b1b82f8ef67cbb3f184c7627ea6cf638c3a32394a03c99e259a8346b2a5 WatchSource:0}: Error finding container ce5a8b1b82f8ef67cbb3f184c7627ea6cf638c3a32394a03c99e259a8346b2a5: Status 404 returned error can't find the container with id ce5a8b1b82f8ef67cbb3f184c7627ea6cf638c3a32394a03c99e259a8346b2a5
Oct 31 19:58:28.074072 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:28.074049    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-26cd59-79c7d4d9c9-j7z8j"]
Oct 31 19:58:28.715501 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:28.715463    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-26cd59-79c7d4d9c9-j7z8j" event={"ID":"29c8935f-7368-41b3-9b04-febe7848fd41","Type":"ContainerStarted","Data":"f5a7416fce3dd9ac31222f9a7274cb7fb000a4cc9bbe6ba13aadb7db4366fdd7"}
Oct 31 19:58:28.716010 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:28.715525    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-26cd59-79c7d4d9c9-j7z8j" event={"ID":"29c8935f-7368-41b3-9b04-febe7848fd41","Type":"ContainerStarted","Data":"ce5a8b1b82f8ef67cbb3f184c7627ea6cf638c3a32394a03c99e259a8346b2a5"}
Oct 31 19:58:28.716010 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:28.715553    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-26cd59-79c7d4d9c9-j7z8j"
Oct 31 19:58:28.717435 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:28.717404    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-26cd59-554cf8bc5f-hfrgd" event={"ID":"28db49c5-ce35-4752-8e9e-11fd054e4884","Type":"ContainerStarted","Data":"ecc3a2c7dddafc62fe88a0dea019adff03ee5b1f8a7a89917487c455745321e7"}
Oct 31 19:58:28.717599 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:28.717448    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-26cd59-554cf8bc5f-hfrgd" event={"ID":"28db49c5-ce35-4752-8e9e-11fd054e4884","Type":"ContainerStarted","Data":"adc2ee8480beba2726414b7cb1689a87c3c880b25ea235e516f6454dfea1048e"}
Oct 31 19:58:28.717599 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:28.717470    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-26cd59-554cf8bc5f-hfrgd"
Oct 31 19:58:28.736554 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:28.736510    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-26cd59-79c7d4d9c9-j7z8j" podStartSLOduration=1.736496903 podStartE2EDuration="1.736496903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:58:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:58:28.732869948 +0000 UTC m=+2339.306104076" watchObservedRunningTime="2026-10-31 19:58:28.736496903 +0000 UTC m=+2339.309731031"
Oct 31 19:58:28.759726 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:28.759678    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-26cd59-554cf8bc5f-hfrgd" podStartSLOduration=1.7596628829999998 podStartE2EDuration="1.759662883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:58:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:58:28.758428924 +0000 UTC m=+2339.331663052" watchObservedRunningTime="2026-10-31 19:58:28.759662883 +0000 UTC m=+2339.332897013"
Oct 31 19:58:28.869222 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:28.869178    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:28.991522 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:28.991386    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:29.241970 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:29.241875    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:29.904637 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:29.904599    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:30.708437 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:30.708399    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:31.466277 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:31.466249    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:32.317267 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:32.317228    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:33.146493 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:33.146452    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:33.878396 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:33.878360    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:34.640861 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:34.640821    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:35.348684 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:35.348646    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:35.922607 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:35.922565    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-f5c1ab/auth-test-model-f5c1ab-kserve-77f7d9ccc-4ljhr" podUID="d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b" containerName="main" containerID="cri-o://a0b2d0bbde02722938141bebe4836a59e1776fd4bd726caa9d65ae7ab6c3896d" gracePeriod=45
Oct 31 19:58:36.111038 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:36.110991    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:36.249972 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:58:36.249940    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 19:58:36.277665 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:36.277628    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-f5c1ab/auth-test-model-f5c1ab-kserve-77f7d9ccc-4ljhr"
Oct 31 19:58:36.301439 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:36.301407    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b-model-cache\") pod \"d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b\" (UID: \"d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b\") "
Oct 31 19:58:36.301604 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:36.301545    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b-home\" (UniqueName: \"kubernetes.io/empty-dir/d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b-home\") pod \"d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b\" (UID: \"d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b\") "
Oct 31 19:58:36.301758 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:36.301738    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b-dshm\") pod \"d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b\" (UID: \"d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b\") "
Oct 31 19:58:36.302014 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:36.301991    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b-model-cache" pod "d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b" (UID: "d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:58:36.302116 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:36.302013    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b-home" pod "d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b" (UID: "d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:58:36.302203 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:36.302003    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b-tmp-dir\") pod \"d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b\" (UID: \"d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b\") "
Oct 31 19:58:36.302317 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:36.302294    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b-tmp-dir" pod "d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b" (UID: "d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:58:36.302516 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:36.302493    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b-kube-api-access-q229g\" (UniqueName: \"kubernetes.io/projected/d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b-kube-api-access-q229g\") pod \"d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b\" (UID: \"d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b\") "
Oct 31 19:58:36.302614 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:36.302594    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b-tls-certs\" (UniqueName: \"kubernetes.io/secret/d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b-tls-certs\") pod \"d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b\" (UID: \"d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b\") "
Oct 31 19:58:36.303473 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:36.303447    4645 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b-tmp-dir\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:58:36.303590 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:36.303478    4645 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b-model-cache\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:58:36.303590 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:36.303493    4645 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b-home\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:58:36.304779 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:36.304751    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b-dshm" pod "d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b" (UID: "d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:58:36.305522 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:36.305500    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b-tls-certs" pod "d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b" (UID: "d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:58:36.305612 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:36.305541    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b-kube-api-access-q229g" pod "d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b" (UID: "d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b"). InnerVolumeSpecName "kube-api-access-q229g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:58:36.404816 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:36.404770    4645 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b-dshm\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:58:36.405005 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:36.404823    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q229g\" (UniqueName: \"kubernetes.io/projected/d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b-kube-api-access-q229g\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:58:36.405005 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:36.404838    4645 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b-tls-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:58:36.774227 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:36.774186    4645 generic.go:356] "Generic (PLEG): container finished" podID="d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b" containerID="a0b2d0bbde02722938141bebe4836a59e1776fd4bd726caa9d65ae7ab6c3896d" exitCode=0
Oct 31 19:58:36.774444 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:36.774247    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-f5c1ab/auth-test-model-f5c1ab-kserve-77f7d9ccc-4ljhr"
Oct 31 19:58:36.774444 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:36.774270    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-f5c1ab/auth-test-model-f5c1ab-kserve-77f7d9ccc-4ljhr" event={"ID":"d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b","Type":"ContainerDied","Data":"a0b2d0bbde02722938141bebe4836a59e1776fd4bd726caa9d65ae7ab6c3896d"}
Oct 31 19:58:36.774444 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:36.774358    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-f5c1ab/auth-test-model-f5c1ab-kserve-77f7d9ccc-4ljhr" event={"ID":"d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b","Type":"ContainerDied","Data":"8aa949bf63335ac9ad85a2b7fc2dbd51eb0988be8535055a3f394ed444d6bc31"}
Oct 31 19:58:36.774444 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:36.774365    4645 scope.go:122] "RemoveContainer" containerID="a0b2d0bbde02722938141bebe4836a59e1776fd4bd726caa9d65ae7ab6c3896d"
Oct 31 19:58:36.787091 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:36.787067    4645 scope.go:122] "RemoveContainer" containerID="a0b2d0bbde02722938141bebe4836a59e1776fd4bd726caa9d65ae7ab6c3896d"
Oct 31 19:58:36.787410 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:58:36.787380    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0b2d0bbde02722938141bebe4836a59e1776fd4bd726caa9d65ae7ab6c3896d\": container with ID starting with a0b2d0bbde02722938141bebe4836a59e1776fd4bd726caa9d65ae7ab6c3896d not found: ID does not exist" containerID="a0b2d0bbde02722938141bebe4836a59e1776fd4bd726caa9d65ae7ab6c3896d"
Oct 31 19:58:36.787492 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:36.787421    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0b2d0bbde02722938141bebe4836a59e1776fd4bd726caa9d65ae7ab6c3896d"} err="failed to get container status \"a0b2d0bbde02722938141bebe4836a59e1776fd4bd726caa9d65ae7ab6c3896d\": rpc error: code = NotFound desc = could not find container \"a0b2d0bbde02722938141bebe4836a59e1776fd4bd726caa9d65ae7ab6c3896d\": container with ID starting with a0b2d0bbde02722938141bebe4836a59e1776fd4bd726caa9d65ae7ab6c3896d not found: ID does not exist"
Oct 31 19:58:36.802408 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:36.802378    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-f5c1ab/auth-test-model-f5c1ab-kserve-77f7d9ccc-4ljhr"]
Oct 31 19:58:36.806182 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:36.806162    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-f5c1ab/auth-test-model-f5c1ab-kserve-77f7d9ccc-4ljhr"]
Oct 31 19:58:36.927999 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:36.927959    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:37.620653 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:37.620617    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b" path="/var/lib/kubelet/pods/d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b/volumes"
Oct 31 19:58:37.741713 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:37.741685    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:38.438174 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:38.438133    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:39.166547 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:39.166516    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-cd9729/auth-test-model-cd9729-kserve-65f895f85c-pcprs"]
Oct 31 19:58:39.168945 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:39.168915    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:39.393171 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:39.393132    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:39.726664 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:39.726639    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-26cd59-554cf8bc5f-hfrgd"
Oct 31 19:58:39.727122 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:39.726704    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-26cd59-79c7d4d9c9-j7z8j"
Oct 31 19:58:40.144438 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:40.144316    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:41.082358 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:41.073201    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:41.870972 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:41.870938    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:42.683366 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:42.671980    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:42.698217 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:42.698185    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-435728/auth-test-model-435728-kserve-76df7cb4d9-t5bn2"]
Oct 31 19:58:42.793365 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:42.791372    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:43.555904 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:43.555864    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:43.711419 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:43.704607    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:44.541650 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:44.541611    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:45.391919 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:45.391882    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:46.136712 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:46.136679    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:46.946255 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:46.946215    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:47.843553 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:47.843519    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:48.654055 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:48.654010    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:49.429484 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:49.429452    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:50.158921 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:50.158890    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:50.387987 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:50.387955    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-01afb5-7956f95fdf-4sgpr"]
Oct 31 19:58:50.471691 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:50.471661    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:50.575465 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:50.575430    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:50.740408 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:50.740301    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:51.132000 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:51.131903    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:51.957869 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:51.957833    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:52.914788 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:52.914747    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:53.757432 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:53.757398    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:54.188447 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:54.188394    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-cd9729/auth-test-model-cd9729-kserve-65f895f85c-pcprs" podUID="96edc0b2-28d0-4058-8aff-c0e179fe5f2e" containerName="main" containerID="cri-o://18e085ad912a1794d78b216d685983baa6868317b6299200b768d5b15dc54cec" gracePeriod=45
Oct 31 19:58:54.475623 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:54.475595    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-cd9729/auth-test-model-cd9729-kserve-65f895f85c-pcprs"
Oct 31 19:58:54.524978 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:54.524949    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96edc0b2-28d0-4058-8aff-c0e179fe5f2e-kube-api-access-rqcgj\" (UniqueName: \"kubernetes.io/projected/96edc0b2-28d0-4058-8aff-c0e179fe5f2e-kube-api-access-rqcgj\") pod \"96edc0b2-28d0-4058-8aff-c0e179fe5f2e\" (UID: \"96edc0b2-28d0-4058-8aff-c0e179fe5f2e\") "
Oct 31 19:58:54.525123 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:54.525020    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96edc0b2-28d0-4058-8aff-c0e179fe5f2e-tls-certs\" (UniqueName: \"kubernetes.io/secret/96edc0b2-28d0-4058-8aff-c0e179fe5f2e-tls-certs\") pod \"96edc0b2-28d0-4058-8aff-c0e179fe5f2e\" (UID: \"96edc0b2-28d0-4058-8aff-c0e179fe5f2e\") "
Oct 31 19:58:54.525123 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:54.525074    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96edc0b2-28d0-4058-8aff-c0e179fe5f2e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96edc0b2-28d0-4058-8aff-c0e179fe5f2e-model-cache\") pod \"96edc0b2-28d0-4058-8aff-c0e179fe5f2e\" (UID: \"96edc0b2-28d0-4058-8aff-c0e179fe5f2e\") "
Oct 31 19:58:54.525123 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:54.525100    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96edc0b2-28d0-4058-8aff-c0e179fe5f2e-home\" (UniqueName: \"kubernetes.io/empty-dir/96edc0b2-28d0-4058-8aff-c0e179fe5f2e-home\") pod \"96edc0b2-28d0-4058-8aff-c0e179fe5f2e\" (UID: \"96edc0b2-28d0-4058-8aff-c0e179fe5f2e\") "
Oct 31 19:58:54.525304 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:54.525277    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96edc0b2-28d0-4058-8aff-c0e179fe5f2e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/96edc0b2-28d0-4058-8aff-c0e179fe5f2e-dshm\") pod \"96edc0b2-28d0-4058-8aff-c0e179fe5f2e\" (UID: \"96edc0b2-28d0-4058-8aff-c0e179fe5f2e\") "
Oct 31 19:58:54.525412 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:54.525377    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96edc0b2-28d0-4058-8aff-c0e179fe5f2e-model-cache" pod "96edc0b2-28d0-4058-8aff-c0e179fe5f2e" (UID: "96edc0b2-28d0-4058-8aff-c0e179fe5f2e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:58:54.525490 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:54.525441    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96edc0b2-28d0-4058-8aff-c0e179fe5f2e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96edc0b2-28d0-4058-8aff-c0e179fe5f2e-tmp-dir\") pod \"96edc0b2-28d0-4058-8aff-c0e179fe5f2e\" (UID: \"96edc0b2-28d0-4058-8aff-c0e179fe5f2e\") "
Oct 31 19:58:54.525655 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:54.525607    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96edc0b2-28d0-4058-8aff-c0e179fe5f2e-home" pod "96edc0b2-28d0-4058-8aff-c0e179fe5f2e" (UID: "96edc0b2-28d0-4058-8aff-c0e179fe5f2e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:58:54.525830 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:54.525806    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96edc0b2-28d0-4058-8aff-c0e179fe5f2e-tmp-dir" pod "96edc0b2-28d0-4058-8aff-c0e179fe5f2e" (UID: "96edc0b2-28d0-4058-8aff-c0e179fe5f2e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:58:54.526186 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:54.526161    4645 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96edc0b2-28d0-4058-8aff-c0e179fe5f2e-tmp-dir\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:58:54.526304 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:54.526190    4645 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96edc0b2-28d0-4058-8aff-c0e179fe5f2e-model-cache\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:58:54.526304 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:54.526205    4645 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96edc0b2-28d0-4058-8aff-c0e179fe5f2e-home\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:58:54.527678 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:54.527651    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96edc0b2-28d0-4058-8aff-c0e179fe5f2e-dshm" pod "96edc0b2-28d0-4058-8aff-c0e179fe5f2e" (UID: "96edc0b2-28d0-4058-8aff-c0e179fe5f2e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:58:54.527858 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:54.527833    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96edc0b2-28d0-4058-8aff-c0e179fe5f2e-tls-certs" pod "96edc0b2-28d0-4058-8aff-c0e179fe5f2e" (UID: "96edc0b2-28d0-4058-8aff-c0e179fe5f2e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:58:54.527929 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:54.527873    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96edc0b2-28d0-4058-8aff-c0e179fe5f2e-kube-api-access-rqcgj" pod "96edc0b2-28d0-4058-8aff-c0e179fe5f2e" (UID: "96edc0b2-28d0-4058-8aff-c0e179fe5f2e"). InnerVolumeSpecName "kube-api-access-rqcgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:58:54.592579 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:54.592552    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:54.639115 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:54.639015    4645 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96edc0b2-28d0-4058-8aff-c0e179fe5f2e-dshm\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:58:54.639115 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:54.639063    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqcgj\" (UniqueName: \"kubernetes.io/projected/96edc0b2-28d0-4058-8aff-c0e179fe5f2e-kube-api-access-rqcgj\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:58:54.639115 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:54.639080    4645 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96edc0b2-28d0-4058-8aff-c0e179fe5f2e-tls-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:58:54.993085 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:54.993045    4645 generic.go:356] "Generic (PLEG): container finished" podID="96edc0b2-28d0-4058-8aff-c0e179fe5f2e" containerID="18e085ad912a1794d78b216d685983baa6868317b6299200b768d5b15dc54cec" exitCode=0
Oct 31 19:58:54.993600 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:54.993096    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-cd9729/auth-test-model-cd9729-kserve-65f895f85c-pcprs"
Oct 31 19:58:54.993600 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:54.993099    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-cd9729/auth-test-model-cd9729-kserve-65f895f85c-pcprs" event={"ID":"96edc0b2-28d0-4058-8aff-c0e179fe5f2e","Type":"ContainerDied","Data":"18e085ad912a1794d78b216d685983baa6868317b6299200b768d5b15dc54cec"}
Oct 31 19:58:54.993600 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:54.993211    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-cd9729/auth-test-model-cd9729-kserve-65f895f85c-pcprs" event={"ID":"96edc0b2-28d0-4058-8aff-c0e179fe5f2e","Type":"ContainerDied","Data":"5d5ba015191ee8ae81c5cb298a225bcd10b7aaa7c500f7f367b350d8d2cd7a84"}
Oct 31 19:58:54.993600 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:54.993232    4645 scope.go:122] "RemoveContainer" containerID="18e085ad912a1794d78b216d685983baa6868317b6299200b768d5b15dc54cec"
Oct 31 19:58:55.004411 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:55.004389    4645 scope.go:122] "RemoveContainer" containerID="18e085ad912a1794d78b216d685983baa6868317b6299200b768d5b15dc54cec"
Oct 31 19:58:55.004685 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:58:55.004669    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18e085ad912a1794d78b216d685983baa6868317b6299200b768d5b15dc54cec\": container with ID starting with 18e085ad912a1794d78b216d685983baa6868317b6299200b768d5b15dc54cec not found: ID does not exist" containerID="18e085ad912a1794d78b216d685983baa6868317b6299200b768d5b15dc54cec"
Oct 31 19:58:55.004736 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:55.004693    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18e085ad912a1794d78b216d685983baa6868317b6299200b768d5b15dc54cec"} err="failed to get container status \"18e085ad912a1794d78b216d685983baa6868317b6299200b768d5b15dc54cec\": rpc error: code = NotFound desc = could not find container \"18e085ad912a1794d78b216d685983baa6868317b6299200b768d5b15dc54cec\": container with ID starting with 18e085ad912a1794d78b216d685983baa6868317b6299200b768d5b15dc54cec not found: ID does not exist"
Oct 31 19:58:55.017196 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:55.017170    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-cd9729/auth-test-model-cd9729-kserve-65f895f85c-pcprs"]
Oct 31 19:58:55.023433 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:55.023411    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-cd9729/auth-test-model-cd9729-kserve-65f895f85c-pcprs"]
Oct 31 19:58:55.332516 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:55.332433    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:55.393351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:55.391539    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-01afb5-7956f95fdf-4sgpr" podUID="d3b92f72-3871-446f-bd45-f279f2e5d712" containerName="payload-processing" containerID="cri-o://4a59b465a65eb1ac3a42110ec801f2f8863183505ff3058a521a80efb5c6a11f" gracePeriod=25
Oct 31 19:58:55.626719 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:55.626637    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96edc0b2-28d0-4058-8aff-c0e179fe5f2e" path="/var/lib/kubelet/pods/96edc0b2-28d0-4058-8aff-c0e179fe5f2e/volumes"
Oct 31 19:58:55.692412 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:55.692385    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-01afb5-7956f95fdf-4sgpr"
Oct 31 19:58:55.755751 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:55.755703    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3b92f72-3871-446f-bd45-f279f2e5d712-kube-api-access-64qnt\" (UniqueName: \"kubernetes.io/projected/d3b92f72-3871-446f-bd45-f279f2e5d712-kube-api-access-64qnt\") pod \"d3b92f72-3871-446f-bd45-f279f2e5d712\" (UID: \"d3b92f72-3871-446f-bd45-f279f2e5d712\") "
Oct 31 19:58:55.756139 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:55.756106    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3b92f72-3871-446f-bd45-f279f2e5d712-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3b92f72-3871-446f-bd45-f279f2e5d712-plugins-config-volume\") pod \"d3b92f72-3871-446f-bd45-f279f2e5d712\" (UID: \"d3b92f72-3871-446f-bd45-f279f2e5d712\") "
Oct 31 19:58:55.756668 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:55.756639    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3b92f72-3871-446f-bd45-f279f2e5d712-plugins-config-volume" pod "d3b92f72-3871-446f-bd45-f279f2e5d712" (UID: "d3b92f72-3871-446f-bd45-f279f2e5d712"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:58:55.756903 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:55.756880    4645 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3b92f72-3871-446f-bd45-f279f2e5d712-plugins-config-volume\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:58:55.758218 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:55.758189    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3b92f72-3871-446f-bd45-f279f2e5d712-kube-api-access-64qnt" pod "d3b92f72-3871-446f-bd45-f279f2e5d712" (UID: "d3b92f72-3871-446f-bd45-f279f2e5d712"). InnerVolumeSpecName "kube-api-access-64qnt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:58:55.858658 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:55.858623    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64qnt\" (UniqueName: \"kubernetes.io/projected/d3b92f72-3871-446f-bd45-f279f2e5d712-kube-api-access-64qnt\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:58:56.001884 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:56.001850    4645 generic.go:356] "Generic (PLEG): container finished" podID="d3b92f72-3871-446f-bd45-f279f2e5d712" containerID="4a59b465a65eb1ac3a42110ec801f2f8863183505ff3058a521a80efb5c6a11f" exitCode=0
Oct 31 19:58:56.002319 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:56.001907    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-01afb5-7956f95fdf-4sgpr"
Oct 31 19:58:56.002319 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:56.001929    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-01afb5-7956f95fdf-4sgpr" event={"ID":"d3b92f72-3871-446f-bd45-f279f2e5d712","Type":"ContainerDied","Data":"4a59b465a65eb1ac3a42110ec801f2f8863183505ff3058a521a80efb5c6a11f"}
Oct 31 19:58:56.002319 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:56.002014    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-01afb5-7956f95fdf-4sgpr" event={"ID":"d3b92f72-3871-446f-bd45-f279f2e5d712","Type":"ContainerDied","Data":"f325e82dfde38427e4787da6aef33c3e4cf5c914d234aee1945c72f32ebdad82"}
Oct 31 19:58:56.002319 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:56.002042    4645 scope.go:122] "RemoveContainer" containerID="4a59b465a65eb1ac3a42110ec801f2f8863183505ff3058a521a80efb5c6a11f"
Oct 31 19:58:56.013116 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:56.013099    4645 scope.go:122] "RemoveContainer" containerID="4a59b465a65eb1ac3a42110ec801f2f8863183505ff3058a521a80efb5c6a11f"
Oct 31 19:58:56.013422 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:58:56.013403    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a59b465a65eb1ac3a42110ec801f2f8863183505ff3058a521a80efb5c6a11f\": container with ID starting with 4a59b465a65eb1ac3a42110ec801f2f8863183505ff3058a521a80efb5c6a11f not found: ID does not exist" containerID="4a59b465a65eb1ac3a42110ec801f2f8863183505ff3058a521a80efb5c6a11f"
Oct 31 19:58:56.013474 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:56.013430    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a59b465a65eb1ac3a42110ec801f2f8863183505ff3058a521a80efb5c6a11f"} err="failed to get container status \"4a59b465a65eb1ac3a42110ec801f2f8863183505ff3058a521a80efb5c6a11f\": rpc error: code = NotFound desc = could not find container \"4a59b465a65eb1ac3a42110ec801f2f8863183505ff3058a521a80efb5c6a11f\": container with ID starting with 4a59b465a65eb1ac3a42110ec801f2f8863183505ff3058a521a80efb5c6a11f not found: ID does not exist"
Oct 31 19:58:56.031561 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:56.031536    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-01afb5-7956f95fdf-4sgpr"]
Oct 31 19:58:56.035139 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:56.035107    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-01afb5-7956f95fdf-4sgpr"]
Oct 31 19:58:56.057472 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:56.057449    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:56.829639 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:56.829597    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:57.112249 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:57.112165    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:57.619073 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:57.619041    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3b92f72-3871-446f-bd45-f279f2e5d712" path="/var/lib/kubelet/pods/d3b92f72-3871-446f-bd45-f279f2e5d712/volumes"
Oct 31 19:58:57.720980 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:57.720940    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-435728/auth-test-model-435728-kserve-76df7cb4d9-t5bn2" podUID="fa08927f-11e4-458e-beac-c96cfa4de425" containerName="main" containerID="cri-o://f5828703961f47a98c706588e1954631d1fbd12aa89119a568e04c13bca9bc77" gracePeriod=45
Oct 31 19:58:57.964903 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:57.964853    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:58.040406 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:58.040375    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-435728/auth-test-model-435728-kserve-76df7cb4d9-t5bn2"
Oct 31 19:58:58.055872 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:58.055834    4645 generic.go:356] "Generic (PLEG): container finished" podID="fa08927f-11e4-458e-beac-c96cfa4de425" containerID="f5828703961f47a98c706588e1954631d1fbd12aa89119a568e04c13bca9bc77" exitCode=0
Oct 31 19:58:58.055872 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:58.055868    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-435728/auth-test-model-435728-kserve-76df7cb4d9-t5bn2"
Oct 31 19:58:58.056134 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:58.055971    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-435728/auth-test-model-435728-kserve-76df7cb4d9-t5bn2" event={"ID":"fa08927f-11e4-458e-beac-c96cfa4de425","Type":"ContainerDied","Data":"f5828703961f47a98c706588e1954631d1fbd12aa89119a568e04c13bca9bc77"}
Oct 31 19:58:58.056134 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:58.056006    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-435728/auth-test-model-435728-kserve-76df7cb4d9-t5bn2" event={"ID":"fa08927f-11e4-458e-beac-c96cfa4de425","Type":"ContainerDied","Data":"c5d25a7313f77443448697588368790a33ded3909fe91a8138c72fe3ad6682fb"}
Oct 31 19:58:58.056134 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:58.056023    4645 scope.go:122] "RemoveContainer" containerID="f5828703961f47a98c706588e1954631d1fbd12aa89119a568e04c13bca9bc77"
Oct 31 19:58:58.073769 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:58.073743    4645 scope.go:122] "RemoveContainer" containerID="f5828703961f47a98c706588e1954631d1fbd12aa89119a568e04c13bca9bc77"
Oct 31 19:58:58.074152 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:58:58.074112    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5828703961f47a98c706588e1954631d1fbd12aa89119a568e04c13bca9bc77\": container with ID starting with f5828703961f47a98c706588e1954631d1fbd12aa89119a568e04c13bca9bc77 not found: ID does not exist" containerID="f5828703961f47a98c706588e1954631d1fbd12aa89119a568e04c13bca9bc77"
Oct 31 19:58:58.074293 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:58.074161    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5828703961f47a98c706588e1954631d1fbd12aa89119a568e04c13bca9bc77"} err="failed to get container status \"f5828703961f47a98c706588e1954631d1fbd12aa89119a568e04c13bca9bc77\": rpc error: code = NotFound desc = could not find container \"f5828703961f47a98c706588e1954631d1fbd12aa89119a568e04c13bca9bc77\": container with ID starting with f5828703961f47a98c706588e1954631d1fbd12aa89119a568e04c13bca9bc77 not found: ID does not exist"
Oct 31 19:58:58.129011 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:58.128981    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa08927f-11e4-458e-beac-c96cfa4de425-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa08927f-11e4-458e-beac-c96cfa4de425-dshm\") pod \"fa08927f-11e4-458e-beac-c96cfa4de425\" (UID: \"fa08927f-11e4-458e-beac-c96cfa4de425\") "
Oct 31 19:58:58.129481 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:58.129071    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa08927f-11e4-458e-beac-c96cfa4de425-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa08927f-11e4-458e-beac-c96cfa4de425-model-cache\") pod \"fa08927f-11e4-458e-beac-c96cfa4de425\" (UID: \"fa08927f-11e4-458e-beac-c96cfa4de425\") "
Oct 31 19:58:58.129481 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:58.129213    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa08927f-11e4-458e-beac-c96cfa4de425-kube-api-access-jcshj\" (UniqueName: \"kubernetes.io/projected/fa08927f-11e4-458e-beac-c96cfa4de425-kube-api-access-jcshj\") pod \"fa08927f-11e4-458e-beac-c96cfa4de425\" (UID: \"fa08927f-11e4-458e-beac-c96cfa4de425\") "
Oct 31 19:58:58.129481 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:58.129235    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa08927f-11e4-458e-beac-c96cfa4de425-home\" (UniqueName: \"kubernetes.io/empty-dir/fa08927f-11e4-458e-beac-c96cfa4de425-home\") pod \"fa08927f-11e4-458e-beac-c96cfa4de425\" (UID: \"fa08927f-11e4-458e-beac-c96cfa4de425\") "
Oct 31 19:58:58.129481 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:58.129322    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa08927f-11e4-458e-beac-c96cfa4de425-tls-certs\" (UniqueName: \"kubernetes.io/secret/fa08927f-11e4-458e-beac-c96cfa4de425-tls-certs\") pod \"fa08927f-11e4-458e-beac-c96cfa4de425\" (UID: \"fa08927f-11e4-458e-beac-c96cfa4de425\") "
Oct 31 19:58:58.129481 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:58.129339    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa08927f-11e4-458e-beac-c96cfa4de425-model-cache" pod "fa08927f-11e4-458e-beac-c96cfa4de425" (UID: "fa08927f-11e4-458e-beac-c96cfa4de425"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:58:58.129481 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:58.129368    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa08927f-11e4-458e-beac-c96cfa4de425-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa08927f-11e4-458e-beac-c96cfa4de425-tmp-dir\") pod \"fa08927f-11e4-458e-beac-c96cfa4de425\" (UID: \"fa08927f-11e4-458e-beac-c96cfa4de425\") "
Oct 31 19:58:58.129811 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:58.129551    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa08927f-11e4-458e-beac-c96cfa4de425-home" pod "fa08927f-11e4-458e-beac-c96cfa4de425" (UID: "fa08927f-11e4-458e-beac-c96cfa4de425"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:58:58.129871 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:58.129806    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa08927f-11e4-458e-beac-c96cfa4de425-tmp-dir" pod "fa08927f-11e4-458e-beac-c96cfa4de425" (UID: "fa08927f-11e4-458e-beac-c96cfa4de425"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:58:58.130101 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:58.130083    4645 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa08927f-11e4-458e-beac-c96cfa4de425-tmp-dir\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:58:58.130160 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:58.130103    4645 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa08927f-11e4-458e-beac-c96cfa4de425-model-cache\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:58:58.130160 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:58.130114    4645 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa08927f-11e4-458e-beac-c96cfa4de425-home\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:58:58.131586 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:58.131563    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa08927f-11e4-458e-beac-c96cfa4de425-tls-certs" pod "fa08927f-11e4-458e-beac-c96cfa4de425" (UID: "fa08927f-11e4-458e-beac-c96cfa4de425"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:58:58.131754 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:58.131732    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa08927f-11e4-458e-beac-c96cfa4de425-dshm" pod "fa08927f-11e4-458e-beac-c96cfa4de425" (UID: "fa08927f-11e4-458e-beac-c96cfa4de425"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:58:58.131818 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:58.131772    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa08927f-11e4-458e-beac-c96cfa4de425-kube-api-access-jcshj" pod "fa08927f-11e4-458e-beac-c96cfa4de425" (UID: "fa08927f-11e4-458e-beac-c96cfa4de425"). InnerVolumeSpecName "kube-api-access-jcshj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:58:58.231685 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:58.231651    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcshj\" (UniqueName: \"kubernetes.io/projected/fa08927f-11e4-458e-beac-c96cfa4de425-kube-api-access-jcshj\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:58:58.231685 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:58.231681    4645 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa08927f-11e4-458e-beac-c96cfa4de425-tls-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:58:58.231857 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:58.231696    4645 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa08927f-11e4-458e-beac-c96cfa4de425-dshm\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:58:58.380642 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:58.380573    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-435728/auth-test-model-435728-kserve-76df7cb4d9-t5bn2"]
Oct 31 19:58:58.384234 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:58.384205    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-435728/auth-test-model-435728-kserve-76df7cb4d9-t5bn2"]
Oct 31 19:58:58.722426 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:58.722390    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:59.466002 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:59.465971    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:58:59.620306 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:58:59.620275    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa08927f-11e4-458e-beac-c96cfa4de425" path="/var/lib/kubelet/pods/fa08927f-11e4-458e-beac-c96cfa4de425/volumes"
Oct 31 19:59:00.503650 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:00.503613    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:01.243120 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:01.243077    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:01.994495 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:01.994451    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:02.720798 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:02.720762    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:03.472391 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:03.472355    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:04.469394 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:04.469355    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:05.254030 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:05.253989    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:05.957791 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:05.957738    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:06.247787 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:59:06.247704    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 19:59:06.787079 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:06.787046    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-26cd59-79c7d4d9c9-j7z8j"]
Oct 31 19:59:06.806674 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:06.806638    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-26cd59-554cf8bc5f-hfrgd"]
Oct 31 19:59:06.835974 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:06.835932    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:06.994693 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:06.994661    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:07.110010 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:07.109910    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:07.491506 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:07.491458    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:07.615424 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:07.615385    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:07.659085 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:07.659044    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-01afb5-6fdd5b76b7-qggmv"]
Oct 31 19:59:07.659712 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:07.659697    4645 cpu_manager.go:414] "containerMap: removing container" podUID="fa08927f-11e4-458e-beac-c96cfa4de425" containerName="main"
Oct 31 19:59:07.659770 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:07.659713    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa08927f-11e4-458e-beac-c96cfa4de425" containerName="main"
Oct 31 19:59:07.659770 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:07.659730    4645 cpu_manager.go:414] "containerMap: removing container" podUID="96edc0b2-28d0-4058-8aff-c0e179fe5f2e" containerName="main"
Oct 31 19:59:07.659770 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:07.659736    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96edc0b2-28d0-4058-8aff-c0e179fe5f2e" containerName="main"
Oct 31 19:59:07.659770 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:07.659746    4645 cpu_manager.go:414] "containerMap: removing container" podUID="d3b92f72-3871-446f-bd45-f279f2e5d712" containerName="payload-processing"
Oct 31 19:59:07.659770 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:07.659751    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3b92f72-3871-446f-bd45-f279f2e5d712" containerName="payload-processing"
Oct 31 19:59:07.659770 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:07.659770    4645 cpu_manager.go:414] "containerMap: removing container" podUID="d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b" containerName="main"
Oct 31 19:59:07.660016 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:07.659775    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b" containerName="main"
Oct 31 19:59:07.660016 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:07.659908    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1063ddb-50eb-4c9e-a37c-1e84e5e1da9b" containerName="main"
Oct 31 19:59:07.660016 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:07.659926    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa08927f-11e4-458e-beac-c96cfa4de425" containerName="main"
Oct 31 19:59:07.660016 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:07.659934    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="96edc0b2-28d0-4058-8aff-c0e179fe5f2e" containerName="main"
Oct 31 19:59:07.660016 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:07.659940    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3b92f72-3871-446f-bd45-f279f2e5d712" containerName="payload-processing"
Oct 31 19:59:07.664996 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:07.664977    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01afb5-6fdd5b76b7-qggmv"
Oct 31 19:59:07.672420 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:07.672389    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-01afb5\""
Oct 31 19:59:07.673446 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:07.673421    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-01afb5-dockercfg-k6lfs\""
Oct 31 19:59:07.683094 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:07.683062    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-01afb5-6fdd5b76b7-qggmv"]
Oct 31 19:59:07.770410 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:07.770301    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-01afb5-db5b465fd-jkvfg"]
Oct 31 19:59:07.800689 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:07.800650    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-01afb5-db5b465fd-jkvfg"
Oct 31 19:59:07.831350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:07.831294    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f35e911-c0c8-45a1-862e-f6a55fb88b77-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-01afb5-6fdd5b76b7-qggmv\" (UID: \"2f35e911-c0c8-45a1-862e-f6a55fb88b77\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01afb5-6fdd5b76b7-qggmv"
Oct 31 19:59:07.831623 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:07.831364    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zw56\" (UniqueName: \"kubernetes.io/projected/2f35e911-c0c8-45a1-862e-f6a55fb88b77-kube-api-access-4zw56\") pod \"payload-pre-processing-e2e-ipp-b-01afb5-6fdd5b76b7-qggmv\" (UID: \"2f35e911-c0c8-45a1-862e-f6a55fb88b77\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01afb5-6fdd5b76b7-qggmv"
Oct 31 19:59:07.851077 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:07.850470    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-01afb5-db5b465fd-jkvfg"]
Oct 31 19:59:08.037574 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:08.037461    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bc15cf8-897f-4501-98f8-1a7abbea383c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-01afb5-db5b465fd-jkvfg\" (UID: \"4bc15cf8-897f-4501-98f8-1a7abbea383c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-01afb5-db5b465fd-jkvfg"
Oct 31 19:59:08.038037 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:08.038010    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfgkl\" (UniqueName: \"kubernetes.io/projected/4bc15cf8-897f-4501-98f8-1a7abbea383c-kube-api-access-jfgkl\") pod \"payload-processing-e2e-ipp-b-01afb5-db5b465fd-jkvfg\" (UID: \"4bc15cf8-897f-4501-98f8-1a7abbea383c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-01afb5-db5b465fd-jkvfg"
Oct 31 19:59:08.038364 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:08.038321    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f35e911-c0c8-45a1-862e-f6a55fb88b77-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-01afb5-6fdd5b76b7-qggmv\" (UID: \"2f35e911-c0c8-45a1-862e-f6a55fb88b77\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01afb5-6fdd5b76b7-qggmv"
Oct 31 19:59:08.038517 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:08.038501    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zw56\" (UniqueName: \"kubernetes.io/projected/2f35e911-c0c8-45a1-862e-f6a55fb88b77-kube-api-access-4zw56\") pod \"payload-pre-processing-e2e-ipp-b-01afb5-6fdd5b76b7-qggmv\" (UID: \"2f35e911-c0c8-45a1-862e-f6a55fb88b77\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01afb5-6fdd5b76b7-qggmv"
Oct 31 19:59:08.040015 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:08.039990    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f35e911-c0c8-45a1-862e-f6a55fb88b77-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-01afb5-6fdd5b76b7-qggmv\" (UID: \"2f35e911-c0c8-45a1-862e-f6a55fb88b77\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01afb5-6fdd5b76b7-qggmv"
Oct 31 19:59:08.051850 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:08.051812    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zw56\" (UniqueName: \"kubernetes.io/projected/2f35e911-c0c8-45a1-862e-f6a55fb88b77-kube-api-access-4zw56\") pod \"payload-pre-processing-e2e-ipp-b-01afb5-6fdd5b76b7-qggmv\" (UID: \"2f35e911-c0c8-45a1-862e-f6a55fb88b77\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01afb5-6fdd5b76b7-qggmv"
Oct 31 19:59:08.154363 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:08.149017    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bc15cf8-897f-4501-98f8-1a7abbea383c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-01afb5-db5b465fd-jkvfg\" (UID: \"4bc15cf8-897f-4501-98f8-1a7abbea383c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-01afb5-db5b465fd-jkvfg"
Oct 31 19:59:08.154363 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:08.149155    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfgkl\" (UniqueName: \"kubernetes.io/projected/4bc15cf8-897f-4501-98f8-1a7abbea383c-kube-api-access-jfgkl\") pod \"payload-processing-e2e-ipp-b-01afb5-db5b465fd-jkvfg\" (UID: \"4bc15cf8-897f-4501-98f8-1a7abbea383c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-01afb5-db5b465fd-jkvfg"
Oct 31 19:59:08.184349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:08.183526    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bc15cf8-897f-4501-98f8-1a7abbea383c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-01afb5-db5b465fd-jkvfg\" (UID: \"4bc15cf8-897f-4501-98f8-1a7abbea383c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-01afb5-db5b465fd-jkvfg"
Oct 31 19:59:08.272353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:08.272136    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfgkl\" (UniqueName: \"kubernetes.io/projected/4bc15cf8-897f-4501-98f8-1a7abbea383c-kube-api-access-jfgkl\") pod \"payload-processing-e2e-ipp-b-01afb5-db5b465fd-jkvfg\" (UID: \"4bc15cf8-897f-4501-98f8-1a7abbea383c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-01afb5-db5b465fd-jkvfg"
Oct 31 19:59:08.284814 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:08.280598    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01afb5-6fdd5b76b7-qggmv"
Oct 31 19:59:08.296918 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:08.295918    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:08.486080 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:08.486038    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-01afb5-db5b465fd-jkvfg"
Oct 31 19:59:08.644715 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:08.644681    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-01afb5-6fdd5b76b7-qggmv"]
Oct 31 19:59:08.646290 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:59:08.646255    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f35e911_c0c8_45a1_862e_f6a55fb88b77.slice/crio-15c2cd42fb65139779f6e36f608bd5363ebdf410c24abfc0079c7bfbed2765f0 WatchSource:0}: Error finding container 15c2cd42fb65139779f6e36f608bd5363ebdf410c24abfc0079c7bfbed2765f0: Status 404 returned error can't find the container with id 15c2cd42fb65139779f6e36f608bd5363ebdf410c24abfc0079c7bfbed2765f0
Oct 31 19:59:08.648963 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:08.648941    4645 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 19:59:08.792484 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:59:08.792438    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bc15cf8_897f_4501_98f8_1a7abbea383c.slice/crio-6f60e313d1c34394d2a01631f56fe3281e5157df4cfcec18449c2bc6ed7b62a1 WatchSource:0}: Error finding container 6f60e313d1c34394d2a01631f56fe3281e5157df4cfcec18449c2bc6ed7b62a1: Status 404 returned error can't find the container with id 6f60e313d1c34394d2a01631f56fe3281e5157df4cfcec18449c2bc6ed7b62a1
Oct 31 19:59:08.793526 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:08.793494    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-01afb5-db5b465fd-jkvfg"]
Oct 31 19:59:09.065648 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:09.065612    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:09.199654 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:09.199612    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01afb5-6fdd5b76b7-qggmv" event={"ID":"2f35e911-c0c8-45a1-862e-f6a55fb88b77","Type":"ContainerStarted","Data":"15c2cd42fb65139779f6e36f608bd5363ebdf410c24abfc0079c7bfbed2765f0"}
Oct 31 19:59:09.200919 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:09.200889    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-01afb5-db5b465fd-jkvfg" event={"ID":"4bc15cf8-897f-4501-98f8-1a7abbea383c","Type":"ContainerStarted","Data":"6f60e313d1c34394d2a01631f56fe3281e5157df4cfcec18449c2bc6ed7b62a1"}
Oct 31 19:59:10.545378 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:10.539351    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:10.978461 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:10.974455    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-e618e1/sub-test-model-e618e1-kserve-584c8d9c65-42m96" podUID="4f876b87-1980-4834-89d7-47b4db24a2c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.218:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:59:11.706358 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:11.607403    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x" podUID="a774e286-dcb8-4a2f-8541-26bf231eb08d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded"
Oct 31 19:59:11.706358 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:11.618225    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669" podUID="9285bee0-bf7c-4fd4-9da7-a1db021cae05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded"
Oct 31 19:59:11.706358 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:11.618789    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw" podUID="d504b2d3-c26d-4dcf-a7eb-efe20732ae4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 31 19:59:11.706358 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:11.619012    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n" podUID="dbb97168-db93-4757-9a1b-f9c5eb704fc5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:59:11.706358 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:11.667421    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw" podUID="1287db54-a587-4c11-9eb6-1606de555894" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:59:11.793555 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:11.793504    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-26cd59-79c7d4d9c9-j7z8j" podUID="29c8935f-7368-41b3-9b04-febe7848fd41" containerName="payload-processing" containerID="cri-o://f5a7416fce3dd9ac31222f9a7274cb7fb000a4cc9bbe6ba13aadb7db4366fdd7" gracePeriod=25
Oct 31 19:59:11.841450 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:11.821574    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:11.842215 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:11.842174    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-26cd59-554cf8bc5f-hfrgd" podUID="28db49c5-ce35-4752-8e9e-11fd054e4884" containerName="payload-pre-processing" containerID="cri-o://ecc3a2c7dddafc62fe88a0dea019adff03ee5b1f8a7a89917487c455745321e7" gracePeriod=25
Oct 31 19:59:11.860550 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:11.858443    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w6-a891ce/auth-test-model-a891ce-kserve-c6cd86cc7-vckq2" podUID="5f6f43f0-380a-45d9-a8ab-324872927234" containerName="main" probeResult="failure" output="Get \"https://10.133.0.206:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:59:11.936637 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:11.936590    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l" podUID="12bea86c-61e5-465e-82fb-b4673a4d1be9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:59:11.977405 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:11.975463    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-e618e1/sub-test-model-e618e1-kserve-584c8d9c65-42m96" podUID="4f876b87-1980-4834-89d7-47b4db24a2c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.218:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:59:11.977405 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:11.975800    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-e618e1/sub-test-model-e618e1-kserve-584c8d9c65-42m96"
Oct 31 19:59:12.047716 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:12.047676    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:12.251113 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:12.217657    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-4bac5a/rate-test-model-4bac5a-kserve-564dc4dfdf-vv676" podUID="f9b4e54a-b150-415f-b266-caa45004d860" containerName="main" probeResult="failure" output="Get \"https://10.133.0.214:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:59:12.252120 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:12.217690    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w6-e1d060/auth-test-model-e1d060-kserve-589d89d7b9-2vkt5" podUID="093bfd46-77e5-45d4-aa2e-d07fd8dafe33" containerName="main" probeResult="failure" output="Get \"https://10.133.0.210:8000/ready\": context deadline exceeded"
Oct 31 19:59:12.254050 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:12.217709    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-c55cf1/auth-test-model-c55cf1-kserve-6bb6c9bd5c-qnh9f" podUID="6ba2c7b6-54c7-4610-8051-8fc89f6f8731" containerName="main" probeResult="failure" output="Get \"https://10.133.0.211:8000/ready\": context deadline exceeded"
Oct 31 19:59:12.254050 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:12.223729    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-d9c048/sub-test-model-d9c048-kserve-6fff8b86d5-2mb99" podUID="2665b4f7-db95-40bf-8027-63b27e28afa3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.219:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:59:12.422110 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:12.419634    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-ae2e60/rate-test-model-ae2e60-kserve-744475655c-xk2vp" podUID="44f86c77-4686-48fe-8cf3-cc0e688af7a4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.216:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:59:12.458375 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:12.454608    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4" podUID="6032a997-b795-4561-805c-110cf67a9d5d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:59:12.833520 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:12.784934    4645 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-b-w4-d9c048/sub-test-model-d9c048-kserve-6fff8b86d5-2mb99" podUID="2665b4f7-db95-40bf-8027-63b27e28afa3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.219:8000/health\": context deadline exceeded"
Oct 31 19:59:12.833520 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:12.785251    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n" podUID="dbb97168-db93-4757-9a1b-f9c5eb704fc5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 31 19:59:12.833520 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:12.787170    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n"
Oct 31 19:59:12.833520 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:12.791398    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x" podUID="a774e286-dcb8-4a2f-8541-26bf231eb08d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:59:12.833520 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:12.791687    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669" podUID="9285bee0-bf7c-4fd4-9da7-a1db021cae05" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:59:12.833520 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:12.791988    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x"
Oct 31 19:59:12.833520 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:12.792039    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669"
Oct 31 19:59:12.833520 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:12.830636    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw" podUID="d504b2d3-c26d-4dcf-a7eb-efe20732ae4e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded"
Oct 31 19:59:12.833520 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:12.832035    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw" podUID="1287db54-a587-4c11-9eb6-1606de555894" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": context deadline exceeded"
Oct 31 19:59:12.833520 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:12.832412    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw"
Oct 31 19:59:12.833520 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:12.832472    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw"
Oct 31 19:59:13.153996 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:13.153572    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-e618e1/sub-test-model-e618e1-kserve-584c8d9c65-42m96" podUID="4f876b87-1980-4834-89d7-47b4db24a2c1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.218:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:59:13.168349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:13.166616    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w6-a891ce/auth-test-model-a891ce-kserve-c6cd86cc7-vckq2" podUID="5f6f43f0-380a-45d9-a8ab-324872927234" containerName="main" probeResult="failure" output="Get \"https://10.133.0.206:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 19:59:13.168349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:13.166988    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w6-a891ce/auth-test-model-a891ce-kserve-c6cd86cc7-vckq2"
Oct 31 19:59:13.168349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:13.168124    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw"
Oct 31 19:59:13.239248 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:13.185314    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669"
Oct 31 19:59:13.239248 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:13.192043    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n"
Oct 31 19:59:13.239248 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:13.221378    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:13.239248 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:13.221961    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x"
Oct 31 19:59:13.273899 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:13.271981    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w6-a891ce/auth-test-model-a891ce-kserve-c6cd86cc7-vckq2"
Oct 31 19:59:13.277295 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:13.277242    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-e618e1/sub-test-model-e618e1-kserve-584c8d9c65-42m96"
Oct 31 19:59:13.280922 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:13.279933    4645 generic.go:356] "Generic (PLEG): container finished" podID="28db49c5-ce35-4752-8e9e-11fd054e4884" containerID="ecc3a2c7dddafc62fe88a0dea019adff03ee5b1f8a7a89917487c455745321e7" exitCode=0
Oct 31 19:59:13.280922 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:13.280060    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-26cd59-554cf8bc5f-hfrgd" event={"ID":"28db49c5-ce35-4752-8e9e-11fd054e4884","Type":"ContainerDied","Data":"ecc3a2c7dddafc62fe88a0dea019adff03ee5b1f8a7a89917487c455745321e7"}
Oct 31 19:59:13.280922 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:13.280105    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-26cd59-554cf8bc5f-hfrgd" event={"ID":"28db49c5-ce35-4752-8e9e-11fd054e4884","Type":"ContainerDied","Data":"adc2ee8480beba2726414b7cb1689a87c3c880b25ea235e516f6454dfea1048e"}
Oct 31 19:59:13.280922 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:13.280130    4645 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="adc2ee8480beba2726414b7cb1689a87c3c880b25ea235e516f6454dfea1048e"
Oct 31 19:59:13.295233 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:13.295215    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw"
Oct 31 19:59:13.313439 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:13.313413    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-26cd59-554cf8bc5f-hfrgd"
Oct 31 19:59:13.322688 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:13.322659    4645 generic.go:356] "Generic (PLEG): container finished" podID="29c8935f-7368-41b3-9b04-febe7848fd41" containerID="f5a7416fce3dd9ac31222f9a7274cb7fb000a4cc9bbe6ba13aadb7db4366fdd7" exitCode=0
Oct 31 19:59:13.324045 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:13.322771    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-26cd59-79c7d4d9c9-j7z8j" event={"ID":"29c8935f-7368-41b3-9b04-febe7848fd41","Type":"ContainerDied","Data":"f5a7416fce3dd9ac31222f9a7274cb7fb000a4cc9bbe6ba13aadb7db4366fdd7"}
Oct 31 19:59:13.429030 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:13.428899    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28db49c5-ce35-4752-8e9e-11fd054e4884-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28db49c5-ce35-4752-8e9e-11fd054e4884-plugins-config-volume\") pod \"28db49c5-ce35-4752-8e9e-11fd054e4884\" (UID: \"28db49c5-ce35-4752-8e9e-11fd054e4884\") "
Oct 31 19:59:13.451220 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:13.451180    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28db49c5-ce35-4752-8e9e-11fd054e4884-plugins-config-volume" pod "28db49c5-ce35-4752-8e9e-11fd054e4884" (UID: "28db49c5-ce35-4752-8e9e-11fd054e4884"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:59:13.536670 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:13.536632    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28db49c5-ce35-4752-8e9e-11fd054e4884-kube-api-access-j7tkq\" (UniqueName: \"kubernetes.io/projected/28db49c5-ce35-4752-8e9e-11fd054e4884-kube-api-access-j7tkq\") pod \"28db49c5-ce35-4752-8e9e-11fd054e4884\" (UID: \"28db49c5-ce35-4752-8e9e-11fd054e4884\") "
Oct 31 19:59:13.537577 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:13.537534    4645 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28db49c5-ce35-4752-8e9e-11fd054e4884-plugins-config-volume\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:13.554309 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:13.554278    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28db49c5-ce35-4752-8e9e-11fd054e4884-kube-api-access-j7tkq" pod "28db49c5-ce35-4752-8e9e-11fd054e4884" (UID: "28db49c5-ce35-4752-8e9e-11fd054e4884"). InnerVolumeSpecName "kube-api-access-j7tkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:59:13.639354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:13.639305    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7tkq\" (UniqueName: \"kubernetes.io/projected/28db49c5-ce35-4752-8e9e-11fd054e4884-kube-api-access-j7tkq\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:13.715669 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:13.715581    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-26cd59-79c7d4d9c9-j7z8j"
Oct 31 19:59:13.752443 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:13.752406    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29c8935f-7368-41b3-9b04-febe7848fd41-kube-api-access-7dcnm\" (UniqueName: \"kubernetes.io/projected/29c8935f-7368-41b3-9b04-febe7848fd41-kube-api-access-7dcnm\") pod \"29c8935f-7368-41b3-9b04-febe7848fd41\" (UID: \"29c8935f-7368-41b3-9b04-febe7848fd41\") "
Oct 31 19:59:13.752751 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:13.752729    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29c8935f-7368-41b3-9b04-febe7848fd41-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29c8935f-7368-41b3-9b04-febe7848fd41-plugins-config-volume\") pod \"29c8935f-7368-41b3-9b04-febe7848fd41\" (UID: \"29c8935f-7368-41b3-9b04-febe7848fd41\") "
Oct 31 19:59:13.787750 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:13.787712    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29c8935f-7368-41b3-9b04-febe7848fd41-plugins-config-volume" pod "29c8935f-7368-41b3-9b04-febe7848fd41" (UID: "29c8935f-7368-41b3-9b04-febe7848fd41"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:59:13.789920 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:13.789885    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29c8935f-7368-41b3-9b04-febe7848fd41-kube-api-access-7dcnm" pod "29c8935f-7368-41b3-9b04-febe7848fd41" (UID: "29c8935f-7368-41b3-9b04-febe7848fd41"). InnerVolumeSpecName "kube-api-access-7dcnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:59:13.856404 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:13.856368    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dcnm\" (UniqueName: \"kubernetes.io/projected/29c8935f-7368-41b3-9b04-febe7848fd41-kube-api-access-7dcnm\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:13.856938 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:13.856914    4645 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29c8935f-7368-41b3-9b04-febe7848fd41-plugins-config-volume\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:13.892669 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:13.892634    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:14.349133 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:14.349087    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-26cd59-79c7d4d9c9-j7z8j" event={"ID":"29c8935f-7368-41b3-9b04-febe7848fd41","Type":"ContainerDied","Data":"ce5a8b1b82f8ef67cbb3f184c7627ea6cf638c3a32394a03c99e259a8346b2a5"}
Oct 31 19:59:14.349343 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:14.349150    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-26cd59-554cf8bc5f-hfrgd"
Oct 31 19:59:14.349343 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:14.349163    4645 scope.go:122] "RemoveContainer" containerID="f5a7416fce3dd9ac31222f9a7274cb7fb000a4cc9bbe6ba13aadb7db4366fdd7"
Oct 31 19:59:14.349343 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:14.349301    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-26cd59-79c7d4d9c9-j7z8j"
Oct 31 19:59:14.408659 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:14.408457    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-26cd59-554cf8bc5f-hfrgd"]
Oct 31 19:59:14.417202 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:14.417172    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-26cd59-554cf8bc5f-hfrgd"]
Oct 31 19:59:14.428923 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:14.428893    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-26cd59-79c7d4d9c9-j7z8j"]
Oct 31 19:59:14.429125 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:14.429113    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-26cd59-79c7d4d9c9-j7z8j"]
Oct 31 19:59:14.783350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:14.782813    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:15.450216 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:15.450180    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e02c2e34-rq6jc"]
Oct 31 19:59:15.450951 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:15.450927    4645 cpu_manager.go:414] "containerMap: removing container" podUID="29c8935f-7368-41b3-9b04-febe7848fd41" containerName="payload-processing"
Oct 31 19:59:15.450951 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:15.450952    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29c8935f-7368-41b3-9b04-febe7848fd41" containerName="payload-processing"
Oct 31 19:59:15.451094 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:15.450978    4645 cpu_manager.go:414] "containerMap: removing container" podUID="28db49c5-ce35-4752-8e9e-11fd054e4884" containerName="payload-pre-processing"
Oct 31 19:59:15.451094 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:15.450986    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28db49c5-ce35-4752-8e9e-11fd054e4884" containerName="payload-pre-processing"
Oct 31 19:59:15.451385 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:15.451366    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="29c8935f-7368-41b3-9b04-febe7848fd41" containerName="payload-processing"
Oct 31 19:59:15.451476 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:15.451390    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="28db49c5-ce35-4752-8e9e-11fd054e4884" containerName="payload-pre-processing"
Oct 31 19:59:15.491020 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:15.489667    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e02c2e34-rq6jc"
Oct 31 19:59:15.491020 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:15.490922    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e02c2e34-rq6jc"]
Oct 31 19:59:15.496462 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:15.496438    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 19:59:15.496694 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:15.496669    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:59:15.538634 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:15.538477    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d14cdb3-49a7-4bed-ac45-b18e6452ebbe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e02c2e34-rq6jc\" (UID: \"1d14cdb3-49a7-4bed-ac45-b18e6452ebbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e02c2e34-rq6jc"
Oct 31 19:59:15.538634 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:15.538614    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d14cdb3-49a7-4bed-ac45-b18e6452ebbe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e02c2e34-rq6jc\" (UID: \"1d14cdb3-49a7-4bed-ac45-b18e6452ebbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e02c2e34-rq6jc"
Oct 31 19:59:15.538953 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:15.538901    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmjpx\" (UniqueName: \"kubernetes.io/projected/1d14cdb3-49a7-4bed-ac45-b18e6452ebbe-kube-api-access-zmjpx\") pod \"maas-api-revoke-sub-e02c2e34-rq6jc\" (UID: \"1d14cdb3-49a7-4bed-ac45-b18e6452ebbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e02c2e34-rq6jc"
Oct 31 19:59:15.611524 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:15.611488    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:15.622803 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:15.622775    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28db49c5-ce35-4752-8e9e-11fd054e4884" path="/var/lib/kubelet/pods/28db49c5-ce35-4752-8e9e-11fd054e4884/volumes"
Oct 31 19:59:15.623353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:15.623304    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29c8935f-7368-41b3-9b04-febe7848fd41" path="/var/lib/kubelet/pods/29c8935f-7368-41b3-9b04-febe7848fd41/volumes"
Oct 31 19:59:15.640051 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:15.640025    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d14cdb3-49a7-4bed-ac45-b18e6452ebbe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e02c2e34-rq6jc\" (UID: \"1d14cdb3-49a7-4bed-ac45-b18e6452ebbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e02c2e34-rq6jc"
Oct 31 19:59:15.640194 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:15.640112    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmjpx\" (UniqueName: \"kubernetes.io/projected/1d14cdb3-49a7-4bed-ac45-b18e6452ebbe-kube-api-access-zmjpx\") pod \"maas-api-revoke-sub-e02c2e34-rq6jc\" (UID: \"1d14cdb3-49a7-4bed-ac45-b18e6452ebbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e02c2e34-rq6jc"
Oct 31 19:59:15.640273 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:15.640257    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d14cdb3-49a7-4bed-ac45-b18e6452ebbe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e02c2e34-rq6jc\" (UID: \"1d14cdb3-49a7-4bed-ac45-b18e6452ebbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e02c2e34-rq6jc"
Oct 31 19:59:15.641444 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:15.641424    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d14cdb3-49a7-4bed-ac45-b18e6452ebbe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e02c2e34-rq6jc\" (UID: \"1d14cdb3-49a7-4bed-ac45-b18e6452ebbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e02c2e34-rq6jc"
Oct 31 19:59:15.672915 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:15.672823    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d14cdb3-49a7-4bed-ac45-b18e6452ebbe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e02c2e34-rq6jc\" (UID: \"1d14cdb3-49a7-4bed-ac45-b18e6452ebbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e02c2e34-rq6jc"
Oct 31 19:59:15.696182 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:15.696082    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmjpx\" (UniqueName: \"kubernetes.io/projected/1d14cdb3-49a7-4bed-ac45-b18e6452ebbe-kube-api-access-zmjpx\") pod \"maas-api-revoke-sub-e02c2e34-rq6jc\" (UID: \"1d14cdb3-49a7-4bed-ac45-b18e6452ebbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e02c2e34-rq6jc"
Oct 31 19:59:15.825514 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:15.821275    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7020-d53f28"]
Oct 31 19:59:15.825514 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:15.821642    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e02c2e34-rq6jc"
Oct 31 19:59:15.832359 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:15.831006    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7020-d53f28"
Oct 31 19:59:15.836370 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:15.832918    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7020-d53f28"]
Oct 31 19:59:15.836370 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:15.835073    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-4pmld\""
Oct 31 19:59:15.984244 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:15.984207    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmj4k\" (UniqueName: \"kubernetes.io/projected/8f0a7362-4c8c-41c4-b8f6-7627cf1e1ef7-kube-api-access-rmj4k\") pod \"test-curl-7020-d53f28\" (UID: \"8f0a7362-4c8c-41c4-b8f6-7627cf1e1ef7\") " pod="opendatahub/test-curl-7020-d53f28"
Oct 31 19:59:16.086748 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:16.086654    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmj4k\" (UniqueName: \"kubernetes.io/projected/8f0a7362-4c8c-41c4-b8f6-7627cf1e1ef7-kube-api-access-rmj4k\") pod \"test-curl-7020-d53f28\" (UID: \"8f0a7362-4c8c-41c4-b8f6-7627cf1e1ef7\") " pod="opendatahub/test-curl-7020-d53f28"
Oct 31 19:59:16.100261 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:16.100235    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmj4k\" (UniqueName: \"kubernetes.io/projected/8f0a7362-4c8c-41c4-b8f6-7627cf1e1ef7-kube-api-access-rmj4k\") pod \"test-curl-7020-d53f28\" (UID: \"8f0a7362-4c8c-41c4-b8f6-7627cf1e1ef7\") " pod="opendatahub/test-curl-7020-d53f28"
Oct 31 19:59:16.155732 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:16.155507    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7020-d53f28"
Oct 31 19:59:16.190783 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:16.190545    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e02c2e34-rq6jc"]
Oct 31 19:59:16.375088 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:16.374957    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:16.452613 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:16.452571    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e02c2e34-rq6jc" event={"ID":"1d14cdb3-49a7-4bed-ac45-b18e6452ebbe","Type":"ContainerStarted","Data":"cbd004846fbd2151b5e80a0cd6ae8a055fe696b4644604c7c9ecbf53bf61e5c5"}
Oct 31 19:59:16.515164 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:16.515082    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7020-d53f28"]
Oct 31 19:59:17.280643 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:17.280607    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:17.416073 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:17.416038    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:17.464574 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:17.464542    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01afb5-6fdd5b76b7-qggmv" event={"ID":"2f35e911-c0c8-45a1-862e-f6a55fb88b77","Type":"ContainerStarted","Data":"11e56de629a037edadaa266a1362d815c4775dbb2090fe98079008197cc0b13d"}
Oct 31 19:59:17.464991 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:17.464694    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01afb5-6fdd5b76b7-qggmv"
Oct 31 19:59:17.466825 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:17.466800    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-01afb5-db5b465fd-jkvfg" event={"ID":"4bc15cf8-897f-4501-98f8-1a7abbea383c","Type":"ContainerStarted","Data":"7363d9ba9ba6afe3327cce6e8961e051eeea2ed963419cf591fdee568ee7a333"}
Oct 31 19:59:17.466965 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:17.466946    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-01afb5-db5b465fd-jkvfg"
Oct 31 19:59:17.468498 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:17.468472    4645 generic.go:356] "Generic (PLEG): container finished" podID="1d14cdb3-49a7-4bed-ac45-b18e6452ebbe" containerID="78768d4731529b62a86b24591f9f089f7f4d08f762f395f752995d8a492dab1c" exitCode=0
Oct 31 19:59:17.468615 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:17.468555    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e02c2e34-rq6jc" event={"ID":"1d14cdb3-49a7-4bed-ac45-b18e6452ebbe","Type":"ContainerDied","Data":"78768d4731529b62a86b24591f9f089f7f4d08f762f395f752995d8a492dab1c"}
Oct 31 19:59:17.470110 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:17.470084    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7020-d53f28" event={"ID":"8f0a7362-4c8c-41c4-b8f6-7627cf1e1ef7","Type":"ContainerStarted","Data":"c86c25d7fdfef1629617143eb06023bb375691ede0b9c058ef58ed2b728392f4"}
Oct 31 19:59:17.470231 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:17.470119    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7020-d53f28" event={"ID":"8f0a7362-4c8c-41c4-b8f6-7627cf1e1ef7","Type":"ContainerStarted","Data":"40978ed6d1f7ad4cf7c3a08ef1583778c3bcebce2c37d9bd6a6735150d4db885"}
Oct 31 19:59:17.544382 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:17.544264    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01afb5-6fdd5b76b7-qggmv" podStartSLOduration=2.260338606 podStartE2EDuration="10.544250207s" totalImagesPullingTime="8.283911601s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:59:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:59:17.542695662 +0000 UTC m=+2388.115929792" watchObservedRunningTime="2026-10-31 19:59:17.544250207 +0000 UTC m=+2388.117484337"
Oct 31 19:59:17.574864 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:17.574812    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-01afb5-db5b465fd-jkvfg" podStartSLOduration=2.542049118 podStartE2EDuration="10.574796447s" totalImagesPullingTime="8.032747329s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:59:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:59:17.571472091 +0000 UTC m=+2388.144706220" watchObservedRunningTime="2026-10-31 19:59:17.574796447 +0000 UTC m=+2388.148030577"
Oct 31 19:59:17.691667 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:17.691591    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7020-d53f28" podStartSLOduration=2.691576551 podStartE2EDuration="2.691576551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:59:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:59:17.687065411 +0000 UTC m=+2388.260299541" watchObservedRunningTime="2026-10-31 19:59:17.691576551 +0000 UTC m=+2388.264810693"
Oct 31 19:59:18.243964 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:18.243927    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:18.456740 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:18.456704    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:18.517048 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:18.516970    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7020-d53f28"]
Oct 31 19:59:18.524922 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:18.524892    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7020-d53f28"]
Oct 31 19:59:18.684828 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:18.684781    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7020-3da8ea"]
Oct 31 19:59:18.685892 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:18.685848    4645 cpu_manager.go:414] "containerMap: removing container" podUID="8f0a7362-4c8c-41c4-b8f6-7627cf1e1ef7" containerName="test-curl-7020-d53f28"
Oct 31 19:59:18.685892 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:18.685881    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f0a7362-4c8c-41c4-b8f6-7627cf1e1ef7" containerName="test-curl-7020-d53f28"
Oct 31 19:59:18.686109 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:18.686096    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f0a7362-4c8c-41c4-b8f6-7627cf1e1ef7" containerName="test-curl-7020-d53f28"
Oct 31 19:59:18.706837 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:18.705229    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7020-3da8ea"
Oct 31 19:59:18.737871 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:18.737386    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7020-3da8ea"]
Oct 31 19:59:18.778460 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:18.778429    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nld74\" (UniqueName: \"kubernetes.io/projected/daaee800-f23a-4154-b55c-7007f2385707-kube-api-access-nld74\") pod \"test-curl-7020-3da8ea\" (UID: \"daaee800-f23a-4154-b55c-7007f2385707\") " pod="opendatahub/test-curl-7020-3da8ea"
Oct 31 19:59:18.787289 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:18.787267    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e02c2e34-rq6jc"
Oct 31 19:59:18.879674 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:18.879636    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d14cdb3-49a7-4bed-ac45-b18e6452ebbe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d14cdb3-49a7-4bed-ac45-b18e6452ebbe-maas-api-xks-ca\") pod \"1d14cdb3-49a7-4bed-ac45-b18e6452ebbe\" (UID: \"1d14cdb3-49a7-4bed-ac45-b18e6452ebbe\") "
Oct 31 19:59:18.880152 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:18.880125    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d14cdb3-49a7-4bed-ac45-b18e6452ebbe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d14cdb3-49a7-4bed-ac45-b18e6452ebbe-maas-api-openshift-ca\") pod \"1d14cdb3-49a7-4bed-ac45-b18e6452ebbe\" (UID: \"1d14cdb3-49a7-4bed-ac45-b18e6452ebbe\") "
Oct 31 19:59:18.880429 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:18.880408    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d14cdb3-49a7-4bed-ac45-b18e6452ebbe-kube-api-access-zmjpx\" (UniqueName: \"kubernetes.io/projected/1d14cdb3-49a7-4bed-ac45-b18e6452ebbe-kube-api-access-zmjpx\") pod \"1d14cdb3-49a7-4bed-ac45-b18e6452ebbe\" (UID: \"1d14cdb3-49a7-4bed-ac45-b18e6452ebbe\") "
Oct 31 19:59:18.880966 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:18.880937    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d14cdb3-49a7-4bed-ac45-b18e6452ebbe-maas-api-openshift-ca" pod "1d14cdb3-49a7-4bed-ac45-b18e6452ebbe" (UID: "1d14cdb3-49a7-4bed-ac45-b18e6452ebbe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:59:18.881450 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:18.881425    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nld74\" (UniqueName: \"kubernetes.io/projected/daaee800-f23a-4154-b55c-7007f2385707-kube-api-access-nld74\") pod \"test-curl-7020-3da8ea\" (UID: \"daaee800-f23a-4154-b55c-7007f2385707\") " pod="opendatahub/test-curl-7020-3da8ea"
Oct 31 19:59:18.881751 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:18.881728    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d14cdb3-49a7-4bed-ac45-b18e6452ebbe-maas-api-openshift-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:18.883345 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:18.882223    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d14cdb3-49a7-4bed-ac45-b18e6452ebbe-maas-api-xks-ca" pod "1d14cdb3-49a7-4bed-ac45-b18e6452ebbe" (UID: "1d14cdb3-49a7-4bed-ac45-b18e6452ebbe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:59:18.885783 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:18.885760    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d14cdb3-49a7-4bed-ac45-b18e6452ebbe-kube-api-access-zmjpx" pod "1d14cdb3-49a7-4bed-ac45-b18e6452ebbe" (UID: "1d14cdb3-49a7-4bed-ac45-b18e6452ebbe"). InnerVolumeSpecName "kube-api-access-zmjpx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:59:18.894085 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:18.894058    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nld74\" (UniqueName: \"kubernetes.io/projected/daaee800-f23a-4154-b55c-7007f2385707-kube-api-access-nld74\") pod \"test-curl-7020-3da8ea\" (UID: \"daaee800-f23a-4154-b55c-7007f2385707\") " pod="opendatahub/test-curl-7020-3da8ea"
Oct 31 19:59:18.984807 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:18.984766    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d14cdb3-49a7-4bed-ac45-b18e6452ebbe-maas-api-xks-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:18.984807 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:18.984805    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmjpx\" (UniqueName: \"kubernetes.io/projected/1d14cdb3-49a7-4bed-ac45-b18e6452ebbe-kube-api-access-zmjpx\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:19.083079 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:19.083000    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:19.084396 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:19.084371    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7020-3da8ea"
Oct 31 19:59:19.194845 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:19.194775    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:19.459734 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:19.459698    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7020-3da8ea"]
Oct 31 19:59:19.463823 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:59:19.463793    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddaaee800_f23a_4154_b55c_7007f2385707.slice/crio-4e4817231f0dc514e718dd6ead344413357f998b94b07086a2803f77b8f98bd1 WatchSource:0}: Error finding container 4e4817231f0dc514e718dd6ead344413357f998b94b07086a2803f77b8f98bd1: Status 404 returned error can't find the container with id 4e4817231f0dc514e718dd6ead344413357f998b94b07086a2803f77b8f98bd1
Oct 31 19:59:19.489370 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:19.489324    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7020-3da8ea" event={"ID":"daaee800-f23a-4154-b55c-7007f2385707","Type":"ContainerStarted","Data":"4e4817231f0dc514e718dd6ead344413357f998b94b07086a2803f77b8f98bd1"}
Oct 31 19:59:19.490870 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:19.490855    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e02c2e34-rq6jc"
Oct 31 19:59:19.490945 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:19.490893    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e02c2e34-rq6jc" event={"ID":"1d14cdb3-49a7-4bed-ac45-b18e6452ebbe","Type":"ContainerDied","Data":"cbd004846fbd2151b5e80a0cd6ae8a055fe696b4644604c7c9ecbf53bf61e5c5"}
Oct 31 19:59:19.490997 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:19.490941    4645 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cbd004846fbd2151b5e80a0cd6ae8a055fe696b4644604c7c9ecbf53bf61e5c5"
Oct 31 19:59:19.491176 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:19.491125    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7020-d53f28" podUID="8f0a7362-4c8c-41c4-b8f6-7627cf1e1ef7" containerName="test-curl-7020-d53f28" containerID="cri-o://c86c25d7fdfef1629617143eb06023bb375691ede0b9c058ef58ed2b728392f4" gracePeriod=2
Oct 31 19:59:19.493792 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:59:19.493752    4645 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7020-d53f28\" is forbidden: User \"system:node:ip-10-0-131-94.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-94.us-east-2.compute.internal' and this object" podUID="8f0a7362-4c8c-41c4-b8f6-7627cf1e1ef7" pod="opendatahub/test-curl-7020-d53f28"
Oct 31 19:59:19.506714 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:59:19.506683    4645 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7020-d53f28\" is forbidden: User \"system:node:ip-10-0-131-94.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-94.us-east-2.compute.internal' and this object" podUID="8f0a7362-4c8c-41c4-b8f6-7627cf1e1ef7" pod="opendatahub/test-curl-7020-d53f28"
Oct 31 19:59:19.616851 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:59:19.616814    4645 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7020-d53f28\" is forbidden: User \"system:node:ip-10-0-131-94.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-94.us-east-2.compute.internal' and this object" podUID="8f0a7362-4c8c-41c4-b8f6-7627cf1e1ef7" pod="opendatahub/test-curl-7020-d53f28"
Oct 31 19:59:19.963892 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:19.963858    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:20.502474 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:20.502442    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7020-3da8ea" event={"ID":"daaee800-f23a-4154-b55c-7007f2385707","Type":"ContainerStarted","Data":"6b9e2a0b98f59563d8e2cffe027bb1dcd1a6c5d6ad098c23f705ee97ca63f747"}
Oct 31 19:59:20.518976 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:20.518918    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7020-3da8ea" podStartSLOduration=2.518901059 podStartE2EDuration="2.518901059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:59:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:59:20.517914791 +0000 UTC m=+2391.091148921" watchObservedRunningTime="2026-10-31 19:59:20.518901059 +0000 UTC m=+2391.092135188"
Oct 31 19:59:20.861295 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:20.861219    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:21.246388 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:21.241638    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7020-3da8ea"]
Oct 31 19:59:21.254233 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:21.254202    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7020-3da8ea"]
Oct 31 19:59:21.512396 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:21.512361    4645 generic.go:356] "Generic (PLEG): container finished" podID="8f0a7362-4c8c-41c4-b8f6-7627cf1e1ef7" containerID="c86c25d7fdfef1629617143eb06023bb375691ede0b9c058ef58ed2b728392f4" exitCode=137
Oct 31 19:59:21.737766 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:21.737733    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7020-d53f28"
Oct 31 19:59:21.759954 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:21.759891    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89846e5e-82ghj"]
Oct 31 19:59:21.760559 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:21.760539    4645 cpu_manager.go:414] "containerMap: removing container" podUID="1d14cdb3-49a7-4bed-ac45-b18e6452ebbe" containerName="revoke-keys"
Oct 31 19:59:21.760559 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:21.760556    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d14cdb3-49a7-4bed-ac45-b18e6452ebbe" containerName="revoke-keys"
Oct 31 19:59:21.760733 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:21.760588    4645 cpu_manager.go:414] "containerMap: removing container" podUID="daaee800-f23a-4154-b55c-7007f2385707" containerName="test-curl-7020-3da8ea"
Oct 31 19:59:21.760733 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:21.760593    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daaee800-f23a-4154-b55c-7007f2385707" containerName="test-curl-7020-3da8ea"
Oct 31 19:59:21.760733 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:21.760704    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d14cdb3-49a7-4bed-ac45-b18e6452ebbe" containerName="revoke-keys"
Oct 31 19:59:21.760733 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:21.760717    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="daaee800-f23a-4154-b55c-7007f2385707" containerName="test-curl-7020-3da8ea"
Oct 31 19:59:21.767076 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:21.767060    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89846e5e-82ghj"
Oct 31 19:59:21.770740 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:21.770702    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:59:21.770897 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:21.770801    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 19:59:21.777424 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:21.775516    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89846e5e-82ghj"]
Oct 31 19:59:21.885278 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:21.885249    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f0a7362-4c8c-41c4-b8f6-7627cf1e1ef7-kube-api-access-rmj4k\" (UniqueName: \"kubernetes.io/projected/8f0a7362-4c8c-41c4-b8f6-7627cf1e1ef7-kube-api-access-rmj4k\") pod \"8f0a7362-4c8c-41c4-b8f6-7627cf1e1ef7\" (UID: \"8f0a7362-4c8c-41c4-b8f6-7627cf1e1ef7\") "
Oct 31 19:59:21.885763 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:21.885586    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwtsx\" (UniqueName: \"kubernetes.io/projected/6d5386a8-12b7-4668-a255-7d0861f01989-kube-api-access-fwtsx\") pod \"maas-api-revoke-sub-89846e5e-82ghj\" (UID: \"6d5386a8-12b7-4668-a255-7d0861f01989\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89846e5e-82ghj"
Oct 31 19:59:21.885763 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:21.885635    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d5386a8-12b7-4668-a255-7d0861f01989-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89846e5e-82ghj\" (UID: \"6d5386a8-12b7-4668-a255-7d0861f01989\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89846e5e-82ghj"
Oct 31 19:59:21.885872 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:21.885849    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d5386a8-12b7-4668-a255-7d0861f01989-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89846e5e-82ghj\" (UID: \"6d5386a8-12b7-4668-a255-7d0861f01989\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89846e5e-82ghj"
Oct 31 19:59:21.888211 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:21.888182    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f0a7362-4c8c-41c4-b8f6-7627cf1e1ef7-kube-api-access-rmj4k" pod "8f0a7362-4c8c-41c4-b8f6-7627cf1e1ef7" (UID: "8f0a7362-4c8c-41c4-b8f6-7627cf1e1ef7"). InnerVolumeSpecName "kube-api-access-rmj4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:59:21.943107 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:21.943080    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:21.989380 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:21.989114    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwtsx\" (UniqueName: \"kubernetes.io/projected/6d5386a8-12b7-4668-a255-7d0861f01989-kube-api-access-fwtsx\") pod \"maas-api-revoke-sub-89846e5e-82ghj\" (UID: \"6d5386a8-12b7-4668-a255-7d0861f01989\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89846e5e-82ghj"
Oct 31 19:59:21.989739 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:21.989704    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d5386a8-12b7-4668-a255-7d0861f01989-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89846e5e-82ghj\" (UID: \"6d5386a8-12b7-4668-a255-7d0861f01989\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89846e5e-82ghj"
Oct 31 19:59:21.990127 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:21.990100    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d5386a8-12b7-4668-a255-7d0861f01989-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89846e5e-82ghj\" (UID: \"6d5386a8-12b7-4668-a255-7d0861f01989\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89846e5e-82ghj"
Oct 31 19:59:21.990411 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:21.990355    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmj4k\" (UniqueName: \"kubernetes.io/projected/8f0a7362-4c8c-41c4-b8f6-7627cf1e1ef7-kube-api-access-rmj4k\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:21.993591 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:21.993443    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d5386a8-12b7-4668-a255-7d0861f01989-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89846e5e-82ghj\" (UID: \"6d5386a8-12b7-4668-a255-7d0861f01989\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89846e5e-82ghj"
Oct 31 19:59:22.002499 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:22.002377    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d5386a8-12b7-4668-a255-7d0861f01989-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89846e5e-82ghj\" (UID: \"6d5386a8-12b7-4668-a255-7d0861f01989\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89846e5e-82ghj"
Oct 31 19:59:22.011908 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:22.009465    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwtsx\" (UniqueName: \"kubernetes.io/projected/6d5386a8-12b7-4668-a255-7d0861f01989-kube-api-access-fwtsx\") pod \"maas-api-revoke-sub-89846e5e-82ghj\" (UID: \"6d5386a8-12b7-4668-a255-7d0861f01989\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89846e5e-82ghj"
Oct 31 19:59:22.087805 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:22.087769    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89846e5e-82ghj"
Oct 31 19:59:22.392230 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:22.392064    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89846e5e-82ghj"]
Oct 31 19:59:22.395624 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:59:22.395594    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d5386a8_12b7_4668_a255_7d0861f01989.slice/crio-f81a59a391f3634d5c84f87bf00789d6979a8e1f26acf45ee6655b9d7382b543 WatchSource:0}: Error finding container f81a59a391f3634d5c84f87bf00789d6979a8e1f26acf45ee6655b9d7382b543: Status 404 returned error can't find the container with id f81a59a391f3634d5c84f87bf00789d6979a8e1f26acf45ee6655b9d7382b543
Oct 31 19:59:22.521165 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:22.521143    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7020-d53f28"
Oct 31 19:59:22.521165 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:22.521159    4645 scope.go:122] "RemoveContainer" containerID="c86c25d7fdfef1629617143eb06023bb375691ede0b9c058ef58ed2b728392f4"
Oct 31 19:59:22.522553 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:22.522525    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89846e5e-82ghj" event={"ID":"6d5386a8-12b7-4668-a255-7d0861f01989","Type":"ContainerStarted","Data":"f81a59a391f3634d5c84f87bf00789d6979a8e1f26acf45ee6655b9d7382b543"}
Oct 31 19:59:22.522711 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:22.522683    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7020-3da8ea" podUID="daaee800-f23a-4154-b55c-7007f2385707" containerName="test-curl-7020-3da8ea" containerID="cri-o://6b9e2a0b98f59563d8e2cffe027bb1dcd1a6c5d6ad098c23f705ee97ca63f747" gracePeriod=2
Oct 31 19:59:22.532730 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:59:22.532704    4645 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7020-d53f28\" is forbidden: User \"system:node:ip-10-0-131-94.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-94.us-east-2.compute.internal' and this object" podUID="8f0a7362-4c8c-41c4-b8f6-7627cf1e1ef7" pod="opendatahub/test-curl-7020-d53f28"
Oct 31 19:59:23.009393 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:23.009358    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:23.348742 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:23.348715    4645 patch_prober.go:29] interesting pod/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body=
Oct 31 19:59:23.348858 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:23.348765    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n" podUID="f54f6ce5-712d-430a-be69-8ec4a29e44d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused"
Oct 31 19:59:23.530594 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:23.530524    4645 generic.go:356] "Generic (PLEG): container finished" podID="6d5386a8-12b7-4668-a255-7d0861f01989" containerID="c7fd818880817cc4b98b12ccddc43c17eb4e4a4bd1b9342eda66520de9f279ee" exitCode=0
Oct 31 19:59:23.530594 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:23.530568    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89846e5e-82ghj" event={"ID":"6d5386a8-12b7-4668-a255-7d0861f01989","Type":"ContainerDied","Data":"c7fd818880817cc4b98b12ccddc43c17eb4e4a4bd1b9342eda66520de9f279ee"}
Oct 31 19:59:23.532280 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:23.532258    4645 generic.go:356] "Generic (PLEG): container finished" podID="f54f6ce5-712d-430a-be69-8ec4a29e44d0" containerID="f7a48a62a3970b16f21af40bb7b30de5f7d063f23813c6ccd68033d4c6e884c9" exitCode=0
Oct 31 19:59:23.532427 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:23.532358    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n" event={"ID":"f54f6ce5-712d-430a-be69-8ec4a29e44d0","Type":"ContainerDied","Data":"f7a48a62a3970b16f21af40bb7b30de5f7d063f23813c6ccd68033d4c6e884c9"}
Oct 31 19:59:23.532820 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:23.532801    4645 scope.go:122] "RemoveContainer" containerID="f7a48a62a3970b16f21af40bb7b30de5f7d063f23813c6ccd68033d4c6e884c9"
Oct 31 19:59:23.534446 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:23.534416    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:59:23.534527 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:23.534487    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:59:23.534568 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:23.534527    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:59:23.548477 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:59:23.548452    4645 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7020-d53f28\" is forbidden: User \"system:node:ip-10-0-131-94.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-94.us-east-2.compute.internal' and this object" podUID="8f0a7362-4c8c-41c4-b8f6-7627cf1e1ef7" pod="opendatahub/test-curl-7020-d53f28"
Oct 31 19:59:23.550230 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:59:23.550193    4645 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7020-d53f28\" is forbidden: User \"system:node:ip-10-0-131-94.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-94.us-east-2.compute.internal' and this object" podUID="8f0a7362-4c8c-41c4-b8f6-7627cf1e1ef7" pod="opendatahub/test-curl-7020-d53f28"
Oct 31 19:59:23.623509 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:23.623476    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f0a7362-4c8c-41c4-b8f6-7627cf1e1ef7" path="/var/lib/kubelet/pods/8f0a7362-4c8c-41c4-b8f6-7627cf1e1ef7/volumes"
Oct 31 19:59:23.722618 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:23.722587    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:24.542556 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:24.542483    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n" event={"ID":"f54f6ce5-712d-430a-be69-8ec4a29e44d0","Type":"ContainerStarted","Data":"4ba94447b82efb89836e1cc98145b9fb1404c061978e32f68989cef941847038"}
Oct 31 19:59:24.654364 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:24.652939    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:24.820362 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:24.820323    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89846e5e-82ghj"
Oct 31 19:59:24.853764 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:24.853733    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d5386a8-12b7-4668-a255-7d0861f01989-kube-api-access-fwtsx\" (UniqueName: \"kubernetes.io/projected/6d5386a8-12b7-4668-a255-7d0861f01989-kube-api-access-fwtsx\") pod \"6d5386a8-12b7-4668-a255-7d0861f01989\" (UID: \"6d5386a8-12b7-4668-a255-7d0861f01989\") "
Oct 31 19:59:24.854018 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:24.853993    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d5386a8-12b7-4668-a255-7d0861f01989-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d5386a8-12b7-4668-a255-7d0861f01989-maas-api-openshift-ca\") pod \"6d5386a8-12b7-4668-a255-7d0861f01989\" (UID: \"6d5386a8-12b7-4668-a255-7d0861f01989\") "
Oct 31 19:59:24.854438 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:24.854413    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d5386a8-12b7-4668-a255-7d0861f01989-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d5386a8-12b7-4668-a255-7d0861f01989-maas-api-xks-ca\") pod \"6d5386a8-12b7-4668-a255-7d0861f01989\" (UID: \"6d5386a8-12b7-4668-a255-7d0861f01989\") "
Oct 31 19:59:24.855006 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:24.854980    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d5386a8-12b7-4668-a255-7d0861f01989-maas-api-openshift-ca" pod "6d5386a8-12b7-4668-a255-7d0861f01989" (UID: "6d5386a8-12b7-4668-a255-7d0861f01989"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:59:24.881348 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:24.855435    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d5386a8-12b7-4668-a255-7d0861f01989-maas-api-openshift-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:24.885391 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:24.874897    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d5386a8-12b7-4668-a255-7d0861f01989-kube-api-access-fwtsx" pod "6d5386a8-12b7-4668-a255-7d0861f01989" (UID: "6d5386a8-12b7-4668-a255-7d0861f01989"). InnerVolumeSpecName "kube-api-access-fwtsx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:59:24.894103 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:24.894072    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d5386a8-12b7-4668-a255-7d0861f01989-maas-api-xks-ca" pod "6d5386a8-12b7-4668-a255-7d0861f01989" (UID: "6d5386a8-12b7-4668-a255-7d0861f01989"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:59:24.918009 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:24.917979    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7020-3da8ea"
Oct 31 19:59:24.982998 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:24.982958    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/daaee800-f23a-4154-b55c-7007f2385707-kube-api-access-nld74\" (UniqueName: \"kubernetes.io/projected/daaee800-f23a-4154-b55c-7007f2385707-kube-api-access-nld74\") pod \"daaee800-f23a-4154-b55c-7007f2385707\" (UID: \"daaee800-f23a-4154-b55c-7007f2385707\") "
Oct 31 19:59:24.983867 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:24.983844    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d5386a8-12b7-4668-a255-7d0861f01989-maas-api-xks-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:24.984003 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:24.983870    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwtsx\" (UniqueName: \"kubernetes.io/projected/6d5386a8-12b7-4668-a255-7d0861f01989-kube-api-access-fwtsx\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:24.985812 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:24.985785    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daaee800-f23a-4154-b55c-7007f2385707-kube-api-access-nld74" pod "daaee800-f23a-4154-b55c-7007f2385707" (UID: "daaee800-f23a-4154-b55c-7007f2385707"). InnerVolumeSpecName "kube-api-access-nld74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:59:25.086168 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:25.086073    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nld74\" (UniqueName: \"kubernetes.io/projected/daaee800-f23a-4154-b55c-7007f2385707-kube-api-access-nld74\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:25.215273 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:25.213650    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:25.317713 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:25.317679    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-26cd59-84899b8c48-bgz8c"]
Oct 31 19:59:25.318323 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:25.318309    4645 cpu_manager.go:414] "containerMap: removing container" podUID="6d5386a8-12b7-4668-a255-7d0861f01989" containerName="revoke-keys"
Oct 31 19:59:25.318323 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:25.318324    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d5386a8-12b7-4668-a255-7d0861f01989" containerName="revoke-keys"
Oct 31 19:59:25.318534 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:25.318522    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d5386a8-12b7-4668-a255-7d0861f01989" containerName="revoke-keys"
Oct 31 19:59:25.322128 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:25.322106    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-26cd59-84899b8c48-bgz8c"
Oct 31 19:59:25.323020 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:25.322975    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:25.327855 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:25.327830    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-26cd59\""
Oct 31 19:59:25.328836 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:25.328565    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-26cd59-dockercfg-h2gxt\""
Oct 31 19:59:25.333007 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:25.332986    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-26cd59-84899b8c48-bgz8c"]
Oct 31 19:59:25.380605 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:25.380505    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n"
Oct 31 19:59:25.380785 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:25.380731    4645 patch_prober.go:29] interesting pod/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body=
Oct 31 19:59:25.380864 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:25.380781    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n" podUID="f54f6ce5-712d-430a-be69-8ec4a29e44d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused"
Oct 31 19:59:25.390169 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:25.390136    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35f95598-1a16-43a3-84c0-f85af4894aeb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-26cd59-84899b8c48-bgz8c\" (UID: \"35f95598-1a16-43a3-84c0-f85af4894aeb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-26cd59-84899b8c48-bgz8c"
Oct 31 19:59:25.390364 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:25.390319    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm56h\" (UniqueName: \"kubernetes.io/projected/35f95598-1a16-43a3-84c0-f85af4894aeb-kube-api-access-fm56h\") pod \"payload-processing-e2e-ipp-a-26cd59-84899b8c48-bgz8c\" (UID: \"35f95598-1a16-43a3-84c0-f85af4894aeb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-26cd59-84899b8c48-bgz8c"
Oct 31 19:59:25.390818 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:25.390793    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-26cd59-585984b9f5-z4pwr"]
Oct 31 19:59:25.395178 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:25.395158    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-26cd59-585984b9f5-z4pwr"
Oct 31 19:59:25.405895 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:25.405871    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-26cd59-585984b9f5-z4pwr"]
Oct 31 19:59:25.513485 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:25.511808    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc5j8\" (UniqueName: \"kubernetes.io/projected/1aa2c43f-d011-4aaf-9f2a-63c63479f3c4-kube-api-access-xc5j8\") pod \"payload-pre-processing-e2e-ipp-a-26cd59-585984b9f5-z4pwr\" (UID: \"1aa2c43f-d011-4aaf-9f2a-63c63479f3c4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-26cd59-585984b9f5-z4pwr"
Oct 31 19:59:25.513485 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:25.511930    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm56h\" (UniqueName: \"kubernetes.io/projected/35f95598-1a16-43a3-84c0-f85af4894aeb-kube-api-access-fm56h\") pod \"payload-processing-e2e-ipp-a-26cd59-84899b8c48-bgz8c\" (UID: \"35f95598-1a16-43a3-84c0-f85af4894aeb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-26cd59-84899b8c48-bgz8c"
Oct 31 19:59:25.513485 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:25.512059    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1aa2c43f-d011-4aaf-9f2a-63c63479f3c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-26cd59-585984b9f5-z4pwr\" (UID: \"1aa2c43f-d011-4aaf-9f2a-63c63479f3c4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-26cd59-585984b9f5-z4pwr"
Oct 31 19:59:25.513485 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:25.512162    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35f95598-1a16-43a3-84c0-f85af4894aeb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-26cd59-84899b8c48-bgz8c\" (UID: \"35f95598-1a16-43a3-84c0-f85af4894aeb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-26cd59-84899b8c48-bgz8c"
Oct 31 19:59:25.513485 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:25.513155    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35f95598-1a16-43a3-84c0-f85af4894aeb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-26cd59-84899b8c48-bgz8c\" (UID: \"35f95598-1a16-43a3-84c0-f85af4894aeb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-26cd59-84899b8c48-bgz8c"
Oct 31 19:59:25.663524 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:25.658815    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc5j8\" (UniqueName: \"kubernetes.io/projected/1aa2c43f-d011-4aaf-9f2a-63c63479f3c4-kube-api-access-xc5j8\") pod \"payload-pre-processing-e2e-ipp-a-26cd59-585984b9f5-z4pwr\" (UID: \"1aa2c43f-d011-4aaf-9f2a-63c63479f3c4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-26cd59-585984b9f5-z4pwr"
Oct 31 19:59:25.663524 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:25.659088    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1aa2c43f-d011-4aaf-9f2a-63c63479f3c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-26cd59-585984b9f5-z4pwr\" (UID: \"1aa2c43f-d011-4aaf-9f2a-63c63479f3c4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-26cd59-585984b9f5-z4pwr"
Oct 31 19:59:25.663524 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:25.660036    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1aa2c43f-d011-4aaf-9f2a-63c63479f3c4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-26cd59-585984b9f5-z4pwr\" (UID: \"1aa2c43f-d011-4aaf-9f2a-63c63479f3c4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-26cd59-585984b9f5-z4pwr"
Oct 31 19:59:25.720257 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:25.720068    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm56h\" (UniqueName: \"kubernetes.io/projected/35f95598-1a16-43a3-84c0-f85af4894aeb-kube-api-access-fm56h\") pod \"payload-processing-e2e-ipp-a-26cd59-84899b8c48-bgz8c\" (UID: \"35f95598-1a16-43a3-84c0-f85af4894aeb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-26cd59-84899b8c48-bgz8c"
Oct 31 19:59:25.738944 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:25.738705    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc5j8\" (UniqueName: \"kubernetes.io/projected/1aa2c43f-d011-4aaf-9f2a-63c63479f3c4-kube-api-access-xc5j8\") pod \"payload-pre-processing-e2e-ipp-a-26cd59-585984b9f5-z4pwr\" (UID: \"1aa2c43f-d011-4aaf-9f2a-63c63479f3c4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-26cd59-585984b9f5-z4pwr"
Oct 31 19:59:25.742266 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:25.742245    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-26cd59-84899b8c48-bgz8c"
Oct 31 19:59:25.759443 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:25.759414    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="daaee800-f23a-4154-b55c-7007f2385707" path="/var/lib/kubelet/pods/daaee800-f23a-4154-b55c-7007f2385707/volumes"
Oct 31 19:59:25.763643 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:25.763611    4645 generic.go:356] "Generic (PLEG): container finished" podID="daaee800-f23a-4154-b55c-7007f2385707" containerID="6b9e2a0b98f59563d8e2cffe027bb1dcd1a6c5d6ad098c23f705ee97ca63f747" exitCode=137
Oct 31 19:59:25.763923 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:25.763904    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7020-3da8ea"
Oct 31 19:59:25.764119 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:25.764100    4645 scope.go:122] "RemoveContainer" containerID="6b9e2a0b98f59563d8e2cffe027bb1dcd1a6c5d6ad098c23f705ee97ca63f747"
Oct 31 19:59:25.785561 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:25.785539    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89846e5e-82ghj"
Oct 31 19:59:25.787493 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:25.787473    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89846e5e-82ghj" event={"ID":"6d5386a8-12b7-4668-a255-7d0861f01989","Type":"ContainerDied","Data":"f81a59a391f3634d5c84f87bf00789d6979a8e1f26acf45ee6655b9d7382b543"}
Oct 31 19:59:25.787633 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:25.787618    4645 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f81a59a391f3634d5c84f87bf00789d6979a8e1f26acf45ee6655b9d7382b543"
Oct 31 19:59:25.823858 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:25.823835    4645 scope.go:122] "RemoveContainer" containerID="6b9e2a0b98f59563d8e2cffe027bb1dcd1a6c5d6ad098c23f705ee97ca63f747"
Oct 31 19:59:25.824230 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:59:25.824202    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b9e2a0b98f59563d8e2cffe027bb1dcd1a6c5d6ad098c23f705ee97ca63f747\": container with ID starting with 6b9e2a0b98f59563d8e2cffe027bb1dcd1a6c5d6ad098c23f705ee97ca63f747 not found: ID does not exist" containerID="6b9e2a0b98f59563d8e2cffe027bb1dcd1a6c5d6ad098c23f705ee97ca63f747"
Oct 31 19:59:25.824295 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:25.824243    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b9e2a0b98f59563d8e2cffe027bb1dcd1a6c5d6ad098c23f705ee97ca63f747"} err="failed to get container status \"6b9e2a0b98f59563d8e2cffe027bb1dcd1a6c5d6ad098c23f705ee97ca63f747\": rpc error: code = NotFound desc = could not find container \"6b9e2a0b98f59563d8e2cffe027bb1dcd1a6c5d6ad098c23f705ee97ca63f747\": container with ID starting with 6b9e2a0b98f59563d8e2cffe027bb1dcd1a6c5d6ad098c23f705ee97ca63f747 not found: ID does not exist"
Oct 31 19:59:25.862350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:25.856766    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b66e50dd-pbgn9"]
Oct 31 19:59:25.879396 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:25.878259    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b66e50dd-pbgn9"
Oct 31 19:59:25.889990 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:25.884225    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b66e50dd-pbgn9"]
Oct 31 19:59:25.894576 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:25.894552    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:59:25.894680 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:25.894632    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 19:59:25.968402 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:25.965649    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e725d80-b1c1-45e1-8036-65313087d735-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b66e50dd-pbgn9\" (UID: \"9e725d80-b1c1-45e1-8036-65313087d735\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b66e50dd-pbgn9"
Oct 31 19:59:25.968402 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:25.965878    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e725d80-b1c1-45e1-8036-65313087d735-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b66e50dd-pbgn9\" (UID: \"9e725d80-b1c1-45e1-8036-65313087d735\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b66e50dd-pbgn9"
Oct 31 19:59:25.968402 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:25.965948    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk7q8\" (UniqueName: \"kubernetes.io/projected/9e725d80-b1c1-45e1-8036-65313087d735-kube-api-access-lk7q8\") pod \"maas-api-revoke-sub-b66e50dd-pbgn9\" (UID: \"9e725d80-b1c1-45e1-8036-65313087d735\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b66e50dd-pbgn9"
Oct 31 19:59:26.027917 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:26.027887    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-26cd59-585984b9f5-z4pwr"
Oct 31 19:59:26.087375 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:26.085849    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e725d80-b1c1-45e1-8036-65313087d735-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b66e50dd-pbgn9\" (UID: \"9e725d80-b1c1-45e1-8036-65313087d735\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b66e50dd-pbgn9"
Oct 31 19:59:26.087375 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:26.086148    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk7q8\" (UniqueName: \"kubernetes.io/projected/9e725d80-b1c1-45e1-8036-65313087d735-kube-api-access-lk7q8\") pod \"maas-api-revoke-sub-b66e50dd-pbgn9\" (UID: \"9e725d80-b1c1-45e1-8036-65313087d735\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b66e50dd-pbgn9"
Oct 31 19:59:26.087375 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:26.086557    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e725d80-b1c1-45e1-8036-65313087d735-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b66e50dd-pbgn9\" (UID: \"9e725d80-b1c1-45e1-8036-65313087d735\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b66e50dd-pbgn9"
Oct 31 19:59:26.088844 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:26.088342    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e725d80-b1c1-45e1-8036-65313087d735-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b66e50dd-pbgn9\" (UID: \"9e725d80-b1c1-45e1-8036-65313087d735\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b66e50dd-pbgn9"
Oct 31 19:59:26.091055 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:26.090986    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e725d80-b1c1-45e1-8036-65313087d735-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b66e50dd-pbgn9\" (UID: \"9e725d80-b1c1-45e1-8036-65313087d735\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b66e50dd-pbgn9"
Oct 31 19:59:26.092973 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:26.092847    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-26cd59-84899b8c48-bgz8c"]
Oct 31 19:59:26.103032 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:26.102993    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk7q8\" (UniqueName: \"kubernetes.io/projected/9e725d80-b1c1-45e1-8036-65313087d735-kube-api-access-lk7q8\") pod \"maas-api-revoke-sub-b66e50dd-pbgn9\" (UID: \"9e725d80-b1c1-45e1-8036-65313087d735\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b66e50dd-pbgn9"
Oct 31 19:59:26.162993 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:26.157640    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:26.260695 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:26.260381    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b66e50dd-pbgn9"
Oct 31 19:59:26.348218 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:26.348194    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-26cd59-585984b9f5-z4pwr"]
Oct 31 19:59:26.349359 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:59:26.349312    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aa2c43f_d011_4aaf_9f2a_63c63479f3c4.slice/crio-13e60e2339ec7db90b1066d419d7f8d23ecacc1e27054afdc268d540a556aa64 WatchSource:0}: Error finding container 13e60e2339ec7db90b1066d419d7f8d23ecacc1e27054afdc268d540a556aa64: Status 404 returned error can't find the container with id 13e60e2339ec7db90b1066d419d7f8d23ecacc1e27054afdc268d540a556aa64
Oct 31 19:59:26.384349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:26.382538    4645 patch_prober.go:29] interesting pod/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body=
Oct 31 19:59:26.384349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:26.382591    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n" podUID="f54f6ce5-712d-430a-be69-8ec4a29e44d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused"
Oct 31 19:59:26.648006 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:26.647975    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b66e50dd-pbgn9"]
Oct 31 19:59:26.648845 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:59:26.648804    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e725d80_b1c1_45e1_8036_65313087d735.slice/crio-f860d3ce5d1ca01a2fee3316fdcf6909b2b8504b7f661e18976eed5b67dbd4c0 WatchSource:0}: Error finding container f860d3ce5d1ca01a2fee3316fdcf6909b2b8504b7f661e18976eed5b67dbd4c0: Status 404 returned error can't find the container with id f860d3ce5d1ca01a2fee3316fdcf6909b2b8504b7f661e18976eed5b67dbd4c0
Oct 31 19:59:26.801636 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:26.801598    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-26cd59-84899b8c48-bgz8c" event={"ID":"35f95598-1a16-43a3-84c0-f85af4894aeb","Type":"ContainerStarted","Data":"999aa10dcd792ff4c7ac1d5dbd7e89e3df8b5a72e5c3070c7202f8ca18796336"}
Oct 31 19:59:26.802081 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:26.801658    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-26cd59-84899b8c48-bgz8c" event={"ID":"35f95598-1a16-43a3-84c0-f85af4894aeb","Type":"ContainerStarted","Data":"e7ec891feee59d75fc0ed13e42eacd6a06114d5f0c55fe3ada2d7464a72cc97e"}
Oct 31 19:59:26.802081 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:26.801688    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-26cd59-84899b8c48-bgz8c"
Oct 31 19:59:26.804790 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:26.804763    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b66e50dd-pbgn9" event={"ID":"9e725d80-b1c1-45e1-8036-65313087d735","Type":"ContainerStarted","Data":"f860d3ce5d1ca01a2fee3316fdcf6909b2b8504b7f661e18976eed5b67dbd4c0"}
Oct 31 19:59:26.806369 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:26.806315    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-26cd59-585984b9f5-z4pwr" event={"ID":"1aa2c43f-d011-4aaf-9f2a-63c63479f3c4","Type":"ContainerStarted","Data":"0f6ed8be296db4fe8f82ed8ba6688325b258d0b07978a2d2ea2fb6c1396d003c"}
Oct 31 19:59:26.806479 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:26.806392    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-26cd59-585984b9f5-z4pwr" event={"ID":"1aa2c43f-d011-4aaf-9f2a-63c63479f3c4","Type":"ContainerStarted","Data":"13e60e2339ec7db90b1066d419d7f8d23ecacc1e27054afdc268d540a556aa64"}
Oct 31 19:59:26.806608 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:26.806590    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-26cd59-585984b9f5-z4pwr"
Oct 31 19:59:26.821084 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:26.821043    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-26cd59-84899b8c48-bgz8c" podStartSLOduration=1.821030422 podStartE2EDuration="1.821030422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:59:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:59:26.819375419 +0000 UTC m=+2397.392609551" watchObservedRunningTime="2026-10-31 19:59:26.821030422 +0000 UTC m=+2397.394264559"
Oct 31 19:59:26.843386 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:26.843340    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-26cd59-585984b9f5-z4pwr" podStartSLOduration=1.843309412 podStartE2EDuration="1.843309412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:59:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:59:26.837020934 +0000 UTC m=+2397.410255056" watchObservedRunningTime="2026-10-31 19:59:26.843309412 +0000 UTC m=+2397.416543543"
Oct 31 19:59:27.065366 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:27.061654    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:27.382236 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:27.382200    4645 patch_prober.go:29] interesting pod/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body=
Oct 31 19:59:27.382546 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:27.382515    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n" podUID="f54f6ce5-712d-430a-be69-8ec4a29e44d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused"
Oct 31 19:59:27.814923 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:27.814889    4645 generic.go:356] "Generic (PLEG): container finished" podID="9e725d80-b1c1-45e1-8036-65313087d735" containerID="51e8b49226dda1ac9378769450aa1a85c3d80b6bd98a50e39643bf27b3d1d95a" exitCode=0
Oct 31 19:59:27.815300 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:27.814995    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b66e50dd-pbgn9" event={"ID":"9e725d80-b1c1-45e1-8036-65313087d735","Type":"ContainerDied","Data":"51e8b49226dda1ac9378769450aa1a85c3d80b6bd98a50e39643bf27b3d1d95a"}
Oct 31 19:59:28.143751 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:28.143661    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:28.384691 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:28.384655    4645 patch_prober.go:29] interesting pod/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body=
Oct 31 19:59:28.385662 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:28.385628    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n" podUID="f54f6ce5-712d-430a-be69-8ec4a29e44d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused"
Oct 31 19:59:28.489709 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:28.489682    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-01afb5-db5b465fd-jkvfg"
Oct 31 19:59:28.489995 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:28.489968    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01afb5-6fdd5b76b7-qggmv"
Oct 31 19:59:28.936489 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:28.936387    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:29.036406 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:29.036377    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b66e50dd-pbgn9"
Oct 31 19:59:29.155258 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:29.155165    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e725d80-b1c1-45e1-8036-65313087d735-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e725d80-b1c1-45e1-8036-65313087d735-maas-api-xks-ca\") pod \"9e725d80-b1c1-45e1-8036-65313087d735\" (UID: \"9e725d80-b1c1-45e1-8036-65313087d735\") "
Oct 31 19:59:29.155258 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:29.155228    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e725d80-b1c1-45e1-8036-65313087d735-kube-api-access-lk7q8\" (UniqueName: \"kubernetes.io/projected/9e725d80-b1c1-45e1-8036-65313087d735-kube-api-access-lk7q8\") pod \"9e725d80-b1c1-45e1-8036-65313087d735\" (UID: \"9e725d80-b1c1-45e1-8036-65313087d735\") "
Oct 31 19:59:29.155572 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:29.155451    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e725d80-b1c1-45e1-8036-65313087d735-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e725d80-b1c1-45e1-8036-65313087d735-maas-api-openshift-ca\") pod \"9e725d80-b1c1-45e1-8036-65313087d735\" (UID: \"9e725d80-b1c1-45e1-8036-65313087d735\") "
Oct 31 19:59:29.167351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:29.157998    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e725d80-b1c1-45e1-8036-65313087d735-maas-api-xks-ca" pod "9e725d80-b1c1-45e1-8036-65313087d735" (UID: "9e725d80-b1c1-45e1-8036-65313087d735"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:59:29.195218 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:29.193515    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e725d80-b1c1-45e1-8036-65313087d735-maas-api-openshift-ca" pod "9e725d80-b1c1-45e1-8036-65313087d735" (UID: "9e725d80-b1c1-45e1-8036-65313087d735"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:59:29.195218 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:29.194618    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e725d80-b1c1-45e1-8036-65313087d735-kube-api-access-lk7q8" pod "9e725d80-b1c1-45e1-8036-65313087d735" (UID: "9e725d80-b1c1-45e1-8036-65313087d735"). InnerVolumeSpecName "kube-api-access-lk7q8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:59:29.281842 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:29.281807    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e725d80-b1c1-45e1-8036-65313087d735-maas-api-xks-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:29.282067 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:29.282051    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lk7q8\" (UniqueName: \"kubernetes.io/projected/9e725d80-b1c1-45e1-8036-65313087d735-kube-api-access-lk7q8\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:29.282157 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:29.282146    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e725d80-b1c1-45e1-8036-65313087d735-maas-api-openshift-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:29.380632 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:29.380591    4645 patch_prober.go:29] interesting pod/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused" start-of-body=
Oct 31 19:59:29.380803 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:29.380643    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n" podUID="f54f6ce5-712d-430a-be69-8ec4a29e44d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.225:15021/healthz/ready\": dial tcp 10.133.0.225:15021: connect: connection refused"
Oct 31 19:59:29.720048 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:29.720008    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:29.847663 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:29.847633    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b66e50dd-pbgn9"
Oct 31 19:59:29.847935 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:29.847654    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b66e50dd-pbgn9" event={"ID":"9e725d80-b1c1-45e1-8036-65313087d735","Type":"ContainerDied","Data":"f860d3ce5d1ca01a2fee3316fdcf6909b2b8504b7f661e18976eed5b67dbd4c0"}
Oct 31 19:59:29.848048 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:29.847963    4645 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f860d3ce5d1ca01a2fee3316fdcf6909b2b8504b7f661e18976eed5b67dbd4c0"
Oct 31 19:59:30.504353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:30.497668    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n"
Oct 31 19:59:30.504353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:30.499377    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n"
Oct 31 19:59:30.510348 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:30.507587    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n"
Oct 31 19:59:30.848692 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:30.848599    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:31.718788 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:31.718696    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:32.075959 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:32.075882    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f70b2521-7sgrn"]
Oct 31 19:59:32.076780 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:32.076756    4645 cpu_manager.go:414] "containerMap: removing container" podUID="9e725d80-b1c1-45e1-8036-65313087d735" containerName="revoke-keys"
Oct 31 19:59:32.076780 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:32.076776    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e725d80-b1c1-45e1-8036-65313087d735" containerName="revoke-keys"
Oct 31 19:59:32.076988 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:32.076906    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e725d80-b1c1-45e1-8036-65313087d735" containerName="revoke-keys"
Oct 31 19:59:32.082546 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:32.082522    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f70b2521-7sgrn"
Oct 31 19:59:32.086662 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:32.086635    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:59:32.086797 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:32.086691    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 19:59:32.092995 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:32.092700    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f70b2521-7sgrn"]
Oct 31 19:59:32.237864 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:32.237827    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18543794-fe65-4afa-8335-54df6a28f6db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f70b2521-7sgrn\" (UID: \"18543794-fe65-4afa-8335-54df6a28f6db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f70b2521-7sgrn"
Oct 31 19:59:32.238038 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:32.237999    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18543794-fe65-4afa-8335-54df6a28f6db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f70b2521-7sgrn\" (UID: \"18543794-fe65-4afa-8335-54df6a28f6db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f70b2521-7sgrn"
Oct 31 19:59:32.238038 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:32.238034    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84c9r\" (UniqueName: \"kubernetes.io/projected/18543794-fe65-4afa-8335-54df6a28f6db-kube-api-access-84c9r\") pod \"maas-api-revoke-sub-f70b2521-7sgrn\" (UID: \"18543794-fe65-4afa-8335-54df6a28f6db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f70b2521-7sgrn"
Oct 31 19:59:32.339205 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:32.339146    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18543794-fe65-4afa-8335-54df6a28f6db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f70b2521-7sgrn\" (UID: \"18543794-fe65-4afa-8335-54df6a28f6db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f70b2521-7sgrn"
Oct 31 19:59:32.339365 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:32.339239    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18543794-fe65-4afa-8335-54df6a28f6db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f70b2521-7sgrn\" (UID: \"18543794-fe65-4afa-8335-54df6a28f6db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f70b2521-7sgrn"
Oct 31 19:59:32.339365 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:32.339262    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84c9r\" (UniqueName: \"kubernetes.io/projected/18543794-fe65-4afa-8335-54df6a28f6db-kube-api-access-84c9r\") pod \"maas-api-revoke-sub-f70b2521-7sgrn\" (UID: \"18543794-fe65-4afa-8335-54df6a28f6db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f70b2521-7sgrn"
Oct 31 19:59:32.339931 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:32.339907    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18543794-fe65-4afa-8335-54df6a28f6db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f70b2521-7sgrn\" (UID: \"18543794-fe65-4afa-8335-54df6a28f6db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f70b2521-7sgrn"
Oct 31 19:59:32.341512 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:32.341491    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18543794-fe65-4afa-8335-54df6a28f6db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f70b2521-7sgrn\" (UID: \"18543794-fe65-4afa-8335-54df6a28f6db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f70b2521-7sgrn"
Oct 31 19:59:32.350951 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:32.350931    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84c9r\" (UniqueName: \"kubernetes.io/projected/18543794-fe65-4afa-8335-54df6a28f6db-kube-api-access-84c9r\") pod \"maas-api-revoke-sub-f70b2521-7sgrn\" (UID: \"18543794-fe65-4afa-8335-54df6a28f6db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f70b2521-7sgrn"
Oct 31 19:59:32.409731 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:32.409710    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f70b2521-7sgrn"
Oct 31 19:59:32.490201 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:32.490119    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:32.698054 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:32.698021    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f70b2521-7sgrn"]
Oct 31 19:59:32.699145 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:59:32.699117    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18543794_fe65_4afa_8335_54df6a28f6db.slice/crio-27b4c5698b0e6bcf61f17c7cada60af1051d61fb728ef6bf6cd6bdc08815d330 WatchSource:0}: Error finding container 27b4c5698b0e6bcf61f17c7cada60af1051d61fb728ef6bf6cd6bdc08815d330: Status 404 returned error can't find the container with id 27b4c5698b0e6bcf61f17c7cada60af1051d61fb728ef6bf6cd6bdc08815d330
Oct 31 19:59:32.869728 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:32.869701    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f70b2521-7sgrn" event={"ID":"18543794-fe65-4afa-8335-54df6a28f6db","Type":"ContainerStarted","Data":"27b4c5698b0e6bcf61f17c7cada60af1051d61fb728ef6bf6cd6bdc08815d330"}
Oct 31 19:59:33.280644 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:33.280511    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:33.878499 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:33.878465    4645 generic.go:356] "Generic (PLEG): container finished" podID="18543794-fe65-4afa-8335-54df6a28f6db" containerID="d703025bf065d3c1c3e0fe60c888c9a997be647b03c85ad9537a93260de88d8a" exitCode=0
Oct 31 19:59:33.878499 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:33.878504    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f70b2521-7sgrn" event={"ID":"18543794-fe65-4afa-8335-54df6a28f6db","Type":"ContainerDied","Data":"d703025bf065d3c1c3e0fe60c888c9a997be647b03c85ad9537a93260de88d8a"}
Oct 31 19:59:34.194104 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:34.194074    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:34.949394 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:34.946772    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:35.085391 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:35.085367    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f70b2521-7sgrn"
Oct 31 19:59:35.194831 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:35.194801    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18543794-fe65-4afa-8335-54df6a28f6db-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18543794-fe65-4afa-8335-54df6a28f6db-maas-api-openshift-ca\") pod \"18543794-fe65-4afa-8335-54df6a28f6db\" (UID: \"18543794-fe65-4afa-8335-54df6a28f6db\") "
Oct 31 19:59:35.194991 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:35.194850    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18543794-fe65-4afa-8335-54df6a28f6db-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18543794-fe65-4afa-8335-54df6a28f6db-maas-api-xks-ca\") pod \"18543794-fe65-4afa-8335-54df6a28f6db\" (UID: \"18543794-fe65-4afa-8335-54df6a28f6db\") "
Oct 31 19:59:35.194991 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:35.194923    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18543794-fe65-4afa-8335-54df6a28f6db-kube-api-access-84c9r\" (UniqueName: \"kubernetes.io/projected/18543794-fe65-4afa-8335-54df6a28f6db-kube-api-access-84c9r\") pod \"18543794-fe65-4afa-8335-54df6a28f6db\" (UID: \"18543794-fe65-4afa-8335-54df6a28f6db\") "
Oct 31 19:59:35.195218 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:35.195193    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18543794-fe65-4afa-8335-54df6a28f6db-maas-api-openshift-ca" pod "18543794-fe65-4afa-8335-54df6a28f6db" (UID: "18543794-fe65-4afa-8335-54df6a28f6db"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:59:35.195597 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:35.195567    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18543794-fe65-4afa-8335-54df6a28f6db-maas-api-openshift-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:35.197088 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:35.197065    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18543794-fe65-4afa-8335-54df6a28f6db-maas-api-xks-ca" pod "18543794-fe65-4afa-8335-54df6a28f6db" (UID: "18543794-fe65-4afa-8335-54df6a28f6db"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:59:35.197618 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:35.197592    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18543794-fe65-4afa-8335-54df6a28f6db-kube-api-access-84c9r" pod "18543794-fe65-4afa-8335-54df6a28f6db" (UID: "18543794-fe65-4afa-8335-54df6a28f6db"). InnerVolumeSpecName "kube-api-access-84c9r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:59:35.296507 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:35.296486    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18543794-fe65-4afa-8335-54df6a28f6db-maas-api-xks-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:35.296615 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:35.296509    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84c9r\" (UniqueName: \"kubernetes.io/projected/18543794-fe65-4afa-8335-54df6a28f6db-kube-api-access-84c9r\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:35.739134 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:35.739091    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:35.893610 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:35.893524    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f70b2521-7sgrn" event={"ID":"18543794-fe65-4afa-8335-54df6a28f6db","Type":"ContainerDied","Data":"27b4c5698b0e6bcf61f17c7cada60af1051d61fb728ef6bf6cd6bdc08815d330"}
Oct 31 19:59:35.893610 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:35.893589    4645 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27b4c5698b0e6bcf61f17c7cada60af1051d61fb728ef6bf6cd6bdc08815d330"
Oct 31 19:59:35.893610 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:35.893549    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f70b2521-7sgrn"
Oct 31 19:59:36.246665 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:59:36.246632    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 19:59:36.408713 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:36.408674    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:37.246198 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:37.246165    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:37.348569 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:37.348527    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr"]
Oct 31 19:59:37.349804 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:37.349771    4645 cpu_manager.go:414] "containerMap: removing container" podUID="18543794-fe65-4afa-8335-54df6a28f6db" containerName="revoke-keys"
Oct 31 19:59:37.349949 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:37.349806    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18543794-fe65-4afa-8335-54df6a28f6db" containerName="revoke-keys"
Oct 31 19:59:37.350070 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:37.350050    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="18543794-fe65-4afa-8335-54df6a28f6db" containerName="revoke-keys"
Oct 31 19:59:37.354565 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:37.354545    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr"
Oct 31 19:59:37.366495 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:37.366471    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr"]
Oct 31 19:59:37.531735 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:37.531654    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52d535c0-6d59-4788-9258-bbee438df233-istio-envoy\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr\" (UID: \"52d535c0-6d59-4788-9258-bbee438df233\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr"
Oct 31 19:59:37.531735 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:37.531728    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52d535c0-6d59-4788-9258-bbee438df233-workload-socket\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr\" (UID: \"52d535c0-6d59-4788-9258-bbee438df233\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr"
Oct 31 19:59:37.531921 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:37.531805    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52d535c0-6d59-4788-9258-bbee438df233-workload-certs\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr\" (UID: \"52d535c0-6d59-4788-9258-bbee438df233\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr"
Oct 31 19:59:37.531967 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:37.531919    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf2lc\" (UniqueName: \"kubernetes.io/projected/52d535c0-6d59-4788-9258-bbee438df233-kube-api-access-qf2lc\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr\" (UID: \"52d535c0-6d59-4788-9258-bbee438df233\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr"
Oct 31 19:59:37.532024 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:37.532008    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52d535c0-6d59-4788-9258-bbee438df233-istio-data\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr\" (UID: \"52d535c0-6d59-4788-9258-bbee438df233\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr"
Oct 31 19:59:37.532059 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:37.532040    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52d535c0-6d59-4788-9258-bbee438df233-istiod-ca-cert\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr\" (UID: \"52d535c0-6d59-4788-9258-bbee438df233\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr"
Oct 31 19:59:37.532125 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:37.532113    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52d535c0-6d59-4788-9258-bbee438df233-istio-podinfo\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr\" (UID: \"52d535c0-6d59-4788-9258-bbee438df233\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr"
Oct 31 19:59:37.532187 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:37.532167    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52d535c0-6d59-4788-9258-bbee438df233-istio-token\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr\" (UID: \"52d535c0-6d59-4788-9258-bbee438df233\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr"
Oct 31 19:59:37.532232 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:37.532223    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52d535c0-6d59-4788-9258-bbee438df233-credential-socket\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr\" (UID: \"52d535c0-6d59-4788-9258-bbee438df233\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr"
Oct 31 19:59:37.633794 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:37.633760    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52d535c0-6d59-4788-9258-bbee438df233-istio-podinfo\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr\" (UID: \"52d535c0-6d59-4788-9258-bbee438df233\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr"
Oct 31 19:59:37.633950 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:37.633827    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52d535c0-6d59-4788-9258-bbee438df233-istio-token\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr\" (UID: \"52d535c0-6d59-4788-9258-bbee438df233\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr"
Oct 31 19:59:37.633950 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:37.633846    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52d535c0-6d59-4788-9258-bbee438df233-credential-socket\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr\" (UID: \"52d535c0-6d59-4788-9258-bbee438df233\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr"
Oct 31 19:59:37.633950 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:37.633886    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52d535c0-6d59-4788-9258-bbee438df233-istio-envoy\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr\" (UID: \"52d535c0-6d59-4788-9258-bbee438df233\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr"
Oct 31 19:59:37.633950 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:37.633929    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52d535c0-6d59-4788-9258-bbee438df233-workload-socket\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr\" (UID: \"52d535c0-6d59-4788-9258-bbee438df233\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr"
Oct 31 19:59:37.633950 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:37.633952    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52d535c0-6d59-4788-9258-bbee438df233-workload-certs\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr\" (UID: \"52d535c0-6d59-4788-9258-bbee438df233\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr"
Oct 31 19:59:37.634255 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:37.634004    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf2lc\" (UniqueName: \"kubernetes.io/projected/52d535c0-6d59-4788-9258-bbee438df233-kube-api-access-qf2lc\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr\" (UID: \"52d535c0-6d59-4788-9258-bbee438df233\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr"
Oct 31 19:59:37.634255 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:37.634089    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52d535c0-6d59-4788-9258-bbee438df233-istio-data\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr\" (UID: \"52d535c0-6d59-4788-9258-bbee438df233\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr"
Oct 31 19:59:37.634255 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:37.634111    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52d535c0-6d59-4788-9258-bbee438df233-istiod-ca-cert\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr\" (UID: \"52d535c0-6d59-4788-9258-bbee438df233\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr"
Oct 31 19:59:37.634455 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:37.634436    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52d535c0-6d59-4788-9258-bbee438df233-workload-socket\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr\" (UID: \"52d535c0-6d59-4788-9258-bbee438df233\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr"
Oct 31 19:59:37.634520 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:37.634483    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52d535c0-6d59-4788-9258-bbee438df233-workload-certs\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr\" (UID: \"52d535c0-6d59-4788-9258-bbee438df233\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr"
Oct 31 19:59:37.634618 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:37.634594    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52d535c0-6d59-4788-9258-bbee438df233-istio-data\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr\" (UID: \"52d535c0-6d59-4788-9258-bbee438df233\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr"
Oct 31 19:59:37.634736 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:37.634715    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52d535c0-6d59-4788-9258-bbee438df233-credential-socket\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr\" (UID: \"52d535c0-6d59-4788-9258-bbee438df233\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr"
Oct 31 19:59:37.634851 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:37.634831    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52d535c0-6d59-4788-9258-bbee438df233-istiod-ca-cert\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr\" (UID: \"52d535c0-6d59-4788-9258-bbee438df233\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr"
Oct 31 19:59:37.636246 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:37.636219    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52d535c0-6d59-4788-9258-bbee438df233-istio-envoy\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr\" (UID: \"52d535c0-6d59-4788-9258-bbee438df233\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr"
Oct 31 19:59:37.636475 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:37.636455    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52d535c0-6d59-4788-9258-bbee438df233-istio-podinfo\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr\" (UID: \"52d535c0-6d59-4788-9258-bbee438df233\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr"
Oct 31 19:59:37.646814 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:37.646790    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52d535c0-6d59-4788-9258-bbee438df233-istio-token\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr\" (UID: \"52d535c0-6d59-4788-9258-bbee438df233\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr"
Oct 31 19:59:37.647106 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:37.647091    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf2lc\" (UniqueName: \"kubernetes.io/projected/52d535c0-6d59-4788-9258-bbee438df233-kube-api-access-qf2lc\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr\" (UID: \"52d535c0-6d59-4788-9258-bbee438df233\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr"
Oct 31 19:59:37.673122 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:37.673101    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr"
Oct 31 19:59:37.817712 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:37.817551    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-26cd59-84899b8c48-bgz8c"
Oct 31 19:59:37.817876 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:37.817732    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-26cd59-585984b9f5-z4pwr"
Oct 31 19:59:37.966975 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:37.966936    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c55cf1/auth-test-model-c55cf1-kserve-6bb6c9bd5c-qnh9f"]
Oct 31 19:59:37.979021 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:37.978988    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr"]
Oct 31 19:59:37.982397 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:37.982361    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:59:37.982601 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:37.982580    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:59:37.982724 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:37.982709    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:59:38.010019 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:38.009986    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:38.299350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:38.294137    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:38.493382 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:38.491622    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:38.498717 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:38.497165    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w6-a891ce/auth-test-model-a891ce-kserve-c6cd86cc7-vckq2"]
Oct 31 19:59:38.926051 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:38.926007    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr" event={"ID":"52d535c0-6d59-4788-9258-bbee438df233","Type":"ContainerStarted","Data":"c17be81049cf163d701039e21a9a7baf8ab010531b946507691e08439cf34bd5"}
Oct 31 19:59:38.926238 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:38.926069    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr" event={"ID":"52d535c0-6d59-4788-9258-bbee438df233","Type":"ContainerStarted","Data":"3d21ae2a7db75457a68c444d8c1c9429e68b403f06211ee15fb34c9853d8560f"}
Oct 31 19:59:38.953797 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:38.953746    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr" podStartSLOduration=1.953727943 podStartE2EDuration="1.953727943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:59:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:59:38.952595616 +0000 UTC m=+2409.525829745" watchObservedRunningTime="2026-10-31 19:59:38.953727943 +0000 UTC m=+2409.526962076"
Oct 31 19:59:39.037263 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:39.037220    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:39.506722 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:39.506689    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:39.542435 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:39.542402    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v"]
Oct 31 19:59:39.549363 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:39.549340    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v"
Oct 31 19:59:39.581280 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:39.581249    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-5f3eff-openshift-default-dockercfg-vw6kf\""
Oct 31 19:59:39.589253 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:39.589228    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v"]
Oct 31 19:59:39.618371 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:39.612642    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59f3365e-9975-44b4-b7c5-02e91e891f8d-istio-token\") pod \"e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v\" (UID: \"59f3365e-9975-44b4-b7c5-02e91e891f8d\") " pod="openshift-ingress/e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v"
Oct 31 19:59:39.618371 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:39.612682    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59f3365e-9975-44b4-b7c5-02e91e891f8d-istiod-ca-cert\") pod \"e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v\" (UID: \"59f3365e-9975-44b4-b7c5-02e91e891f8d\") " pod="openshift-ingress/e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v"
Oct 31 19:59:39.618371 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:39.612726    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59f3365e-9975-44b4-b7c5-02e91e891f8d-istio-data\") pod \"e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v\" (UID: \"59f3365e-9975-44b4-b7c5-02e91e891f8d\") " pod="openshift-ingress/e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v"
Oct 31 19:59:39.618371 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:39.612743    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v786v\" (UniqueName: \"kubernetes.io/projected/59f3365e-9975-44b4-b7c5-02e91e891f8d-kube-api-access-v786v\") pod \"e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v\" (UID: \"59f3365e-9975-44b4-b7c5-02e91e891f8d\") " pod="openshift-ingress/e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v"
Oct 31 19:59:39.618371 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:39.612769    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f3365e-9975-44b4-b7c5-02e91e891f8d-credential-socket\") pod \"e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v\" (UID: \"59f3365e-9975-44b4-b7c5-02e91e891f8d\") " pod="openshift-ingress/e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v"
Oct 31 19:59:39.618371 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:39.612908    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59f3365e-9975-44b4-b7c5-02e91e891f8d-workload-certs\") pod \"e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v\" (UID: \"59f3365e-9975-44b4-b7c5-02e91e891f8d\") " pod="openshift-ingress/e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v"
Oct 31 19:59:39.618371 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:39.612929    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59f3365e-9975-44b4-b7c5-02e91e891f8d-istio-podinfo\") pod \"e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v\" (UID: \"59f3365e-9975-44b4-b7c5-02e91e891f8d\") " pod="openshift-ingress/e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v"
Oct 31 19:59:39.618371 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:39.612982    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f3365e-9975-44b4-b7c5-02e91e891f8d-workload-socket\") pod \"e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v\" (UID: \"59f3365e-9975-44b4-b7c5-02e91e891f8d\") " pod="openshift-ingress/e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v"
Oct 31 19:59:39.618371 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:39.613038    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59f3365e-9975-44b4-b7c5-02e91e891f8d-istio-envoy\") pod \"e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v\" (UID: \"59f3365e-9975-44b4-b7c5-02e91e891f8d\") " pod="openshift-ingress/e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v"
Oct 31 19:59:39.680608 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:39.680568    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr"
Oct 31 19:59:39.682409 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:39.682374    4645 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" start-of-body=
Oct 31 19:59:39.682599 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:39.682428    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr" podUID="52d535c0-6d59-4788-9258-bbee438df233" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused"
Oct 31 19:59:39.717353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:39.715061    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59f3365e-9975-44b4-b7c5-02e91e891f8d-workload-certs\") pod \"e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v\" (UID: \"59f3365e-9975-44b4-b7c5-02e91e891f8d\") " pod="openshift-ingress/e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v"
Oct 31 19:59:39.717353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:39.715110    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59f3365e-9975-44b4-b7c5-02e91e891f8d-istio-podinfo\") pod \"e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v\" (UID: \"59f3365e-9975-44b4-b7c5-02e91e891f8d\") " pod="openshift-ingress/e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v"
Oct 31 19:59:39.717353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:39.715236    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f3365e-9975-44b4-b7c5-02e91e891f8d-workload-socket\") pod \"e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v\" (UID: \"59f3365e-9975-44b4-b7c5-02e91e891f8d\") " pod="openshift-ingress/e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v"
Oct 31 19:59:39.717353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:39.715423    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59f3365e-9975-44b4-b7c5-02e91e891f8d-istio-envoy\") pod \"e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v\" (UID: \"59f3365e-9975-44b4-b7c5-02e91e891f8d\") " pod="openshift-ingress/e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v"
Oct 31 19:59:39.717353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:39.715524    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59f3365e-9975-44b4-b7c5-02e91e891f8d-istio-token\") pod \"e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v\" (UID: \"59f3365e-9975-44b4-b7c5-02e91e891f8d\") " pod="openshift-ingress/e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v"
Oct 31 19:59:39.717353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:39.715574    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59f3365e-9975-44b4-b7c5-02e91e891f8d-istiod-ca-cert\") pod \"e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v\" (UID: \"59f3365e-9975-44b4-b7c5-02e91e891f8d\") " pod="openshift-ingress/e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v"
Oct 31 19:59:39.717353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:39.715648    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59f3365e-9975-44b4-b7c5-02e91e891f8d-istio-data\") pod \"e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v\" (UID: \"59f3365e-9975-44b4-b7c5-02e91e891f8d\") " pod="openshift-ingress/e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v"
Oct 31 19:59:39.717353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:39.715674    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v786v\" (UniqueName: \"kubernetes.io/projected/59f3365e-9975-44b4-b7c5-02e91e891f8d-kube-api-access-v786v\") pod \"e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v\" (UID: \"59f3365e-9975-44b4-b7c5-02e91e891f8d\") " pod="openshift-ingress/e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v"
Oct 31 19:59:39.717353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:39.715714    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f3365e-9975-44b4-b7c5-02e91e891f8d-credential-socket\") pod \"e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v\" (UID: \"59f3365e-9975-44b4-b7c5-02e91e891f8d\") " pod="openshift-ingress/e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v"
Oct 31 19:59:39.728323 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:39.718700    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59f3365e-9975-44b4-b7c5-02e91e891f8d-workload-certs\") pod \"e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v\" (UID: \"59f3365e-9975-44b4-b7c5-02e91e891f8d\") " pod="openshift-ingress/e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v"
Oct 31 19:59:39.732348 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:39.729246    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59f3365e-9975-44b4-b7c5-02e91e891f8d-istiod-ca-cert\") pod \"e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v\" (UID: \"59f3365e-9975-44b4-b7c5-02e91e891f8d\") " pod="openshift-ingress/e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v"
Oct 31 19:59:39.732348 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:39.729544    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f3365e-9975-44b4-b7c5-02e91e891f8d-workload-socket\") pod \"e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v\" (UID: \"59f3365e-9975-44b4-b7c5-02e91e891f8d\") " pod="openshift-ingress/e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v"
Oct 31 19:59:39.732348 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:39.731711    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f3365e-9975-44b4-b7c5-02e91e891f8d-credential-socket\") pod \"e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v\" (UID: \"59f3365e-9975-44b4-b7c5-02e91e891f8d\") " pod="openshift-ingress/e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v"
Oct 31 19:59:39.732660 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:39.732377    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59f3365e-9975-44b4-b7c5-02e91e891f8d-istio-data\") pod \"e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v\" (UID: \"59f3365e-9975-44b4-b7c5-02e91e891f8d\") " pod="openshift-ingress/e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v"
Oct 31 19:59:39.770398 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:39.770141    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59f3365e-9975-44b4-b7c5-02e91e891f8d-istio-podinfo\") pod \"e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v\" (UID: \"59f3365e-9975-44b4-b7c5-02e91e891f8d\") " pod="openshift-ingress/e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v"
Oct 31 19:59:39.788347 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:39.787660    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59f3365e-9975-44b4-b7c5-02e91e891f8d-istio-envoy\") pod \"e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v\" (UID: \"59f3365e-9975-44b4-b7c5-02e91e891f8d\") " pod="openshift-ingress/e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v"
Oct 31 19:59:39.800357 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:39.788807    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:39.942396 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:39.941809    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59f3365e-9975-44b4-b7c5-02e91e891f8d-istio-token\") pod \"e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v\" (UID: \"59f3365e-9975-44b4-b7c5-02e91e891f8d\") " pod="openshift-ingress/e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v"
Oct 31 19:59:39.954350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:39.943554    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:39.998282 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:39.998198    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v786v\" (UniqueName: \"kubernetes.io/projected/59f3365e-9975-44b4-b7c5-02e91e891f8d-kube-api-access-v786v\") pod \"e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v\" (UID: \"59f3365e-9975-44b4-b7c5-02e91e891f8d\") " pod="openshift-ingress/e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v"
Oct 31 19:59:40.189006 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:40.188973    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v"
Oct 31 19:59:40.479400 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:40.479289    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:40.521641 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:40.515281    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v"]
Oct 31 19:59:40.540073 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:59:40.539032    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59f3365e_9975_44b4_b7c5_02e91e891f8d.slice/crio-34f65d220f0917e69d90e700100b1b49a08d3573dcf70aa992cb4ed2724c8d25 WatchSource:0}: Error finding container 34f65d220f0917e69d90e700100b1b49a08d3573dcf70aa992cb4ed2724c8d25: Status 404 returned error can't find the container with id 34f65d220f0917e69d90e700100b1b49a08d3573dcf70aa992cb4ed2724c8d25
Oct 31 19:59:40.546562 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:40.546512    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:59:40.546846 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:40.546806    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:59:40.546993 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:40.546976    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:59:40.677349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:40.673787    4645 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" start-of-body=
Oct 31 19:59:40.677349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:40.673841    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr" podUID="52d535c0-6d59-4788-9258-bbee438df233" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused"
Oct 31 19:59:41.049063 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:41.049021    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v" event={"ID":"59f3365e-9975-44b4-b7c5-02e91e891f8d","Type":"ContainerStarted","Data":"3ed7967e7fa87c094576a3a4207749c0f93597da2f202741498966e30ff1a490"}
Oct 31 19:59:41.049223 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:41.049091    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v" event={"ID":"59f3365e-9975-44b4-b7c5-02e91e891f8d","Type":"ContainerStarted","Data":"34f65d220f0917e69d90e700100b1b49a08d3573dcf70aa992cb4ed2724c8d25"}
Oct 31 19:59:41.194305 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:41.192550    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v"
Oct 31 19:59:41.254913 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:41.254878    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v"
Oct 31 19:59:41.285999 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:41.285931    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v" podStartSLOduration=2.285883191 podStartE2EDuration="2.285883191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:59:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:59:41.088879165 +0000 UTC m=+2411.662113295" watchObservedRunningTime="2026-10-31 19:59:41.285883191 +0000 UTC m=+2411.859117323"
Oct 31 19:59:41.376405 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:41.376140    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:41.678351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:41.677420    4645 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" start-of-body=
Oct 31 19:59:41.678351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:41.677480    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr" podUID="52d535c0-6d59-4788-9258-bbee438df233" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused"
Oct 31 19:59:42.084407 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:42.072040    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v"
Oct 31 19:59:42.084407 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:42.081733    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v"
Oct 31 19:59:42.281109 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:42.281072    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:42.677352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:42.676660    4645 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" start-of-body=
Oct 31 19:59:42.677352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:42.676720    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr" podUID="52d535c0-6d59-4788-9258-bbee438df233" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused"
Oct 31 19:59:42.954234 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:42.954154    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b1ba135-ndbx5"]
Oct 31 19:59:42.985351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:42.978641    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1ba135-ndbx5"
Oct 31 19:59:42.985351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:42.984585    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 19:59:42.985351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:42.984658    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:59:42.994354 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:42.989543    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b1ba135-ndbx5"]
Oct 31 19:59:43.094282 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:43.094249    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1ea88a5-c67d-4dc3-818b-e93c61a03a82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b1ba135-ndbx5\" (UID: \"f1ea88a5-c67d-4dc3-818b-e93c61a03a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1ba135-ndbx5"
Oct 31 19:59:43.094555 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:43.094535    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qglnl\" (UniqueName: \"kubernetes.io/projected/f1ea88a5-c67d-4dc3-818b-e93c61a03a82-kube-api-access-qglnl\") pod \"maas-api-revoke-sub-1b1ba135-ndbx5\" (UID: \"f1ea88a5-c67d-4dc3-818b-e93c61a03a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1ba135-ndbx5"
Oct 31 19:59:43.094667 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:43.094596    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1ea88a5-c67d-4dc3-818b-e93c61a03a82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b1ba135-ndbx5\" (UID: \"f1ea88a5-c67d-4dc3-818b-e93c61a03a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1ba135-ndbx5"
Oct 31 19:59:43.196503 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:43.196467    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qglnl\" (UniqueName: \"kubernetes.io/projected/f1ea88a5-c67d-4dc3-818b-e93c61a03a82-kube-api-access-qglnl\") pod \"maas-api-revoke-sub-1b1ba135-ndbx5\" (UID: \"f1ea88a5-c67d-4dc3-818b-e93c61a03a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1ba135-ndbx5"
Oct 31 19:59:43.196691 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:43.196545    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1ea88a5-c67d-4dc3-818b-e93c61a03a82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b1ba135-ndbx5\" (UID: \"f1ea88a5-c67d-4dc3-818b-e93c61a03a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1ba135-ndbx5"
Oct 31 19:59:43.196691 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:43.196646    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1ea88a5-c67d-4dc3-818b-e93c61a03a82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b1ba135-ndbx5\" (UID: \"f1ea88a5-c67d-4dc3-818b-e93c61a03a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1ba135-ndbx5"
Oct 31 19:59:43.197944 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:43.197915    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1ea88a5-c67d-4dc3-818b-e93c61a03a82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b1ba135-ndbx5\" (UID: \"f1ea88a5-c67d-4dc3-818b-e93c61a03a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1ba135-ndbx5"
Oct 31 19:59:43.199657 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:43.199631    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1ea88a5-c67d-4dc3-818b-e93c61a03a82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b1ba135-ndbx5\" (UID: \"f1ea88a5-c67d-4dc3-818b-e93c61a03a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1ba135-ndbx5"
Oct 31 19:59:43.208530 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:43.208475    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qglnl\" (UniqueName: \"kubernetes.io/projected/f1ea88a5-c67d-4dc3-818b-e93c61a03a82-kube-api-access-qglnl\") pod \"maas-api-revoke-sub-1b1ba135-ndbx5\" (UID: \"f1ea88a5-c67d-4dc3-818b-e93c61a03a82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1ba135-ndbx5"
Oct 31 19:59:43.304346 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:43.298981    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w6-e1d060/auth-test-model-e1d060-kserve-589d89d7b9-2vkt5"]
Oct 31 19:59:43.313346 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:43.310495    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1ba135-ndbx5"
Oct 31 19:59:43.381372 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:43.362936    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:43.550185 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:43.543703    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:43.681350 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:43.667190    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:43.718376 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:43.710622    4645 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" start-of-body=
Oct 31 19:59:43.718376 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:43.710744    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr" podUID="52d535c0-6d59-4788-9258-bbee438df233" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused"
Oct 31 19:59:43.932783 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:43.883301    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b1ba135-ndbx5"]
Oct 31 19:59:44.099761 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:44.099731    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1ba135-ndbx5" event={"ID":"f1ea88a5-c67d-4dc3-818b-e93c61a03a82","Type":"ContainerStarted","Data":"f00846a15755732839e7196b8f623412b73aea57351f7e7df1102f98e27fd583"}
Oct 31 19:59:44.673671 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:44.673635    4645 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused" start-of-body=
Oct 31 19:59:44.673845 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:44.673684    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr" podUID="52d535c0-6d59-4788-9258-bbee438df233" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.239:15021/healthz/ready\": dial tcp 10.133.0.239:15021: connect: connection refused"
Oct 31 19:59:45.031349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:45.030442    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:45.032868 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:45.032839    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5f3eff-6f85d65cf9-2c4l7"]
Oct 31 19:59:45.037641 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:45.037622    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5f3eff-6f85d65cf9-2c4l7"
Oct 31 19:59:45.042595 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:45.042552    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-5f3eff\""
Oct 31 19:59:45.043281 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:45.043245    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-5f3eff-dockercfg-89bpq\""
Oct 31 19:59:45.047299 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:45.047277    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5f3eff-6f85d65cf9-2c4l7"]
Oct 31 19:59:45.096517 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:45.096488    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5f3eff-7655b4885c-k6dn9"]
Oct 31 19:59:45.101868 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:45.101834    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5f3eff-7655b4885c-k6dn9"
Oct 31 19:59:45.123607 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:45.123202    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5f3eff-7655b4885c-k6dn9"]
Oct 31 19:59:45.135886 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:45.135854    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b4ca125-b910-4fe9-91ed-0a8e1dfa6e3e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5f3eff-6f85d65cf9-2c4l7\" (UID: \"7b4ca125-b910-4fe9-91ed-0a8e1dfa6e3e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5f3eff-6f85d65cf9-2c4l7"
Oct 31 19:59:45.136021 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:45.136001    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49zt2\" (UniqueName: \"kubernetes.io/projected/7b4ca125-b910-4fe9-91ed-0a8e1dfa6e3e-kube-api-access-49zt2\") pod \"payload-pre-processing-e2e-ipp-b-5f3eff-6f85d65cf9-2c4l7\" (UID: \"7b4ca125-b910-4fe9-91ed-0a8e1dfa6e3e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5f3eff-6f85d65cf9-2c4l7"
Oct 31 19:59:45.141628 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:45.141533    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:45.148611 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:45.148586    4645 generic.go:356] "Generic (PLEG): container finished" podID="f1ea88a5-c67d-4dc3-818b-e93c61a03a82" containerID="16875b9e9e74219d6de1c3dda4aedcc4945145f5b8df793486eab95cf93a33b7" exitCode=0
Oct 31 19:59:45.149202 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:45.148768    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1ba135-ndbx5" event={"ID":"f1ea88a5-c67d-4dc3-818b-e93c61a03a82","Type":"ContainerDied","Data":"16875b9e9e74219d6de1c3dda4aedcc4945145f5b8df793486eab95cf93a33b7"}
Oct 31 19:59:45.249495 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:45.240909    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgpcg\" (UniqueName: \"kubernetes.io/projected/71f14650-d480-4e47-90c3-2ca4afac56d9-kube-api-access-jgpcg\") pod \"payload-processing-e2e-ipp-b-5f3eff-7655b4885c-k6dn9\" (UID: \"71f14650-d480-4e47-90c3-2ca4afac56d9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5f3eff-7655b4885c-k6dn9"
Oct 31 19:59:45.249495 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:45.241012    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49zt2\" (UniqueName: \"kubernetes.io/projected/7b4ca125-b910-4fe9-91ed-0a8e1dfa6e3e-kube-api-access-49zt2\") pod \"payload-pre-processing-e2e-ipp-b-5f3eff-6f85d65cf9-2c4l7\" (UID: \"7b4ca125-b910-4fe9-91ed-0a8e1dfa6e3e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5f3eff-6f85d65cf9-2c4l7"
Oct 31 19:59:45.249495 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:45.241163    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71f14650-d480-4e47-90c3-2ca4afac56d9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5f3eff-7655b4885c-k6dn9\" (UID: \"71f14650-d480-4e47-90c3-2ca4afac56d9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5f3eff-7655b4885c-k6dn9"
Oct 31 19:59:45.249495 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:45.241547    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b4ca125-b910-4fe9-91ed-0a8e1dfa6e3e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5f3eff-6f85d65cf9-2c4l7\" (UID: \"7b4ca125-b910-4fe9-91ed-0a8e1dfa6e3e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5f3eff-6f85d65cf9-2c4l7"
Oct 31 19:59:45.249495 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:45.242835    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b4ca125-b910-4fe9-91ed-0a8e1dfa6e3e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5f3eff-6f85d65cf9-2c4l7\" (UID: \"7b4ca125-b910-4fe9-91ed-0a8e1dfa6e3e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5f3eff-6f85d65cf9-2c4l7"
Oct 31 19:59:45.303479 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:45.303437    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49zt2\" (UniqueName: \"kubernetes.io/projected/7b4ca125-b910-4fe9-91ed-0a8e1dfa6e3e-kube-api-access-49zt2\") pod \"payload-pre-processing-e2e-ipp-b-5f3eff-6f85d65cf9-2c4l7\" (UID: \"7b4ca125-b910-4fe9-91ed-0a8e1dfa6e3e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5f3eff-6f85d65cf9-2c4l7"
Oct 31 19:59:45.304127 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:45.304062    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:45.365351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:45.362224    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgpcg\" (UniqueName: \"kubernetes.io/projected/71f14650-d480-4e47-90c3-2ca4afac56d9-kube-api-access-jgpcg\") pod \"payload-processing-e2e-ipp-b-5f3eff-7655b4885c-k6dn9\" (UID: \"71f14650-d480-4e47-90c3-2ca4afac56d9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5f3eff-7655b4885c-k6dn9"
Oct 31 19:59:45.365351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:45.362871    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71f14650-d480-4e47-90c3-2ca4afac56d9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5f3eff-7655b4885c-k6dn9\" (UID: \"71f14650-d480-4e47-90c3-2ca4afac56d9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5f3eff-7655b4885c-k6dn9"
Oct 31 19:59:45.365351 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:45.364343    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71f14650-d480-4e47-90c3-2ca4afac56d9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5f3eff-7655b4885c-k6dn9\" (UID: \"71f14650-d480-4e47-90c3-2ca4afac56d9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5f3eff-7655b4885c-k6dn9"
Oct 31 19:59:45.391361 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:45.389074    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5f3eff-6f85d65cf9-2c4l7"
Oct 31 19:59:45.396268 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:45.396222    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgpcg\" (UniqueName: \"kubernetes.io/projected/71f14650-d480-4e47-90c3-2ca4afac56d9-kube-api-access-jgpcg\") pod \"payload-processing-e2e-ipp-b-5f3eff-7655b4885c-k6dn9\" (UID: \"71f14650-d480-4e47-90c3-2ca4afac56d9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5f3eff-7655b4885c-k6dn9"
Oct 31 19:59:45.467439 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:45.455661    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5f3eff-7655b4885c-k6dn9"
Oct 31 19:59:45.680145 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:45.680110    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr"
Oct 31 19:59:45.681088 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:45.681061    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr"
Oct 31 19:59:45.681737 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:45.681714    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr"
Oct 31 19:59:45.787340 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:45.787290    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5f3eff-7655b4885c-k6dn9"]
Oct 31 19:59:45.790279 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:59:45.788693    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71f14650_d480_4e47_90c3_2ca4afac56d9.slice/crio-69a6a3c4cd4b1114574ade0bcd60a9a1b2cfe87c9be1b3d2632ca8349b5bdfe8 WatchSource:0}: Error finding container 69a6a3c4cd4b1114574ade0bcd60a9a1b2cfe87c9be1b3d2632ca8349b5bdfe8: Status 404 returned error can't find the container with id 69a6a3c4cd4b1114574ade0bcd60a9a1b2cfe87c9be1b3d2632ca8349b5bdfe8
Oct 31 19:59:45.792118 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:59:45.792091    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b4ca125_b910_4fe9_91ed_0a8e1dfa6e3e.slice/crio-fb4355d6bf25a03a442260e2dd4a346484bab67555d2ee1c63c23544e8d8083a WatchSource:0}: Error finding container fb4355d6bf25a03a442260e2dd4a346484bab67555d2ee1c63c23544e8d8083a: Status 404 returned error can't find the container with id fb4355d6bf25a03a442260e2dd4a346484bab67555d2ee1c63c23544e8d8083a
Oct 31 19:59:45.794108 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:45.794084    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5f3eff-6f85d65cf9-2c4l7"]
Oct 31 19:59:46.158595 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:46.158518    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5f3eff-7655b4885c-k6dn9" event={"ID":"71f14650-d480-4e47-90c3-2ca4afac56d9","Type":"ContainerStarted","Data":"8b995ebe402bf29905f5a735fa29d3d12377326b2dacf40b683150b80dfdec79"}
Oct 31 19:59:46.158595 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:46.158573    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5f3eff-7655b4885c-k6dn9" event={"ID":"71f14650-d480-4e47-90c3-2ca4afac56d9","Type":"ContainerStarted","Data":"69a6a3c4cd4b1114574ade0bcd60a9a1b2cfe87c9be1b3d2632ca8349b5bdfe8"}
Oct 31 19:59:46.159062 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:46.158602    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-5f3eff-7655b4885c-k6dn9"
Oct 31 19:59:46.160465 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:46.160438    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5f3eff-6f85d65cf9-2c4l7" event={"ID":"7b4ca125-b910-4fe9-91ed-0a8e1dfa6e3e","Type":"ContainerStarted","Data":"39eed9102bca75571d7403dfd57f986de07d4b85ca7f4be5f42d6c3eb346a3c0"}
Oct 31 19:59:46.160593 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:46.160488    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5f3eff-6f85d65cf9-2c4l7" event={"ID":"7b4ca125-b910-4fe9-91ed-0a8e1dfa6e3e","Type":"ContainerStarted","Data":"fb4355d6bf25a03a442260e2dd4a346484bab67555d2ee1c63c23544e8d8083a"}
Oct 31 19:59:46.181023 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:46.180978    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-5f3eff-7655b4885c-k6dn9" podStartSLOduration=1.180965545 podStartE2EDuration="1.180965545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:59:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:59:46.17563383 +0000 UTC m=+2416.748867959" watchObservedRunningTime="2026-10-31 19:59:46.180965545 +0000 UTC m=+2416.754199674"
Oct 31 19:59:46.182100 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:46.182035    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:46.201207 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:46.201125    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5f3eff-6f85d65cf9-2c4l7" podStartSLOduration=1.2011117 podStartE2EDuration="1.2011117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:59:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:59:46.193289409 +0000 UTC m=+2416.766523543" watchObservedRunningTime="2026-10-31 19:59:46.2011117 +0000 UTC m=+2416.774345828"
Oct 31 19:59:46.367951 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:46.367913    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1ba135-ndbx5"
Oct 31 19:59:46.422942 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:46.422916    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1ea88a5-c67d-4dc3-818b-e93c61a03a82-kube-api-access-qglnl\" (UniqueName: \"kubernetes.io/projected/f1ea88a5-c67d-4dc3-818b-e93c61a03a82-kube-api-access-qglnl\") pod \"f1ea88a5-c67d-4dc3-818b-e93c61a03a82\" (UID: \"f1ea88a5-c67d-4dc3-818b-e93c61a03a82\") "
Oct 31 19:59:46.423113 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:46.423061    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1ea88a5-c67d-4dc3-818b-e93c61a03a82-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1ea88a5-c67d-4dc3-818b-e93c61a03a82-maas-api-openshift-ca\") pod \"f1ea88a5-c67d-4dc3-818b-e93c61a03a82\" (UID: \"f1ea88a5-c67d-4dc3-818b-e93c61a03a82\") "
Oct 31 19:59:46.423113 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:46.423105    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1ea88a5-c67d-4dc3-818b-e93c61a03a82-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1ea88a5-c67d-4dc3-818b-e93c61a03a82-maas-api-xks-ca\") pod \"f1ea88a5-c67d-4dc3-818b-e93c61a03a82\" (UID: \"f1ea88a5-c67d-4dc3-818b-e93c61a03a82\") "
Oct 31 19:59:46.423543 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:46.423501    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1ea88a5-c67d-4dc3-818b-e93c61a03a82-maas-api-openshift-ca" pod "f1ea88a5-c67d-4dc3-818b-e93c61a03a82" (UID: "f1ea88a5-c67d-4dc3-818b-e93c61a03a82"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:59:46.424077 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:46.424049    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1ea88a5-c67d-4dc3-818b-e93c61a03a82-maas-api-openshift-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:46.425141 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:46.425110    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1ea88a5-c67d-4dc3-818b-e93c61a03a82-kube-api-access-qglnl" pod "f1ea88a5-c67d-4dc3-818b-e93c61a03a82" (UID: "f1ea88a5-c67d-4dc3-818b-e93c61a03a82"). InnerVolumeSpecName "kube-api-access-qglnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:59:46.425720 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:46.425696    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1ea88a5-c67d-4dc3-818b-e93c61a03a82-maas-api-xks-ca" pod "f1ea88a5-c67d-4dc3-818b-e93c61a03a82" (UID: "f1ea88a5-c67d-4dc3-818b-e93c61a03a82"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:59:46.525428 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:46.525399    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qglnl\" (UniqueName: \"kubernetes.io/projected/f1ea88a5-c67d-4dc3-818b-e93c61a03a82-kube-api-access-qglnl\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:46.525428 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:46.525425    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1ea88a5-c67d-4dc3-818b-e93c61a03a82-maas-api-xks-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:47.160650 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:47.160616    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:47.169344 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:47.169313    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1ba135-ndbx5"
Oct 31 19:59:47.169552 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:47.169383    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b1ba135-ndbx5" event={"ID":"f1ea88a5-c67d-4dc3-818b-e93c61a03a82","Type":"ContainerDied","Data":"f00846a15755732839e7196b8f623412b73aea57351f7e7df1102f98e27fd583"}
Oct 31 19:59:47.169552 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:47.169434    4645 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f00846a15755732839e7196b8f623412b73aea57351f7e7df1102f98e27fd583"
Oct 31 19:59:47.169700 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:47.169681    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5f3eff-6f85d65cf9-2c4l7"
Oct 31 19:59:48.422709 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:48.422670    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:49.253593 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:49.253555    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:50.062570 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:50.062512    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:50.934533 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:50.934496    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:51.953355 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:51.953300    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:52.413438 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:52.413393    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph"]
Oct 31 19:59:52.414258 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:52.414235    4645 cpu_manager.go:414] "containerMap: removing container" podUID="f1ea88a5-c67d-4dc3-818b-e93c61a03a82" containerName="revoke-keys"
Oct 31 19:59:52.414258 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:52.414260    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1ea88a5-c67d-4dc3-818b-e93c61a03a82" containerName="revoke-keys"
Oct 31 19:59:52.414454 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:52.414441    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1ea88a5-c67d-4dc3-818b-e93c61a03a82" containerName="revoke-keys"
Oct 31 19:59:52.418531 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:52.418508    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph"
Oct 31 19:59:52.435221 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:52.435188    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph"]
Oct 31 19:59:52.533096 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:52.533050    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b24f33df-8939-405b-83a6-8d7af872e7da-istio-data\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph\" (UID: \"b24f33df-8939-405b-83a6-8d7af872e7da\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph"
Oct 31 19:59:52.533289 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:52.533156    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b24f33df-8939-405b-83a6-8d7af872e7da-istiod-ca-cert\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph\" (UID: \"b24f33df-8939-405b-83a6-8d7af872e7da\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph"
Oct 31 19:59:52.533289 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:52.533236    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24f33df-8939-405b-83a6-8d7af872e7da-workload-socket\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph\" (UID: \"b24f33df-8939-405b-83a6-8d7af872e7da\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph"
Oct 31 19:59:52.533416 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:52.533402    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b24f33df-8939-405b-83a6-8d7af872e7da-istio-podinfo\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph\" (UID: \"b24f33df-8939-405b-83a6-8d7af872e7da\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph"
Oct 31 19:59:52.533460 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:52.533427    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b24f33df-8939-405b-83a6-8d7af872e7da-istio-envoy\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph\" (UID: \"b24f33df-8939-405b-83a6-8d7af872e7da\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph"
Oct 31 19:59:52.533495 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:52.533468    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24f33df-8939-405b-83a6-8d7af872e7da-credential-socket\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph\" (UID: \"b24f33df-8939-405b-83a6-8d7af872e7da\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph"
Oct 31 19:59:52.533566 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:52.533541    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w87jq\" (UniqueName: \"kubernetes.io/projected/b24f33df-8939-405b-83a6-8d7af872e7da-kube-api-access-w87jq\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph\" (UID: \"b24f33df-8939-405b-83a6-8d7af872e7da\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph"
Oct 31 19:59:52.533645 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:52.533627    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b24f33df-8939-405b-83a6-8d7af872e7da-workload-certs\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph\" (UID: \"b24f33df-8939-405b-83a6-8d7af872e7da\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph"
Oct 31 19:59:52.533692 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:52.533677    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b24f33df-8939-405b-83a6-8d7af872e7da-istio-token\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph\" (UID: \"b24f33df-8939-405b-83a6-8d7af872e7da\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph"
Oct 31 19:59:52.635049 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:52.634982    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b24f33df-8939-405b-83a6-8d7af872e7da-istio-podinfo\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph\" (UID: \"b24f33df-8939-405b-83a6-8d7af872e7da\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph"
Oct 31 19:59:52.635239 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:52.635149    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b24f33df-8939-405b-83a6-8d7af872e7da-istio-envoy\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph\" (UID: \"b24f33df-8939-405b-83a6-8d7af872e7da\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph"
Oct 31 19:59:52.635239 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:52.635220    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24f33df-8939-405b-83a6-8d7af872e7da-credential-socket\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph\" (UID: \"b24f33df-8939-405b-83a6-8d7af872e7da\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph"
Oct 31 19:59:52.635400 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:52.635286    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w87jq\" (UniqueName: \"kubernetes.io/projected/b24f33df-8939-405b-83a6-8d7af872e7da-kube-api-access-w87jq\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph\" (UID: \"b24f33df-8939-405b-83a6-8d7af872e7da\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph"
Oct 31 19:59:52.635400 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:52.635385    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b24f33df-8939-405b-83a6-8d7af872e7da-workload-certs\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph\" (UID: \"b24f33df-8939-405b-83a6-8d7af872e7da\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph"
Oct 31 19:59:52.635481 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:52.635418    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b24f33df-8939-405b-83a6-8d7af872e7da-istio-token\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph\" (UID: \"b24f33df-8939-405b-83a6-8d7af872e7da\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph"
Oct 31 19:59:52.635545 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:52.635535    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b24f33df-8939-405b-83a6-8d7af872e7da-istio-data\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph\" (UID: \"b24f33df-8939-405b-83a6-8d7af872e7da\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph"
Oct 31 19:59:52.635600 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:52.635573    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b24f33df-8939-405b-83a6-8d7af872e7da-istiod-ca-cert\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph\" (UID: \"b24f33df-8939-405b-83a6-8d7af872e7da\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph"
Oct 31 19:59:52.635664 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:52.635637    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24f33df-8939-405b-83a6-8d7af872e7da-workload-socket\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph\" (UID: \"b24f33df-8939-405b-83a6-8d7af872e7da\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph"
Oct 31 19:59:52.635955 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:52.635928    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24f33df-8939-405b-83a6-8d7af872e7da-credential-socket\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph\" (UID: \"b24f33df-8939-405b-83a6-8d7af872e7da\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph"
Oct 31 19:59:52.636063 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:52.636004    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b24f33df-8939-405b-83a6-8d7af872e7da-istio-data\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph\" (UID: \"b24f33df-8939-405b-83a6-8d7af872e7da\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph"
Oct 31 19:59:52.636165 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:52.636145    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b24f33df-8939-405b-83a6-8d7af872e7da-workload-certs\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph\" (UID: \"b24f33df-8939-405b-83a6-8d7af872e7da\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph"
Oct 31 19:59:52.636267 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:52.636206    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24f33df-8939-405b-83a6-8d7af872e7da-workload-socket\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph\" (UID: \"b24f33df-8939-405b-83a6-8d7af872e7da\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph"
Oct 31 19:59:52.636431 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:52.636409    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b24f33df-8939-405b-83a6-8d7af872e7da-istiod-ca-cert\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph\" (UID: \"b24f33df-8939-405b-83a6-8d7af872e7da\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph"
Oct 31 19:59:52.637682 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:52.637658    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b24f33df-8939-405b-83a6-8d7af872e7da-istio-envoy\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph\" (UID: \"b24f33df-8939-405b-83a6-8d7af872e7da\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph"
Oct 31 19:59:52.637833 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:52.637816    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b24f33df-8939-405b-83a6-8d7af872e7da-istio-podinfo\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph\" (UID: \"b24f33df-8939-405b-83a6-8d7af872e7da\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph"
Oct 31 19:59:52.648122 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:52.648081    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b24f33df-8939-405b-83a6-8d7af872e7da-istio-token\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph\" (UID: \"b24f33df-8939-405b-83a6-8d7af872e7da\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph"
Oct 31 19:59:52.648298 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:52.648244    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w87jq\" (UniqueName: \"kubernetes.io/projected/b24f33df-8939-405b-83a6-8d7af872e7da-kube-api-access-w87jq\") pod \"e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph\" (UID: \"b24f33df-8939-405b-83a6-8d7af872e7da\") " pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph"
Oct 31 19:59:52.734686 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:52.734579    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph"
Oct 31 19:59:53.000303 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:53.000213    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-c55cf1/auth-test-model-c55cf1-kserve-6bb6c9bd5c-qnh9f" podUID="6ba2c7b6-54c7-4610-8051-8fc89f6f8731" containerName="main" containerID="cri-o://2ceeb3d7f440c730b397144ebfbf2d63313b7797cf0919686437ecf2507e674d" gracePeriod=45
Oct 31 19:59:53.032360 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:53.032309    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph"]
Oct 31 19:59:53.053621 ip-10-0-131-94 kubenswrapper[4645]: W1031 19:59:53.053583    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb24f33df_8939_405b_83a6_8d7af872e7da.slice/crio-634c3a98781b99971bc36b5fa61c77cb20e62c33e29ec55ce6f80ecd4750559f WatchSource:0}: Error finding container 634c3a98781b99971bc36b5fa61c77cb20e62c33e29ec55ce6f80ecd4750559f: Status 404 returned error can't find the container with id 634c3a98781b99971bc36b5fa61c77cb20e62c33e29ec55ce6f80ecd4750559f
Oct 31 19:59:53.107344 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:53.107288    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:59:53.107478 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:53.107403    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:59:53.107478 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:53.107456    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:59:53.132492 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:53.132449    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-c55cf1/auth-test-model-c55cf1-kserve-6bb6c9bd5c-qnh9f" podUID="6ba2c7b6-54c7-4610-8051-8fc89f6f8731" containerName="main" probeResult="failure" output="Get \"https://10.133.0.211:8000/ready\": dial tcp 10.133.0.211:8000: connect: connection refused"
Oct 31 19:59:53.188193 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:53.188155    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:53.329470 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:53.327034    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:53.564631 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:53.564582    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w6-a891ce/auth-test-model-a891ce-kserve-c6cd86cc7-vckq2" podUID="5f6f43f0-380a-45d9-a8ab-324872927234" containerName="main" containerID="cri-o://e6425167d186986409fc9f751f86fc01d76cb8e70fc4c95b8e65b6e3ff58da83" gracePeriod=45
Oct 31 19:59:53.574801 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:53.569242    4645 generic.go:356] "Generic (PLEG): container finished" podID="6ba2c7b6-54c7-4610-8051-8fc89f6f8731" containerID="2ceeb3d7f440c730b397144ebfbf2d63313b7797cf0919686437ecf2507e674d" exitCode=0
Oct 31 19:59:53.574801 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:53.569627    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c55cf1/auth-test-model-c55cf1-kserve-6bb6c9bd5c-qnh9f" event={"ID":"6ba2c7b6-54c7-4610-8051-8fc89f6f8731","Type":"ContainerDied","Data":"2ceeb3d7f440c730b397144ebfbf2d63313b7797cf0919686437ecf2507e674d"}
Oct 31 19:59:53.574801 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:53.569713    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf94617d-rhhp2"]
Oct 31 19:59:53.598495 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:53.598448    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf94617d-rhhp2"]
Oct 31 19:59:53.598661 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:53.598583    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf94617d-rhhp2"
Oct 31 19:59:53.599150 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:53.599107    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph" event={"ID":"b24f33df-8939-405b-83a6-8d7af872e7da","Type":"ContainerStarted","Data":"b3422fc5e1f05bc8edd9545342a5050b0eb6813883c69fb12fe63c3bbc81ae57"}
Oct 31 19:59:53.599268 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:53.599177    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph" event={"ID":"b24f33df-8939-405b-83a6-8d7af872e7da","Type":"ContainerStarted","Data":"634c3a98781b99971bc36b5fa61c77cb20e62c33e29ec55ce6f80ecd4750559f"}
Oct 31 19:59:53.603358 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:53.603303    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 19:59:53.603524 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:53.603506    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 19:59:53.665323 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:53.665283    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43db5dd0-3e61-485e-9d44-b141096e887a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf94617d-rhhp2\" (UID: \"43db5dd0-3e61-485e-9d44-b141096e887a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf94617d-rhhp2"
Oct 31 19:59:53.665567 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:53.665542    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmh6x\" (UniqueName: \"kubernetes.io/projected/43db5dd0-3e61-485e-9d44-b141096e887a-kube-api-access-zmh6x\") pod \"maas-api-revoke-sub-bf94617d-rhhp2\" (UID: \"43db5dd0-3e61-485e-9d44-b141096e887a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf94617d-rhhp2"
Oct 31 19:59:53.665865 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:53.665844    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43db5dd0-3e61-485e-9d44-b141096e887a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf94617d-rhhp2\" (UID: \"43db5dd0-3e61-485e-9d44-b141096e887a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf94617d-rhhp2"
Oct 31 19:59:53.674848 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:53.674815    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c55cf1/auth-test-model-c55cf1-kserve-6bb6c9bd5c-qnh9f"
Oct 31 19:59:53.704502 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:53.704435    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph" podStartSLOduration=1.704413689 podStartE2EDuration="1.704413689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:59:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:59:53.664239358 +0000 UTC m=+2424.237473488" watchObservedRunningTime="2026-10-31 19:59:53.704413689 +0000 UTC m=+2424.277647821"
Oct 31 19:59:53.736009 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:53.735707    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph"
Oct 31 19:59:53.767424 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:53.767384    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ba2c7b6-54c7-4610-8051-8fc89f6f8731-tls-certs\" (UniqueName: \"kubernetes.io/secret/6ba2c7b6-54c7-4610-8051-8fc89f6f8731-tls-certs\") pod \"6ba2c7b6-54c7-4610-8051-8fc89f6f8731\" (UID: \"6ba2c7b6-54c7-4610-8051-8fc89f6f8731\") "
Oct 31 19:59:53.767851 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:53.767823    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ba2c7b6-54c7-4610-8051-8fc89f6f8731-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ba2c7b6-54c7-4610-8051-8fc89f6f8731-tmp-dir\") pod \"6ba2c7b6-54c7-4610-8051-8fc89f6f8731\" (UID: \"6ba2c7b6-54c7-4610-8051-8fc89f6f8731\") "
Oct 31 19:59:53.768036 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:53.768019    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ba2c7b6-54c7-4610-8051-8fc89f6f8731-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ba2c7b6-54c7-4610-8051-8fc89f6f8731-model-cache\") pod \"6ba2c7b6-54c7-4610-8051-8fc89f6f8731\" (UID: \"6ba2c7b6-54c7-4610-8051-8fc89f6f8731\") "
Oct 31 19:59:53.768293 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:53.768273    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ba2c7b6-54c7-4610-8051-8fc89f6f8731-home\" (UniqueName: \"kubernetes.io/empty-dir/6ba2c7b6-54c7-4610-8051-8fc89f6f8731-home\") pod \"6ba2c7b6-54c7-4610-8051-8fc89f6f8731\" (UID: \"6ba2c7b6-54c7-4610-8051-8fc89f6f8731\") "
Oct 31 19:59:53.768541 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:53.768522    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ba2c7b6-54c7-4610-8051-8fc89f6f8731-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ba2c7b6-54c7-4610-8051-8fc89f6f8731-dshm\") pod \"6ba2c7b6-54c7-4610-8051-8fc89f6f8731\" (UID: \"6ba2c7b6-54c7-4610-8051-8fc89f6f8731\") "
Oct 31 19:59:53.768736 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:53.768716    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ba2c7b6-54c7-4610-8051-8fc89f6f8731-kube-api-access-q6kjl\" (UniqueName: \"kubernetes.io/projected/6ba2c7b6-54c7-4610-8051-8fc89f6f8731-kube-api-access-q6kjl\") pod \"6ba2c7b6-54c7-4610-8051-8fc89f6f8731\" (UID: \"6ba2c7b6-54c7-4610-8051-8fc89f6f8731\") "
Oct 31 19:59:53.769153 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:53.769122    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ba2c7b6-54c7-4610-8051-8fc89f6f8731-model-cache" pod "6ba2c7b6-54c7-4610-8051-8fc89f6f8731" (UID: "6ba2c7b6-54c7-4610-8051-8fc89f6f8731"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:59:53.769266 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:53.769189    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ba2c7b6-54c7-4610-8051-8fc89f6f8731-home" pod "6ba2c7b6-54c7-4610-8051-8fc89f6f8731" (UID: "6ba2c7b6-54c7-4610-8051-8fc89f6f8731"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:59:53.769454 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:53.769428    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ba2c7b6-54c7-4610-8051-8fc89f6f8731-tmp-dir" pod "6ba2c7b6-54c7-4610-8051-8fc89f6f8731" (UID: "6ba2c7b6-54c7-4610-8051-8fc89f6f8731"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:59:53.769531 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:53.769428    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43db5dd0-3e61-485e-9d44-b141096e887a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf94617d-rhhp2\" (UID: \"43db5dd0-3e61-485e-9d44-b141096e887a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf94617d-rhhp2"
Oct 31 19:59:53.769770 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:53.769746    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43db5dd0-3e61-485e-9d44-b141096e887a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf94617d-rhhp2\" (UID: \"43db5dd0-3e61-485e-9d44-b141096e887a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf94617d-rhhp2"
Oct 31 19:59:53.770949 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:53.770692    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43db5dd0-3e61-485e-9d44-b141096e887a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf94617d-rhhp2\" (UID: \"43db5dd0-3e61-485e-9d44-b141096e887a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf94617d-rhhp2"
Oct 31 19:59:53.771060 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:53.770975    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmh6x\" (UniqueName: \"kubernetes.io/projected/43db5dd0-3e61-485e-9d44-b141096e887a-kube-api-access-zmh6x\") pod \"maas-api-revoke-sub-bf94617d-rhhp2\" (UID: \"43db5dd0-3e61-485e-9d44-b141096e887a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf94617d-rhhp2"
Oct 31 19:59:53.773623 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:53.771562    4645 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ba2c7b6-54c7-4610-8051-8fc89f6f8731-tmp-dir\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:53.773623 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:53.771590    4645 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ba2c7b6-54c7-4610-8051-8fc89f6f8731-model-cache\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:53.773623 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:53.771605    4645 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ba2c7b6-54c7-4610-8051-8fc89f6f8731-home\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:53.777467 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:53.774845    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ba2c7b6-54c7-4610-8051-8fc89f6f8731-tls-certs" pod "6ba2c7b6-54c7-4610-8051-8fc89f6f8731" (UID: "6ba2c7b6-54c7-4610-8051-8fc89f6f8731"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:59:53.777467 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:53.775035    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ba2c7b6-54c7-4610-8051-8fc89f6f8731-kube-api-access-q6kjl" pod "6ba2c7b6-54c7-4610-8051-8fc89f6f8731" (UID: "6ba2c7b6-54c7-4610-8051-8fc89f6f8731"). InnerVolumeSpecName "kube-api-access-q6kjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:59:53.777467 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:53.777414    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ba2c7b6-54c7-4610-8051-8fc89f6f8731-dshm" pod "6ba2c7b6-54c7-4610-8051-8fc89f6f8731" (UID: "6ba2c7b6-54c7-4610-8051-8fc89f6f8731"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:59:53.777467 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:53.777424    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43db5dd0-3e61-485e-9d44-b141096e887a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf94617d-rhhp2\" (UID: \"43db5dd0-3e61-485e-9d44-b141096e887a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf94617d-rhhp2"
Oct 31 19:59:53.793412 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:53.793251    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmh6x\" (UniqueName: \"kubernetes.io/projected/43db5dd0-3e61-485e-9d44-b141096e887a-kube-api-access-zmh6x\") pod \"maas-api-revoke-sub-bf94617d-rhhp2\" (UID: \"43db5dd0-3e61-485e-9d44-b141096e887a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf94617d-rhhp2"
Oct 31 19:59:53.873684 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:53.873595    4645 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ba2c7b6-54c7-4610-8051-8fc89f6f8731-dshm\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:53.873684 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:53.873633    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6kjl\" (UniqueName: \"kubernetes.io/projected/6ba2c7b6-54c7-4610-8051-8fc89f6f8731-kube-api-access-q6kjl\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:53.873684 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:53.873644    4645 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ba2c7b6-54c7-4610-8051-8fc89f6f8731-tls-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:53.928867 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:53.928829    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:53.938464 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:53.938435    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-a891ce/auth-test-model-a891ce-kserve-c6cd86cc7-vckq2"
Oct 31 19:59:53.990123 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:53.990060    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf94617d-rhhp2"
Oct 31 19:59:54.085563 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:54.084933    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f6f43f0-380a-45d9-a8ab-324872927234-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f6f43f0-380a-45d9-a8ab-324872927234-tmp-dir\") pod \"5f6f43f0-380a-45d9-a8ab-324872927234\" (UID: \"5f6f43f0-380a-45d9-a8ab-324872927234\") "
Oct 31 19:59:54.085563 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:54.084986    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f6f43f0-380a-45d9-a8ab-324872927234-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f6f43f0-380a-45d9-a8ab-324872927234-dshm\") pod \"5f6f43f0-380a-45d9-a8ab-324872927234\" (UID: \"5f6f43f0-380a-45d9-a8ab-324872927234\") "
Oct 31 19:59:54.085563 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:54.085045    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f6f43f0-380a-45d9-a8ab-324872927234-tls-certs\" (UniqueName: \"kubernetes.io/secret/5f6f43f0-380a-45d9-a8ab-324872927234-tls-certs\") pod \"5f6f43f0-380a-45d9-a8ab-324872927234\" (UID: \"5f6f43f0-380a-45d9-a8ab-324872927234\") "
Oct 31 19:59:54.085563 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:54.085131    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f6f43f0-380a-45d9-a8ab-324872927234-home\" (UniqueName: \"kubernetes.io/empty-dir/5f6f43f0-380a-45d9-a8ab-324872927234-home\") pod \"5f6f43f0-380a-45d9-a8ab-324872927234\" (UID: \"5f6f43f0-380a-45d9-a8ab-324872927234\") "
Oct 31 19:59:54.085563 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:54.085305    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f6f43f0-380a-45d9-a8ab-324872927234-kube-api-access-sfqh7\" (UniqueName: \"kubernetes.io/projected/5f6f43f0-380a-45d9-a8ab-324872927234-kube-api-access-sfqh7\") pod \"5f6f43f0-380a-45d9-a8ab-324872927234\" (UID: \"5f6f43f0-380a-45d9-a8ab-324872927234\") "
Oct 31 19:59:54.086156 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:54.085593    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f6f43f0-380a-45d9-a8ab-324872927234-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f6f43f0-380a-45d9-a8ab-324872927234-model-cache\") pod \"5f6f43f0-380a-45d9-a8ab-324872927234\" (UID: \"5f6f43f0-380a-45d9-a8ab-324872927234\") "
Oct 31 19:59:54.098406 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:54.089401    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f6f43f0-380a-45d9-a8ab-324872927234-tmp-dir" pod "5f6f43f0-380a-45d9-a8ab-324872927234" (UID: "5f6f43f0-380a-45d9-a8ab-324872927234"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:59:54.098406 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:54.094626    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f6f43f0-380a-45d9-a8ab-324872927234-tls-certs" pod "5f6f43f0-380a-45d9-a8ab-324872927234" (UID: "5f6f43f0-380a-45d9-a8ab-324872927234"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:59:54.098406 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:54.094888    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f6f43f0-380a-45d9-a8ab-324872927234-home" pod "5f6f43f0-380a-45d9-a8ab-324872927234" (UID: "5f6f43f0-380a-45d9-a8ab-324872927234"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:59:54.098406 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:54.096418    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f6f43f0-380a-45d9-a8ab-324872927234-model-cache" pod "5f6f43f0-380a-45d9-a8ab-324872927234" (UID: "5f6f43f0-380a-45d9-a8ab-324872927234"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:59:54.098406 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:54.097879    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f6f43f0-380a-45d9-a8ab-324872927234-kube-api-access-sfqh7" pod "5f6f43f0-380a-45d9-a8ab-324872927234" (UID: "5f6f43f0-380a-45d9-a8ab-324872927234"). InnerVolumeSpecName "kube-api-access-sfqh7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:59:54.098406 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:54.098171    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f6f43f0-380a-45d9-a8ab-324872927234-dshm" pod "5f6f43f0-380a-45d9-a8ab-324872927234" (UID: "5f6f43f0-380a-45d9-a8ab-324872927234"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:59:54.188156 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:54.188102    4645 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f6f43f0-380a-45d9-a8ab-324872927234-tmp-dir\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:54.188156 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:54.188145    4645 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f6f43f0-380a-45d9-a8ab-324872927234-dshm\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:54.188156 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:54.188155    4645 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f6f43f0-380a-45d9-a8ab-324872927234-tls-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:54.188156 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:54.188164    4645 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f6f43f0-380a-45d9-a8ab-324872927234-home\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:54.188697 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:54.188182    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfqh7\" (UniqueName: \"kubernetes.io/projected/5f6f43f0-380a-45d9-a8ab-324872927234-kube-api-access-sfqh7\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:54.188697 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:54.188196    4645 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f6f43f0-380a-45d9-a8ab-324872927234-model-cache\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:54.251902 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:54.251852    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:54.317130 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:54.317043    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf94617d-rhhp2"]
Oct 31 19:59:54.337587 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:54.337529    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b"]
Oct 31 19:59:54.339079 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:54.339045    4645 cpu_manager.go:414] "containerMap: removing container" podUID="5f6f43f0-380a-45d9-a8ab-324872927234" containerName="main"
Oct 31 19:59:54.339079 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:54.339075    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f6f43f0-380a-45d9-a8ab-324872927234" containerName="main"
Oct 31 19:59:54.339396 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:54.339104    4645 cpu_manager.go:414] "containerMap: removing container" podUID="6ba2c7b6-54c7-4610-8051-8fc89f6f8731" containerName="main"
Oct 31 19:59:54.339396 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:54.339111    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ba2c7b6-54c7-4610-8051-8fc89f6f8731" containerName="main"
Oct 31 19:59:54.339520 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:54.339410    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f6f43f0-380a-45d9-a8ab-324872927234" containerName="main"
Oct 31 19:59:54.339520 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:54.339431    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ba2c7b6-54c7-4610-8051-8fc89f6f8731" containerName="main"
Oct 31 19:59:54.345214 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:54.345179    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b"
Oct 31 19:59:54.350572 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:54.350541    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-c173d8-openshift-default-dockercfg-skncb\""
Oct 31 19:59:54.398666 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:54.398620    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:54.398900 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:54.398781    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/710e3436-3828-45fb-b090-e28d1147cbfe-istio-token\") pod \"e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b\" (UID: \"710e3436-3828-45fb-b090-e28d1147cbfe\") " pod="openshift-ingress/e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b"
Oct 31 19:59:54.398989 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:54.398939    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/710e3436-3828-45fb-b090-e28d1147cbfe-istio-envoy\") pod \"e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b\" (UID: \"710e3436-3828-45fb-b090-e28d1147cbfe\") " pod="openshift-ingress/e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b"
Oct 31 19:59:54.399095 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:54.399073    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/710e3436-3828-45fb-b090-e28d1147cbfe-istio-podinfo\") pod \"e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b\" (UID: \"710e3436-3828-45fb-b090-e28d1147cbfe\") " pod="openshift-ingress/e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b"
Oct 31 19:59:54.399180 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:54.399142    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/710e3436-3828-45fb-b090-e28d1147cbfe-istiod-ca-cert\") pod \"e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b\" (UID: \"710e3436-3828-45fb-b090-e28d1147cbfe\") " pod="openshift-ingress/e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b"
Oct 31 19:59:54.399180 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:54.399170    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldsq7\" (UniqueName: \"kubernetes.io/projected/710e3436-3828-45fb-b090-e28d1147cbfe-kube-api-access-ldsq7\") pod \"e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b\" (UID: \"710e3436-3828-45fb-b090-e28d1147cbfe\") " pod="openshift-ingress/e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b"
Oct 31 19:59:54.399301 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:54.399272    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/710e3436-3828-45fb-b090-e28d1147cbfe-workload-socket\") pod \"e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b\" (UID: \"710e3436-3828-45fb-b090-e28d1147cbfe\") " pod="openshift-ingress/e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b"
Oct 31 19:59:54.399447 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:54.399312    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/710e3436-3828-45fb-b090-e28d1147cbfe-credential-socket\") pod \"e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b\" (UID: \"710e3436-3828-45fb-b090-e28d1147cbfe\") " pod="openshift-ingress/e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b"
Oct 31 19:59:54.399447 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:54.399409    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/710e3436-3828-45fb-b090-e28d1147cbfe-istio-data\") pod \"e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b\" (UID: \"710e3436-3828-45fb-b090-e28d1147cbfe\") " pod="openshift-ingress/e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b"
Oct 31 19:59:54.399664 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:54.399644    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/710e3436-3828-45fb-b090-e28d1147cbfe-workload-certs\") pod \"e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b\" (UID: \"710e3436-3828-45fb-b090-e28d1147cbfe\") " pod="openshift-ingress/e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b"
Oct 31 19:59:54.401129 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:54.401039    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b"]
Oct 31 19:59:54.544435 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:54.541678    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:54.544435 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:54.542884    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/710e3436-3828-45fb-b090-e28d1147cbfe-istio-data\") pod \"e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b\" (UID: \"710e3436-3828-45fb-b090-e28d1147cbfe\") " pod="openshift-ingress/e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b"
Oct 31 19:59:54.544435 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:54.543106    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/710e3436-3828-45fb-b090-e28d1147cbfe-workload-certs\") pod \"e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b\" (UID: \"710e3436-3828-45fb-b090-e28d1147cbfe\") " pod="openshift-ingress/e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b"
Oct 31 19:59:54.544435 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:54.543143    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/710e3436-3828-45fb-b090-e28d1147cbfe-istio-token\") pod \"e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b\" (UID: \"710e3436-3828-45fb-b090-e28d1147cbfe\") " pod="openshift-ingress/e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b"
Oct 31 19:59:54.544435 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:54.543220    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/710e3436-3828-45fb-b090-e28d1147cbfe-istio-envoy\") pod \"e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b\" (UID: \"710e3436-3828-45fb-b090-e28d1147cbfe\") " pod="openshift-ingress/e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b"
Oct 31 19:59:54.544435 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:54.543320    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/710e3436-3828-45fb-b090-e28d1147cbfe-istio-podinfo\") pod \"e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b\" (UID: \"710e3436-3828-45fb-b090-e28d1147cbfe\") " pod="openshift-ingress/e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b"
Oct 31 19:59:54.544435 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:54.543396    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/710e3436-3828-45fb-b090-e28d1147cbfe-istiod-ca-cert\") pod \"e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b\" (UID: \"710e3436-3828-45fb-b090-e28d1147cbfe\") " pod="openshift-ingress/e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b"
Oct 31 19:59:54.544435 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:54.543420    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldsq7\" (UniqueName: \"kubernetes.io/projected/710e3436-3828-45fb-b090-e28d1147cbfe-kube-api-access-ldsq7\") pod \"e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b\" (UID: \"710e3436-3828-45fb-b090-e28d1147cbfe\") " pod="openshift-ingress/e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b"
Oct 31 19:59:54.544435 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:54.543502    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/710e3436-3828-45fb-b090-e28d1147cbfe-workload-socket\") pod \"e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b\" (UID: \"710e3436-3828-45fb-b090-e28d1147cbfe\") " pod="openshift-ingress/e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b"
Oct 31 19:59:54.544435 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:54.543538    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/710e3436-3828-45fb-b090-e28d1147cbfe-credential-socket\") pod \"e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b\" (UID: \"710e3436-3828-45fb-b090-e28d1147cbfe\") " pod="openshift-ingress/e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b"
Oct 31 19:59:54.544435 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:54.544084    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/710e3436-3828-45fb-b090-e28d1147cbfe-credential-socket\") pod \"e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b\" (UID: \"710e3436-3828-45fb-b090-e28d1147cbfe\") " pod="openshift-ingress/e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b"
Oct 31 19:59:54.544435 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:54.544419    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/710e3436-3828-45fb-b090-e28d1147cbfe-istio-data\") pod \"e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b\" (UID: \"710e3436-3828-45fb-b090-e28d1147cbfe\") " pod="openshift-ingress/e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b"
Oct 31 19:59:54.545237 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:54.544666    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/710e3436-3828-45fb-b090-e28d1147cbfe-workload-certs\") pod \"e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b\" (UID: \"710e3436-3828-45fb-b090-e28d1147cbfe\") " pod="openshift-ingress/e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b"
Oct 31 19:59:54.547172 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:54.547136    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/710e3436-3828-45fb-b090-e28d1147cbfe-istiod-ca-cert\") pod \"e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b\" (UID: \"710e3436-3828-45fb-b090-e28d1147cbfe\") " pod="openshift-ingress/e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b"
Oct 31 19:59:54.548876 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:54.547976    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/710e3436-3828-45fb-b090-e28d1147cbfe-workload-socket\") pod \"e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b\" (UID: \"710e3436-3828-45fb-b090-e28d1147cbfe\") " pod="openshift-ingress/e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b"
Oct 31 19:59:54.577234 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:54.576166    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldsq7\" (UniqueName: \"kubernetes.io/projected/710e3436-3828-45fb-b090-e28d1147cbfe-kube-api-access-ldsq7\") pod \"e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b\" (UID: \"710e3436-3828-45fb-b090-e28d1147cbfe\") " pod="openshift-ingress/e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b"
Oct 31 19:59:54.579053 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:54.579009    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/710e3436-3828-45fb-b090-e28d1147cbfe-istio-podinfo\") pod \"e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b\" (UID: \"710e3436-3828-45fb-b090-e28d1147cbfe\") " pod="openshift-ingress/e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b"
Oct 31 19:59:54.590139 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:54.590103    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/710e3436-3828-45fb-b090-e28d1147cbfe-istio-token\") pod \"e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b\" (UID: \"710e3436-3828-45fb-b090-e28d1147cbfe\") " pod="openshift-ingress/e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b"
Oct 31 19:59:54.668356 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:54.634751    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/710e3436-3828-45fb-b090-e28d1147cbfe-istio-envoy\") pod \"e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b\" (UID: \"710e3436-3828-45fb-b090-e28d1147cbfe\") " pod="openshift-ingress/e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b"
Oct 31 19:59:54.734547 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:54.727502    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b"
Oct 31 19:59:54.749665 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:54.748548    4645 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" start-of-body=
Oct 31 19:59:54.749665 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:54.748610    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph" podUID="b24f33df-8939-405b-83a6-8d7af872e7da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused"
Oct 31 19:59:54.820550 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:54.817798    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-c55cf1/auth-test-model-c55cf1-kserve-6bb6c9bd5c-qnh9f"
Oct 31 19:59:54.820550 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:54.818426    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-c55cf1/auth-test-model-c55cf1-kserve-6bb6c9bd5c-qnh9f" event={"ID":"6ba2c7b6-54c7-4610-8051-8fc89f6f8731","Type":"ContainerDied","Data":"3a493ca9dcc9b9111d50e08fa80c02770c9fa4702f882c3ba4b62492f76848be"}
Oct 31 19:59:54.820550 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:54.818485    4645 scope.go:122] "RemoveContainer" containerID="2ceeb3d7f440c730b397144ebfbf2d63313b7797cf0919686437ecf2507e674d"
Oct 31 19:59:54.824096 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:54.824056    4645 generic.go:356] "Generic (PLEG): container finished" podID="5f6f43f0-380a-45d9-a8ab-324872927234" containerID="e6425167d186986409fc9f751f86fc01d76cb8e70fc4c95b8e65b6e3ff58da83" exitCode=0
Oct 31 19:59:54.824294 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:54.824272    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-a891ce/auth-test-model-a891ce-kserve-c6cd86cc7-vckq2" event={"ID":"5f6f43f0-380a-45d9-a8ab-324872927234","Type":"ContainerDied","Data":"e6425167d186986409fc9f751f86fc01d76cb8e70fc4c95b8e65b6e3ff58da83"}
Oct 31 19:59:54.824409 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:54.824318    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-a891ce/auth-test-model-a891ce-kserve-c6cd86cc7-vckq2" event={"ID":"5f6f43f0-380a-45d9-a8ab-324872927234","Type":"ContainerDied","Data":"b6db8efcb580fb5bda9909cb12a1c38debd1a3903bb302860f33061355060a5a"}
Oct 31 19:59:54.824476 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:54.824411    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-a891ce/auth-test-model-a891ce-kserve-c6cd86cc7-vckq2"
Oct 31 19:59:54.828288 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:54.828258    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf94617d-rhhp2" event={"ID":"43db5dd0-3e61-485e-9d44-b141096e887a","Type":"ContainerStarted","Data":"650f2d1cf4d865887fc82a96f7bfa4ad467451c93a89beb0c3a24229459d4009"}
Oct 31 19:59:54.908617 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:54.908588    4645 scope.go:122] "RemoveContainer" containerID="e6425167d186986409fc9f751f86fc01d76cb8e70fc4c95b8e65b6e3ff58da83"
Oct 31 19:59:54.930241 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:54.930158    4645 scope.go:122] "RemoveContainer" containerID="e6425167d186986409fc9f751f86fc01d76cb8e70fc4c95b8e65b6e3ff58da83"
Oct 31 19:59:54.932491 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:59:54.931665    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6425167d186986409fc9f751f86fc01d76cb8e70fc4c95b8e65b6e3ff58da83\": container with ID starting with e6425167d186986409fc9f751f86fc01d76cb8e70fc4c95b8e65b6e3ff58da83 not found: ID does not exist" containerID="e6425167d186986409fc9f751f86fc01d76cb8e70fc4c95b8e65b6e3ff58da83"
Oct 31 19:59:54.932491 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:54.931713    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6425167d186986409fc9f751f86fc01d76cb8e70fc4c95b8e65b6e3ff58da83"} err="failed to get container status \"e6425167d186986409fc9f751f86fc01d76cb8e70fc4c95b8e65b6e3ff58da83\": rpc error: code = NotFound desc = could not find container \"e6425167d186986409fc9f751f86fc01d76cb8e70fc4c95b8e65b6e3ff58da83\": container with ID starting with e6425167d186986409fc9f751f86fc01d76cb8e70fc4c95b8e65b6e3ff58da83 not found: ID does not exist"
Oct 31 19:59:54.934349 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:54.934308    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w6-a891ce/auth-test-model-a891ce-kserve-c6cd86cc7-vckq2"]
Oct 31 19:59:54.949598 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:54.949517    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w6-a891ce/auth-test-model-a891ce-kserve-c6cd86cc7-vckq2"]
Oct 31 19:59:54.974353 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:59:54.969888    4645 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5f6f43f0_380a_45d9_a8ab_324872927234.slice/crio-b6db8efcb580fb5bda9909cb12a1c38debd1a3903bb302860f33061355060a5a\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5f6f43f0_380a_45d9_a8ab_324872927234.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6ba2c7b6_54c7_4610_8051_8fc89f6f8731.slice/crio-3a493ca9dcc9b9111d50e08fa80c02770c9fa4702f882c3ba4b62492f76848be\": RecentStats: unable to find data in memory cache]"
Oct 31 19:59:54.996884 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:54.995822    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c55cf1/auth-test-model-c55cf1-kserve-6bb6c9bd5c-qnh9f"]
Oct 31 19:59:55.001788 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:55.001729    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-c55cf1/auth-test-model-c55cf1-kserve-6bb6c9bd5c-qnh9f"]
Oct 31 19:59:55.150353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:55.150238    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:55.190497 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:55.190411    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b"]
Oct 31 19:59:55.241381 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:55.234763    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:59:55.241381 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:55.234863    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:59:55.241381 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:55.234905    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 19:59:55.625049 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:55.625006    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f6f43f0-380a-45d9-a8ab-324872927234" path="/var/lib/kubelet/pods/5f6f43f0-380a-45d9-a8ab-324872927234/volumes"
Oct 31 19:59:55.625856 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:55.625825    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ba2c7b6-54c7-4610-8051-8fc89f6f8731" path="/var/lib/kubelet/pods/6ba2c7b6-54c7-4610-8051-8fc89f6f8731/volumes"
Oct 31 19:59:55.735562 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:55.735467    4645 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" start-of-body=
Oct 31 19:59:55.735562 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:55.735542    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph" podUID="b24f33df-8939-405b-83a6-8d7af872e7da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused"
Oct 31 19:59:55.816959 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:55.816899    4645 pod_container_manager_linux.go:218] "Failed to delete cgroup paths" cgroupName=["kubepods","besteffort","poddaaee800-f23a-4154-b55c-7007f2385707"] err="unable to destroy cgroup paths for cgroup [kubepods besteffort poddaaee800-f23a-4154-b55c-7007f2385707] : Timed out while waiting for systemd to remove kubepods-besteffort-poddaaee800_f23a_4154_b55c_7007f2385707.slice"
Oct 31 19:59:55.817154 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:59:55.816967    4645 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to delete cgroup paths for [kubepods besteffort poddaaee800-f23a-4154-b55c-7007f2385707] : unable to destroy cgroup paths for cgroup [kubepods besteffort poddaaee800-f23a-4154-b55c-7007f2385707] : Timed out while waiting for systemd to remove kubepods-besteffort-poddaaee800_f23a_4154_b55c_7007f2385707.slice" pod="opendatahub/test-curl-7020-3da8ea" podUID="daaee800-f23a-4154-b55c-7007f2385707"
Oct 31 19:59:55.837714 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:55.837675    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b" event={"ID":"710e3436-3828-45fb-b090-e28d1147cbfe","Type":"ContainerStarted","Data":"0af901541f97b5801a6cbce660bb1a1c5085864c764964cb4ce45babf30fafa0"}
Oct 31 19:59:55.837926 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:55.837744    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b" event={"ID":"710e3436-3828-45fb-b090-e28d1147cbfe","Type":"ContainerStarted","Data":"4e21813536ba9b6b815b6943f43ff7def10cd87aef5801e707d3b12b2806ecdd"}
Oct 31 19:59:55.841289 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:55.841247    4645 generic.go:356] "Generic (PLEG): container finished" podID="43db5dd0-3e61-485e-9d44-b141096e887a" containerID="fae17cbac3f49413a152b889ed3b03a9aae46d411281dea981aff1511abd7be4" exitCode=0
Oct 31 19:59:55.841542 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:55.841322    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf94617d-rhhp2" event={"ID":"43db5dd0-3e61-485e-9d44-b141096e887a","Type":"ContainerDied","Data":"fae17cbac3f49413a152b889ed3b03a9aae46d411281dea981aff1511abd7be4"}
Oct 31 19:59:55.842676 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:55.842652    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7020-3da8ea"
Oct 31 19:59:55.887565 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:59:55.887510    4645 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7020-3da8ea\" is forbidden: User \"system:node:ip-10-0-131-94.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-94.us-east-2.compute.internal' and this object" podUID="daaee800-f23a-4154-b55c-7007f2385707" pod="opendatahub/test-curl-7020-3da8ea"
Oct 31 19:59:55.901658 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:55.901616    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:55.907303 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:55.907239    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b" podStartSLOduration=1.907220275 podStartE2EDuration="1.907220275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:59:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 19:59:55.885461719 +0000 UTC m=+2426.458695852" watchObservedRunningTime="2026-10-31 19:59:55.907220275 +0000 UTC m=+2426.480454444"
Oct 31 19:59:56.735352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:56.733054    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:56.748149 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:56.741924    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b"
Oct 31 19:59:56.748149 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:56.742945    4645 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" start-of-body=
Oct 31 19:59:56.748149 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:56.742996    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph" podUID="b24f33df-8939-405b-83a6-8d7af872e7da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused"
Oct 31 19:59:56.785175 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:56.784984    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b"
Oct 31 19:59:56.815366 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:59:56.815300    4645 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7020-3da8ea\" is forbidden: User \"system:node:ip-10-0-131-94.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-94.us-east-2.compute.internal' and this object" podUID="daaee800-f23a-4154-b55c-7007f2385707" pod="opendatahub/test-curl-7020-3da8ea"
Oct 31 19:59:56.852741 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:56.852256    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b"
Oct 31 19:59:56.853229 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:56.853201    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b"
Oct 31 19:59:56.889909 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:59:56.889864    4645 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7020-3da8ea\" is forbidden: User \"system:node:ip-10-0-131-94.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-94.us-east-2.compute.internal' and this object" podUID="daaee800-f23a-4154-b55c-7007f2385707" pod="opendatahub/test-curl-7020-3da8ea"
Oct 31 19:59:57.135672 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:57.135640    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf94617d-rhhp2"
Oct 31 19:59:57.179352 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:59:57.175648    4645 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7020-3da8ea\" is forbidden: User \"system:node:ip-10-0-131-94.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-94.us-east-2.compute.internal' and this object" podUID="daaee800-f23a-4154-b55c-7007f2385707" pod="opendatahub/test-curl-7020-3da8ea"
Oct 31 19:59:57.191365 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:57.189832    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-5f3eff-7655b4885c-k6dn9"
Oct 31 19:59:57.192882 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:59:57.192846    4645 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7020-3da8ea\" is forbidden: User \"system:node:ip-10-0-131-94.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-94.us-east-2.compute.internal' and this object" podUID="daaee800-f23a-4154-b55c-7007f2385707" pod="opendatahub/test-curl-7020-3da8ea"
Oct 31 19:59:57.319294 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:57.299557    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43db5dd0-3e61-485e-9d44-b141096e887a-kube-api-access-zmh6x\" (UniqueName: \"kubernetes.io/projected/43db5dd0-3e61-485e-9d44-b141096e887a-kube-api-access-zmh6x\") pod \"43db5dd0-3e61-485e-9d44-b141096e887a\" (UID: \"43db5dd0-3e61-485e-9d44-b141096e887a\") "
Oct 31 19:59:57.319774 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:57.319748    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43db5dd0-3e61-485e-9d44-b141096e887a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43db5dd0-3e61-485e-9d44-b141096e887a-maas-api-openshift-ca\") pod \"43db5dd0-3e61-485e-9d44-b141096e887a\" (UID: \"43db5dd0-3e61-485e-9d44-b141096e887a\") "
Oct 31 19:59:57.319911 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:57.319896    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43db5dd0-3e61-485e-9d44-b141096e887a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43db5dd0-3e61-485e-9d44-b141096e887a-maas-api-xks-ca\") pod \"43db5dd0-3e61-485e-9d44-b141096e887a\" (UID: \"43db5dd0-3e61-485e-9d44-b141096e887a\") "
Oct 31 19:59:57.334621 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:57.315518    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43db5dd0-3e61-485e-9d44-b141096e887a-kube-api-access-zmh6x" pod "43db5dd0-3e61-485e-9d44-b141096e887a" (UID: "43db5dd0-3e61-485e-9d44-b141096e887a"). InnerVolumeSpecName "kube-api-access-zmh6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:59:57.334834 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:57.334579    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43db5dd0-3e61-485e-9d44-b141096e887a-maas-api-openshift-ca" pod "43db5dd0-3e61-485e-9d44-b141096e887a" (UID: "43db5dd0-3e61-485e-9d44-b141096e887a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 19:59:57.336082 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:57.336018    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmh6x\" (UniqueName: \"kubernetes.io/projected/43db5dd0-3e61-485e-9d44-b141096e887a-kube-api-access-zmh6x\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:57.336082 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:57.336052    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43db5dd0-3e61-485e-9d44-b141096e887a-maas-api-openshift-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:57.337138 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:57.337110    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43db5dd0-3e61-485e-9d44-b141096e887a-maas-api-xks-ca" pod "43db5dd0-3e61-485e-9d44-b141096e887a" (UID: "43db5dd0-3e61-485e-9d44-b141096e887a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:59:57.437565 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:57.437533    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43db5dd0-3e61-485e-9d44-b141096e887a-maas-api-xks-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:57.735506 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:57.735467    4645 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" start-of-body=
Oct 31 19:59:57.735911 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:57.735539    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph" podUID="b24f33df-8939-405b-83a6-8d7af872e7da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused"
Oct 31 19:59:57.795433 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:57.795392    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:57.925475 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:57.925425    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf94617d-rhhp2" event={"ID":"43db5dd0-3e61-485e-9d44-b141096e887a","Type":"ContainerDied","Data":"650f2d1cf4d865887fc82a96f7bfa4ad467451c93a89beb0c3a24229459d4009"}
Oct 31 19:59:57.925667 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:57.925554    4645 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="650f2d1cf4d865887fc82a96f7bfa4ad467451c93a89beb0c3a24229459d4009"
Oct 31 19:59:57.925885 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:57.925858    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf94617d-rhhp2"
Oct 31 19:59:58.187578 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:58.187485    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5f3eff-6f85d65cf9-2c4l7"
Oct 31 19:59:58.209126 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:58.209094    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:58.341514 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:58.341452    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w6-e1d060/auth-test-model-e1d060-kserve-589d89d7b9-2vkt5" podUID="093bfd46-77e5-45d4-aa2e-d07fd8dafe33" containerName="main" containerID="cri-o://935cd178a0f34fbf2f41990252513cd18c71ed1dbbc9fda6bafa11c08daf08ea" gracePeriod=45
Oct 31 19:59:58.435559 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:58.435496    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:58.734974 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:58.734905    4645 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" start-of-body=
Oct 31 19:59:58.734974 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:58.734956    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph" podUID="b24f33df-8939-405b-83a6-8d7af872e7da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused"
Oct 31 19:59:58.737121 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:58.737100    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-e1d060/auth-test-model-e1d060-kserve-589d89d7b9-2vkt5"
Oct 31 19:59:58.820262 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:58.820221    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/093bfd46-77e5-45d4-aa2e-d07fd8dafe33-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/093bfd46-77e5-45d4-aa2e-d07fd8dafe33-tmp-dir\") pod \"093bfd46-77e5-45d4-aa2e-d07fd8dafe33\" (UID: \"093bfd46-77e5-45d4-aa2e-d07fd8dafe33\") "
Oct 31 19:59:58.820484 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:58.820292    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/093bfd46-77e5-45d4-aa2e-d07fd8dafe33-dshm\" (UniqueName: \"kubernetes.io/empty-dir/093bfd46-77e5-45d4-aa2e-d07fd8dafe33-dshm\") pod \"093bfd46-77e5-45d4-aa2e-d07fd8dafe33\" (UID: \"093bfd46-77e5-45d4-aa2e-d07fd8dafe33\") "
Oct 31 19:59:58.820484 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:58.820391    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/093bfd46-77e5-45d4-aa2e-d07fd8dafe33-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/093bfd46-77e5-45d4-aa2e-d07fd8dafe33-model-cache\") pod \"093bfd46-77e5-45d4-aa2e-d07fd8dafe33\" (UID: \"093bfd46-77e5-45d4-aa2e-d07fd8dafe33\") "
Oct 31 19:59:58.820484 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:58.820475    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/093bfd46-77e5-45d4-aa2e-d07fd8dafe33-home\" (UniqueName: \"kubernetes.io/empty-dir/093bfd46-77e5-45d4-aa2e-d07fd8dafe33-home\") pod \"093bfd46-77e5-45d4-aa2e-d07fd8dafe33\" (UID: \"093bfd46-77e5-45d4-aa2e-d07fd8dafe33\") "
Oct 31 19:59:58.820662 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:58.820491    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/093bfd46-77e5-45d4-aa2e-d07fd8dafe33-tmp-dir" pod "093bfd46-77e5-45d4-aa2e-d07fd8dafe33" (UID: "093bfd46-77e5-45d4-aa2e-d07fd8dafe33"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:59:58.820662 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:58.820571    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/093bfd46-77e5-45d4-aa2e-d07fd8dafe33-kube-api-access-rh6k6\" (UniqueName: \"kubernetes.io/projected/093bfd46-77e5-45d4-aa2e-d07fd8dafe33-kube-api-access-rh6k6\") pod \"093bfd46-77e5-45d4-aa2e-d07fd8dafe33\" (UID: \"093bfd46-77e5-45d4-aa2e-d07fd8dafe33\") "
Oct 31 19:59:58.820752 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:58.820659    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/093bfd46-77e5-45d4-aa2e-d07fd8dafe33-model-cache" pod "093bfd46-77e5-45d4-aa2e-d07fd8dafe33" (UID: "093bfd46-77e5-45d4-aa2e-d07fd8dafe33"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:59:58.820752 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:58.820719    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/093bfd46-77e5-45d4-aa2e-d07fd8dafe33-home" pod "093bfd46-77e5-45d4-aa2e-d07fd8dafe33" (UID: "093bfd46-77e5-45d4-aa2e-d07fd8dafe33"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:59:58.820854 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:58.820765    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/093bfd46-77e5-45d4-aa2e-d07fd8dafe33-tls-certs\" (UniqueName: \"kubernetes.io/secret/093bfd46-77e5-45d4-aa2e-d07fd8dafe33-tls-certs\") pod \"093bfd46-77e5-45d4-aa2e-d07fd8dafe33\" (UID: \"093bfd46-77e5-45d4-aa2e-d07fd8dafe33\") "
Oct 31 19:59:58.821693 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:58.821664    4645 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/093bfd46-77e5-45d4-aa2e-d07fd8dafe33-tmp-dir\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:58.821693 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:58.821694    4645 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/093bfd46-77e5-45d4-aa2e-d07fd8dafe33-model-cache\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:58.821906 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:58.821707    4645 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/093bfd46-77e5-45d4-aa2e-d07fd8dafe33-home\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:58.823316 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:58.823290    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/093bfd46-77e5-45d4-aa2e-d07fd8dafe33-kube-api-access-rh6k6" pod "093bfd46-77e5-45d4-aa2e-d07fd8dafe33" (UID: "093bfd46-77e5-45d4-aa2e-d07fd8dafe33"). InnerVolumeSpecName "kube-api-access-rh6k6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 19:59:58.823456 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:58.823432    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/093bfd46-77e5-45d4-aa2e-d07fd8dafe33-tls-certs" pod "093bfd46-77e5-45d4-aa2e-d07fd8dafe33" (UID: "093bfd46-77e5-45d4-aa2e-d07fd8dafe33"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 19:59:58.824225 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:58.824105    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/093bfd46-77e5-45d4-aa2e-d07fd8dafe33-dshm" pod "093bfd46-77e5-45d4-aa2e-d07fd8dafe33" (UID: "093bfd46-77e5-45d4-aa2e-d07fd8dafe33"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 19:59:58.900389 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:58.900360    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:58.923561 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:58.923528    4645 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/093bfd46-77e5-45d4-aa2e-d07fd8dafe33-tls-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:58.923561 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:58.923560    4645 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/093bfd46-77e5-45d4-aa2e-d07fd8dafe33-dshm\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:58.923775 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:58.923574    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rh6k6\" (UniqueName: \"kubernetes.io/projected/093bfd46-77e5-45d4-aa2e-d07fd8dafe33-kube-api-access-rh6k6\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 19:59:58.953357 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:58.953269    4645 generic.go:356] "Generic (PLEG): container finished" podID="093bfd46-77e5-45d4-aa2e-d07fd8dafe33" containerID="935cd178a0f34fbf2f41990252513cd18c71ed1dbbc9fda6bafa11c08daf08ea" exitCode=0
Oct 31 19:59:58.953679 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:58.953654    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-e1d060/auth-test-model-e1d060-kserve-589d89d7b9-2vkt5"
Oct 31 19:59:58.963359 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:58.954573    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-e1d060/auth-test-model-e1d060-kserve-589d89d7b9-2vkt5" event={"ID":"093bfd46-77e5-45d4-aa2e-d07fd8dafe33","Type":"ContainerDied","Data":"935cd178a0f34fbf2f41990252513cd18c71ed1dbbc9fda6bafa11c08daf08ea"}
Oct 31 19:59:58.963359 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:58.954625    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-e1d060/auth-test-model-e1d060-kserve-589d89d7b9-2vkt5" event={"ID":"093bfd46-77e5-45d4-aa2e-d07fd8dafe33","Type":"ContainerDied","Data":"49894448c9647abda62f0169d4c690a4ac03fed55f641f0c38c0afe921effd6b"}
Oct 31 19:59:58.963359 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:58.954647    4645 scope.go:122] "RemoveContainer" containerID="935cd178a0f34fbf2f41990252513cd18c71ed1dbbc9fda6bafa11c08daf08ea"
Oct 31 19:59:59.032247 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:59.032226    4645 scope.go:122] "RemoveContainer" containerID="935cd178a0f34fbf2f41990252513cd18c71ed1dbbc9fda6bafa11c08daf08ea"
Oct 31 19:59:59.032777 ip-10-0-131-94 kubenswrapper[4645]: E1031 19:59:59.032751    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"935cd178a0f34fbf2f41990252513cd18c71ed1dbbc9fda6bafa11c08daf08ea\": container with ID starting with 935cd178a0f34fbf2f41990252513cd18c71ed1dbbc9fda6bafa11c08daf08ea not found: ID does not exist" containerID="935cd178a0f34fbf2f41990252513cd18c71ed1dbbc9fda6bafa11c08daf08ea"
Oct 31 19:59:59.032871 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:59.032784    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"935cd178a0f34fbf2f41990252513cd18c71ed1dbbc9fda6bafa11c08daf08ea"} err="failed to get container status \"935cd178a0f34fbf2f41990252513cd18c71ed1dbbc9fda6bafa11c08daf08ea\": rpc error: code = NotFound desc = could not find container \"935cd178a0f34fbf2f41990252513cd18c71ed1dbbc9fda6bafa11c08daf08ea\": container with ID starting with 935cd178a0f34fbf2f41990252513cd18c71ed1dbbc9fda6bafa11c08daf08ea not found: ID does not exist"
Oct 31 19:59:59.046497 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:59.046462    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w6-e1d060/auth-test-model-e1d060-kserve-589d89d7b9-2vkt5"]
Oct 31 19:59:59.050984 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:59.050962    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w6-e1d060/auth-test-model-e1d060-kserve-589d89d7b9-2vkt5"]
Oct 31 19:59:59.413925 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:59.413892    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:59.486209 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:59.486169    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c173d8-56dc894d7f-pgf8r"]
Oct 31 19:59:59.486873 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:59.486856    4645 cpu_manager.go:414] "containerMap: removing container" podUID="43db5dd0-3e61-485e-9d44-b141096e887a" containerName="revoke-keys"
Oct 31 19:59:59.486873 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:59.486873    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43db5dd0-3e61-485e-9d44-b141096e887a" containerName="revoke-keys"
Oct 31 19:59:59.487005 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:59.486901    4645 cpu_manager.go:414] "containerMap: removing container" podUID="093bfd46-77e5-45d4-aa2e-d07fd8dafe33" containerName="main"
Oct 31 19:59:59.487005 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:59.486907    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="093bfd46-77e5-45d4-aa2e-d07fd8dafe33" containerName="main"
Oct 31 19:59:59.487074 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:59.487047    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="093bfd46-77e5-45d4-aa2e-d07fd8dafe33" containerName="main"
Oct 31 19:59:59.487074 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:59.487067    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="43db5dd0-3e61-485e-9d44-b141096e887a" containerName="revoke-keys"
Oct 31 19:59:59.492779 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:59.492758    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c173d8-56dc894d7f-pgf8r"
Oct 31 19:59:59.498682 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:59.498664    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-c173d8\""
Oct 31 19:59:59.501532 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:59.501509    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-c173d8-dockercfg-twmmb\""
Oct 31 19:59:59.505388 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:59.505321    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c173d8-56dc894d7f-pgf8r"]
Oct 31 19:59:59.529082 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:59.529053    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c173d8-8c8c9bf69-d6xbz"]
Oct 31 19:59:59.535389 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:59.535359    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a5d740e-e1fd-4461-976e-a3a49ab9232f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c173d8-56dc894d7f-pgf8r\" (UID: \"4a5d740e-e1fd-4461-976e-a3a49ab9232f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c173d8-56dc894d7f-pgf8r"
Oct 31 19:59:59.535697 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:59.535669    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7ckh\" (UniqueName: \"kubernetes.io/projected/4a5d740e-e1fd-4461-976e-a3a49ab9232f-kube-api-access-q7ckh\") pod \"payload-pre-processing-e2e-ipp-a-c173d8-56dc894d7f-pgf8r\" (UID: \"4a5d740e-e1fd-4461-976e-a3a49ab9232f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c173d8-56dc894d7f-pgf8r"
Oct 31 19:59:59.535841 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:59.535697    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c173d8-8c8c9bf69-d6xbz"
Oct 31 19:59:59.540001 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:59.539979    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c173d8-8c8c9bf69-d6xbz"]
Oct 31 19:59:59.548882 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:59.548854    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:59.640352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:59.639876    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7ckh\" (UniqueName: \"kubernetes.io/projected/4a5d740e-e1fd-4461-976e-a3a49ab9232f-kube-api-access-q7ckh\") pod \"payload-pre-processing-e2e-ipp-a-c173d8-56dc894d7f-pgf8r\" (UID: \"4a5d740e-e1fd-4461-976e-a3a49ab9232f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c173d8-56dc894d7f-pgf8r"
Oct 31 19:59:59.640352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:59.640110    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a5d740e-e1fd-4461-976e-a3a49ab9232f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c173d8-56dc894d7f-pgf8r\" (UID: \"4a5d740e-e1fd-4461-976e-a3a49ab9232f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c173d8-56dc894d7f-pgf8r"
Oct 31 19:59:59.721406 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:59.641205    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a5d740e-e1fd-4461-976e-a3a49ab9232f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c173d8-56dc894d7f-pgf8r\" (UID: \"4a5d740e-e1fd-4461-976e-a3a49ab9232f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c173d8-56dc894d7f-pgf8r"
Oct 31 19:59:59.845353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:59.839591    4645 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused" start-of-body=
Oct 31 19:59:59.845353 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:59.839665    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph" podUID="b24f33df-8939-405b-83a6-8d7af872e7da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.244:15021/healthz/ready\": dial tcp 10.133.0.244:15021: connect: connection refused"
Oct 31 19:59:59.899352 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:59.893490    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 19:59:59.918091 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:59.908813    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ckrk\" (UniqueName: \"kubernetes.io/projected/0c35e9d4-0b05-42c6-b07b-d76a793639cb-kube-api-access-2ckrk\") pod \"payload-processing-e2e-ipp-a-c173d8-8c8c9bf69-d6xbz\" (UID: \"0c35e9d4-0b05-42c6-b07b-d76a793639cb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c173d8-8c8c9bf69-d6xbz"
Oct 31 19:59:59.918091 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:59.908948    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c35e9d4-0b05-42c6-b07b-d76a793639cb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c173d8-8c8c9bf69-d6xbz\" (UID: \"0c35e9d4-0b05-42c6-b07b-d76a793639cb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c173d8-8c8c9bf69-d6xbz"
Oct 31 19:59:59.918091 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:59.911228    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7ckh\" (UniqueName: \"kubernetes.io/projected/4a5d740e-e1fd-4461-976e-a3a49ab9232f-kube-api-access-q7ckh\") pod \"payload-pre-processing-e2e-ipp-a-c173d8-56dc894d7f-pgf8r\" (UID: \"4a5d740e-e1fd-4461-976e-a3a49ab9232f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c173d8-56dc894d7f-pgf8r"
Oct 31 20:00:00.001401 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:59.995485    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="093bfd46-77e5-45d4-aa2e-d07fd8dafe33" path="/var/lib/kubelet/pods/093bfd46-77e5-45d4-aa2e-d07fd8dafe33/volumes"
Oct 31 20:00:00.001401 ip-10-0-131-94 kubenswrapper[4645]: I1031 19:59:59.996025    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w6-a891ce/rate-test-model-a891ce-kserve-55f654969c-tf5cj"]
Oct 31 20:00:00.065348 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.061663    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w6-a891ce/rate-test-model-a891ce-kserve-55f654969c-tf5cj"]
Oct 31 20:00:00.065348 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.061786    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-a891ce/rate-test-model-a891ce-kserve-55f654969c-tf5cj"
Oct 31 20:00:00.069752 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.067769    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w6-a891ce\"/\"rate-test-model-a891ce-kserve-self-signed-certs\""
Oct 31 20:00:00.070612 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.070548    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w6-a891ce\"/\"kube-root-ca.crt\""
Oct 31 20:00:00.071085 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.070861    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ckrk\" (UniqueName: \"kubernetes.io/projected/0c35e9d4-0b05-42c6-b07b-d76a793639cb-kube-api-access-2ckrk\") pod \"payload-processing-e2e-ipp-a-c173d8-8c8c9bf69-d6xbz\" (UID: \"0c35e9d4-0b05-42c6-b07b-d76a793639cb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c173d8-8c8c9bf69-d6xbz"
Oct 31 20:00:00.071085 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.071030    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c35e9d4-0b05-42c6-b07b-d76a793639cb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c173d8-8c8c9bf69-d6xbz\" (UID: \"0c35e9d4-0b05-42c6-b07b-d76a793639cb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c173d8-8c8c9bf69-d6xbz"
Oct 31 20:00:00.073007 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.072966    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w6-a891ce\"/\"openshift-service-ca.crt\""
Oct 31 20:00:00.074033 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.074009    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c35e9d4-0b05-42c6-b07b-d76a793639cb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c173d8-8c8c9bf69-d6xbz\" (UID: \"0c35e9d4-0b05-42c6-b07b-d76a793639cb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c173d8-8c8c9bf69-d6xbz"
Oct 31 20:00:00.105282 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.105007    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w6-a891ce\"/\"default-dockercfg-m26fx\""
Oct 31 20:00:00.112380 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.108725    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ckrk\" (UniqueName: \"kubernetes.io/projected/0c35e9d4-0b05-42c6-b07b-d76a793639cb-kube-api-access-2ckrk\") pod \"payload-processing-e2e-ipp-a-c173d8-8c8c9bf69-d6xbz\" (UID: \"0c35e9d4-0b05-42c6-b07b-d76a793639cb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c173d8-8c8c9bf69-d6xbz"
Oct 31 20:00:00.112380 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.110582    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c173d8-56dc894d7f-pgf8r"
Oct 31 20:00:00.124378 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.123235    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:00.169350 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.168994    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c173d8-8c8c9bf69-d6xbz"
Oct 31 20:00:00.194015 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.172654    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1b1d76a-8047-46ad-b08d-fdfb89ea668c-model-cache\") pod \"rate-test-model-a891ce-kserve-55f654969c-tf5cj\" (UID: \"f1b1d76a-8047-46ad-b08d-fdfb89ea668c\") " pod="ai-tenant-e2e-shared-a-w6-a891ce/rate-test-model-a891ce-kserve-55f654969c-tf5cj"
Oct 31 20:00:00.194015 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.172715    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1b1d76a-8047-46ad-b08d-fdfb89ea668c-tmp-dir\") pod \"rate-test-model-a891ce-kserve-55f654969c-tf5cj\" (UID: \"f1b1d76a-8047-46ad-b08d-fdfb89ea668c\") " pod="ai-tenant-e2e-shared-a-w6-a891ce/rate-test-model-a891ce-kserve-55f654969c-tf5cj"
Oct 31 20:00:00.194015 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.172824    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1b1d76a-8047-46ad-b08d-fdfb89ea668c-dshm\") pod \"rate-test-model-a891ce-kserve-55f654969c-tf5cj\" (UID: \"f1b1d76a-8047-46ad-b08d-fdfb89ea668c\") " pod="ai-tenant-e2e-shared-a-w6-a891ce/rate-test-model-a891ce-kserve-55f654969c-tf5cj"
Oct 31 20:00:00.194015 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.172972    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1b1d76a-8047-46ad-b08d-fdfb89ea668c-home\") pod \"rate-test-model-a891ce-kserve-55f654969c-tf5cj\" (UID: \"f1b1d76a-8047-46ad-b08d-fdfb89ea668c\") " pod="ai-tenant-e2e-shared-a-w6-a891ce/rate-test-model-a891ce-kserve-55f654969c-tf5cj"
Oct 31 20:00:00.194015 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.173066    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cn5zv\" (UniqueName: \"kubernetes.io/projected/f1b1d76a-8047-46ad-b08d-fdfb89ea668c-kube-api-access-cn5zv\") pod \"rate-test-model-a891ce-kserve-55f654969c-tf5cj\" (UID: \"f1b1d76a-8047-46ad-b08d-fdfb89ea668c\") " pod="ai-tenant-e2e-shared-a-w6-a891ce/rate-test-model-a891ce-kserve-55f654969c-tf5cj"
Oct 31 20:00:00.194015 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.173114    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1b1d76a-8047-46ad-b08d-fdfb89ea668c-tls-certs\") pod \"rate-test-model-a891ce-kserve-55f654969c-tf5cj\" (UID: \"f1b1d76a-8047-46ad-b08d-fdfb89ea668c\") " pod="ai-tenant-e2e-shared-a-w6-a891ce/rate-test-model-a891ce-kserve-55f654969c-tf5cj"
Oct 31 20:00:00.237556 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.234434    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d9c048-29891280-wsjnm"]
Oct 31 20:00:00.253609 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.253533    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d9c048-29891280-wsjnm"
Oct 31 20:00:00.262529 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.261912    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e618e1-29891280-c679x"]
Oct 31 20:00:00.274963 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.274933    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn5zv\" (UniqueName: \"kubernetes.io/projected/f1b1d76a-8047-46ad-b08d-fdfb89ea668c-kube-api-access-cn5zv\") pod \"rate-test-model-a891ce-kserve-55f654969c-tf5cj\" (UID: \"f1b1d76a-8047-46ad-b08d-fdfb89ea668c\") " pod="ai-tenant-e2e-shared-a-w6-a891ce/rate-test-model-a891ce-kserve-55f654969c-tf5cj"
Oct 31 20:00:00.275218 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.275194    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1b1d76a-8047-46ad-b08d-fdfb89ea668c-tls-certs\") pod \"rate-test-model-a891ce-kserve-55f654969c-tf5cj\" (UID: \"f1b1d76a-8047-46ad-b08d-fdfb89ea668c\") " pod="ai-tenant-e2e-shared-a-w6-a891ce/rate-test-model-a891ce-kserve-55f654969c-tf5cj"
Oct 31 20:00:00.275444 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.275425    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1b1d76a-8047-46ad-b08d-fdfb89ea668c-model-cache\") pod \"rate-test-model-a891ce-kserve-55f654969c-tf5cj\" (UID: \"f1b1d76a-8047-46ad-b08d-fdfb89ea668c\") " pod="ai-tenant-e2e-shared-a-w6-a891ce/rate-test-model-a891ce-kserve-55f654969c-tf5cj"
Oct 31 20:00:00.275595 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.275579    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1b1d76a-8047-46ad-b08d-fdfb89ea668c-tmp-dir\") pod \"rate-test-model-a891ce-kserve-55f654969c-tf5cj\" (UID: \"f1b1d76a-8047-46ad-b08d-fdfb89ea668c\") " pod="ai-tenant-e2e-shared-a-w6-a891ce/rate-test-model-a891ce-kserve-55f654969c-tf5cj"
Oct 31 20:00:00.275851 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.275833    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1b1d76a-8047-46ad-b08d-fdfb89ea668c-dshm\") pod \"rate-test-model-a891ce-kserve-55f654969c-tf5cj\" (UID: \"f1b1d76a-8047-46ad-b08d-fdfb89ea668c\") " pod="ai-tenant-e2e-shared-a-w6-a891ce/rate-test-model-a891ce-kserve-55f654969c-tf5cj"
Oct 31 20:00:00.276128 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.276106    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1b1d76a-8047-46ad-b08d-fdfb89ea668c-home\") pod \"rate-test-model-a891ce-kserve-55f654969c-tf5cj\" (UID: \"f1b1d76a-8047-46ad-b08d-fdfb89ea668c\") " pod="ai-tenant-e2e-shared-a-w6-a891ce/rate-test-model-a891ce-kserve-55f654969c-tf5cj"
Oct 31 20:00:00.276790 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.276768    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1b1d76a-8047-46ad-b08d-fdfb89ea668c-home\") pod \"rate-test-model-a891ce-kserve-55f654969c-tf5cj\" (UID: \"f1b1d76a-8047-46ad-b08d-fdfb89ea668c\") " pod="ai-tenant-e2e-shared-a-w6-a891ce/rate-test-model-a891ce-kserve-55f654969c-tf5cj"
Oct 31 20:00:00.278190 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.278161    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1b1d76a-8047-46ad-b08d-fdfb89ea668c-tmp-dir\") pod \"rate-test-model-a891ce-kserve-55f654969c-tf5cj\" (UID: \"f1b1d76a-8047-46ad-b08d-fdfb89ea668c\") " pod="ai-tenant-e2e-shared-a-w6-a891ce/rate-test-model-a891ce-kserve-55f654969c-tf5cj"
Oct 31 20:00:00.278625 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.278599    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1b1d76a-8047-46ad-b08d-fdfb89ea668c-model-cache\") pod \"rate-test-model-a891ce-kserve-55f654969c-tf5cj\" (UID: \"f1b1d76a-8047-46ad-b08d-fdfb89ea668c\") " pod="ai-tenant-e2e-shared-a-w6-a891ce/rate-test-model-a891ce-kserve-55f654969c-tf5cj"
Oct 31 20:00:00.292353 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.278852    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-01afb5-29891280-4hgmt"]
Oct 31 20:00:00.292353 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.279986    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e618e1-29891280-c679x"
Oct 31 20:00:00.352801 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.352771    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 20:00:00.357205 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.357133    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1b1d76a-8047-46ad-b08d-fdfb89ea668c-tls-certs\") pod \"rate-test-model-a891ce-kserve-55f654969c-tf5cj\" (UID: \"f1b1d76a-8047-46ad-b08d-fdfb89ea668c\") " pod="ai-tenant-e2e-shared-a-w6-a891ce/rate-test-model-a891ce-kserve-55f654969c-tf5cj"
Oct 31 20:00:00.378929 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.378320    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xprgj\" (UniqueName: \"kubernetes.io/projected/4360bbcf-669f-4d6e-933c-0dcdfa76071a-kube-api-access-xprgj\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-e618e1-29891280-c679x\" (UID: \"4360bbcf-669f-4d6e-933c-0dcdfa76071a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e618e1-29891280-c679x"
Oct 31 20:00:00.378929 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.378467    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpmbw\" (UniqueName: \"kubernetes.io/projected/0b226e15-ab1f-4f6c-9b08-1ce863756e43-kube-api-access-vpmbw\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-d9c048-29891280-wsjnm\" (UID: \"0b226e15-ab1f-4f6c-9b08-1ce863756e43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d9c048-29891280-wsjnm"
Oct 31 20:00:00.378929 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.378539    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b226e15-ab1f-4f6c-9b08-1ce863756e43-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-d9c048-29891280-wsjnm\" (UID: \"0b226e15-ab1f-4f6c-9b08-1ce863756e43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d9c048-29891280-wsjnm"
Oct 31 20:00:00.378929 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.378710    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4360bbcf-669f-4d6e-933c-0dcdfa76071a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-e618e1-29891280-c679x\" (UID: \"4360bbcf-669f-4d6e-933c-0dcdfa76071a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e618e1-29891280-c679x"
Oct 31 20:00:00.391456 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.387076    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cn5zv\" (UniqueName: \"kubernetes.io/projected/f1b1d76a-8047-46ad-b08d-fdfb89ea668c-kube-api-access-cn5zv\") pod \"rate-test-model-a891ce-kserve-55f654969c-tf5cj\" (UID: \"f1b1d76a-8047-46ad-b08d-fdfb89ea668c\") " pod="ai-tenant-e2e-shared-a-w6-a891ce/rate-test-model-a891ce-kserve-55f654969c-tf5cj"
Oct 31 20:00:00.391456 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.387119    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1b1d76a-8047-46ad-b08d-fdfb89ea668c-dshm\") pod \"rate-test-model-a891ce-kserve-55f654969c-tf5cj\" (UID: \"f1b1d76a-8047-46ad-b08d-fdfb89ea668c\") " pod="ai-tenant-e2e-shared-a-w6-a891ce/rate-test-model-a891ce-kserve-55f654969c-tf5cj"
Oct 31 20:00:00.391456 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.389770    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d9c048-29891280-wsjnm"]
Oct 31 20:00:00.391456 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.389820    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1f2781-29891280-948jm"]
Oct 31 20:00:00.401555 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.399526    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-01afb5-29891280-4hgmt"
Oct 31 20:00:00.402835 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.402710    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a891ce-29891280-7sxwj"]
Oct 31 20:00:00.403079 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.402807    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1f2781-29891280-948jm"
Oct 31 20:00:00.420050 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.418232    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-01afb5-29891280-4hgmt"]
Oct 31 20:00:00.420050 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.418285    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a891ce-29891280-7sxwj"]
Oct 31 20:00:00.420050 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.418312    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1f2781-29891280-948jm"]
Oct 31 20:00:00.420050 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.418358    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-fd1c91-29891280-xbpn9"]
Oct 31 20:00:00.420050 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.418574    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a891ce-29891280-7sxwj"
Oct 31 20:00:00.421459 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.421440    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-a891ce/rate-test-model-a891ce-kserve-55f654969c-tf5cj"
Oct 31 20:00:00.426914 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.426768    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-fd1c91-29891280-xbpn9"
Oct 31 20:00:00.443487 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.441441    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aeb292-29891280-dzgjd"]
Oct 31 20:00:00.474352 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.473949    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23814d-29891280-4zhlc"]
Oct 31 20:00:00.474352 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.474048    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aeb292-29891280-dzgjd"
Oct 31 20:00:00.481741 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.481550    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aeb292-29891280-dzgjd"]
Oct 31 20:00:00.481741 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.481595    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-fd1c91-29891280-xbpn9"]
Oct 31 20:00:00.481741 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.481627    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b083a-29891280-nq9lm"]
Oct 31 20:00:00.486512 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.486482    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a9b3262-5fdf-4375-8411-a0640050c34e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-fd1c91-29891280-xbpn9\" (UID: \"2a9b3262-5fdf-4375-8411-a0640050c34e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-fd1c91-29891280-xbpn9"
Oct 31 20:00:00.486636 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.486533    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b3e85d8-1b99-40b3-87ed-7e0d9c78bb86-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-a891ce-29891280-7sxwj\" (UID: \"3b3e85d8-1b99-40b3-87ed-7e0d9c78bb86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a891ce-29891280-7sxwj"
Oct 31 20:00:00.486636 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.486599    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr7s7\" (UniqueName: \"kubernetes.io/projected/674684f6-15f9-4e2e-a5d6-136098c534d3-kube-api-access-wr7s7\") pod \"maas-api-key-cleanup-e2e-ipp-b-01afb5-29891280-4hgmt\" (UID: \"674684f6-15f9-4e2e-a5d6-136098c534d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-01afb5-29891280-4hgmt"
Oct 31 20:00:00.486765 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.486643    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xprgj\" (UniqueName: \"kubernetes.io/projected/4360bbcf-669f-4d6e-933c-0dcdfa76071a-kube-api-access-xprgj\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-e618e1-29891280-c679x\" (UID: \"4360bbcf-669f-4d6e-933c-0dcdfa76071a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e618e1-29891280-c679x"
Oct 31 20:00:00.486765 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.486702    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63f3555a-1396-4563-abee-5ed20eda9e11-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-1f2781-29891280-948jm\" (UID: \"63f3555a-1396-4563-abee-5ed20eda9e11\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1f2781-29891280-948jm"
Oct 31 20:00:00.486897 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.486769    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mn5dn\" (UniqueName: \"kubernetes.io/projected/2a9b3262-5fdf-4375-8411-a0640050c34e-kube-api-access-mn5dn\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-fd1c91-29891280-xbpn9\" (UID: \"2a9b3262-5fdf-4375-8411-a0640050c34e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-fd1c91-29891280-xbpn9"
Oct 31 20:00:00.486897 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.486810    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpmbw\" (UniqueName: \"kubernetes.io/projected/0b226e15-ab1f-4f6c-9b08-1ce863756e43-kube-api-access-vpmbw\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-d9c048-29891280-wsjnm\" (UID: \"0b226e15-ab1f-4f6c-9b08-1ce863756e43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d9c048-29891280-wsjnm"
Oct 31 20:00:00.486897 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.486875    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b226e15-ab1f-4f6c-9b08-1ce863756e43-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-d9c048-29891280-wsjnm\" (UID: \"0b226e15-ab1f-4f6c-9b08-1ce863756e43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d9c048-29891280-wsjnm"
Oct 31 20:00:00.487095 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.486980    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsfh5\" (UniqueName: \"kubernetes.io/projected/3b3e85d8-1b99-40b3-87ed-7e0d9c78bb86-kube-api-access-bsfh5\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-a891ce-29891280-7sxwj\" (UID: \"3b3e85d8-1b99-40b3-87ed-7e0d9c78bb86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a891ce-29891280-7sxwj"
Oct 31 20:00:00.487095 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.487025    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8ghl\" (UniqueName: \"kubernetes.io/projected/63f3555a-1396-4563-abee-5ed20eda9e11-kube-api-access-m8ghl\") pod \"maas-api-key-cleanup-e2e-ipp-a-1f2781-29891280-948jm\" (UID: \"63f3555a-1396-4563-abee-5ed20eda9e11\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1f2781-29891280-948jm"
Oct 31 20:00:00.487095 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.487065    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/674684f6-15f9-4e2e-a5d6-136098c534d3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-01afb5-29891280-4hgmt\" (UID: \"674684f6-15f9-4e2e-a5d6-136098c534d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-01afb5-29891280-4hgmt"
Oct 31 20:00:00.487278 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.487122    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4360bbcf-669f-4d6e-933c-0dcdfa76071a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-e618e1-29891280-c679x\" (UID: \"4360bbcf-669f-4d6e-933c-0dcdfa76071a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e618e1-29891280-c679x"
Oct 31 20:00:00.489251 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.487940    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4360bbcf-669f-4d6e-933c-0dcdfa76071a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-e618e1-29891280-c679x\" (UID: \"4360bbcf-669f-4d6e-933c-0dcdfa76071a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e618e1-29891280-c679x"
Oct 31 20:00:00.489251 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.489159    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b226e15-ab1f-4f6c-9b08-1ce863756e43-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-d9c048-29891280-wsjnm\" (UID: \"0b226e15-ab1f-4f6c-9b08-1ce863756e43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d9c048-29891280-wsjnm"
Oct 31 20:00:00.489572 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.489544    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23814d-29891280-4zhlc"
Oct 31 20:00:00.510839 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.510704    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b083a-29891280-nq9lm"
Oct 31 20:00:00.510985 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.510607    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-996c32-29891280-8phmw"]
Oct 31 20:00:00.522510 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.522479    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xprgj\" (UniqueName: \"kubernetes.io/projected/4360bbcf-669f-4d6e-933c-0dcdfa76071a-kube-api-access-xprgj\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-e618e1-29891280-c679x\" (UID: \"4360bbcf-669f-4d6e-933c-0dcdfa76071a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e618e1-29891280-c679x"
Oct 31 20:00:00.531552 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.531524    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpmbw\" (UniqueName: \"kubernetes.io/projected/0b226e15-ab1f-4f6c-9b08-1ce863756e43-kube-api-access-vpmbw\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-d9c048-29891280-wsjnm\" (UID: \"0b226e15-ab1f-4f6c-9b08-1ce863756e43\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d9c048-29891280-wsjnm"
Oct 31 20:00:00.533639 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.533522    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b083a-29891280-nq9lm"]
Oct 31 20:00:00.533968 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.533621    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-996c32-29891280-8phmw"
Oct 31 20:00:00.534076 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.533823    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-996c32-29891280-8phmw"]
Oct 31 20:00:00.534193 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.534181    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-435728-29891280-6sfnt"]
Oct 31 20:00:00.551198 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.551177    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23814d-29891280-4zhlc"]
Oct 31 20:00:00.551653 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.551521    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-435728-29891280-6sfnt"
Oct 31 20:00:00.555210 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.554591    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-435728-29891280-6sfnt"]
Oct 31 20:00:00.576566 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.576501    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e618e1-29891280-c679x"]
Oct 31 20:00:00.600374 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.596789    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn5dn\" (UniqueName: \"kubernetes.io/projected/2a9b3262-5fdf-4375-8411-a0640050c34e-kube-api-access-mn5dn\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-fd1c91-29891280-xbpn9\" (UID: \"2a9b3262-5fdf-4375-8411-a0640050c34e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-fd1c91-29891280-xbpn9"
Oct 31 20:00:00.600374 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.596913    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46fe6436-5347-408b-b173-aa153f96236e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-435728-29891280-6sfnt\" (UID: \"46fe6436-5347-408b-b173-aa153f96236e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-435728-29891280-6sfnt"
Oct 31 20:00:00.600374 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.596973    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c8e3489-4fca-4756-a7a0-0c494eab1ba5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-996c32-29891280-8phmw\" (UID: \"1c8e3489-4fca-4756-a7a0-0c494eab1ba5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-996c32-29891280-8phmw"
Oct 31 20:00:00.600374 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.597024    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j56zz\" (UniqueName: \"kubernetes.io/projected/f1c79dcb-63c1-4844-9527-5ef43f45eb95-kube-api-access-j56zz\") pod \"maas-api-key-cleanup-e2e-worker-w2-aeb292-29891280-dzgjd\" (UID: \"f1c79dcb-63c1-4844-9527-5ef43f45eb95\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aeb292-29891280-dzgjd"
Oct 31 20:00:00.600374 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.597061    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0a6fdf8-881f-4993-9515-dfc51f685500-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-8b083a-29891280-nq9lm\" (UID: \"d0a6fdf8-881f-4993-9515-dfc51f685500\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b083a-29891280-nq9lm"
Oct 31 20:00:00.600374 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.597109    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f88b769-7931-40c2-8cf6-ef92c2a85327-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-23814d-29891280-4zhlc\" (UID: \"4f88b769-7931-40c2-8cf6-ef92c2a85327\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23814d-29891280-4zhlc"
Oct 31 20:00:00.600374 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.597169    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsfh5\" (UniqueName: \"kubernetes.io/projected/3b3e85d8-1b99-40b3-87ed-7e0d9c78bb86-kube-api-access-bsfh5\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-a891ce-29891280-7sxwj\" (UID: \"3b3e85d8-1b99-40b3-87ed-7e0d9c78bb86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a891ce-29891280-7sxwj"
Oct 31 20:00:00.600374 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.597226    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8ghl\" (UniqueName: \"kubernetes.io/projected/63f3555a-1396-4563-abee-5ed20eda9e11-kube-api-access-m8ghl\") pod \"maas-api-key-cleanup-e2e-ipp-a-1f2781-29891280-948jm\" (UID: \"63f3555a-1396-4563-abee-5ed20eda9e11\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1f2781-29891280-948jm"
Oct 31 20:00:00.600374 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.597278    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/674684f6-15f9-4e2e-a5d6-136098c534d3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-01afb5-29891280-4hgmt\" (UID: \"674684f6-15f9-4e2e-a5d6-136098c534d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-01afb5-29891280-4hgmt"
Oct 31 20:00:00.600374 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.597316    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppv87\" (UniqueName: \"kubernetes.io/projected/46fe6436-5347-408b-b173-aa153f96236e-kube-api-access-ppv87\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-435728-29891280-6sfnt\" (UID: \"46fe6436-5347-408b-b173-aa153f96236e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-435728-29891280-6sfnt"
Oct 31 20:00:00.600374 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.597453    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfrch\" (UniqueName: \"kubernetes.io/projected/d0a6fdf8-881f-4993-9515-dfc51f685500-kube-api-access-nfrch\") pod \"maas-api-key-cleanup-e2e-worker-w1-8b083a-29891280-nq9lm\" (UID: \"d0a6fdf8-881f-4993-9515-dfc51f685500\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b083a-29891280-nq9lm"
Oct 31 20:00:00.600374 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.597491    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1c79dcb-63c1-4844-9527-5ef43f45eb95-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-aeb292-29891280-dzgjd\" (UID: \"f1c79dcb-63c1-4844-9527-5ef43f45eb95\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aeb292-29891280-dzgjd"
Oct 31 20:00:00.600374 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.597564    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a9b3262-5fdf-4375-8411-a0640050c34e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-fd1c91-29891280-xbpn9\" (UID: \"2a9b3262-5fdf-4375-8411-a0640050c34e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-fd1c91-29891280-xbpn9"
Oct 31 20:00:00.600374 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.597599    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7976z\" (UniqueName: \"kubernetes.io/projected/4f88b769-7931-40c2-8cf6-ef92c2a85327-kube-api-access-7976z\") pod \"maas-api-key-cleanup-e2e-worker-w3-23814d-29891280-4zhlc\" (UID: \"4f88b769-7931-40c2-8cf6-ef92c2a85327\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23814d-29891280-4zhlc"
Oct 31 20:00:00.600374 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.597653    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b3e85d8-1b99-40b3-87ed-7e0d9c78bb86-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-a891ce-29891280-7sxwj\" (UID: \"3b3e85d8-1b99-40b3-87ed-7e0d9c78bb86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a891ce-29891280-7sxwj"
Oct 31 20:00:00.600374 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.597706    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjkvc\" (UniqueName: \"kubernetes.io/projected/1c8e3489-4fca-4756-a7a0-0c494eab1ba5-kube-api-access-mjkvc\") pod \"maas-api-key-cleanup-e2e-worker-w4-996c32-29891280-8phmw\" (UID: \"1c8e3489-4fca-4756-a7a0-0c494eab1ba5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-996c32-29891280-8phmw"
Oct 31 20:00:00.600374 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.597772    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr7s7\" (UniqueName: \"kubernetes.io/projected/674684f6-15f9-4e2e-a5d6-136098c534d3-kube-api-access-wr7s7\") pod \"maas-api-key-cleanup-e2e-ipp-b-01afb5-29891280-4hgmt\" (UID: \"674684f6-15f9-4e2e-a5d6-136098c534d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-01afb5-29891280-4hgmt"
Oct 31 20:00:00.600374 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.597871    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63f3555a-1396-4563-abee-5ed20eda9e11-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-1f2781-29891280-948jm\" (UID: \"63f3555a-1396-4563-abee-5ed20eda9e11\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1f2781-29891280-948jm"
Oct 31 20:00:00.604865 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.602550    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/674684f6-15f9-4e2e-a5d6-136098c534d3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-01afb5-29891280-4hgmt\" (UID: \"674684f6-15f9-4e2e-a5d6-136098c534d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-01afb5-29891280-4hgmt"
Oct 31 20:00:00.604865 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.602878    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a9b3262-5fdf-4375-8411-a0640050c34e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-fd1c91-29891280-xbpn9\" (UID: \"2a9b3262-5fdf-4375-8411-a0640050c34e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-fd1c91-29891280-xbpn9"
Oct 31 20:00:00.604865 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.604388    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b3e85d8-1b99-40b3-87ed-7e0d9c78bb86-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-a891ce-29891280-7sxwj\" (UID: \"3b3e85d8-1b99-40b3-87ed-7e0d9c78bb86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a891ce-29891280-7sxwj"
Oct 31 20:00:00.604865 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.604727    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d9c048-29891280-wsjnm"
Oct 31 20:00:00.612934 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.607200    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63f3555a-1396-4563-abee-5ed20eda9e11-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-1f2781-29891280-948jm\" (UID: \"63f3555a-1396-4563-abee-5ed20eda9e11\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1f2781-29891280-948jm"
Oct 31 20:00:00.622998 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.622970    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mn5dn\" (UniqueName: \"kubernetes.io/projected/2a9b3262-5fdf-4375-8411-a0640050c34e-kube-api-access-mn5dn\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-fd1c91-29891280-xbpn9\" (UID: \"2a9b3262-5fdf-4375-8411-a0640050c34e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-fd1c91-29891280-xbpn9"
Oct 31 20:00:00.624804 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.624729    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8ghl\" (UniqueName: \"kubernetes.io/projected/63f3555a-1396-4563-abee-5ed20eda9e11-kube-api-access-m8ghl\") pod \"maas-api-key-cleanup-e2e-ipp-a-1f2781-29891280-948jm\" (UID: \"63f3555a-1396-4563-abee-5ed20eda9e11\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1f2781-29891280-948jm"
Oct 31 20:00:00.641371 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.637488    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr7s7\" (UniqueName: \"kubernetes.io/projected/674684f6-15f9-4e2e-a5d6-136098c534d3-kube-api-access-wr7s7\") pod \"maas-api-key-cleanup-e2e-ipp-b-01afb5-29891280-4hgmt\" (UID: \"674684f6-15f9-4e2e-a5d6-136098c534d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-01afb5-29891280-4hgmt"
Oct 31 20:00:00.660131 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.659997    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c173d8-56dc894d7f-pgf8r"]
Oct 31 20:00:00.660455 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.660294    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e618e1-29891280-c679x"
Oct 31 20:00:00.668698 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.668421    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsfh5\" (UniqueName: \"kubernetes.io/projected/3b3e85d8-1b99-40b3-87ed-7e0d9c78bb86-kube-api-access-bsfh5\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-a891ce-29891280-7sxwj\" (UID: \"3b3e85d8-1b99-40b3-87ed-7e0d9c78bb86\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a891ce-29891280-7sxwj"
Oct 31 20:00:00.701911 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.701869    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfrch\" (UniqueName: \"kubernetes.io/projected/d0a6fdf8-881f-4993-9515-dfc51f685500-kube-api-access-nfrch\") pod \"maas-api-key-cleanup-e2e-worker-w1-8b083a-29891280-nq9lm\" (UID: \"d0a6fdf8-881f-4993-9515-dfc51f685500\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b083a-29891280-nq9lm"
Oct 31 20:00:00.702090 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.701924    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1c79dcb-63c1-4844-9527-5ef43f45eb95-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-aeb292-29891280-dzgjd\" (UID: \"f1c79dcb-63c1-4844-9527-5ef43f45eb95\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aeb292-29891280-dzgjd"
Oct 31 20:00:00.702090 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.701978    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7976z\" (UniqueName: \"kubernetes.io/projected/4f88b769-7931-40c2-8cf6-ef92c2a85327-kube-api-access-7976z\") pod \"maas-api-key-cleanup-e2e-worker-w3-23814d-29891280-4zhlc\" (UID: \"4f88b769-7931-40c2-8cf6-ef92c2a85327\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23814d-29891280-4zhlc"
Oct 31 20:00:00.702090 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.702023    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjkvc\" (UniqueName: \"kubernetes.io/projected/1c8e3489-4fca-4756-a7a0-0c494eab1ba5-kube-api-access-mjkvc\") pod \"maas-api-key-cleanup-e2e-worker-w4-996c32-29891280-8phmw\" (UID: \"1c8e3489-4fca-4756-a7a0-0c494eab1ba5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-996c32-29891280-8phmw"
Oct 31 20:00:00.702287 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.702201    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46fe6436-5347-408b-b173-aa153f96236e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-435728-29891280-6sfnt\" (UID: \"46fe6436-5347-408b-b173-aa153f96236e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-435728-29891280-6sfnt"
Oct 31 20:00:00.702287 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.702235    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c8e3489-4fca-4756-a7a0-0c494eab1ba5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-996c32-29891280-8phmw\" (UID: \"1c8e3489-4fca-4756-a7a0-0c494eab1ba5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-996c32-29891280-8phmw"
Oct 31 20:00:00.702287 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.702277    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j56zz\" (UniqueName: \"kubernetes.io/projected/f1c79dcb-63c1-4844-9527-5ef43f45eb95-kube-api-access-j56zz\") pod \"maas-api-key-cleanup-e2e-worker-w2-aeb292-29891280-dzgjd\" (UID: \"f1c79dcb-63c1-4844-9527-5ef43f45eb95\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aeb292-29891280-dzgjd"
Oct 31 20:00:00.702491 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.702306    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0a6fdf8-881f-4993-9515-dfc51f685500-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-8b083a-29891280-nq9lm\" (UID: \"d0a6fdf8-881f-4993-9515-dfc51f685500\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b083a-29891280-nq9lm"
Oct 31 20:00:00.702491 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.702363    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f88b769-7931-40c2-8cf6-ef92c2a85327-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-23814d-29891280-4zhlc\" (UID: \"4f88b769-7931-40c2-8cf6-ef92c2a85327\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23814d-29891280-4zhlc"
Oct 31 20:00:00.702491 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.702445    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppv87\" (UniqueName: \"kubernetes.io/projected/46fe6436-5347-408b-b173-aa153f96236e-kube-api-access-ppv87\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-435728-29891280-6sfnt\" (UID: \"46fe6436-5347-408b-b173-aa153f96236e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-435728-29891280-6sfnt"
Oct 31 20:00:00.703671 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.703644    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1c79dcb-63c1-4844-9527-5ef43f45eb95-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-aeb292-29891280-dzgjd\" (UID: \"f1c79dcb-63c1-4844-9527-5ef43f45eb95\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aeb292-29891280-dzgjd"
Oct 31 20:00:00.704646 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.704620    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46fe6436-5347-408b-b173-aa153f96236e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-435728-29891280-6sfnt\" (UID: \"46fe6436-5347-408b-b173-aa153f96236e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-435728-29891280-6sfnt"
Oct 31 20:00:00.705225 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.705202    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c8e3489-4fca-4756-a7a0-0c494eab1ba5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-996c32-29891280-8phmw\" (UID: \"1c8e3489-4fca-4756-a7a0-0c494eab1ba5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-996c32-29891280-8phmw"
Oct 31 20:00:00.706026 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.706004    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0a6fdf8-881f-4993-9515-dfc51f685500-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-8b083a-29891280-nq9lm\" (UID: \"d0a6fdf8-881f-4993-9515-dfc51f685500\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b083a-29891280-nq9lm"
Oct 31 20:00:00.706771 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.706750    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f88b769-7931-40c2-8cf6-ef92c2a85327-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-23814d-29891280-4zhlc\" (UID: \"4f88b769-7931-40c2-8cf6-ef92c2a85327\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23814d-29891280-4zhlc"
Oct 31 20:00:00.719866 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.719840    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppv87\" (UniqueName: \"kubernetes.io/projected/46fe6436-5347-408b-b173-aa153f96236e-kube-api-access-ppv87\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-435728-29891280-6sfnt\" (UID: \"46fe6436-5347-408b-b173-aa153f96236e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-435728-29891280-6sfnt"
Oct 31 20:00:00.723523 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.723110    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7976z\" (UniqueName: \"kubernetes.io/projected/4f88b769-7931-40c2-8cf6-ef92c2a85327-kube-api-access-7976z\") pod \"maas-api-key-cleanup-e2e-worker-w3-23814d-29891280-4zhlc\" (UID: \"4f88b769-7931-40c2-8cf6-ef92c2a85327\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23814d-29891280-4zhlc"
Oct 31 20:00:00.731340 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.731304    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfrch\" (UniqueName: \"kubernetes.io/projected/d0a6fdf8-881f-4993-9515-dfc51f685500-kube-api-access-nfrch\") pod \"maas-api-key-cleanup-e2e-worker-w1-8b083a-29891280-nq9lm\" (UID: \"d0a6fdf8-881f-4993-9515-dfc51f685500\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b083a-29891280-nq9lm"
Oct 31 20:00:00.739165 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.739118    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjkvc\" (UniqueName: \"kubernetes.io/projected/1c8e3489-4fca-4756-a7a0-0c494eab1ba5-kube-api-access-mjkvc\") pod \"maas-api-key-cleanup-e2e-worker-w4-996c32-29891280-8phmw\" (UID: \"1c8e3489-4fca-4756-a7a0-0c494eab1ba5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-996c32-29891280-8phmw"
Oct 31 20:00:00.740379 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.740309    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j56zz\" (UniqueName: \"kubernetes.io/projected/f1c79dcb-63c1-4844-9527-5ef43f45eb95-kube-api-access-j56zz\") pod \"maas-api-key-cleanup-e2e-worker-w2-aeb292-29891280-dzgjd\" (UID: \"f1c79dcb-63c1-4844-9527-5ef43f45eb95\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aeb292-29891280-dzgjd"
Oct 31 20:00:00.774294 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.774195    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-01afb5-29891280-4hgmt"
Oct 31 20:00:00.786420 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.786177    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1f2781-29891280-948jm"
Oct 31 20:00:00.824407 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.824121    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a891ce-29891280-7sxwj"
Oct 31 20:00:00.840083 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.840038    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-fd1c91-29891280-xbpn9"
Oct 31 20:00:00.851906 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.851878    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aeb292-29891280-dzgjd"
Oct 31 20:00:00.865894 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.865831    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23814d-29891280-4zhlc"
Oct 31 20:00:00.944581 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.944504    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b083a-29891280-nq9lm"
Oct 31 20:00:00.949121 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.948025    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-996c32-29891280-8phmw"
Oct 31 20:00:00.965989 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.964969    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-435728-29891280-6sfnt"
Oct 31 20:00:00.973491 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:00.973439    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c173d8-8c8c9bf69-d6xbz"]
Oct 31 20:00:01.039770 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:01.035159    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph"
Oct 31 20:00:01.039770 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:01.037317    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph"
Oct 31 20:00:01.041513 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:01.041074    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph"
Oct 31 20:00:01.063779 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:01.063744    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w6-a891ce/rate-test-model-a891ce-kserve-55f654969c-tf5cj"]
Oct 31 20:00:01.083937 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:00:01.083891    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf1b1d76a_8047_46ad_b08d_fdfb89ea668c.slice/crio-eb361ad5c7ffe721b80c4c39ae1334fe6474b3ded8dc2587fa831d8a62c91775 WatchSource:0}: Error finding container eb361ad5c7ffe721b80c4c39ae1334fe6474b3ded8dc2587fa831d8a62c91775: Status 404 returned error can't find the container with id eb361ad5c7ffe721b80c4c39ae1334fe6474b3ded8dc2587fa831d8a62c91775
Oct 31 20:00:01.184728 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:01.184692    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c173d8-56dc894d7f-pgf8r" event={"ID":"4a5d740e-e1fd-4461-976e-a3a49ab9232f","Type":"ContainerStarted","Data":"b792558802b97a658d3b5871c37fed245d99b065ddcbbffc0ec5977396e42389"}
Oct 31 20:00:01.184950 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:01.184924    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c173d8-56dc894d7f-pgf8r" event={"ID":"4a5d740e-e1fd-4461-976e-a3a49ab9232f","Type":"ContainerStarted","Data":"e4c01f105f75d5bf352b812edb6bf86db37327d4ff067e719d2bce52b3ce6a3d"}
Oct 31 20:00:01.187170 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:01.187003    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c173d8-56dc894d7f-pgf8r"
Oct 31 20:00:01.233102 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:01.232778    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-a891ce/rate-test-model-a891ce-kserve-55f654969c-tf5cj" event={"ID":"f1b1d76a-8047-46ad-b08d-fdfb89ea668c","Type":"ContainerStarted","Data":"eb361ad5c7ffe721b80c4c39ae1334fe6474b3ded8dc2587fa831d8a62c91775"}
Oct 31 20:00:01.234073 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:01.234018    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c173d8-56dc894d7f-pgf8r" podStartSLOduration=2.233998695 podStartE2EDuration="2.233998695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:59:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:00:01.228819713 +0000 UTC m=+2431.802053848" watchObservedRunningTime="2026-10-31 20:00:01.233998695 +0000 UTC m=+2431.807232859"
Oct 31 20:00:01.239592 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:01.239409    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c173d8-8c8c9bf69-d6xbz" event={"ID":"0c35e9d4-0b05-42c6-b07b-d76a793639cb","Type":"ContainerStarted","Data":"0bd0d31befe08b8f394fa54b9121fcac173921a83ff3de016055f8694e874979"}
Oct 31 20:00:01.276145 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:00:01.276071    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b226e15_ab1f_4f6c_9b08_1ce863756e43.slice/crio-15e968361e43087799e9c910fcce5eafbfbb94c7ca337fd470c956222b151b6b WatchSource:0}: Error finding container 15e968361e43087799e9c910fcce5eafbfbb94c7ca337fd470c956222b151b6b: Status 404 returned error can't find the container with id 15e968361e43087799e9c910fcce5eafbfbb94c7ca337fd470c956222b151b6b
Oct 31 20:00:01.282464 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:01.280565    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d9c048-29891280-wsjnm"]
Oct 31 20:00:01.355353 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:01.336878    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e618e1-29891280-c679x"]
Oct 31 20:00:01.388439 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:00:01.388406    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4360bbcf_669f_4d6e_933c_0dcdfa76071a.slice/crio-f52521742dd1ae8933779cdcd1a89d1f9016ed72d833c74dcbeaba72281d3571 WatchSource:0}: Error finding container f52521742dd1ae8933779cdcd1a89d1f9016ed72d833c74dcbeaba72281d3571: Status 404 returned error can't find the container with id f52521742dd1ae8933779cdcd1a89d1f9016ed72d833c74dcbeaba72281d3571
Oct 31 20:00:01.412548 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:01.412525    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1f2781-29891280-948jm"]
Oct 31 20:00:01.426387 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:00:01.426285    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63f3555a_1396_4563_abee_5ed20eda9e11.slice/crio-a37055257e9b04277e8e19945f207884e2e7e5e7e6caaff22a9aace800a50a98 WatchSource:0}: Error finding container a37055257e9b04277e8e19945f207884e2e7e5e7e6caaff22a9aace800a50a98: Status 404 returned error can't find the container with id a37055257e9b04277e8e19945f207884e2e7e5e7e6caaff22a9aace800a50a98
Oct 31 20:00:01.499002 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:01.498172    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:01.637675 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:01.637643    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-01afb5-29891280-4hgmt"]
Oct 31 20:00:01.665317 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:00:01.665284    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod674684f6_15f9_4e2e_a5d6_136098c534d3.slice/crio-a368b7b0c93357620aaac3f185de29c12b38e676d5896964693736619da081d6 WatchSource:0}: Error finding container a368b7b0c93357620aaac3f185de29c12b38e676d5896964693736619da081d6: Status 404 returned error can't find the container with id a368b7b0c93357620aaac3f185de29c12b38e676d5896964693736619da081d6
Oct 31 20:00:01.698832 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:00:01.698800    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f88b769_7931_40c2_8cf6_ef92c2a85327.slice/crio-2419b38cd0686b3a59bcc14ca1660255b425d2c655dbcbcfe7bcd1bc2a598e73 WatchSource:0}: Error finding container 2419b38cd0686b3a59bcc14ca1660255b425d2c655dbcbcfe7bcd1bc2a598e73: Status 404 returned error can't find the container with id 2419b38cd0686b3a59bcc14ca1660255b425d2c655dbcbcfe7bcd1bc2a598e73
Oct 31 20:00:01.704185 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:01.703934    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23814d-29891280-4zhlc"]
Oct 31 20:00:01.722518 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:01.722490    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a891ce-29891280-7sxwj"]
Oct 31 20:00:01.728366 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:01.728322    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-435728-29891280-6sfnt"]
Oct 31 20:00:01.732182 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:00:01.732139    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b3e85d8_1b99_40b3_87ed_7e0d9c78bb86.slice/crio-988dcd25bbe7e5e9a03ecfa4f53582f2ad7a422cad520cd1db01d04c7cc06b92 WatchSource:0}: Error finding container 988dcd25bbe7e5e9a03ecfa4f53582f2ad7a422cad520cd1db01d04c7cc06b92: Status 404 returned error can't find the container with id 988dcd25bbe7e5e9a03ecfa4f53582f2ad7a422cad520cd1db01d04c7cc06b92
Oct 31 20:00:01.733620 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:00:01.733600    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46fe6436_5347_408b_b173_aa153f96236e.slice/crio-b4e31678ec941a374580e8182f380e48ee4ca82d950bb6dfe3c73835f91db560 WatchSource:0}: Error finding container b4e31678ec941a374580e8182f380e48ee4ca82d950bb6dfe3c73835f91db560: Status 404 returned error can't find the container with id b4e31678ec941a374580e8182f380e48ee4ca82d950bb6dfe3c73835f91db560
Oct 31 20:00:01.741721 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:00:01.741695    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c8e3489_4fca_4756_a7a0_0c494eab1ba5.slice/crio-44fae0895a32eaf64af09c1ee6441516d486b5331b448ec8953604345447ce74 WatchSource:0}: Error finding container 44fae0895a32eaf64af09c1ee6441516d486b5331b448ec8953604345447ce74: Status 404 returned error can't find the container with id 44fae0895a32eaf64af09c1ee6441516d486b5331b448ec8953604345447ce74
Oct 31 20:00:01.742629 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:01.742605    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-996c32-29891280-8phmw"]
Oct 31 20:00:01.758695 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:01.758672    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b083a-29891280-nq9lm"]
Oct 31 20:00:01.759854 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:00:01.759830    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0a6fdf8_881f_4993_9515_dfc51f685500.slice/crio-0d92c04f086bcaa3d8d53d0b5511f1b22d16ec278ba173a4ffb50da1bd92b6c3 WatchSource:0}: Error finding container 0d92c04f086bcaa3d8d53d0b5511f1b22d16ec278ba173a4ffb50da1bd92b6c3: Status 404 returned error can't find the container with id 0d92c04f086bcaa3d8d53d0b5511f1b22d16ec278ba173a4ffb50da1bd92b6c3
Oct 31 20:00:01.777690 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:01.777667    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aeb292-29891280-dzgjd"]
Oct 31 20:00:01.864556 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:01.864518    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-fd1c91-29891280-xbpn9"]
Oct 31 20:00:02.250489 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:02.250450    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-01afb5-29891280-4hgmt" event={"ID":"674684f6-15f9-4e2e-a5d6-136098c534d3","Type":"ContainerStarted","Data":"a368b7b0c93357620aaac3f185de29c12b38e676d5896964693736619da081d6"}
Oct 31 20:00:02.252489 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:02.252457    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-a891ce/rate-test-model-a891ce-kserve-55f654969c-tf5cj" event={"ID":"f1b1d76a-8047-46ad-b08d-fdfb89ea668c","Type":"ContainerStarted","Data":"799fdc8bd35fa54f1a4abc49ccd072bf42032a9145b8a73c718f3472c8a3c25a"}
Oct 31 20:00:02.254302 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:02.254272    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-996c32-29891280-8phmw" event={"ID":"1c8e3489-4fca-4756-a7a0-0c494eab1ba5","Type":"ContainerStarted","Data":"44fae0895a32eaf64af09c1ee6441516d486b5331b448ec8953604345447ce74"}
Oct 31 20:00:02.262134 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:02.262085    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-fd1c91-29891280-xbpn9" event={"ID":"2a9b3262-5fdf-4375-8411-a0640050c34e","Type":"ContainerStarted","Data":"3f3717157fa17a25f681c840bb19a703b60b1821e02bfba4b37e5d83baac0192"}
Oct 31 20:00:02.265165 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:02.265134    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c173d8-8c8c9bf69-d6xbz" event={"ID":"0c35e9d4-0b05-42c6-b07b-d76a793639cb","Type":"ContainerStarted","Data":"edb36725084bb94870c72378333e38f2da88ab6fe953881380b7dabeb25bc227"}
Oct 31 20:00:02.266104 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:02.266082    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c173d8-8c8c9bf69-d6xbz"
Oct 31 20:00:02.267813 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:02.267790    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23814d-29891280-4zhlc" event={"ID":"4f88b769-7931-40c2-8cf6-ef92c2a85327","Type":"ContainerStarted","Data":"2419b38cd0686b3a59bcc14ca1660255b425d2c655dbcbcfe7bcd1bc2a598e73"}
Oct 31 20:00:02.270015 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:02.269992    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a891ce-29891280-7sxwj" event={"ID":"3b3e85d8-1b99-40b3-87ed-7e0d9c78bb86","Type":"ContainerStarted","Data":"988dcd25bbe7e5e9a03ecfa4f53582f2ad7a422cad520cd1db01d04c7cc06b92"}
Oct 31 20:00:02.273571 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:02.272632    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d9c048-29891280-wsjnm" event={"ID":"0b226e15-ab1f-4f6c-9b08-1ce863756e43","Type":"ContainerStarted","Data":"7db8ff2cf4cc0ff6e3f78ae8ef606bffab37d3f6d6d4bb20aefcfaef14a9c614"}
Oct 31 20:00:02.273571 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:02.272675    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d9c048-29891280-wsjnm" event={"ID":"0b226e15-ab1f-4f6c-9b08-1ce863756e43","Type":"ContainerStarted","Data":"15e968361e43087799e9c910fcce5eafbfbb94c7ca337fd470c956222b151b6b"}
Oct 31 20:00:02.279200 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:02.278302    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e618e1-29891280-c679x" event={"ID":"4360bbcf-669f-4d6e-933c-0dcdfa76071a","Type":"ContainerStarted","Data":"b96c258c869df878dd3bc4793fd1ab79acc4e2070823bb97e00a9c3a8b254e65"}
Oct 31 20:00:02.279200 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:02.278391    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e618e1-29891280-c679x" event={"ID":"4360bbcf-669f-4d6e-933c-0dcdfa76071a","Type":"ContainerStarted","Data":"f52521742dd1ae8933779cdcd1a89d1f9016ed72d833c74dcbeaba72281d3571"}
Oct 31 20:00:02.283259 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:02.283223    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w6-a891ce/rate-test-model-a891ce-kserve-55f654969c-tf5cj" podStartSLOduration=3.283212015 podStartE2EDuration="3.283212015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:59:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:00:02.275733907 +0000 UTC m=+2432.848968028" watchObservedRunningTime="2026-10-31 20:00:02.283212015 +0000 UTC m=+2432.856446145"
Oct 31 20:00:02.284711 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:02.284683    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b083a-29891280-nq9lm" event={"ID":"d0a6fdf8-881f-4993-9515-dfc51f685500","Type":"ContainerStarted","Data":"0d92c04f086bcaa3d8d53d0b5511f1b22d16ec278ba173a4ffb50da1bd92b6c3"}
Oct 31 20:00:02.294385 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:02.294361    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aeb292-29891280-dzgjd" event={"ID":"f1c79dcb-63c1-4844-9527-5ef43f45eb95","Type":"ContainerStarted","Data":"a0752e04e485a3624a72daabdfeb87161226587a4d1936f686477fb965827a3c"}
Oct 31 20:00:02.296241 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:02.295996    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-435728-29891280-6sfnt" event={"ID":"46fe6436-5347-408b-b173-aa153f96236e","Type":"ContainerStarted","Data":"b4e31678ec941a374580e8182f380e48ee4ca82d950bb6dfe3c73835f91db560"}
Oct 31 20:00:02.298256 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:02.298229    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1f2781-29891280-948jm" event={"ID":"63f3555a-1396-4563-abee-5ed20eda9e11","Type":"ContainerStarted","Data":"6430b4717f1a950e548e04de4b5e4245f2e4b845792d755c296128249fc8ec47"}
Oct 31 20:00:02.298370 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:02.298275    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1f2781-29891280-948jm" event={"ID":"63f3555a-1396-4563-abee-5ed20eda9e11","Type":"ContainerStarted","Data":"a37055257e9b04277e8e19945f207884e2e7e5e7e6caaff22a9aace800a50a98"}
Oct 31 20:00:02.314950 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:02.314896    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-c173d8-8c8c9bf69-d6xbz" podStartSLOduration=3.314882497 podStartE2EDuration="3.314882497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 19:59:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:00:02.312669303 +0000 UTC m=+2432.885903435" watchObservedRunningTime="2026-10-31 20:00:02.314882497 +0000 UTC m=+2432.888116628"
Oct 31 20:00:02.341448 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:02.341399    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d9c048-29891280-wsjnm" podStartSLOduration=2.341387169 podStartE2EDuration="2.341387169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:00:02.336787593 +0000 UTC m=+2432.910021724" watchObservedRunningTime="2026-10-31 20:00:02.341387169 +0000 UTC m=+2432.914621328"
Oct 31 20:00:02.359942 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:02.359895    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1f2781-29891280-948jm" podStartSLOduration=2.359879948 podStartE2EDuration="2.359879948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:00:02.359496723 +0000 UTC m=+2432.932730855" watchObservedRunningTime="2026-10-31 20:00:02.359879948 +0000 UTC m=+2432.933114077"
Oct 31 20:00:02.386632 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:02.386583    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e618e1-29891280-c679x" podStartSLOduration=2.386566552 podStartE2EDuration="2.386566552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:00:02.386107495 +0000 UTC m=+2432.959341627" watchObservedRunningTime="2026-10-31 20:00:02.386566552 +0000 UTC m=+2432.959800683"
Oct 31 20:00:02.423321 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:02.423261    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w6-a891ce/rate-test-model-a891ce-kserve-55f654969c-tf5cj"
Oct 31 20:00:02.600640 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:02.600541    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:03.312001 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:03.311971    4645 generic.go:356] "Generic (PLEG): container finished" podID="674684f6-15f9-4e2e-a5d6-136098c534d3" containerID="087b6c750309568477d588cd8bf9d985af97ae7abe967cab617e6e7e87e23f47" exitCode=60
Oct 31 20:00:03.312445 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:03.312107    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-01afb5-29891280-4hgmt" event={"ID":"674684f6-15f9-4e2e-a5d6-136098c534d3","Type":"ContainerDied","Data":"087b6c750309568477d588cd8bf9d985af97ae7abe967cab617e6e7e87e23f47"}
Oct 31 20:00:03.312653 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:03.312626    4645 scope.go:122] "RemoveContainer" containerID="087b6c750309568477d588cd8bf9d985af97ae7abe967cab617e6e7e87e23f47"
Oct 31 20:00:03.315481 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:03.315460    4645 generic.go:356] "Generic (PLEG): container finished" podID="1c8e3489-4fca-4756-a7a0-0c494eab1ba5" containerID="2de10796e8af216b71a38170be5a682d84dd565168c4ad0d1be595c4d8a84584" exitCode=60
Oct 31 20:00:03.315996 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:03.315615    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-996c32-29891280-8phmw" event={"ID":"1c8e3489-4fca-4756-a7a0-0c494eab1ba5","Type":"ContainerDied","Data":"2de10796e8af216b71a38170be5a682d84dd565168c4ad0d1be595c4d8a84584"}
Oct 31 20:00:03.316091 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:03.315902    4645 scope.go:122] "RemoveContainer" containerID="2de10796e8af216b71a38170be5a682d84dd565168c4ad0d1be595c4d8a84584"
Oct 31 20:00:03.318230 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:03.318213    4645 generic.go:356] "Generic (PLEG): container finished" podID="2a9b3262-5fdf-4375-8411-a0640050c34e" containerID="70c020663fec8abfb7fc4e2d9d9311074a230b16faf6ffdfba34f5f369e3aaef" exitCode=60
Oct 31 20:00:03.318305 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:03.318296    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-fd1c91-29891280-xbpn9" event={"ID":"2a9b3262-5fdf-4375-8411-a0640050c34e","Type":"ContainerDied","Data":"70c020663fec8abfb7fc4e2d9d9311074a230b16faf6ffdfba34f5f369e3aaef"}
Oct 31 20:00:03.318635 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:03.318613    4645 scope.go:122] "RemoveContainer" containerID="70c020663fec8abfb7fc4e2d9d9311074a230b16faf6ffdfba34f5f369e3aaef"
Oct 31 20:00:03.319861 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:03.319842    4645 generic.go:356] "Generic (PLEG): container finished" podID="4f88b769-7931-40c2-8cf6-ef92c2a85327" containerID="866f9a5e61d74f575b169f432972227a0ae7ea50a930dc2b560d98cbd6493ec1" exitCode=60
Oct 31 20:00:03.319950 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:03.319902    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23814d-29891280-4zhlc" event={"ID":"4f88b769-7931-40c2-8cf6-ef92c2a85327","Type":"ContainerDied","Data":"866f9a5e61d74f575b169f432972227a0ae7ea50a930dc2b560d98cbd6493ec1"}
Oct 31 20:00:03.320192 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:03.320165    4645 scope.go:122] "RemoveContainer" containerID="866f9a5e61d74f575b169f432972227a0ae7ea50a930dc2b560d98cbd6493ec1"
Oct 31 20:00:03.322241 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:03.322217    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a891ce-29891280-7sxwj" event={"ID":"3b3e85d8-1b99-40b3-87ed-7e0d9c78bb86","Type":"ContainerStarted","Data":"4afbe3f0a1d943f1a6f8af9ffed4e6895850b791dd04fb2873c3afc0d0a52ab3"}
Oct 31 20:00:03.324007 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:03.323984    4645 generic.go:356] "Generic (PLEG): container finished" podID="0b226e15-ab1f-4f6c-9b08-1ce863756e43" containerID="7db8ff2cf4cc0ff6e3f78ae8ef606bffab37d3f6d6d4bb20aefcfaef14a9c614" exitCode=60
Oct 31 20:00:03.324111 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:03.324084    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d9c048-29891280-wsjnm" event={"ID":"0b226e15-ab1f-4f6c-9b08-1ce863756e43","Type":"ContainerDied","Data":"7db8ff2cf4cc0ff6e3f78ae8ef606bffab37d3f6d6d4bb20aefcfaef14a9c614"}
Oct 31 20:00:03.324394 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:03.324375    4645 scope.go:122] "RemoveContainer" containerID="7db8ff2cf4cc0ff6e3f78ae8ef606bffab37d3f6d6d4bb20aefcfaef14a9c614"
Oct 31 20:00:03.334301 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:03.334065    4645 generic.go:356] "Generic (PLEG): container finished" podID="4360bbcf-669f-4d6e-933c-0dcdfa76071a" containerID="b96c258c869df878dd3bc4793fd1ab79acc4e2070823bb97e00a9c3a8b254e65" exitCode=60
Oct 31 20:00:03.334778 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:03.334497    4645 scope.go:122] "RemoveContainer" containerID="b96c258c869df878dd3bc4793fd1ab79acc4e2070823bb97e00a9c3a8b254e65"
Oct 31 20:00:03.336478 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:03.335410    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e618e1-29891280-c679x" event={"ID":"4360bbcf-669f-4d6e-933c-0dcdfa76071a","Type":"ContainerDied","Data":"b96c258c869df878dd3bc4793fd1ab79acc4e2070823bb97e00a9c3a8b254e65"}
Oct 31 20:00:03.339163 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:03.339134    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b083a-29891280-nq9lm" event={"ID":"d0a6fdf8-881f-4993-9515-dfc51f685500","Type":"ContainerStarted","Data":"dbb337c6a702ee58906992ae5344d5215ee166f02512199475c05c93db535046"}
Oct 31 20:00:03.341265 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:03.341242    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aeb292-29891280-dzgjd" event={"ID":"f1c79dcb-63c1-4844-9527-5ef43f45eb95","Type":"ContainerStarted","Data":"8c8a056c2981f858b0e6874e821c2b447e77efe134a24bea0c1a85e84225ebd1"}
Oct 31 20:00:03.343931 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:03.343910    4645 generic.go:356] "Generic (PLEG): container finished" podID="46fe6436-5347-408b-b173-aa153f96236e" containerID="9d691ed3edb6ab2c725e4ddfbcef665046b37dba66ab89a05c85574b9d8186b7" exitCode=60
Oct 31 20:00:03.344128 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:03.343979    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-435728-29891280-6sfnt" event={"ID":"46fe6436-5347-408b-b173-aa153f96236e","Type":"ContainerDied","Data":"9d691ed3edb6ab2c725e4ddfbcef665046b37dba66ab89a05c85574b9d8186b7"}
Oct 31 20:00:03.344224 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:03.344181    4645 scope.go:122] "RemoveContainer" containerID="9d691ed3edb6ab2c725e4ddfbcef665046b37dba66ab89a05c85574b9d8186b7"
Oct 31 20:00:03.347950 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:03.346600    4645 generic.go:356] "Generic (PLEG): container finished" podID="63f3555a-1396-4563-abee-5ed20eda9e11" containerID="6430b4717f1a950e548e04de4b5e4245f2e4b845792d755c296128249fc8ec47" exitCode=60
Oct 31 20:00:03.347950 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:03.346764    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1f2781-29891280-948jm" event={"ID":"63f3555a-1396-4563-abee-5ed20eda9e11","Type":"ContainerDied","Data":"6430b4717f1a950e548e04de4b5e4245f2e4b845792d755c296128249fc8ec47"}
Oct 31 20:00:03.348291 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:03.348253    4645 scope.go:122] "RemoveContainer" containerID="6430b4717f1a950e548e04de4b5e4245f2e4b845792d755c296128249fc8ec47"
Oct 31 20:00:03.544962 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:03.544859    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a891ce-29891280-7sxwj" podStartSLOduration=3.544840836 podStartE2EDuration="3.544840836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:00:03.507682719 +0000 UTC m=+2434.080916847" watchObservedRunningTime="2026-10-31 20:00:03.544840836 +0000 UTC m=+2434.118074966"
Oct 31 20:00:03.585783 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:03.585728    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aeb292-29891280-dzgjd" podStartSLOduration=3.585711538 podStartE2EDuration="3.585711538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:00:03.581610413 +0000 UTC m=+2434.154844541" watchObservedRunningTime="2026-10-31 20:00:03.585711538 +0000 UTC m=+2434.158945669"
Oct 31 20:00:03.622553 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:03.622496    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b083a-29891280-nq9lm" podStartSLOduration=3.622478851 podStartE2EDuration="3.622478851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:00:03.618378224 +0000 UTC m=+2434.191612353" watchObservedRunningTime="2026-10-31 20:00:03.622478851 +0000 UTC m=+2434.195712982"
Oct 31 20:00:03.634703 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:03.634670    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:04.362017 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:04.361979    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23814d-29891280-4zhlc" event={"ID":"4f88b769-7931-40c2-8cf6-ef92c2a85327","Type":"ContainerStarted","Data":"9d5ace7819fcfebd365628660e1dc762bf44cba542ca7debcf2f0dfb99e9f643"}
Oct 31 20:00:04.364126 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:04.364100    4645 generic.go:356] "Generic (PLEG): container finished" podID="3b3e85d8-1b99-40b3-87ed-7e0d9c78bb86" containerID="4afbe3f0a1d943f1a6f8af9ffed4e6895850b791dd04fb2873c3afc0d0a52ab3" exitCode=60
Oct 31 20:00:04.364280 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:04.364189    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a891ce-29891280-7sxwj" event={"ID":"3b3e85d8-1b99-40b3-87ed-7e0d9c78bb86","Type":"ContainerDied","Data":"4afbe3f0a1d943f1a6f8af9ffed4e6895850b791dd04fb2873c3afc0d0a52ab3"}
Oct 31 20:00:04.364528 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:04.364510    4645 scope.go:122] "RemoveContainer" containerID="4afbe3f0a1d943f1a6f8af9ffed4e6895850b791dd04fb2873c3afc0d0a52ab3"
Oct 31 20:00:04.366556 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:04.366534    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d9c048-29891280-wsjnm" event={"ID":"0b226e15-ab1f-4f6c-9b08-1ce863756e43","Type":"ContainerStarted","Data":"ea2cc04c33f026cb5dadb5dbbede0cf2798296d3c0feadeed0297608d6ea138c"}
Oct 31 20:00:04.368677 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:04.368651    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e618e1-29891280-c679x" event={"ID":"4360bbcf-669f-4d6e-933c-0dcdfa76071a","Type":"ContainerStarted","Data":"01416cf9170be03b405b3302f801235bb45b1bd7f58b7af58363e1b48fd3dbf1"}
Oct 31 20:00:04.370264 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:04.370240    4645 generic.go:356] "Generic (PLEG): container finished" podID="d0a6fdf8-881f-4993-9515-dfc51f685500" containerID="dbb337c6a702ee58906992ae5344d5215ee166f02512199475c05c93db535046" exitCode=60
Oct 31 20:00:04.370392 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:04.370307    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b083a-29891280-nq9lm" event={"ID":"d0a6fdf8-881f-4993-9515-dfc51f685500","Type":"ContainerDied","Data":"dbb337c6a702ee58906992ae5344d5215ee166f02512199475c05c93db535046"}
Oct 31 20:00:04.370584 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:04.370562    4645 scope.go:122] "RemoveContainer" containerID="dbb337c6a702ee58906992ae5344d5215ee166f02512199475c05c93db535046"
Oct 31 20:00:04.372014 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:04.371993    4645 generic.go:356] "Generic (PLEG): container finished" podID="f1c79dcb-63c1-4844-9527-5ef43f45eb95" containerID="8c8a056c2981f858b0e6874e821c2b447e77efe134a24bea0c1a85e84225ebd1" exitCode=60
Oct 31 20:00:04.372123 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:04.372054    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aeb292-29891280-dzgjd" event={"ID":"f1c79dcb-63c1-4844-9527-5ef43f45eb95","Type":"ContainerDied","Data":"8c8a056c2981f858b0e6874e821c2b447e77efe134a24bea0c1a85e84225ebd1"}
Oct 31 20:00:04.372282 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:04.372263    4645 scope.go:122] "RemoveContainer" containerID="8c8a056c2981f858b0e6874e821c2b447e77efe134a24bea0c1a85e84225ebd1"
Oct 31 20:00:04.374375 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:04.374352    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-435728-29891280-6sfnt" event={"ID":"46fe6436-5347-408b-b173-aa153f96236e","Type":"ContainerStarted","Data":"0fdc7ba7cc5c0284a6c91c205002b8ca15f0dae68e0a628b3a2e79709b16411d"}
Oct 31 20:00:04.376374 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:04.376351    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1f2781-29891280-948jm" event={"ID":"63f3555a-1396-4563-abee-5ed20eda9e11","Type":"ContainerStarted","Data":"8301cf5cf9aef41c97cb97f1087b4d84505f66693556da17ab1765f97f215b06"}
Oct 31 20:00:04.380523 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:04.380497    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-01afb5-29891280-4hgmt" event={"ID":"674684f6-15f9-4e2e-a5d6-136098c534d3","Type":"ContainerStarted","Data":"87192f9f259eee05cee1b63489fc95549b2de558500303455d9d5f712f271b99"}
Oct 31 20:00:04.383575 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:04.383550    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-996c32-29891280-8phmw" event={"ID":"1c8e3489-4fca-4756-a7a0-0c494eab1ba5","Type":"ContainerStarted","Data":"8b6be9df817c236194dac0156f793ab1c54c74e6635e19a63c28635c166867c5"}
Oct 31 20:00:04.385793 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:04.385767    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-fd1c91-29891280-xbpn9" event={"ID":"2a9b3262-5fdf-4375-8411-a0640050c34e","Type":"ContainerStarted","Data":"3a809686fb9edfe71427326fdb6119e036bef6ca1abff9e8a273c004646d23cf"}
Oct 31 20:00:04.386050 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:04.386002    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23814d-29891280-4zhlc" podStartSLOduration=4.385991352 podStartE2EDuration="4.385991352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:00:04.382069331 +0000 UTC m=+2434.955303462" watchObservedRunningTime="2026-10-31 20:00:04.385991352 +0000 UTC m=+2434.959225481"
Oct 31 20:00:04.421422 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:04.421364    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-01afb5-29891280-4hgmt" podStartSLOduration=4.421346299 podStartE2EDuration="4.421346299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:00:04.417817257 +0000 UTC m=+2434.991051396" watchObservedRunningTime="2026-10-31 20:00:04.421346299 +0000 UTC m=+2434.994580428"
Oct 31 20:00:04.504804 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:04.504774    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:04.581387 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:04.580555    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-435728-29891280-6sfnt" podStartSLOduration=4.580540414 podStartE2EDuration="4.580540414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:00:04.572788608 +0000 UTC m=+2435.146022737" watchObservedRunningTime="2026-10-31 20:00:04.580540414 +0000 UTC m=+2435.153774543"
Oct 31 20:00:04.595053 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:04.595006    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-996c32-29891280-8phmw" podStartSLOduration=4.594993671 podStartE2EDuration="4.594993671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:00:04.592549123 +0000 UTC m=+2435.165783252" watchObservedRunningTime="2026-10-31 20:00:04.594993671 +0000 UTC m=+2435.168227839"
Oct 31 20:00:04.613443 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:04.613368    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-fd1c91-29891280-xbpn9" podStartSLOduration=4.613313654 podStartE2EDuration="4.613313654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:00:04.610831204 +0000 UTC m=+2435.184065332" watchObservedRunningTime="2026-10-31 20:00:04.613313654 +0000 UTC m=+2435.186547822"
Oct 31 20:00:05.397483 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:05.397446    4645 generic.go:356] "Generic (PLEG): container finished" podID="4f88b769-7931-40c2-8cf6-ef92c2a85327" containerID="9d5ace7819fcfebd365628660e1dc762bf44cba542ca7debcf2f0dfb99e9f643" exitCode=60
Oct 31 20:00:05.397868 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:05.397525    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23814d-29891280-4zhlc" event={"ID":"4f88b769-7931-40c2-8cf6-ef92c2a85327","Type":"ContainerDied","Data":"9d5ace7819fcfebd365628660e1dc762bf44cba542ca7debcf2f0dfb99e9f643"}
Oct 31 20:00:05.397868 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:05.397620    4645 scope.go:122] "RemoveContainer" containerID="866f9a5e61d74f575b169f432972227a0ae7ea50a930dc2b560d98cbd6493ec1"
Oct 31 20:00:05.397983 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:05.397925    4645 scope.go:122] "RemoveContainer" containerID="9d5ace7819fcfebd365628660e1dc762bf44cba542ca7debcf2f0dfb99e9f643"
Oct 31 20:00:05.398304 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:00:05.398283    4645 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-23814d-29891280-4zhlc_odh-ai-gateway-infra(4f88b769-7931-40c2-8cf6-ef92c2a85327)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23814d-29891280-4zhlc" podUID="4f88b769-7931-40c2-8cf6-ef92c2a85327"
Oct 31 20:00:05.399928 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:05.399904    4645 generic.go:356] "Generic (PLEG): container finished" podID="3b3e85d8-1b99-40b3-87ed-7e0d9c78bb86" containerID="cefc70a1b2ab61aa7f3a2233016287bd22cea3ea2b40801f18879f1dc75b493b" exitCode=60
Oct 31 20:00:05.400045 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:05.399980    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a891ce-29891280-7sxwj" event={"ID":"3b3e85d8-1b99-40b3-87ed-7e0d9c78bb86","Type":"ContainerDied","Data":"cefc70a1b2ab61aa7f3a2233016287bd22cea3ea2b40801f18879f1dc75b493b"}
Oct 31 20:00:05.400286 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:05.400265    4645 scope.go:122] "RemoveContainer" containerID="cefc70a1b2ab61aa7f3a2233016287bd22cea3ea2b40801f18879f1dc75b493b"
Oct 31 20:00:05.400641 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:00:05.400621    4645 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-a891ce-29891280-7sxwj_odh-ai-gateway-infra(3b3e85d8-1b99-40b3-87ed-7e0d9c78bb86)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a891ce-29891280-7sxwj" podUID="3b3e85d8-1b99-40b3-87ed-7e0d9c78bb86"
Oct 31 20:00:05.402348 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:05.402310    4645 generic.go:356] "Generic (PLEG): container finished" podID="0b226e15-ab1f-4f6c-9b08-1ce863756e43" containerID="ea2cc04c33f026cb5dadb5dbbede0cf2798296d3c0feadeed0297608d6ea138c" exitCode=60
Oct 31 20:00:05.402456 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:05.402357    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d9c048-29891280-wsjnm" event={"ID":"0b226e15-ab1f-4f6c-9b08-1ce863756e43","Type":"ContainerDied","Data":"ea2cc04c33f026cb5dadb5dbbede0cf2798296d3c0feadeed0297608d6ea138c"}
Oct 31 20:00:05.402682 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:05.402666    4645 scope.go:122] "RemoveContainer" containerID="ea2cc04c33f026cb5dadb5dbbede0cf2798296d3c0feadeed0297608d6ea138c"
Oct 31 20:00:05.403024 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:00:05.402999    4645 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-d9c048-29891280-wsjnm_odh-ai-gateway-infra(0b226e15-ab1f-4f6c-9b08-1ce863756e43)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d9c048-29891280-wsjnm" podUID="0b226e15-ab1f-4f6c-9b08-1ce863756e43"
Oct 31 20:00:05.404544 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:05.404524    4645 generic.go:356] "Generic (PLEG): container finished" podID="4360bbcf-669f-4d6e-933c-0dcdfa76071a" containerID="01416cf9170be03b405b3302f801235bb45b1bd7f58b7af58363e1b48fd3dbf1" exitCode=60
Oct 31 20:00:05.404626 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:05.404601    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e618e1-29891280-c679x" event={"ID":"4360bbcf-669f-4d6e-933c-0dcdfa76071a","Type":"ContainerDied","Data":"01416cf9170be03b405b3302f801235bb45b1bd7f58b7af58363e1b48fd3dbf1"}
Oct 31 20:00:05.404812 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:05.404794    4645 scope.go:122] "RemoveContainer" containerID="01416cf9170be03b405b3302f801235bb45b1bd7f58b7af58363e1b48fd3dbf1"
Oct 31 20:00:05.405096 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:00:05.405065    4645 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-e618e1-29891280-c679x_odh-ai-gateway-infra(4360bbcf-669f-4d6e-933c-0dcdfa76071a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e618e1-29891280-c679x" podUID="4360bbcf-669f-4d6e-933c-0dcdfa76071a"
Oct 31 20:00:05.406662 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:05.406637    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b083a-29891280-nq9lm" event={"ID":"d0a6fdf8-881f-4993-9515-dfc51f685500","Type":"ContainerStarted","Data":"1c16bf1f54cd5da34db9aafdcff7e4629767b86f458ebca4d4974acc5450b8fd"}
Oct 31 20:00:05.408691 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:05.408667    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aeb292-29891280-dzgjd" event={"ID":"f1c79dcb-63c1-4844-9527-5ef43f45eb95","Type":"ContainerStarted","Data":"a83ef4ce41ef26cae8d81102213b586b9f38c5813b0d8ebb322763c18e2aae03"}
Oct 31 20:00:05.410569 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:05.410551    4645 generic.go:356] "Generic (PLEG): container finished" podID="46fe6436-5347-408b-b173-aa153f96236e" containerID="0fdc7ba7cc5c0284a6c91c205002b8ca15f0dae68e0a628b3a2e79709b16411d" exitCode=60
Oct 31 20:00:05.410652 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:05.410628    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-435728-29891280-6sfnt" event={"ID":"46fe6436-5347-408b-b173-aa153f96236e","Type":"ContainerDied","Data":"0fdc7ba7cc5c0284a6c91c205002b8ca15f0dae68e0a628b3a2e79709b16411d"}
Oct 31 20:00:05.410831 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:05.410806    4645 scope.go:122] "RemoveContainer" containerID="0fdc7ba7cc5c0284a6c91c205002b8ca15f0dae68e0a628b3a2e79709b16411d"
Oct 31 20:00:05.411218 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:00:05.411192    4645 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-435728-29891280-6sfnt_odh-ai-gateway-infra(46fe6436-5347-408b-b173-aa153f96236e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-435728-29891280-6sfnt" podUID="46fe6436-5347-408b-b173-aa153f96236e"
Oct 31 20:00:05.412723 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:05.412702    4645 generic.go:356] "Generic (PLEG): container finished" podID="63f3555a-1396-4563-abee-5ed20eda9e11" containerID="8301cf5cf9aef41c97cb97f1087b4d84505f66693556da17ab1765f97f215b06" exitCode=60
Oct 31 20:00:05.412848 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:05.412813    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1f2781-29891280-948jm" event={"ID":"63f3555a-1396-4563-abee-5ed20eda9e11","Type":"ContainerDied","Data":"8301cf5cf9aef41c97cb97f1087b4d84505f66693556da17ab1765f97f215b06"}
Oct 31 20:00:05.413090 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:05.412999    4645 scope.go:122] "RemoveContainer" containerID="8301cf5cf9aef41c97cb97f1087b4d84505f66693556da17ab1765f97f215b06"
Oct 31 20:00:05.413277 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:00:05.413253    4645 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-1f2781-29891280-948jm_odh-ai-gateway-infra(63f3555a-1396-4563-abee-5ed20eda9e11)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1f2781-29891280-948jm" podUID="63f3555a-1396-4563-abee-5ed20eda9e11"
Oct 31 20:00:05.414858 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:05.414836    4645 generic.go:356] "Generic (PLEG): container finished" podID="674684f6-15f9-4e2e-a5d6-136098c534d3" containerID="87192f9f259eee05cee1b63489fc95549b2de558500303455d9d5f712f271b99" exitCode=60
Oct 31 20:00:05.414954 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:05.414915    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-01afb5-29891280-4hgmt" event={"ID":"674684f6-15f9-4e2e-a5d6-136098c534d3","Type":"ContainerDied","Data":"87192f9f259eee05cee1b63489fc95549b2de558500303455d9d5f712f271b99"}
Oct 31 20:00:05.415106 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:05.415087    4645 scope.go:122] "RemoveContainer" containerID="87192f9f259eee05cee1b63489fc95549b2de558500303455d9d5f712f271b99"
Oct 31 20:00:05.415409 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:00:05.415385    4645 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-01afb5-29891280-4hgmt_odh-ai-gateway-infra(674684f6-15f9-4e2e-a5d6-136098c534d3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-01afb5-29891280-4hgmt" podUID="674684f6-15f9-4e2e-a5d6-136098c534d3"
Oct 31 20:00:05.416985 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:05.416966    4645 generic.go:356] "Generic (PLEG): container finished" podID="1c8e3489-4fca-4756-a7a0-0c494eab1ba5" containerID="8b6be9df817c236194dac0156f793ab1c54c74e6635e19a63c28635c166867c5" exitCode=60
Oct 31 20:00:05.417080 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:05.417023    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-996c32-29891280-8phmw" event={"ID":"1c8e3489-4fca-4756-a7a0-0c494eab1ba5","Type":"ContainerDied","Data":"8b6be9df817c236194dac0156f793ab1c54c74e6635e19a63c28635c166867c5"}
Oct 31 20:00:05.417235 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:05.417219    4645 scope.go:122] "RemoveContainer" containerID="8b6be9df817c236194dac0156f793ab1c54c74e6635e19a63c28635c166867c5"
Oct 31 20:00:05.417517 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:00:05.417489    4645 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-996c32-29891280-8phmw_odh-ai-gateway-infra(1c8e3489-4fca-4756-a7a0-0c494eab1ba5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-996c32-29891280-8phmw" podUID="1c8e3489-4fca-4756-a7a0-0c494eab1ba5"
Oct 31 20:00:05.419092 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:05.419038    4645 generic.go:356] "Generic (PLEG): container finished" podID="2a9b3262-5fdf-4375-8411-a0640050c34e" containerID="3a809686fb9edfe71427326fdb6119e036bef6ca1abff9e8a273c004646d23cf" exitCode=60
Oct 31 20:00:05.419092 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:05.419066    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-fd1c91-29891280-xbpn9" event={"ID":"2a9b3262-5fdf-4375-8411-a0640050c34e","Type":"ContainerDied","Data":"3a809686fb9edfe71427326fdb6119e036bef6ca1abff9e8a273c004646d23cf"}
Oct 31 20:00:05.419393 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:05.419372    4645 scope.go:122] "RemoveContainer" containerID="3a809686fb9edfe71427326fdb6119e036bef6ca1abff9e8a273c004646d23cf"
Oct 31 20:00:05.419679 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:00:05.419653    4645 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-fd1c91-29891280-xbpn9_odh-ai-gateway-infra(2a9b3262-5fdf-4375-8411-a0640050c34e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-fd1c91-29891280-xbpn9" podUID="2a9b3262-5fdf-4375-8411-a0640050c34e"
Oct 31 20:00:05.422393 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:05.422313    4645 scope.go:122] "RemoveContainer" containerID="4afbe3f0a1d943f1a6f8af9ffed4e6895850b791dd04fb2873c3afc0d0a52ab3"
Oct 31 20:00:05.438792 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:05.438772    4645 scope.go:122] "RemoveContainer" containerID="7db8ff2cf4cc0ff6e3f78ae8ef606bffab37d3f6d6d4bb20aefcfaef14a9c614"
Oct 31 20:00:05.453784 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:05.453766    4645 scope.go:122] "RemoveContainer" containerID="b96c258c869df878dd3bc4793fd1ab79acc4e2070823bb97e00a9c3a8b254e65"
Oct 31 20:00:05.469445 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:05.469421    4645 scope.go:122] "RemoveContainer" containerID="9d691ed3edb6ab2c725e4ddfbcef665046b37dba66ab89a05c85574b9d8186b7"
Oct 31 20:00:05.483436 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:05.483414    4645 scope.go:122] "RemoveContainer" containerID="6430b4717f1a950e548e04de4b5e4245f2e4b845792d755c296128249fc8ec47"
Oct 31 20:00:05.527556 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:05.527524    4645 scope.go:122] "RemoveContainer" containerID="087b6c750309568477d588cd8bf9d985af97ae7abe967cab617e6e7e87e23f47"
Oct 31 20:00:05.533862 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:05.533626    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:05.554723 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:05.554696    4645 scope.go:122] "RemoveContainer" containerID="2de10796e8af216b71a38170be5a682d84dd565168c4ad0d1be595c4d8a84584"
Oct 31 20:00:05.573382 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:05.573359    4645 scope.go:122] "RemoveContainer" containerID="70c020663fec8abfb7fc4e2d9d9311074a230b16faf6ffdfba34f5f369e3aaef"
Oct 31 20:00:06.246715 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:00:06.246680    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 20:00:06.356843 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:06.356811    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:06.480726 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:06.480688    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:06.483699 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:06.483665    4645 scope.go:122] "RemoveContainer" containerID="87192f9f259eee05cee1b63489fc95549b2de558500303455d9d5f712f271b99"
Oct 31 20:00:06.484231 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:00:06.484201    4645 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-01afb5-29891280-4hgmt_odh-ai-gateway-infra(674684f6-15f9-4e2e-a5d6-136098c534d3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-01afb5-29891280-4hgmt" podUID="674684f6-15f9-4e2e-a5d6-136098c534d3"
Oct 31 20:00:06.486739 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:06.486718    4645 scope.go:122] "RemoveContainer" containerID="8b6be9df817c236194dac0156f793ab1c54c74e6635e19a63c28635c166867c5"
Oct 31 20:00:06.487323 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:00:06.487299    4645 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-996c32-29891280-8phmw_odh-ai-gateway-infra(1c8e3489-4fca-4756-a7a0-0c494eab1ba5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-996c32-29891280-8phmw" podUID="1c8e3489-4fca-4756-a7a0-0c494eab1ba5"
Oct 31 20:00:06.488879 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:06.488830    4645 scope.go:122] "RemoveContainer" containerID="3a809686fb9edfe71427326fdb6119e036bef6ca1abff9e8a273c004646d23cf"
Oct 31 20:00:06.489178 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:00:06.489149    4645 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-fd1c91-29891280-xbpn9_odh-ai-gateway-infra(2a9b3262-5fdf-4375-8411-a0640050c34e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-fd1c91-29891280-xbpn9" podUID="2a9b3262-5fdf-4375-8411-a0640050c34e"
Oct 31 20:00:06.491697 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:06.491680    4645 scope.go:122] "RemoveContainer" containerID="9d5ace7819fcfebd365628660e1dc762bf44cba542ca7debcf2f0dfb99e9f643"
Oct 31 20:00:06.492001 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:00:06.491973    4645 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-23814d-29891280-4zhlc_odh-ai-gateway-infra(4f88b769-7931-40c2-8cf6-ef92c2a85327)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23814d-29891280-4zhlc" podUID="4f88b769-7931-40c2-8cf6-ef92c2a85327"
Oct 31 20:00:06.493242 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:06.493222    4645 scope.go:122] "RemoveContainer" containerID="cefc70a1b2ab61aa7f3a2233016287bd22cea3ea2b40801f18879f1dc75b493b"
Oct 31 20:00:06.493563 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:00:06.493542    4645 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-a891ce-29891280-7sxwj_odh-ai-gateway-infra(3b3e85d8-1b99-40b3-87ed-7e0d9c78bb86)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a891ce-29891280-7sxwj" podUID="3b3e85d8-1b99-40b3-87ed-7e0d9c78bb86"
Oct 31 20:00:06.494730 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:06.494713    4645 scope.go:122] "RemoveContainer" containerID="ea2cc04c33f026cb5dadb5dbbede0cf2798296d3c0feadeed0297608d6ea138c"
Oct 31 20:00:06.494974 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:00:06.494954    4645 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-d9c048-29891280-wsjnm_odh-ai-gateway-infra(0b226e15-ab1f-4f6c-9b08-1ce863756e43)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d9c048-29891280-wsjnm" podUID="0b226e15-ab1f-4f6c-9b08-1ce863756e43"
Oct 31 20:00:06.496503 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:06.496472    4645 scope.go:122] "RemoveContainer" containerID="01416cf9170be03b405b3302f801235bb45b1bd7f58b7af58363e1b48fd3dbf1"
Oct 31 20:00:06.496821 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:00:06.496764    4645 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-e618e1-29891280-c679x_odh-ai-gateway-infra(4360bbcf-669f-4d6e-933c-0dcdfa76071a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e618e1-29891280-c679x" podUID="4360bbcf-669f-4d6e-933c-0dcdfa76071a"
Oct 31 20:00:06.498030 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:06.498008    4645 generic.go:356] "Generic (PLEG): container finished" podID="d0a6fdf8-881f-4993-9515-dfc51f685500" containerID="1c16bf1f54cd5da34db9aafdcff7e4629767b86f458ebca4d4974acc5450b8fd" exitCode=60
Oct 31 20:00:06.498126 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:06.498080    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b083a-29891280-nq9lm" event={"ID":"d0a6fdf8-881f-4993-9515-dfc51f685500","Type":"ContainerDied","Data":"1c16bf1f54cd5da34db9aafdcff7e4629767b86f458ebca4d4974acc5450b8fd"}
Oct 31 20:00:06.498190 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:06.498164    4645 scope.go:122] "RemoveContainer" containerID="dbb337c6a702ee58906992ae5344d5215ee166f02512199475c05c93db535046"
Oct 31 20:00:06.498285 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:06.498269    4645 scope.go:122] "RemoveContainer" containerID="1c16bf1f54cd5da34db9aafdcff7e4629767b86f458ebca4d4974acc5450b8fd"
Oct 31 20:00:06.498552 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:00:06.498528    4645 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-8b083a-29891280-nq9lm_odh-ai-gateway-infra(d0a6fdf8-881f-4993-9515-dfc51f685500)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b083a-29891280-nq9lm" podUID="d0a6fdf8-881f-4993-9515-dfc51f685500"
Oct 31 20:00:06.500099 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:06.500048    4645 generic.go:356] "Generic (PLEG): container finished" podID="f1c79dcb-63c1-4844-9527-5ef43f45eb95" containerID="a83ef4ce41ef26cae8d81102213b586b9f38c5813b0d8ebb322763c18e2aae03" exitCode=60
Oct 31 20:00:06.500099 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:06.500074    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aeb292-29891280-dzgjd" event={"ID":"f1c79dcb-63c1-4844-9527-5ef43f45eb95","Type":"ContainerDied","Data":"a83ef4ce41ef26cae8d81102213b586b9f38c5813b0d8ebb322763c18e2aae03"}
Oct 31 20:00:06.500879 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:06.500426    4645 scope.go:122] "RemoveContainer" containerID="a83ef4ce41ef26cae8d81102213b586b9f38c5813b0d8ebb322763c18e2aae03"
Oct 31 20:00:06.500879 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:00:06.500649    4645 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-aeb292-29891280-dzgjd_odh-ai-gateway-infra(f1c79dcb-63c1-4844-9527-5ef43f45eb95)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aeb292-29891280-dzgjd" podUID="f1c79dcb-63c1-4844-9527-5ef43f45eb95"
Oct 31 20:00:06.502635 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:06.502614    4645 scope.go:122] "RemoveContainer" containerID="0fdc7ba7cc5c0284a6c91c205002b8ca15f0dae68e0a628b3a2e79709b16411d"
Oct 31 20:00:06.502886 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:00:06.502862    4645 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-435728-29891280-6sfnt_odh-ai-gateway-infra(46fe6436-5347-408b-b173-aa153f96236e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-435728-29891280-6sfnt" podUID="46fe6436-5347-408b-b173-aa153f96236e"
Oct 31 20:00:06.505599 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:06.505581    4645 scope.go:122] "RemoveContainer" containerID="8301cf5cf9aef41c97cb97f1087b4d84505f66693556da17ab1765f97f215b06"
Oct 31 20:00:06.505829 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:00:06.505809    4645 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-1f2781-29891280-948jm_odh-ai-gateway-infra(63f3555a-1396-4563-abee-5ed20eda9e11)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1f2781-29891280-948jm" podUID="63f3555a-1396-4563-abee-5ed20eda9e11"
Oct 31 20:00:06.513565 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:06.513549    4645 scope.go:122] "RemoveContainer" containerID="8c8a056c2981f858b0e6874e821c2b447e77efe134a24bea0c1a85e84225ebd1"
Oct 31 20:00:06.598680 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:06.598637    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:07.256681 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:07.256645    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1f2781-29891280-948jm"]
Oct 31 20:00:07.286879 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:07.286840    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:07.431258 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:07.431224    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:07.515825 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:07.515741    4645 scope.go:122] "RemoveContainer" containerID="1c16bf1f54cd5da34db9aafdcff7e4629767b86f458ebca4d4974acc5450b8fd"
Oct 31 20:00:07.516284 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:00:07.516161    4645 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-8b083a-29891280-nq9lm_odh-ai-gateway-infra(d0a6fdf8-881f-4993-9515-dfc51f685500)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b083a-29891280-nq9lm" podUID="d0a6fdf8-881f-4993-9515-dfc51f685500"
Oct 31 20:00:07.517473 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:07.517447    4645 scope.go:122] "RemoveContainer" containerID="a83ef4ce41ef26cae8d81102213b586b9f38c5813b0d8ebb322763c18e2aae03"
Oct 31 20:00:07.517908 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:00:07.517790    4645 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-aeb292-29891280-dzgjd_odh-ai-gateway-infra(f1c79dcb-63c1-4844-9527-5ef43f45eb95)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aeb292-29891280-dzgjd" podUID="f1c79dcb-63c1-4844-9527-5ef43f45eb95"
Oct 31 20:00:07.585374 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:07.566915    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:07.686389 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:07.685480    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:07.945181 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:07.945107    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:08.053530 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:08.053493    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1f2781-29891280-948jm"
Oct 31 20:00:08.144157 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:08.144123    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63f3555a-1396-4563-abee-5ed20eda9e11-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63f3555a-1396-4563-abee-5ed20eda9e11-maas-api-service-ca\") pod \"63f3555a-1396-4563-abee-5ed20eda9e11\" (UID: \"63f3555a-1396-4563-abee-5ed20eda9e11\") "
Oct 31 20:00:08.144365 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:08.144323    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63f3555a-1396-4563-abee-5ed20eda9e11-kube-api-access-m8ghl\" (UniqueName: \"kubernetes.io/projected/63f3555a-1396-4563-abee-5ed20eda9e11-kube-api-access-m8ghl\") pod \"63f3555a-1396-4563-abee-5ed20eda9e11\" (UID: \"63f3555a-1396-4563-abee-5ed20eda9e11\") "
Oct 31 20:00:08.144539 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:08.144514    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63f3555a-1396-4563-abee-5ed20eda9e11-maas-api-service-ca" pod "63f3555a-1396-4563-abee-5ed20eda9e11" (UID: "63f3555a-1396-4563-abee-5ed20eda9e11"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:00:08.144945 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:08.144927    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63f3555a-1396-4563-abee-5ed20eda9e11-maas-api-service-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:08.146513 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:08.146490    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63f3555a-1396-4563-abee-5ed20eda9e11-kube-api-access-m8ghl" pod "63f3555a-1396-4563-abee-5ed20eda9e11" (UID: "63f3555a-1396-4563-abee-5ed20eda9e11"). InnerVolumeSpecName "kube-api-access-m8ghl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:00:08.245215 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:08.245117    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:08.246092 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:08.246068    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8ghl\" (UniqueName: \"kubernetes.io/projected/63f3555a-1396-4563-abee-5ed20eda9e11-kube-api-access-m8ghl\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:08.528121 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:08.528042    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1f2781-29891280-948jm"
Oct 31 20:00:08.528121 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:08.528082    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1f2781-29891280-948jm" event={"ID":"63f3555a-1396-4563-abee-5ed20eda9e11","Type":"ContainerDied","Data":"a37055257e9b04277e8e19945f207884e2e7e5e7e6caaff22a9aace800a50a98"}
Oct 31 20:00:08.528662 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:08.528145    4645 scope.go:122] "RemoveContainer" containerID="8301cf5cf9aef41c97cb97f1087b4d84505f66693556da17ab1765f97f215b06"
Oct 31 20:00:08.556165 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:08.556138    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1f2781-29891280-948jm"]
Oct 31 20:00:08.561260 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:08.561234    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1f2781-29891280-948jm"]
Oct 31 20:00:08.955166 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:08.955131    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:09.621448 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:09.621414    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63f3555a-1396-4563-abee-5ed20eda9e11" path="/var/lib/kubelet/pods/63f3555a-1396-4563-abee-5ed20eda9e11/volumes"
Oct 31 20:00:09.790074 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:09.790033    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:10.423109 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:10.423073    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w6-a891ce/rate-test-model-a891ce-kserve-55f654969c-tf5cj"
Oct 31 20:00:10.436131 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:10.436106    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w6-a891ce/rate-test-model-a891ce-kserve-55f654969c-tf5cj"
Oct 31 20:00:10.534488 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:10.534457    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:10.556961 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:10.556939    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w6-a891ce/rate-test-model-a891ce-kserve-55f654969c-tf5cj"
Oct 31 20:00:11.529866 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:11.529816    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:12.262136 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:12.262101    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:12.749082 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:12.749046    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-e618e1/sub-test-model-e618e1-kserve-584c8d9c65-42m96"]
Oct 31 20:00:12.843365 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:12.842493    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:13.351509 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:13.351477    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c173d8-56dc894d7f-pgf8r"
Oct 31 20:00:13.455826 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:13.455790    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:13.729605 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:13.729564    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-d9c048/sub-test-model-d9c048-kserve-6fff8b86d5-2mb99"]
Oct 31 20:00:13.770059 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:13.770026    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:14.345062 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:14.345027    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:14.389703 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:14.389679    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c173d8-8c8c9bf69-d6xbz"
Oct 31 20:00:14.462674 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:14.462640    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:14.872065 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:14.872026    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:15.039351 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:15.034706    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:15.348322 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:15.348285    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:15.616197 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:15.616109    4645 scope.go:122] "RemoveContainer" containerID="8b6be9df817c236194dac0156f793ab1c54c74e6635e19a63c28635c166867c5"
Oct 31 20:00:16.355714 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:16.355684    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:16.610774 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:16.610710    4645 scope.go:122] "RemoveContainer" containerID="3a809686fb9edfe71427326fdb6119e036bef6ca1abff9e8a273c004646d23cf"
Oct 31 20:00:16.610918 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:16.610889    4645 scope.go:122] "RemoveContainer" containerID="0fdc7ba7cc5c0284a6c91c205002b8ca15f0dae68e0a628b3a2e79709b16411d"
Oct 31 20:00:16.616123 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:16.616099    4645 generic.go:356] "Generic (PLEG): container finished" podID="1c8e3489-4fca-4756-a7a0-0c494eab1ba5" containerID="3acdf9e0a1868f38a4531579013b380c7f94a92324b08f106428ca407ff1efa4" exitCode=60
Oct 31 20:00:16.616213 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:16.616169    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-996c32-29891280-8phmw" event={"ID":"1c8e3489-4fca-4756-a7a0-0c494eab1ba5","Type":"ContainerDied","Data":"3acdf9e0a1868f38a4531579013b380c7f94a92324b08f106428ca407ff1efa4"}
Oct 31 20:00:16.616281 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:16.616228    4645 scope.go:122] "RemoveContainer" containerID="8b6be9df817c236194dac0156f793ab1c54c74e6635e19a63c28635c166867c5"
Oct 31 20:00:16.616617 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:16.616594    4645 scope.go:122] "RemoveContainer" containerID="3acdf9e0a1868f38a4531579013b380c7f94a92324b08f106428ca407ff1efa4"
Oct 31 20:00:16.616867 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:00:16.616845    4645 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-996c32-29891280-8phmw_odh-ai-gateway-infra(1c8e3489-4fca-4756-a7a0-0c494eab1ba5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-996c32-29891280-8phmw" podUID="1c8e3489-4fca-4756-a7a0-0c494eab1ba5"
Oct 31 20:00:17.083307 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:17.083277    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:17.218201 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:17.218171    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:17.451128 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:17.451044    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs"]
Oct 31 20:00:17.451652 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:17.451249    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs" podUID="48ab68ad-2abf-44c8-98dc-c3ec72d36362" containerName="istio-proxy" containerID="cri-o://bcbcd6e75fa56be2a3f5a818fd449b59252d483240db33b92cf8d944e34b716e" gracePeriod=30
Oct 31 20:00:17.494587 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:17.494554    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:17.631354 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:17.624314    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:17.631354 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:17.624948    4645 scope.go:122] "RemoveContainer" containerID="9d5ace7819fcfebd365628660e1dc762bf44cba542ca7debcf2f0dfb99e9f643"
Oct 31 20:00:17.631354 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:17.625984    4645 scope.go:122] "RemoveContainer" containerID="ea2cc04c33f026cb5dadb5dbbede0cf2798296d3c0feadeed0297608d6ea138c"
Oct 31 20:00:17.631354 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:17.626798    4645 scope.go:122] "RemoveContainer" containerID="a83ef4ce41ef26cae8d81102213b586b9f38c5813b0d8ebb322763c18e2aae03"
Oct 31 20:00:17.822696 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:17.822618    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-996c32-29891280-8phmw"]
Oct 31 20:00:17.822988 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:17.822963    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-fd1c91-29891280-xbpn9" event={"ID":"2a9b3262-5fdf-4375-8411-a0640050c34e","Type":"ContainerStarted","Data":"21fbb4bda3ea9a5a3b7a5ead340288b6d9984868826dc0caa6038f9365a4c79c"}
Oct 31 20:00:17.860821 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:17.860741    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-435728-29891280-6sfnt" event={"ID":"46fe6436-5347-408b-b173-aa153f96236e","Type":"ContainerStarted","Data":"2a69be78a116b70e13fee263b067b4f00be192e69f39767496b9e53e838c0fc1"}
Oct 31 20:00:18.112688 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:18.112665    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-996c32-29891280-8phmw"
Oct 31 20:00:18.216389 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:18.216312    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c8e3489-4fca-4756-a7a0-0c494eab1ba5-kube-api-access-mjkvc\" (UniqueName: \"kubernetes.io/projected/1c8e3489-4fca-4756-a7a0-0c494eab1ba5-kube-api-access-mjkvc\") pod \"1c8e3489-4fca-4756-a7a0-0c494eab1ba5\" (UID: \"1c8e3489-4fca-4756-a7a0-0c494eab1ba5\") "
Oct 31 20:00:18.216781 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:18.216752    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c8e3489-4fca-4756-a7a0-0c494eab1ba5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c8e3489-4fca-4756-a7a0-0c494eab1ba5-maas-api-service-ca\") pod \"1c8e3489-4fca-4756-a7a0-0c494eab1ba5\" (UID: \"1c8e3489-4fca-4756-a7a0-0c494eab1ba5\") "
Oct 31 20:00:18.218912 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:18.218886    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c8e3489-4fca-4756-a7a0-0c494eab1ba5-maas-api-service-ca" pod "1c8e3489-4fca-4756-a7a0-0c494eab1ba5" (UID: "1c8e3489-4fca-4756-a7a0-0c494eab1ba5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:00:18.220193 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:18.220153    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c8e3489-4fca-4756-a7a0-0c494eab1ba5-kube-api-access-mjkvc" pod "1c8e3489-4fca-4756-a7a0-0c494eab1ba5" (UID: "1c8e3489-4fca-4756-a7a0-0c494eab1ba5"). InnerVolumeSpecName "kube-api-access-mjkvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:00:18.319437 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:18.319315    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjkvc\" (UniqueName: \"kubernetes.io/projected/1c8e3489-4fca-4756-a7a0-0c494eab1ba5-kube-api-access-mjkvc\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:18.319437 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:18.319391    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c8e3489-4fca-4756-a7a0-0c494eab1ba5-maas-api-service-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:18.611477 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:18.611366    4645 scope.go:122] "RemoveContainer" containerID="01416cf9170be03b405b3302f801235bb45b1bd7f58b7af58363e1b48fd3dbf1"
Oct 31 20:00:18.611917 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:18.611554    4645 scope.go:122] "RemoveContainer" containerID="1c16bf1f54cd5da34db9aafdcff7e4629767b86f458ebca4d4974acc5450b8fd"
Oct 31 20:00:18.611917 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:18.611709    4645 scope.go:122] "RemoveContainer" containerID="cefc70a1b2ab61aa7f3a2233016287bd22cea3ea2b40801f18879f1dc75b493b"
Oct 31 20:00:18.611917 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:18.611807    4645 scope.go:122] "RemoveContainer" containerID="87192f9f259eee05cee1b63489fc95549b2de558500303455d9d5f712f271b99"
Oct 31 20:00:18.665541 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:18.665270    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:18.869007 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:18.868923    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-fd1c91-29891280-xbpn9"]
Oct 31 20:00:18.879374 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:18.878930    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aeb292-29891280-dzgjd" event={"ID":"f1c79dcb-63c1-4844-9527-5ef43f45eb95","Type":"ContainerStarted","Data":"5e86435dda5e8a4a34bdff53bb1db7cdb9f06da6749cc51de89a383eaaee58cb"}
Oct 31 20:00:18.881545 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:18.881510    4645 generic.go:356] "Generic (PLEG): container finished" podID="46fe6436-5347-408b-b173-aa153f96236e" containerID="2a69be78a116b70e13fee263b067b4f00be192e69f39767496b9e53e838c0fc1" exitCode=60
Oct 31 20:00:18.881763 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:18.881726    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-435728-29891280-6sfnt" event={"ID":"46fe6436-5347-408b-b173-aa153f96236e","Type":"ContainerDied","Data":"2a69be78a116b70e13fee263b067b4f00be192e69f39767496b9e53e838c0fc1"}
Oct 31 20:00:18.881854 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:18.881801    4645 scope.go:122] "RemoveContainer" containerID="0fdc7ba7cc5c0284a6c91c205002b8ca15f0dae68e0a628b3a2e79709b16411d"
Oct 31 20:00:18.882834 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:18.882813    4645 scope.go:122] "RemoveContainer" containerID="2a69be78a116b70e13fee263b067b4f00be192e69f39767496b9e53e838c0fc1"
Oct 31 20:00:18.883203 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:00:18.883180    4645 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-435728-29891280-6sfnt_odh-ai-gateway-infra(46fe6436-5347-408b-b173-aa153f96236e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-435728-29891280-6sfnt" podUID="46fe6436-5347-408b-b173-aa153f96236e"
Oct 31 20:00:18.883863 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:18.883835    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-996c32-29891280-8phmw" event={"ID":"1c8e3489-4fca-4756-a7a0-0c494eab1ba5","Type":"ContainerDied","Data":"44fae0895a32eaf64af09c1ee6441516d486b5331b448ec8953604345447ce74"}
Oct 31 20:00:18.883863 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:18.883829    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-996c32-29891280-8phmw"
Oct 31 20:00:18.886057 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:18.886031    4645 generic.go:356] "Generic (PLEG): container finished" podID="2a9b3262-5fdf-4375-8411-a0640050c34e" containerID="21fbb4bda3ea9a5a3b7a5ead340288b6d9984868826dc0caa6038f9365a4c79c" exitCode=60
Oct 31 20:00:18.886193 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:18.886151    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-fd1c91-29891280-xbpn9" event={"ID":"2a9b3262-5fdf-4375-8411-a0640050c34e","Type":"ContainerDied","Data":"21fbb4bda3ea9a5a3b7a5ead340288b6d9984868826dc0caa6038f9365a4c79c"}
Oct 31 20:00:18.889077 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:18.889045    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23814d-29891280-4zhlc" event={"ID":"4f88b769-7931-40c2-8cf6-ef92c2a85327","Type":"ContainerStarted","Data":"f681f99d8a466e50d5cb6c3416bd1203fe0ee5afe16c7239e4af5262a4b625fd"}
Oct 31 20:00:18.891461 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:18.891433    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d9c048-29891280-wsjnm" event={"ID":"0b226e15-ab1f-4f6c-9b08-1ce863756e43","Type":"ContainerStarted","Data":"78a6262fdd9e13e3acf8d62437c55c04f733676cd86ec9f65e2ff1024b2ed4ab"}
Oct 31 20:00:18.904552 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:18.902732    4645 scope.go:122] "RemoveContainer" containerID="3acdf9e0a1868f38a4531579013b380c7f94a92324b08f106428ca407ff1efa4"
Oct 31 20:00:18.921087 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:18.921060    4645 scope.go:122] "RemoveContainer" containerID="3a809686fb9edfe71427326fdb6119e036bef6ca1abff9e8a273c004646d23cf"
Oct 31 20:00:18.928025 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:18.927994    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-435728-29891280-6sfnt"]
Oct 31 20:00:19.006929 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:19.006867    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-996c32-29891280-8phmw"]
Oct 31 20:00:19.015695 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:19.015666    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-996c32-29891280-8phmw"]
Oct 31 20:00:19.175762 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:19.175672    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:19.467427 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:19.412984    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:19.519381 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:19.519005    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-fd1c91-29891280-xbpn9"
Oct 31 20:00:19.611366 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:19.591451    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a9b3262-5fdf-4375-8411-a0640050c34e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a9b3262-5fdf-4375-8411-a0640050c34e-maas-api-service-ca\") pod \"2a9b3262-5fdf-4375-8411-a0640050c34e\" (UID: \"2a9b3262-5fdf-4375-8411-a0640050c34e\") "
Oct 31 20:00:19.611366 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:19.591580    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a9b3262-5fdf-4375-8411-a0640050c34e-kube-api-access-mn5dn\" (UniqueName: \"kubernetes.io/projected/2a9b3262-5fdf-4375-8411-a0640050c34e-kube-api-access-mn5dn\") pod \"2a9b3262-5fdf-4375-8411-a0640050c34e\" (UID: \"2a9b3262-5fdf-4375-8411-a0640050c34e\") "
Oct 31 20:00:19.637763 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:19.637627    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:19.648361 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:19.647554    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a9b3262-5fdf-4375-8411-a0640050c34e-maas-api-service-ca" pod "2a9b3262-5fdf-4375-8411-a0640050c34e" (UID: "2a9b3262-5fdf-4375-8411-a0640050c34e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:00:19.738410 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:19.702651    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a9b3262-5fdf-4375-8411-a0640050c34e-maas-api-service-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:19.738763 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:19.732006    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a9b3262-5fdf-4375-8411-a0640050c34e-kube-api-access-mn5dn" pod "2a9b3262-5fdf-4375-8411-a0640050c34e" (UID: "2a9b3262-5fdf-4375-8411-a0640050c34e"). InnerVolumeSpecName "kube-api-access-mn5dn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:00:19.764519 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:19.753740    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c8e3489-4fca-4756-a7a0-0c494eab1ba5" path="/var/lib/kubelet/pods/1c8e3489-4fca-4756-a7a0-0c494eab1ba5/volumes"
Oct 31 20:00:19.905637 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:19.905601    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mn5dn\" (UniqueName: \"kubernetes.io/projected/2a9b3262-5fdf-4375-8411-a0640050c34e-kube-api-access-mn5dn\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:19.972555 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:19.972515    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aeb292-29891280-dzgjd"]
Oct 31 20:00:20.582355 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:20.537759    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-01afb5-29891280-4hgmt" event={"ID":"674684f6-15f9-4e2e-a5d6-136098c534d3","Type":"ContainerStarted","Data":"2bda60b8bba123eabb76a3fd481266f0e7dee3166c78b00ed5ece76919047f0c"}
Oct 31 20:00:20.582355 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:20.550917    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-fd1c91-29891280-xbpn9" event={"ID":"2a9b3262-5fdf-4375-8411-a0640050c34e","Type":"ContainerDied","Data":"3f3717157fa17a25f681c840bb19a703b60b1821e02bfba4b37e5d83baac0192"}
Oct 31 20:00:20.582355 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:20.550976    4645 scope.go:122] "RemoveContainer" containerID="21fbb4bda3ea9a5a3b7a5ead340288b6d9984868826dc0caa6038f9365a4c79c"
Oct 31 20:00:20.582355 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:20.551085    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-fd1c91-29891280-xbpn9"
Oct 31 20:00:20.582355 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:20.555859    4645 generic.go:356] "Generic (PLEG): container finished" podID="4f88b769-7931-40c2-8cf6-ef92c2a85327" containerID="f681f99d8a466e50d5cb6c3416bd1203fe0ee5afe16c7239e4af5262a4b625fd" exitCode=60
Oct 31 20:00:20.582355 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:20.555933    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23814d-29891280-4zhlc" event={"ID":"4f88b769-7931-40c2-8cf6-ef92c2a85327","Type":"ContainerDied","Data":"f681f99d8a466e50d5cb6c3416bd1203fe0ee5afe16c7239e4af5262a4b625fd"}
Oct 31 20:00:20.582355 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:20.556474    4645 scope.go:122] "RemoveContainer" containerID="f681f99d8a466e50d5cb6c3416bd1203fe0ee5afe16c7239e4af5262a4b625fd"
Oct 31 20:00:20.582355 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:00:20.556905    4645 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-23814d-29891280-4zhlc_odh-ai-gateway-infra(4f88b769-7931-40c2-8cf6-ef92c2a85327)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23814d-29891280-4zhlc" podUID="4f88b769-7931-40c2-8cf6-ef92c2a85327"
Oct 31 20:00:20.582355 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:20.566080    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a891ce-29891280-7sxwj" event={"ID":"3b3e85d8-1b99-40b3-87ed-7e0d9c78bb86","Type":"ContainerStarted","Data":"089f6b79e343f23d1d1b76fcc95a2c82a94b168ba99b5111134607bb2bdf41e5"}
Oct 31 20:00:20.582355 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:20.579509    4645 generic.go:356] "Generic (PLEG): container finished" podID="0b226e15-ab1f-4f6c-9b08-1ce863756e43" containerID="78a6262fdd9e13e3acf8d62437c55c04f733676cd86ec9f65e2ff1024b2ed4ab" exitCode=60
Oct 31 20:00:20.582355 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:20.579615    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d9c048-29891280-wsjnm" event={"ID":"0b226e15-ab1f-4f6c-9b08-1ce863756e43","Type":"ContainerDied","Data":"78a6262fdd9e13e3acf8d62437c55c04f733676cd86ec9f65e2ff1024b2ed4ab"}
Oct 31 20:00:20.582355 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:20.580050    4645 scope.go:122] "RemoveContainer" containerID="78a6262fdd9e13e3acf8d62437c55c04f733676cd86ec9f65e2ff1024b2ed4ab"
Oct 31 20:00:20.582355 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:00:20.580453    4645 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-d9c048-29891280-wsjnm_odh-ai-gateway-infra(0b226e15-ab1f-4f6c-9b08-1ce863756e43)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d9c048-29891280-wsjnm" podUID="0b226e15-ab1f-4f6c-9b08-1ce863756e43"
Oct 31 20:00:20.638210 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:20.607591    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b083a-29891280-nq9lm" event={"ID":"d0a6fdf8-881f-4993-9515-dfc51f685500","Type":"ContainerStarted","Data":"e3a70fe5e23f7fda764a18b428272dfc7b6938e45e0150854bd025d3dfc94366"}
Oct 31 20:00:20.638210 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:20.608135    4645 scope.go:122] "RemoveContainer" containerID="e3a70fe5e23f7fda764a18b428272dfc7b6938e45e0150854bd025d3dfc94366"
Oct 31 20:00:20.638210 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:00:20.608636    4645 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-8b083a-29891280-nq9lm_odh-ai-gateway-infra(d0a6fdf8-881f-4993-9515-dfc51f685500)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b083a-29891280-nq9lm" podUID="d0a6fdf8-881f-4993-9515-dfc51f685500"
Oct 31 20:00:20.638210 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:20.611655    4645 generic.go:356] "Generic (PLEG): container finished" podID="f1c79dcb-63c1-4844-9527-5ef43f45eb95" containerID="5e86435dda5e8a4a34bdff53bb1db7cdb9f06da6749cc51de89a383eaaee58cb" exitCode=60
Oct 31 20:00:20.638210 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:20.611748    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aeb292-29891280-dzgjd" event={"ID":"f1c79dcb-63c1-4844-9527-5ef43f45eb95","Type":"ContainerDied","Data":"5e86435dda5e8a4a34bdff53bb1db7cdb9f06da6749cc51de89a383eaaee58cb"}
Oct 31 20:00:20.653639 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:20.653608    4645 scope.go:122] "RemoveContainer" containerID="9d5ace7819fcfebd365628660e1dc762bf44cba542ca7debcf2f0dfb99e9f643"
Oct 31 20:00:20.691724 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:20.691664    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:20.694691 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:20.694670    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23814d-29891280-4zhlc"]
Oct 31 20:00:20.781021 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:20.780910    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d9c048-29891280-wsjnm"]
Oct 31 20:00:20.821667 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:20.821626    4645 scope.go:122] "RemoveContainer" containerID="ea2cc04c33f026cb5dadb5dbbede0cf2798296d3c0feadeed0297608d6ea138c"
Oct 31 20:00:21.043661 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:21.043627    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:21.162467 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:21.162421    4645 scope.go:122] "RemoveContainer" containerID="a83ef4ce41ef26cae8d81102213b586b9f38c5813b0d8ebb322763c18e2aae03"
Oct 31 20:00:21.610833 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:21.610804    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-435728-29891280-6sfnt"
Oct 31 20:00:21.651296 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:21.651264    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aeb292-29891280-dzgjd"
Oct 31 20:00:21.683354 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:21.679902    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46fe6436-5347-408b-b173-aa153f96236e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46fe6436-5347-408b-b173-aa153f96236e-maas-api-service-ca\") pod \"46fe6436-5347-408b-b173-aa153f96236e\" (UID: \"46fe6436-5347-408b-b173-aa153f96236e\") "
Oct 31 20:00:21.683354 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:21.680369    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46fe6436-5347-408b-b173-aa153f96236e-kube-api-access-ppv87\" (UniqueName: \"kubernetes.io/projected/46fe6436-5347-408b-b173-aa153f96236e-kube-api-access-ppv87\") pod \"46fe6436-5347-408b-b173-aa153f96236e\" (UID: \"46fe6436-5347-408b-b173-aa153f96236e\") "
Oct 31 20:00:21.722062 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:21.721940    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46fe6436-5347-408b-b173-aa153f96236e-maas-api-service-ca" pod "46fe6436-5347-408b-b173-aa153f96236e" (UID: "46fe6436-5347-408b-b173-aa153f96236e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:00:21.754775 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:21.754730    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46fe6436-5347-408b-b173-aa153f96236e-kube-api-access-ppv87" pod "46fe6436-5347-408b-b173-aa153f96236e" (UID: "46fe6436-5347-408b-b173-aa153f96236e"). InnerVolumeSpecName "kube-api-access-ppv87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:00:21.796638 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:21.796557    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1c79dcb-63c1-4844-9527-5ef43f45eb95-kube-api-access-j56zz\" (UniqueName: \"kubernetes.io/projected/f1c79dcb-63c1-4844-9527-5ef43f45eb95-kube-api-access-j56zz\") pod \"f1c79dcb-63c1-4844-9527-5ef43f45eb95\" (UID: \"f1c79dcb-63c1-4844-9527-5ef43f45eb95\") "
Oct 31 20:00:21.798087 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:21.798028    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1c79dcb-63c1-4844-9527-5ef43f45eb95-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1c79dcb-63c1-4844-9527-5ef43f45eb95-maas-api-service-ca\") pod \"f1c79dcb-63c1-4844-9527-5ef43f45eb95\" (UID: \"f1c79dcb-63c1-4844-9527-5ef43f45eb95\") "
Oct 31 20:00:21.801523 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:21.799652    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46fe6436-5347-408b-b173-aa153f96236e-maas-api-service-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:21.802480 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:21.802461    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppv87\" (UniqueName: \"kubernetes.io/projected/46fe6436-5347-408b-b173-aa153f96236e-kube-api-access-ppv87\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:21.802935 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:21.802912    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1c79dcb-63c1-4844-9527-5ef43f45eb95-maas-api-service-ca" pod "f1c79dcb-63c1-4844-9527-5ef43f45eb95" (UID: "f1c79dcb-63c1-4844-9527-5ef43f45eb95"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:00:21.803957 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:21.797233    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-fd1c91-29891280-xbpn9"]
Oct 31 20:00:21.806959 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:21.806923    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1c79dcb-63c1-4844-9527-5ef43f45eb95-kube-api-access-j56zz" pod "f1c79dcb-63c1-4844-9527-5ef43f45eb95" (UID: "f1c79dcb-63c1-4844-9527-5ef43f45eb95"). InnerVolumeSpecName "kube-api-access-j56zz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:00:21.834862 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:21.831745    4645 generic.go:356] "Generic (PLEG): container finished" podID="4360bbcf-669f-4d6e-933c-0dcdfa76071a" containerID="2eb1f35ef38eb45ffcbdc41bdcf643bc5f51e2b3115c76aa065162595570da9a" exitCode=60
Oct 31 20:00:21.834862 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:21.831879    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e618e1-29891280-c679x" event={"ID":"4360bbcf-669f-4d6e-933c-0dcdfa76071a","Type":"ContainerDied","Data":"2eb1f35ef38eb45ffcbdc41bdcf643bc5f51e2b3115c76aa065162595570da9a"}
Oct 31 20:00:21.834862 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:21.831934    4645 scope.go:122] "RemoveContainer" containerID="01416cf9170be03b405b3302f801235bb45b1bd7f58b7af58363e1b48fd3dbf1"
Oct 31 20:00:21.834862 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:21.832433    4645 scope.go:122] "RemoveContainer" containerID="2eb1f35ef38eb45ffcbdc41bdcf643bc5f51e2b3115c76aa065162595570da9a"
Oct 31 20:00:21.834862 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:00:21.832848    4645 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-e618e1-29891280-c679x_odh-ai-gateway-infra(4360bbcf-669f-4d6e-933c-0dcdfa76071a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e618e1-29891280-c679x" podUID="4360bbcf-669f-4d6e-933c-0dcdfa76071a"
Oct 31 20:00:21.835977 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:21.835928    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-fd1c91-29891280-xbpn9"]
Oct 31 20:00:21.842229 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:21.841790    4645 generic.go:356] "Generic (PLEG): container finished" podID="d0a6fdf8-881f-4993-9515-dfc51f685500" containerID="e3a70fe5e23f7fda764a18b428272dfc7b6938e45e0150854bd025d3dfc94366" exitCode=60
Oct 31 20:00:21.842229 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:21.841883    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b083a-29891280-nq9lm" event={"ID":"d0a6fdf8-881f-4993-9515-dfc51f685500","Type":"ContainerDied","Data":"e3a70fe5e23f7fda764a18b428272dfc7b6938e45e0150854bd025d3dfc94366"}
Oct 31 20:00:21.846395 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:21.846039    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aeb292-29891280-dzgjd" event={"ID":"f1c79dcb-63c1-4844-9527-5ef43f45eb95","Type":"ContainerDied","Data":"a0752e04e485a3624a72daabdfeb87161226587a4d1936f686477fb965827a3c"}
Oct 31 20:00:21.846395 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:21.846141    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aeb292-29891280-dzgjd"
Oct 31 20:00:21.856355 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:21.853585    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-435728-29891280-6sfnt" event={"ID":"46fe6436-5347-408b-b173-aa153f96236e","Type":"ContainerDied","Data":"b4e31678ec941a374580e8182f380e48ee4ca82d950bb6dfe3c73835f91db560"}
Oct 31 20:00:21.856355 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:21.853679    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-435728-29891280-6sfnt"
Oct 31 20:00:21.858796 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:21.858056    4645 scope.go:122] "RemoveContainer" containerID="e3a70fe5e23f7fda764a18b428272dfc7b6938e45e0150854bd025d3dfc94366"
Oct 31 20:00:21.858796 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:00:21.858439    4645 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-8b083a-29891280-nq9lm_odh-ai-gateway-infra(d0a6fdf8-881f-4993-9515-dfc51f685500)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b083a-29891280-nq9lm" podUID="d0a6fdf8-881f-4993-9515-dfc51f685500"
Oct 31 20:00:21.896391 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:21.896366    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c734f5-699f7df769-c5swh"]
Oct 31 20:00:21.897300 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:21.897278    4645 cpu_manager.go:414] "containerMap: removing container" podUID="1c8e3489-4fca-4756-a7a0-0c494eab1ba5" containerName="cleanup"
Oct 31 20:00:21.897460 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:21.897300    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c8e3489-4fca-4756-a7a0-0c494eab1ba5" containerName="cleanup"
Oct 31 20:00:21.897460 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:21.897318    4645 cpu_manager.go:414] "containerMap: removing container" podUID="46fe6436-5347-408b-b173-aa153f96236e" containerName="cleanup"
Oct 31 20:00:21.897460 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:21.897340    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46fe6436-5347-408b-b173-aa153f96236e" containerName="cleanup"
Oct 31 20:00:21.897460 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:21.897371    4645 cpu_manager.go:414] "containerMap: removing container" podUID="f1c79dcb-63c1-4844-9527-5ef43f45eb95" containerName="cleanup"
Oct 31 20:00:21.897460 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:21.897378    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1c79dcb-63c1-4844-9527-5ef43f45eb95" containerName="cleanup"
Oct 31 20:00:21.897460 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:21.897398    4645 cpu_manager.go:414] "containerMap: removing container" podUID="63f3555a-1396-4563-abee-5ed20eda9e11" containerName="cleanup"
Oct 31 20:00:21.897460 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:21.897405    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63f3555a-1396-4563-abee-5ed20eda9e11" containerName="cleanup"
Oct 31 20:00:21.897460 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:21.897422    4645 cpu_manager.go:414] "containerMap: removing container" podUID="2a9b3262-5fdf-4375-8411-a0640050c34e" containerName="cleanup"
Oct 31 20:00:21.897460 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:21.897429    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a9b3262-5fdf-4375-8411-a0640050c34e" containerName="cleanup"
Oct 31 20:00:21.897460 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:21.897442    4645 cpu_manager.go:414] "containerMap: removing container" podUID="f1c79dcb-63c1-4844-9527-5ef43f45eb95" containerName="cleanup"
Oct 31 20:00:21.897460 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:21.897451    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1c79dcb-63c1-4844-9527-5ef43f45eb95" containerName="cleanup"
Oct 31 20:00:21.897994 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:21.897481    4645 cpu_manager.go:414] "containerMap: removing container" podUID="1c8e3489-4fca-4756-a7a0-0c494eab1ba5" containerName="cleanup"
Oct 31 20:00:21.897994 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:21.897489    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c8e3489-4fca-4756-a7a0-0c494eab1ba5" containerName="cleanup"
Oct 31 20:00:21.897994 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:21.897498    4645 cpu_manager.go:414] "containerMap: removing container" podUID="46fe6436-5347-408b-b173-aa153f96236e" containerName="cleanup"
Oct 31 20:00:21.897994 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:21.897505    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46fe6436-5347-408b-b173-aa153f96236e" containerName="cleanup"
Oct 31 20:00:21.897994 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:21.897520    4645 cpu_manager.go:414] "containerMap: removing container" podUID="2a9b3262-5fdf-4375-8411-a0640050c34e" containerName="cleanup"
Oct 31 20:00:21.897994 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:21.897528    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a9b3262-5fdf-4375-8411-a0640050c34e" containerName="cleanup"
Oct 31 20:00:21.897994 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:21.897556    4645 cpu_manager.go:414] "containerMap: removing container" podUID="63f3555a-1396-4563-abee-5ed20eda9e11" containerName="cleanup"
Oct 31 20:00:21.897994 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:21.897563    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63f3555a-1396-4563-abee-5ed20eda9e11" containerName="cleanup"
Oct 31 20:00:21.897994 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:21.897755    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1c79dcb-63c1-4844-9527-5ef43f45eb95" containerName="cleanup"
Oct 31 20:00:21.897994 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:21.897770    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="46fe6436-5347-408b-b173-aa153f96236e" containerName="cleanup"
Oct 31 20:00:21.897994 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:21.897781    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c8e3489-4fca-4756-a7a0-0c494eab1ba5" containerName="cleanup"
Oct 31 20:00:21.897994 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:21.897791    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="46fe6436-5347-408b-b173-aa153f96236e" containerName="cleanup"
Oct 31 20:00:21.897994 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:21.897799    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a9b3262-5fdf-4375-8411-a0640050c34e" containerName="cleanup"
Oct 31 20:00:21.897994 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:21.897809    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a9b3262-5fdf-4375-8411-a0640050c34e" containerName="cleanup"
Oct 31 20:00:21.897994 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:21.897822    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1c79dcb-63c1-4844-9527-5ef43f45eb95" containerName="cleanup"
Oct 31 20:00:21.897994 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:21.897834    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="63f3555a-1396-4563-abee-5ed20eda9e11" containerName="cleanup"
Oct 31 20:00:21.897994 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:21.897853    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c8e3489-4fca-4756-a7a0-0c494eab1ba5" containerName="cleanup"
Oct 31 20:00:21.897994 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:21.897861    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="63f3555a-1396-4563-abee-5ed20eda9e11" containerName="cleanup"
Oct 31 20:00:21.905118 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:21.904590    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j56zz\" (UniqueName: \"kubernetes.io/projected/f1c79dcb-63c1-4844-9527-5ef43f45eb95-kube-api-access-j56zz\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:21.905118 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:21.904618    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1c79dcb-63c1-4844-9527-5ef43f45eb95-maas-api-service-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:21.922437 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:21.921640    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c734f5-699f7df769-c5swh"
Oct 31 20:00:21.932664 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:21.923184    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:21.932664 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:21.927726    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-c734f5\""
Oct 31 20:00:21.947622 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:21.947429    4645 scope.go:122] "RemoveContainer" containerID="1c16bf1f54cd5da34db9aafdcff7e4629767b86f458ebca4d4974acc5450b8fd"
Oct 31 20:00:21.947779 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:21.947701    4645 generic.go:356] "Generic (PLEG): container finished" podID="674684f6-15f9-4e2e-a5d6-136098c534d3" containerID="2bda60b8bba123eabb76a3fd481266f0e7dee3166c78b00ed5ece76919047f0c" exitCode=60
Oct 31 20:00:21.968574 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:21.921539    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c734f5-699f7df769-c5swh"]
Oct 31 20:00:21.968859 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:21.968833    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-01afb5-29891280-4hgmt" event={"ID":"674684f6-15f9-4e2e-a5d6-136098c534d3","Type":"ContainerDied","Data":"2bda60b8bba123eabb76a3fd481266f0e7dee3166c78b00ed5ece76919047f0c"}
Oct 31 20:00:21.969492 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:21.969466    4645 scope.go:122] "RemoveContainer" containerID="2bda60b8bba123eabb76a3fd481266f0e7dee3166c78b00ed5ece76919047f0c"
Oct 31 20:00:21.970139 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:00:21.970109    4645 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-01afb5-29891280-4hgmt_odh-ai-gateway-infra(674684f6-15f9-4e2e-a5d6-136098c534d3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-01afb5-29891280-4hgmt" podUID="674684f6-15f9-4e2e-a5d6-136098c534d3"
Oct 31 20:00:22.013804 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:22.013738    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5tdl\" (UniqueName: \"kubernetes.io/projected/7a2fa9c5-2e84-4531-b72c-183909b077a7-kube-api-access-s5tdl\") pod \"payload-processing-e2e-ipp-a-c734f5-699f7df769-c5swh\" (UID: \"7a2fa9c5-2e84-4531-b72c-183909b077a7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c734f5-699f7df769-c5swh"
Oct 31 20:00:22.014185 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:22.014153    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a2fa9c5-2e84-4531-b72c-183909b077a7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c734f5-699f7df769-c5swh\" (UID: \"7a2fa9c5-2e84-4531-b72c-183909b077a7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c734f5-699f7df769-c5swh"
Oct 31 20:00:22.051716 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:22.051618    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-01afb5-29891280-4hgmt"]
Oct 31 20:00:22.051716 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:22.051698    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a891ce-29891280-7sxwj"]
Oct 31 20:00:22.092514 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:22.092415    4645 scope.go:122] "RemoveContainer" containerID="5e86435dda5e8a4a34bdff53bb1db7cdb9f06da6749cc51de89a383eaaee58cb"
Oct 31 20:00:22.094604 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:22.094120    4645 generic.go:356] "Generic (PLEG): container finished" podID="3b3e85d8-1b99-40b3-87ed-7e0d9c78bb86" containerID="089f6b79e343f23d1d1b76fcc95a2c82a94b168ba99b5111134607bb2bdf41e5" exitCode=60
Oct 31 20:00:22.094604 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:22.094201    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a891ce-29891280-7sxwj" event={"ID":"3b3e85d8-1b99-40b3-87ed-7e0d9c78bb86","Type":"ContainerDied","Data":"089f6b79e343f23d1d1b76fcc95a2c82a94b168ba99b5111134607bb2bdf41e5"}
Oct 31 20:00:22.119369 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:22.118170    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5tdl\" (UniqueName: \"kubernetes.io/projected/7a2fa9c5-2e84-4531-b72c-183909b077a7-kube-api-access-s5tdl\") pod \"payload-processing-e2e-ipp-a-c734f5-699f7df769-c5swh\" (UID: \"7a2fa9c5-2e84-4531-b72c-183909b077a7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c734f5-699f7df769-c5swh"
Oct 31 20:00:22.119369 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:22.118365    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a2fa9c5-2e84-4531-b72c-183909b077a7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c734f5-699f7df769-c5swh\" (UID: \"7a2fa9c5-2e84-4531-b72c-183909b077a7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c734f5-699f7df769-c5swh"
Oct 31 20:00:22.120354 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:22.119715    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a2fa9c5-2e84-4531-b72c-183909b077a7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c734f5-699f7df769-c5swh\" (UID: \"7a2fa9c5-2e84-4531-b72c-183909b077a7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c734f5-699f7df769-c5swh"
Oct 31 20:00:22.153223 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:22.153193    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b083a-29891280-nq9lm"]
Oct 31 20:00:22.155795 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:22.155775    4645 scope.go:122] "RemoveContainer" containerID="2a69be78a116b70e13fee263b067b4f00be192e69f39767496b9e53e838c0fc1"
Oct 31 20:00:22.201365 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:22.169299    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5tdl\" (UniqueName: \"kubernetes.io/projected/7a2fa9c5-2e84-4531-b72c-183909b077a7-kube-api-access-s5tdl\") pod \"payload-processing-e2e-ipp-a-c734f5-699f7df769-c5swh\" (UID: \"7a2fa9c5-2e84-4531-b72c-183909b077a7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c734f5-699f7df769-c5swh"
Oct 31 20:00:22.214804 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:22.214748    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aeb292-29891280-dzgjd"]
Oct 31 20:00:22.268230 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:22.268144    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-aeb292-29891280-dzgjd"]
Oct 31 20:00:22.294359 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:22.287282    4645 scope.go:122] "RemoveContainer" containerID="87192f9f259eee05cee1b63489fc95549b2de558500303455d9d5f712f271b99"
Oct 31 20:00:22.315716 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:22.315195    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-435728-29891280-6sfnt"]
Oct 31 20:00:22.315716 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:22.315486    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c734f5-699f7df769-c5swh"
Oct 31 20:00:22.334980 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:22.334951    4645 scope.go:122] "RemoveContainer" containerID="cefc70a1b2ab61aa7f3a2233016287bd22cea3ea2b40801f18879f1dc75b493b"
Oct 31 20:00:22.335077 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:22.335065    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-435728-29891280-6sfnt"]
Oct 31 20:00:22.495716 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:22.495682    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23814d-29891280-4zhlc"
Oct 31 20:00:22.575831 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:22.575803    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a891ce-29891280-7sxwj"
Oct 31 20:00:22.581483 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:22.581456    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d9c048-29891280-wsjnm"
Oct 31 20:00:22.686204 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:22.684539    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b226e15-ab1f-4f6c-9b08-1ce863756e43-kube-api-access-vpmbw\" (UniqueName: \"kubernetes.io/projected/0b226e15-ab1f-4f6c-9b08-1ce863756e43-kube-api-access-vpmbw\") pod \"0b226e15-ab1f-4f6c-9b08-1ce863756e43\" (UID: \"0b226e15-ab1f-4f6c-9b08-1ce863756e43\") "
Oct 31 20:00:22.686204 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:22.684694    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f88b769-7931-40c2-8cf6-ef92c2a85327-kube-api-access-7976z\" (UniqueName: \"kubernetes.io/projected/4f88b769-7931-40c2-8cf6-ef92c2a85327-kube-api-access-7976z\") pod \"4f88b769-7931-40c2-8cf6-ef92c2a85327\" (UID: \"4f88b769-7931-40c2-8cf6-ef92c2a85327\") "
Oct 31 20:00:22.686204 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:22.684815    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b226e15-ab1f-4f6c-9b08-1ce863756e43-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b226e15-ab1f-4f6c-9b08-1ce863756e43-maas-api-service-ca\") pod \"0b226e15-ab1f-4f6c-9b08-1ce863756e43\" (UID: \"0b226e15-ab1f-4f6c-9b08-1ce863756e43\") "
Oct 31 20:00:22.686204 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:22.684914    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f88b769-7931-40c2-8cf6-ef92c2a85327-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f88b769-7931-40c2-8cf6-ef92c2a85327-maas-api-service-ca\") pod \"4f88b769-7931-40c2-8cf6-ef92c2a85327\" (UID: \"4f88b769-7931-40c2-8cf6-ef92c2a85327\") "
Oct 31 20:00:22.686204 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:22.684997    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b3e85d8-1b99-40b3-87ed-7e0d9c78bb86-kube-api-access-bsfh5\" (UniqueName: \"kubernetes.io/projected/3b3e85d8-1b99-40b3-87ed-7e0d9c78bb86-kube-api-access-bsfh5\") pod \"3b3e85d8-1b99-40b3-87ed-7e0d9c78bb86\" (UID: \"3b3e85d8-1b99-40b3-87ed-7e0d9c78bb86\") "
Oct 31 20:00:22.686204 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:22.685055    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b3e85d8-1b99-40b3-87ed-7e0d9c78bb86-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b3e85d8-1b99-40b3-87ed-7e0d9c78bb86-maas-api-service-ca\") pod \"3b3e85d8-1b99-40b3-87ed-7e0d9c78bb86\" (UID: \"3b3e85d8-1b99-40b3-87ed-7e0d9c78bb86\") "
Oct 31 20:00:22.687155 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:22.687130    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b3e85d8-1b99-40b3-87ed-7e0d9c78bb86-maas-api-service-ca" pod "3b3e85d8-1b99-40b3-87ed-7e0d9c78bb86" (UID: "3b3e85d8-1b99-40b3-87ed-7e0d9c78bb86"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:00:22.688126 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:22.688103    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f88b769-7931-40c2-8cf6-ef92c2a85327-maas-api-service-ca" pod "4f88b769-7931-40c2-8cf6-ef92c2a85327" (UID: "4f88b769-7931-40c2-8cf6-ef92c2a85327"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:00:22.688685 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:22.688662    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b226e15-ab1f-4f6c-9b08-1ce863756e43-maas-api-service-ca" pod "0b226e15-ab1f-4f6c-9b08-1ce863756e43" (UID: "0b226e15-ab1f-4f6c-9b08-1ce863756e43"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:00:22.690253 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:22.690231    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b226e15-ab1f-4f6c-9b08-1ce863756e43-kube-api-access-vpmbw" pod "0b226e15-ab1f-4f6c-9b08-1ce863756e43" (UID: "0b226e15-ab1f-4f6c-9b08-1ce863756e43"). InnerVolumeSpecName "kube-api-access-vpmbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:00:22.693581 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:22.693557    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b3e85d8-1b99-40b3-87ed-7e0d9c78bb86-kube-api-access-bsfh5" pod "3b3e85d8-1b99-40b3-87ed-7e0d9c78bb86" (UID: "3b3e85d8-1b99-40b3-87ed-7e0d9c78bb86"). InnerVolumeSpecName "kube-api-access-bsfh5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:00:22.694565 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:22.694547    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f88b769-7931-40c2-8cf6-ef92c2a85327-kube-api-access-7976z" pod "4f88b769-7931-40c2-8cf6-ef92c2a85327" (UID: "4f88b769-7931-40c2-8cf6-ef92c2a85327"). InnerVolumeSpecName "kube-api-access-7976z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:00:22.753392 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:00:22.753361    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a2fa9c5_2e84_4531_b72c_183909b077a7.slice/crio-ab3830e1b609ad1234b79611a422832e9c85338a2a18cd156d374fb6da038a81 WatchSource:0}: Error finding container ab3830e1b609ad1234b79611a422832e9c85338a2a18cd156d374fb6da038a81: Status 404 returned error can't find the container with id ab3830e1b609ad1234b79611a422832e9c85338a2a18cd156d374fb6da038a81
Oct 31 20:00:22.762502 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:22.762467    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c734f5-699f7df769-c5swh"]
Oct 31 20:00:22.778444 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:22.778410    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs"
Oct 31 20:00:22.799836 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:22.799712    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b226e15-ab1f-4f6c-9b08-1ce863756e43-maas-api-service-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:22.799836 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:22.799746    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f88b769-7931-40c2-8cf6-ef92c2a85327-maas-api-service-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:22.799836 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:22.799764    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsfh5\" (UniqueName: \"kubernetes.io/projected/3b3e85d8-1b99-40b3-87ed-7e0d9c78bb86-kube-api-access-bsfh5\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:22.799836 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:22.799785    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b3e85d8-1b99-40b3-87ed-7e0d9c78bb86-maas-api-service-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:22.799836 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:22.799800    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpmbw\" (UniqueName: \"kubernetes.io/projected/0b226e15-ab1f-4f6c-9b08-1ce863756e43-kube-api-access-vpmbw\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:22.799836 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:22.799813    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7976z\" (UniqueName: \"kubernetes.io/projected/4f88b769-7931-40c2-8cf6-ef92c2a85327-kube-api-access-7976z\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:22.900554 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:22.900519    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48ab68ad-2abf-44c8-98dc-c3ec72d36362-istio-token\" (UniqueName: \"kubernetes.io/projected/48ab68ad-2abf-44c8-98dc-c3ec72d36362-istio-token\") pod \"48ab68ad-2abf-44c8-98dc-c3ec72d36362\" (UID: \"48ab68ad-2abf-44c8-98dc-c3ec72d36362\") "
Oct 31 20:00:22.900707 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:22.900588    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48ab68ad-2abf-44c8-98dc-c3ec72d36362-kube-api-access-xtkxc\" (UniqueName: \"kubernetes.io/projected/48ab68ad-2abf-44c8-98dc-c3ec72d36362-kube-api-access-xtkxc\") pod \"48ab68ad-2abf-44c8-98dc-c3ec72d36362\" (UID: \"48ab68ad-2abf-44c8-98dc-c3ec72d36362\") "
Oct 31 20:00:22.900707 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:22.900699    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/48ab68ad-2abf-44c8-98dc-c3ec72d36362-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48ab68ad-2abf-44c8-98dc-c3ec72d36362-istio-podinfo\") pod \"48ab68ad-2abf-44c8-98dc-c3ec72d36362\" (UID: \"48ab68ad-2abf-44c8-98dc-c3ec72d36362\") "
Oct 31 20:00:22.900814 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:22.900739    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48ab68ad-2abf-44c8-98dc-c3ec72d36362-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48ab68ad-2abf-44c8-98dc-c3ec72d36362-workload-certs\") pod \"48ab68ad-2abf-44c8-98dc-c3ec72d36362\" (UID: \"48ab68ad-2abf-44c8-98dc-c3ec72d36362\") "
Oct 31 20:00:22.900814 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:22.900807    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48ab68ad-2abf-44c8-98dc-c3ec72d36362-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48ab68ad-2abf-44c8-98dc-c3ec72d36362-workload-socket\") pod \"48ab68ad-2abf-44c8-98dc-c3ec72d36362\" (UID: \"48ab68ad-2abf-44c8-98dc-c3ec72d36362\") "
Oct 31 20:00:22.900940 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:22.900916    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48ab68ad-2abf-44c8-98dc-c3ec72d36362-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48ab68ad-2abf-44c8-98dc-c3ec72d36362-istiod-ca-cert\") pod \"48ab68ad-2abf-44c8-98dc-c3ec72d36362\" (UID: \"48ab68ad-2abf-44c8-98dc-c3ec72d36362\") "
Oct 31 20:00:22.900993 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:22.900969    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48ab68ad-2abf-44c8-98dc-c3ec72d36362-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48ab68ad-2abf-44c8-98dc-c3ec72d36362-istio-envoy\") pod \"48ab68ad-2abf-44c8-98dc-c3ec72d36362\" (UID: \"48ab68ad-2abf-44c8-98dc-c3ec72d36362\") "
Oct 31 20:00:22.901042 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:22.901011    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48ab68ad-2abf-44c8-98dc-c3ec72d36362-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48ab68ad-2abf-44c8-98dc-c3ec72d36362-istio-data\") pod \"48ab68ad-2abf-44c8-98dc-c3ec72d36362\" (UID: \"48ab68ad-2abf-44c8-98dc-c3ec72d36362\") "
Oct 31 20:00:22.901158 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:22.901140    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48ab68ad-2abf-44c8-98dc-c3ec72d36362-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48ab68ad-2abf-44c8-98dc-c3ec72d36362-credential-socket\") pod \"48ab68ad-2abf-44c8-98dc-c3ec72d36362\" (UID: \"48ab68ad-2abf-44c8-98dc-c3ec72d36362\") "
Oct 31 20:00:22.901596 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:22.901562    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48ab68ad-2abf-44c8-98dc-c3ec72d36362-istiod-ca-cert" pod "48ab68ad-2abf-44c8-98dc-c3ec72d36362" (UID: "48ab68ad-2abf-44c8-98dc-c3ec72d36362"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:00:22.901837 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:22.901811    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48ab68ad-2abf-44c8-98dc-c3ec72d36362-credential-socket" pod "48ab68ad-2abf-44c8-98dc-c3ec72d36362" (UID: "48ab68ad-2abf-44c8-98dc-c3ec72d36362"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:00:22.902046 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:22.902022    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48ab68ad-2abf-44c8-98dc-c3ec72d36362-workload-socket" pod "48ab68ad-2abf-44c8-98dc-c3ec72d36362" (UID: "48ab68ad-2abf-44c8-98dc-c3ec72d36362"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:00:22.902216 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:22.902191    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48ab68ad-2abf-44c8-98dc-c3ec72d36362-istio-data" pod "48ab68ad-2abf-44c8-98dc-c3ec72d36362" (UID: "48ab68ad-2abf-44c8-98dc-c3ec72d36362"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:00:22.902292 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:22.902207    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48ab68ad-2abf-44c8-98dc-c3ec72d36362-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:22.902292 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:22.902255    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48ab68ad-2abf-44c8-98dc-c3ec72d36362-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:22.902292 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:22.902278    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48ab68ad-2abf-44c8-98dc-c3ec72d36362-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:22.902550 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:22.902352    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48ab68ad-2abf-44c8-98dc-c3ec72d36362-workload-certs" pod "48ab68ad-2abf-44c8-98dc-c3ec72d36362" (UID: "48ab68ad-2abf-44c8-98dc-c3ec72d36362"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:00:22.904232 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:22.904206    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48ab68ad-2abf-44c8-98dc-c3ec72d36362-istio-podinfo" pod "48ab68ad-2abf-44c8-98dc-c3ec72d36362" (UID: "48ab68ad-2abf-44c8-98dc-c3ec72d36362"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 20:00:22.904514 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:22.904492    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48ab68ad-2abf-44c8-98dc-c3ec72d36362-istio-envoy" pod "48ab68ad-2abf-44c8-98dc-c3ec72d36362" (UID: "48ab68ad-2abf-44c8-98dc-c3ec72d36362"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:00:22.904670 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:22.904636    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48ab68ad-2abf-44c8-98dc-c3ec72d36362-istio-token" pod "48ab68ad-2abf-44c8-98dc-c3ec72d36362" (UID: "48ab68ad-2abf-44c8-98dc-c3ec72d36362"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:00:22.905161 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:22.905142    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48ab68ad-2abf-44c8-98dc-c3ec72d36362-kube-api-access-xtkxc" pod "48ab68ad-2abf-44c8-98dc-c3ec72d36362" (UID: "48ab68ad-2abf-44c8-98dc-c3ec72d36362"). InnerVolumeSpecName "kube-api-access-xtkxc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:00:22.912849 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:22.912825    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e618e1-29891280-c679x"]
Oct 31 20:00:23.003477 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:23.003438    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48ab68ad-2abf-44c8-98dc-c3ec72d36362-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:23.003477 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:23.003477    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48ab68ad-2abf-44c8-98dc-c3ec72d36362-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:23.003735 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:23.003493    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48ab68ad-2abf-44c8-98dc-c3ec72d36362-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:23.003735 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:23.003504    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtkxc\" (UniqueName: \"kubernetes.io/projected/48ab68ad-2abf-44c8-98dc-c3ec72d36362-kube-api-access-xtkxc\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:23.003735 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:23.003515    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48ab68ad-2abf-44c8-98dc-c3ec72d36362-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:23.003735 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:23.003523    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48ab68ad-2abf-44c8-98dc-c3ec72d36362-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:23.122788 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:23.122013    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c734f5-699f7df769-c5swh" event={"ID":"7a2fa9c5-2e84-4531-b72c-183909b077a7","Type":"ContainerStarted","Data":"c2865b8413714dbde4532183ec36e0ff878923494dc50387570ff56c433c89c0"}
Oct 31 20:00:23.122788 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:23.122077    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c734f5-699f7df769-c5swh" event={"ID":"7a2fa9c5-2e84-4531-b72c-183909b077a7","Type":"ContainerStarted","Data":"ab3830e1b609ad1234b79611a422832e9c85338a2a18cd156d374fb6da038a81"}
Oct 31 20:00:23.123294 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:23.123271    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c734f5-699f7df769-c5swh"
Oct 31 20:00:23.124095 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:23.124070    4645 generic.go:356] "Generic (PLEG): container finished" podID="48ab68ad-2abf-44c8-98dc-c3ec72d36362" containerID="bcbcd6e75fa56be2a3f5a818fd449b59252d483240db33b92cf8d944e34b716e" exitCode=0
Oct 31 20:00:23.124277 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:23.124254    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs"
Oct 31 20:00:23.124451 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:23.124138    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs" event={"ID":"48ab68ad-2abf-44c8-98dc-c3ec72d36362","Type":"ContainerDied","Data":"bcbcd6e75fa56be2a3f5a818fd449b59252d483240db33b92cf8d944e34b716e"}
Oct 31 20:00:23.124451 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:23.124424    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs" event={"ID":"48ab68ad-2abf-44c8-98dc-c3ec72d36362","Type":"ContainerDied","Data":"72ebcb4a54b2abdc3415279aa576d28255fbbe5533649092e67ac8e42e37e339"}
Oct 31 20:00:23.124577 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:23.124477    4645 scope.go:122] "RemoveContainer" containerID="bcbcd6e75fa56be2a3f5a818fd449b59252d483240db33b92cf8d944e34b716e"
Oct 31 20:00:23.128714 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:23.128691    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23814d-29891280-4zhlc" event={"ID":"4f88b769-7931-40c2-8cf6-ef92c2a85327","Type":"ContainerDied","Data":"2419b38cd0686b3a59bcc14ca1660255b425d2c655dbcbcfe7bcd1bc2a598e73"}
Oct 31 20:00:23.128823 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:23.128773    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23814d-29891280-4zhlc"
Oct 31 20:00:23.141055 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:23.133795    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a891ce-29891280-7sxwj"
Oct 31 20:00:23.141055 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:23.133806    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a891ce-29891280-7sxwj" event={"ID":"3b3e85d8-1b99-40b3-87ed-7e0d9c78bb86","Type":"ContainerDied","Data":"988dcd25bbe7e5e9a03ecfa4f53582f2ad7a422cad520cd1db01d04c7cc06b92"}
Oct 31 20:00:23.141055 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:23.135817    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d9c048-29891280-wsjnm" event={"ID":"0b226e15-ab1f-4f6c-9b08-1ce863756e43","Type":"ContainerDied","Data":"15e968361e43087799e9c910fcce5eafbfbb94c7ca337fd470c956222b151b6b"}
Oct 31 20:00:23.141055 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:23.135900    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d9c048-29891280-wsjnm"
Oct 31 20:00:23.167875 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:23.167823    4645 scope.go:122] "RemoveContainer" containerID="1d7aeedb1b68c93d708d6174b24a7c7cdd3b72f8761a9c34d5da7b479223a126"
Oct 31 20:00:23.228689 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:23.205710    4645 scope.go:122] "RemoveContainer" containerID="bcbcd6e75fa56be2a3f5a818fd449b59252d483240db33b92cf8d944e34b716e"
Oct 31 20:00:23.228689 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:00:23.214799    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcbcd6e75fa56be2a3f5a818fd449b59252d483240db33b92cf8d944e34b716e\": container with ID starting with bcbcd6e75fa56be2a3f5a818fd449b59252d483240db33b92cf8d944e34b716e not found: ID does not exist" containerID="bcbcd6e75fa56be2a3f5a818fd449b59252d483240db33b92cf8d944e34b716e"
Oct 31 20:00:23.228689 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:23.214841    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcbcd6e75fa56be2a3f5a818fd449b59252d483240db33b92cf8d944e34b716e"} err="failed to get container status \"bcbcd6e75fa56be2a3f5a818fd449b59252d483240db33b92cf8d944e34b716e\": rpc error: code = NotFound desc = could not find container \"bcbcd6e75fa56be2a3f5a818fd449b59252d483240db33b92cf8d944e34b716e\": container with ID starting with bcbcd6e75fa56be2a3f5a818fd449b59252d483240db33b92cf8d944e34b716e not found: ID does not exist"
Oct 31 20:00:23.228689 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:23.214868    4645 scope.go:122] "RemoveContainer" containerID="1d7aeedb1b68c93d708d6174b24a7c7cdd3b72f8761a9c34d5da7b479223a126"
Oct 31 20:00:23.228689 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:00:23.215624    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d7aeedb1b68c93d708d6174b24a7c7cdd3b72f8761a9c34d5da7b479223a126\": container with ID starting with 1d7aeedb1b68c93d708d6174b24a7c7cdd3b72f8761a9c34d5da7b479223a126 not found: ID does not exist" containerID="1d7aeedb1b68c93d708d6174b24a7c7cdd3b72f8761a9c34d5da7b479223a126"
Oct 31 20:00:23.228689 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:23.215662    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d7aeedb1b68c93d708d6174b24a7c7cdd3b72f8761a9c34d5da7b479223a126"} err="failed to get container status \"1d7aeedb1b68c93d708d6174b24a7c7cdd3b72f8761a9c34d5da7b479223a126\": rpc error: code = NotFound desc = could not find container \"1d7aeedb1b68c93d708d6174b24a7c7cdd3b72f8761a9c34d5da7b479223a126\": container with ID starting with 1d7aeedb1b68c93d708d6174b24a7c7cdd3b72f8761a9c34d5da7b479223a126 not found: ID does not exist"
Oct 31 20:00:23.228689 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:23.215686    4645 scope.go:122] "RemoveContainer" containerID="f681f99d8a466e50d5cb6c3416bd1203fe0ee5afe16c7239e4af5262a4b625fd"
Oct 31 20:00:23.243971 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:23.242448    4645 scope.go:122] "RemoveContainer" containerID="089f6b79e343f23d1d1b76fcc95a2c82a94b168ba99b5111134607bb2bdf41e5"
Oct 31 20:00:23.280504 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:23.280474    4645 scope.go:122] "RemoveContainer" containerID="78a6262fdd9e13e3acf8d62437c55c04f733676cd86ec9f65e2ff1024b2ed4ab"
Oct 31 20:00:23.339905 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:23.339826    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-c734f5-699f7df769-c5swh" podStartSLOduration=3.33979538 podStartE2EDuration="3.33979538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:00:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:00:23.330673052 +0000 UTC m=+2453.903907182" watchObservedRunningTime="2026-10-31 20:00:23.33979538 +0000 UTC m=+2453.913029507"
Oct 31 20:00:23.350684 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:23.350655    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:23.378934 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:23.378845    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs"]
Oct 31 20:00:23.381541 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:23.381517    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-1f2781-openshift-default-6c5fbfd5f7-w7jcs"]
Oct 31 20:00:23.432206 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:23.432085    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23814d-29891280-4zhlc"]
Oct 31 20:00:23.438357 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:23.437702    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-23814d-29891280-4zhlc"]
Oct 31 20:00:23.448716 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:23.448691    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a891ce-29891280-7sxwj"]
Oct 31 20:00:23.460554 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:23.457727    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-a891ce-29891280-7sxwj"]
Oct 31 20:00:23.496351 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:23.493351    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:23.578370 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:23.573950    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d9c048-29891280-wsjnm"]
Oct 31 20:00:23.591795 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:23.591762    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-d9c048-29891280-wsjnm"]
Oct 31 20:00:23.592986 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:23.592695    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b083a-29891280-nq9lm"
Oct 31 20:00:23.642664 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:23.642297    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b226e15-ab1f-4f6c-9b08-1ce863756e43" path="/var/lib/kubelet/pods/0b226e15-ab1f-4f6c-9b08-1ce863756e43/volumes"
Oct 31 20:00:23.643720 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:23.643627    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a9b3262-5fdf-4375-8411-a0640050c34e" path="/var/lib/kubelet/pods/2a9b3262-5fdf-4375-8411-a0640050c34e/volumes"
Oct 31 20:00:23.644544 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:23.644523    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b3e85d8-1b99-40b3-87ed-7e0d9c78bb86" path="/var/lib/kubelet/pods/3b3e85d8-1b99-40b3-87ed-7e0d9c78bb86/volumes"
Oct 31 20:00:23.645821 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:23.645807    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46fe6436-5347-408b-b173-aa153f96236e" path="/var/lib/kubelet/pods/46fe6436-5347-408b-b173-aa153f96236e/volumes"
Oct 31 20:00:23.646456 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:23.646438    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48ab68ad-2abf-44c8-98dc-c3ec72d36362" path="/var/lib/kubelet/pods/48ab68ad-2abf-44c8-98dc-c3ec72d36362/volumes"
Oct 31 20:00:23.647172 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:23.647140    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f88b769-7931-40c2-8cf6-ef92c2a85327" path="/var/lib/kubelet/pods/4f88b769-7931-40c2-8cf6-ef92c2a85327/volumes"
Oct 31 20:00:23.648290 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:23.648258    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1c79dcb-63c1-4844-9527-5ef43f45eb95" path="/var/lib/kubelet/pods/f1c79dcb-63c1-4844-9527-5ef43f45eb95/volumes"
Oct 31 20:00:23.662708 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:23.662686    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-01afb5-29891280-4hgmt"
Oct 31 20:00:23.664622 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:23.664597    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0a6fdf8-881f-4993-9515-dfc51f685500-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0a6fdf8-881f-4993-9515-dfc51f685500-maas-api-service-ca\") pod \"d0a6fdf8-881f-4993-9515-dfc51f685500\" (UID: \"d0a6fdf8-881f-4993-9515-dfc51f685500\") "
Oct 31 20:00:23.665431 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:23.665407    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0a6fdf8-881f-4993-9515-dfc51f685500-kube-api-access-nfrch\" (UniqueName: \"kubernetes.io/projected/d0a6fdf8-881f-4993-9515-dfc51f685500-kube-api-access-nfrch\") pod \"d0a6fdf8-881f-4993-9515-dfc51f685500\" (UID: \"d0a6fdf8-881f-4993-9515-dfc51f685500\") "
Oct 31 20:00:23.665995 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:23.665971    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e618e1-29891280-c679x"
Oct 31 20:00:23.667544 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:23.665132    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0a6fdf8-881f-4993-9515-dfc51f685500-maas-api-service-ca" pod "d0a6fdf8-881f-4993-9515-dfc51f685500" (UID: "d0a6fdf8-881f-4993-9515-dfc51f685500"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:00:23.669516 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:23.669492    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0a6fdf8-881f-4993-9515-dfc51f685500-maas-api-service-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:23.674525 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:23.674421    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0a6fdf8-881f-4993-9515-dfc51f685500-kube-api-access-nfrch" pod "d0a6fdf8-881f-4993-9515-dfc51f685500" (UID: "d0a6fdf8-881f-4993-9515-dfc51f685500"). InnerVolumeSpecName "kube-api-access-nfrch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:00:23.770900 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:23.770863    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/674684f6-15f9-4e2e-a5d6-136098c534d3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/674684f6-15f9-4e2e-a5d6-136098c534d3-maas-api-service-ca\") pod \"674684f6-15f9-4e2e-a5d6-136098c534d3\" (UID: \"674684f6-15f9-4e2e-a5d6-136098c534d3\") "
Oct 31 20:00:23.771444 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:23.770940    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4360bbcf-669f-4d6e-933c-0dcdfa76071a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4360bbcf-669f-4d6e-933c-0dcdfa76071a-maas-api-service-ca\") pod \"4360bbcf-669f-4d6e-933c-0dcdfa76071a\" (UID: \"4360bbcf-669f-4d6e-933c-0dcdfa76071a\") "
Oct 31 20:00:23.771444 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:23.771239    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4360bbcf-669f-4d6e-933c-0dcdfa76071a-kube-api-access-xprgj\" (UniqueName: \"kubernetes.io/projected/4360bbcf-669f-4d6e-933c-0dcdfa76071a-kube-api-access-xprgj\") pod \"4360bbcf-669f-4d6e-933c-0dcdfa76071a\" (UID: \"4360bbcf-669f-4d6e-933c-0dcdfa76071a\") "
Oct 31 20:00:23.771444 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:23.771354    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/674684f6-15f9-4e2e-a5d6-136098c534d3-kube-api-access-wr7s7\" (UniqueName: \"kubernetes.io/projected/674684f6-15f9-4e2e-a5d6-136098c534d3-kube-api-access-wr7s7\") pod \"674684f6-15f9-4e2e-a5d6-136098c534d3\" (UID: \"674684f6-15f9-4e2e-a5d6-136098c534d3\") "
Oct 31 20:00:23.772032 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:23.772009    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfrch\" (UniqueName: \"kubernetes.io/projected/d0a6fdf8-881f-4993-9515-dfc51f685500-kube-api-access-nfrch\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:23.772587 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:23.772554    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4360bbcf-669f-4d6e-933c-0dcdfa76071a-maas-api-service-ca" pod "4360bbcf-669f-4d6e-933c-0dcdfa76071a" (UID: "4360bbcf-669f-4d6e-933c-0dcdfa76071a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:00:23.772676 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:23.772631    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/674684f6-15f9-4e2e-a5d6-136098c534d3-maas-api-service-ca" pod "674684f6-15f9-4e2e-a5d6-136098c534d3" (UID: "674684f6-15f9-4e2e-a5d6-136098c534d3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:00:23.774678 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:23.774651    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4360bbcf-669f-4d6e-933c-0dcdfa76071a-kube-api-access-xprgj" pod "4360bbcf-669f-4d6e-933c-0dcdfa76071a" (UID: "4360bbcf-669f-4d6e-933c-0dcdfa76071a"). InnerVolumeSpecName "kube-api-access-xprgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:00:23.776962 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:23.776934    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/674684f6-15f9-4e2e-a5d6-136098c534d3-kube-api-access-wr7s7" pod "674684f6-15f9-4e2e-a5d6-136098c534d3" (UID: "674684f6-15f9-4e2e-a5d6-136098c534d3"). InnerVolumeSpecName "kube-api-access-wr7s7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:00:23.873405 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:23.873367    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xprgj\" (UniqueName: \"kubernetes.io/projected/4360bbcf-669f-4d6e-933c-0dcdfa76071a-kube-api-access-xprgj\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:23.873405 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:23.873400    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wr7s7\" (UniqueName: \"kubernetes.io/projected/674684f6-15f9-4e2e-a5d6-136098c534d3-kube-api-access-wr7s7\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:23.873405 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:23.873410    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/674684f6-15f9-4e2e-a5d6-136098c534d3-maas-api-service-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:23.873609 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:23.873420    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4360bbcf-669f-4d6e-933c-0dcdfa76071a-maas-api-service-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:24.167647 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:24.167607    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-01afb5-29891280-4hgmt" event={"ID":"674684f6-15f9-4e2e-a5d6-136098c534d3","Type":"ContainerDied","Data":"a368b7b0c93357620aaac3f185de29c12b38e676d5896964693736619da081d6"}
Oct 31 20:00:24.167841 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:24.167695    4645 scope.go:122] "RemoveContainer" containerID="2bda60b8bba123eabb76a3fd481266f0e7dee3166c78b00ed5ece76919047f0c"
Oct 31 20:00:24.168053 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:24.168034    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-01afb5-29891280-4hgmt"
Oct 31 20:00:24.263392 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:24.261838    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e618e1-29891280-c679x" event={"ID":"4360bbcf-669f-4d6e-933c-0dcdfa76071a","Type":"ContainerDied","Data":"f52521742dd1ae8933779cdcd1a89d1f9016ed72d833c74dcbeaba72281d3571"}
Oct 31 20:00:24.263392 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:24.262023    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e618e1-29891280-c679x"
Oct 31 20:00:24.266149 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:24.266114    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b083a-29891280-nq9lm"
Oct 31 20:00:24.266284 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:24.266169    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b083a-29891280-nq9lm" event={"ID":"d0a6fdf8-881f-4993-9515-dfc51f685500","Type":"ContainerDied","Data":"0d92c04f086bcaa3d8d53d0b5511f1b22d16ec278ba173a4ffb50da1bd92b6c3"}
Oct 31 20:00:24.392351 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:24.391901    4645 scope.go:122] "RemoveContainer" containerID="2eb1f35ef38eb45ffcbdc41bdcf643bc5f51e2b3115c76aa065162595570da9a"
Oct 31 20:00:24.434401 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:24.430846    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-01afb5-29891280-4hgmt"]
Oct 31 20:00:24.441540 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:24.441424    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-01afb5-29891280-4hgmt"]
Oct 31 20:00:24.451964 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:24.451930    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e618e1-29891280-c679x"]
Oct 31 20:00:24.459811 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:24.459787    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e618e1-29891280-c679x"]
Oct 31 20:00:24.465183 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:24.465158    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b083a-29891280-nq9lm"]
Oct 31 20:00:24.470521 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:24.470500    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8b083a-29891280-nq9lm"]
Oct 31 20:00:24.470617 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:24.470504    4645 scope.go:122] "RemoveContainer" containerID="e3a70fe5e23f7fda764a18b428272dfc7b6938e45e0150854bd025d3dfc94366"
Oct 31 20:00:24.979011 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:24.978973    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:25.622172 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:25.622138    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4360bbcf-669f-4d6e-933c-0dcdfa76071a" path="/var/lib/kubelet/pods/4360bbcf-669f-4d6e-933c-0dcdfa76071a/volumes"
Oct 31 20:00:25.622726 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:25.622709    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="674684f6-15f9-4e2e-a5d6-136098c534d3" path="/var/lib/kubelet/pods/674684f6-15f9-4e2e-a5d6-136098c534d3/volumes"
Oct 31 20:00:25.623150 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:25.623136    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0a6fdf8-881f-4993-9515-dfc51f685500" path="/var/lib/kubelet/pods/d0a6fdf8-881f-4993-9515-dfc51f685500/volumes"
Oct 31 20:00:25.946146 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:25.946112    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:26.761742 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:26.761704    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:27.450143 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:27.450106    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-4bac5a/rate-test-model-4bac5a-kserve-564dc4dfdf-vv676"]
Oct 31 20:00:27.500314 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:27.500284    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:27.764021 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:27.763935    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:27.770264 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:27.770233    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-e618e1/sub-test-model-e618e1-kserve-584c8d9c65-42m96" podUID="4f876b87-1980-4834-89d7-47b4db24a2c1" containerName="main" containerID="cri-o://43a30004ad65415df37e1ec6f8998895fabb74123a46f998c67fb358c9687ffc" gracePeriod=45
Oct 31 20:00:28.066527 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:28.066502    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-e618e1/sub-test-model-e618e1-kserve-584c8d9c65-42m96"
Oct 31 20:00:28.215568 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:28.215523    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f876b87-1980-4834-89d7-47b4db24a2c1-kube-api-access-dkvlq\" (UniqueName: \"kubernetes.io/projected/4f876b87-1980-4834-89d7-47b4db24a2c1-kube-api-access-dkvlq\") pod \"4f876b87-1980-4834-89d7-47b4db24a2c1\" (UID: \"4f876b87-1980-4834-89d7-47b4db24a2c1\") "
Oct 31 20:00:28.215732 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:28.215700    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f876b87-1980-4834-89d7-47b4db24a2c1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f876b87-1980-4834-89d7-47b4db24a2c1-tmp-dir\") pod \"4f876b87-1980-4834-89d7-47b4db24a2c1\" (UID: \"4f876b87-1980-4834-89d7-47b4db24a2c1\") "
Oct 31 20:00:28.215777 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:28.215747    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f876b87-1980-4834-89d7-47b4db24a2c1-home\" (UniqueName: \"kubernetes.io/empty-dir/4f876b87-1980-4834-89d7-47b4db24a2c1-home\") pod \"4f876b87-1980-4834-89d7-47b4db24a2c1\" (UID: \"4f876b87-1980-4834-89d7-47b4db24a2c1\") "
Oct 31 20:00:28.215896 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:28.215872    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f876b87-1980-4834-89d7-47b4db24a2c1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f876b87-1980-4834-89d7-47b4db24a2c1-dshm\") pod \"4f876b87-1980-4834-89d7-47b4db24a2c1\" (UID: \"4f876b87-1980-4834-89d7-47b4db24a2c1\") "
Oct 31 20:00:28.216049 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:28.215953    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f876b87-1980-4834-89d7-47b4db24a2c1-tls-certs\" (UniqueName: \"kubernetes.io/secret/4f876b87-1980-4834-89d7-47b4db24a2c1-tls-certs\") pod \"4f876b87-1980-4834-89d7-47b4db24a2c1\" (UID: \"4f876b87-1980-4834-89d7-47b4db24a2c1\") "
Oct 31 20:00:28.216049 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:28.215973    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f876b87-1980-4834-89d7-47b4db24a2c1-tmp-dir" pod "4f876b87-1980-4834-89d7-47b4db24a2c1" (UID: "4f876b87-1980-4834-89d7-47b4db24a2c1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:00:28.216049 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:28.216018    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f876b87-1980-4834-89d7-47b4db24a2c1-home" pod "4f876b87-1980-4834-89d7-47b4db24a2c1" (UID: "4f876b87-1980-4834-89d7-47b4db24a2c1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:00:28.216247 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:28.216105    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f876b87-1980-4834-89d7-47b4db24a2c1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f876b87-1980-4834-89d7-47b4db24a2c1-model-cache\") pod \"4f876b87-1980-4834-89d7-47b4db24a2c1\" (UID: \"4f876b87-1980-4834-89d7-47b4db24a2c1\") "
Oct 31 20:00:28.216438 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:28.216411    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f876b87-1980-4834-89d7-47b4db24a2c1-model-cache" pod "4f876b87-1980-4834-89d7-47b4db24a2c1" (UID: "4f876b87-1980-4834-89d7-47b4db24a2c1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:00:28.216780 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:28.216762    4645 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f876b87-1980-4834-89d7-47b4db24a2c1-tmp-dir\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:28.216780 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:28.216780    4645 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f876b87-1980-4834-89d7-47b4db24a2c1-home\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:28.216922 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:28.216789    4645 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f876b87-1980-4834-89d7-47b4db24a2c1-model-cache\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:28.217809 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:28.217786    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f876b87-1980-4834-89d7-47b4db24a2c1-kube-api-access-dkvlq" pod "4f876b87-1980-4834-89d7-47b4db24a2c1" (UID: "4f876b87-1980-4834-89d7-47b4db24a2c1"). InnerVolumeSpecName "kube-api-access-dkvlq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:00:28.217926 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:28.217904    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f876b87-1980-4834-89d7-47b4db24a2c1-tls-certs" pod "4f876b87-1980-4834-89d7-47b4db24a2c1" (UID: "4f876b87-1980-4834-89d7-47b4db24a2c1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:00:28.218006 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:28.217980    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f876b87-1980-4834-89d7-47b4db24a2c1-dshm" pod "4f876b87-1980-4834-89d7-47b4db24a2c1" (UID: "4f876b87-1980-4834-89d7-47b4db24a2c1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:00:28.309712 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:28.309654    4645 generic.go:356] "Generic (PLEG): container finished" podID="4f876b87-1980-4834-89d7-47b4db24a2c1" containerID="43a30004ad65415df37e1ec6f8998895fabb74123a46f998c67fb358c9687ffc" exitCode=0
Oct 31 20:00:28.309712 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:28.309700    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-e618e1/sub-test-model-e618e1-kserve-584c8d9c65-42m96"
Oct 31 20:00:28.309841 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:28.309735    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-e618e1/sub-test-model-e618e1-kserve-584c8d9c65-42m96" event={"ID":"4f876b87-1980-4834-89d7-47b4db24a2c1","Type":"ContainerDied","Data":"43a30004ad65415df37e1ec6f8998895fabb74123a46f998c67fb358c9687ffc"}
Oct 31 20:00:28.309841 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:28.309776    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-e618e1/sub-test-model-e618e1-kserve-584c8d9c65-42m96" event={"ID":"4f876b87-1980-4834-89d7-47b4db24a2c1","Type":"ContainerDied","Data":"ccc4bda146ca580969e2a7841362360c68c022df65d86b8fc839da86cbf6f28c"}
Oct 31 20:00:28.309841 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:28.309794    4645 scope.go:122] "RemoveContainer" containerID="43a30004ad65415df37e1ec6f8998895fabb74123a46f998c67fb358c9687ffc"
Oct 31 20:00:28.317903 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:28.317875    4645 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f876b87-1980-4834-89d7-47b4db24a2c1-dshm\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:28.317995 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:28.317905    4645 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f876b87-1980-4834-89d7-47b4db24a2c1-tls-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:28.317995 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:28.317919    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkvlq\" (UniqueName: \"kubernetes.io/projected/4f876b87-1980-4834-89d7-47b4db24a2c1-kube-api-access-dkvlq\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:28.321975 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:28.321956    4645 scope.go:122] "RemoveContainer" containerID="43a30004ad65415df37e1ec6f8998895fabb74123a46f998c67fb358c9687ffc"
Oct 31 20:00:28.322279 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:00:28.322258    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43a30004ad65415df37e1ec6f8998895fabb74123a46f998c67fb358c9687ffc\": container with ID starting with 43a30004ad65415df37e1ec6f8998895fabb74123a46f998c67fb358c9687ffc not found: ID does not exist" containerID="43a30004ad65415df37e1ec6f8998895fabb74123a46f998c67fb358c9687ffc"
Oct 31 20:00:28.322362 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:28.322296    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43a30004ad65415df37e1ec6f8998895fabb74123a46f998c67fb358c9687ffc"} err="failed to get container status \"43a30004ad65415df37e1ec6f8998895fabb74123a46f998c67fb358c9687ffc\": rpc error: code = NotFound desc = could not find container \"43a30004ad65415df37e1ec6f8998895fabb74123a46f998c67fb358c9687ffc\": container with ID starting with 43a30004ad65415df37e1ec6f8998895fabb74123a46f998c67fb358c9687ffc not found: ID does not exist"
Oct 31 20:00:28.337340 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:28.337308    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-e618e1/sub-test-model-e618e1-kserve-584c8d9c65-42m96"]
Oct 31 20:00:28.341631 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:28.341610    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-e618e1/sub-test-model-e618e1-kserve-584c8d9c65-42m96"]
Oct 31 20:00:28.702683 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:28.702652    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:28.755356 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:28.754446    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-d9c048/sub-test-model-d9c048-kserve-6fff8b86d5-2mb99" podUID="2665b4f7-db95-40bf-8027-63b27e28afa3" containerName="main" containerID="cri-o://05237464e141ec8e599962cfdb22f91f992cc45c86421c1f0fae5b386a60debc" gracePeriod=45
Oct 31 20:00:29.077302 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:29.077276    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-d9c048/sub-test-model-d9c048-kserve-6fff8b86d5-2mb99"
Oct 31 20:00:29.230704 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:29.230628    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2665b4f7-db95-40bf-8027-63b27e28afa3-kube-api-access-6mpn4\" (UniqueName: \"kubernetes.io/projected/2665b4f7-db95-40bf-8027-63b27e28afa3-kube-api-access-6mpn4\") pod \"2665b4f7-db95-40bf-8027-63b27e28afa3\" (UID: \"2665b4f7-db95-40bf-8027-63b27e28afa3\") "
Oct 31 20:00:29.230704 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:29.230671    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2665b4f7-db95-40bf-8027-63b27e28afa3-tls-certs\" (UniqueName: \"kubernetes.io/secret/2665b4f7-db95-40bf-8027-63b27e28afa3-tls-certs\") pod \"2665b4f7-db95-40bf-8027-63b27e28afa3\" (UID: \"2665b4f7-db95-40bf-8027-63b27e28afa3\") "
Oct 31 20:00:29.230889 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:29.230761    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2665b4f7-db95-40bf-8027-63b27e28afa3-home\" (UniqueName: \"kubernetes.io/empty-dir/2665b4f7-db95-40bf-8027-63b27e28afa3-home\") pod \"2665b4f7-db95-40bf-8027-63b27e28afa3\" (UID: \"2665b4f7-db95-40bf-8027-63b27e28afa3\") "
Oct 31 20:00:29.230937 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:29.230898    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2665b4f7-db95-40bf-8027-63b27e28afa3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2665b4f7-db95-40bf-8027-63b27e28afa3-model-cache\") pod \"2665b4f7-db95-40bf-8027-63b27e28afa3\" (UID: \"2665b4f7-db95-40bf-8027-63b27e28afa3\") "
Oct 31 20:00:29.230991 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:29.230937    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2665b4f7-db95-40bf-8027-63b27e28afa3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2665b4f7-db95-40bf-8027-63b27e28afa3-dshm\") pod \"2665b4f7-db95-40bf-8027-63b27e28afa3\" (UID: \"2665b4f7-db95-40bf-8027-63b27e28afa3\") "
Oct 31 20:00:29.231057 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:29.231004    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2665b4f7-db95-40bf-8027-63b27e28afa3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2665b4f7-db95-40bf-8027-63b27e28afa3-tmp-dir\") pod \"2665b4f7-db95-40bf-8027-63b27e28afa3\" (UID: \"2665b4f7-db95-40bf-8027-63b27e28afa3\") "
Oct 31 20:00:29.231057 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:29.231020    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2665b4f7-db95-40bf-8027-63b27e28afa3-home" pod "2665b4f7-db95-40bf-8027-63b27e28afa3" (UID: "2665b4f7-db95-40bf-8027-63b27e28afa3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:00:29.231419 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:29.231188    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2665b4f7-db95-40bf-8027-63b27e28afa3-model-cache" pod "2665b4f7-db95-40bf-8027-63b27e28afa3" (UID: "2665b4f7-db95-40bf-8027-63b27e28afa3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:00:29.231419 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:29.231386    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2665b4f7-db95-40bf-8027-63b27e28afa3-tmp-dir" pod "2665b4f7-db95-40bf-8027-63b27e28afa3" (UID: "2665b4f7-db95-40bf-8027-63b27e28afa3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:00:29.231736 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:29.231712    4645 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2665b4f7-db95-40bf-8027-63b27e28afa3-model-cache\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:29.231820 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:29.231744    4645 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2665b4f7-db95-40bf-8027-63b27e28afa3-tmp-dir\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:29.231820 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:29.231760    4645 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2665b4f7-db95-40bf-8027-63b27e28afa3-home\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:29.233022 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:29.232998    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2665b4f7-db95-40bf-8027-63b27e28afa3-kube-api-access-6mpn4" pod "2665b4f7-db95-40bf-8027-63b27e28afa3" (UID: "2665b4f7-db95-40bf-8027-63b27e28afa3"). InnerVolumeSpecName "kube-api-access-6mpn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:00:29.233158 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:29.233135    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2665b4f7-db95-40bf-8027-63b27e28afa3-dshm" pod "2665b4f7-db95-40bf-8027-63b27e28afa3" (UID: "2665b4f7-db95-40bf-8027-63b27e28afa3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:00:29.233243 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:29.233195    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2665b4f7-db95-40bf-8027-63b27e28afa3-tls-certs" pod "2665b4f7-db95-40bf-8027-63b27e28afa3" (UID: "2665b4f7-db95-40bf-8027-63b27e28afa3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:00:29.320407 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:29.320380    4645 generic.go:356] "Generic (PLEG): container finished" podID="2665b4f7-db95-40bf-8027-63b27e28afa3" containerID="05237464e141ec8e599962cfdb22f91f992cc45c86421c1f0fae5b386a60debc" exitCode=0
Oct 31 20:00:29.320519 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:29.320429    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-d9c048/sub-test-model-d9c048-kserve-6fff8b86d5-2mb99"
Oct 31 20:00:29.320519 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:29.320469    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-d9c048/sub-test-model-d9c048-kserve-6fff8b86d5-2mb99" event={"ID":"2665b4f7-db95-40bf-8027-63b27e28afa3","Type":"ContainerDied","Data":"05237464e141ec8e599962cfdb22f91f992cc45c86421c1f0fae5b386a60debc"}
Oct 31 20:00:29.320519 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:29.320509    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-d9c048/sub-test-model-d9c048-kserve-6fff8b86d5-2mb99" event={"ID":"2665b4f7-db95-40bf-8027-63b27e28afa3","Type":"ContainerDied","Data":"cffc36185c0f78c5fe6fda6bb360d1d7e09f846e4ee567e947a64b332b7a6cd7"}
Oct 31 20:00:29.320670 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:29.320527    4645 scope.go:122] "RemoveContainer" containerID="05237464e141ec8e599962cfdb22f91f992cc45c86421c1f0fae5b386a60debc"
Oct 31 20:00:29.332119 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:29.332101    4645 scope.go:122] "RemoveContainer" containerID="05237464e141ec8e599962cfdb22f91f992cc45c86421c1f0fae5b386a60debc"
Oct 31 20:00:29.332418 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:00:29.332396    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05237464e141ec8e599962cfdb22f91f992cc45c86421c1f0fae5b386a60debc\": container with ID starting with 05237464e141ec8e599962cfdb22f91f992cc45c86421c1f0fae5b386a60debc not found: ID does not exist" containerID="05237464e141ec8e599962cfdb22f91f992cc45c86421c1f0fae5b386a60debc"
Oct 31 20:00:29.332487 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:29.332426    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05237464e141ec8e599962cfdb22f91f992cc45c86421c1f0fae5b386a60debc"} err="failed to get container status \"05237464e141ec8e599962cfdb22f91f992cc45c86421c1f0fae5b386a60debc\": rpc error: code = NotFound desc = could not find container \"05237464e141ec8e599962cfdb22f91f992cc45c86421c1f0fae5b386a60debc\": container with ID starting with 05237464e141ec8e599962cfdb22f91f992cc45c86421c1f0fae5b386a60debc not found: ID does not exist"
Oct 31 20:00:29.332836 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:29.332817    4645 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2665b4f7-db95-40bf-8027-63b27e28afa3-dshm\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:29.332877 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:29.332842    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mpn4\" (UniqueName: \"kubernetes.io/projected/2665b4f7-db95-40bf-8027-63b27e28afa3-kube-api-access-6mpn4\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:29.332877 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:29.332855    4645 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2665b4f7-db95-40bf-8027-63b27e28afa3-tls-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:29.345107 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:29.345082    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-d9c048/sub-test-model-d9c048-kserve-6fff8b86d5-2mb99"]
Oct 31 20:00:29.349699 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:29.349678    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-d9c048/sub-test-model-d9c048-kserve-6fff8b86d5-2mb99"]
Oct 31 20:00:29.533059 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:29.533000    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:29.625624 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:29.625590    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2665b4f7-db95-40bf-8027-63b27e28afa3" path="/var/lib/kubelet/pods/2665b4f7-db95-40bf-8027-63b27e28afa3/volumes"
Oct 31 20:00:29.626092 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:29.626067    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f876b87-1980-4834-89d7-47b4db24a2c1" path="/var/lib/kubelet/pods/4f876b87-1980-4834-89d7-47b4db24a2c1/volumes"
Oct 31 20:00:30.056891 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:30.056855    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-d9c048/sub-test-model-d9c048-kserve-6fff8b86d5-2mb99" podUID="2665b4f7-db95-40bf-8027-63b27e28afa3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.219:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 20:00:30.357404 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:30.357317    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:31.032473 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.032437    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:31.691143 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.691115    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w6-e1d060/rate-test-model-e1d060-kserve-5969df469-4mfjn"]
Oct 31 20:00:31.691800 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.691786    4645 cpu_manager.go:414] "containerMap: removing container" podUID="4f876b87-1980-4834-89d7-47b4db24a2c1" containerName="main"
Oct 31 20:00:31.691846 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.691801    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f876b87-1980-4834-89d7-47b4db24a2c1" containerName="main"
Oct 31 20:00:31.691846 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.691818    4645 cpu_manager.go:414] "containerMap: removing container" podUID="4f88b769-7931-40c2-8cf6-ef92c2a85327" containerName="cleanup"
Oct 31 20:00:31.691846 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.691823    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f88b769-7931-40c2-8cf6-ef92c2a85327" containerName="cleanup"
Oct 31 20:00:31.691846 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.691833    4645 cpu_manager.go:414] "containerMap: removing container" podUID="4f88b769-7931-40c2-8cf6-ef92c2a85327" containerName="cleanup"
Oct 31 20:00:31.691846 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.691838    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f88b769-7931-40c2-8cf6-ef92c2a85327" containerName="cleanup"
Oct 31 20:00:31.691846 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.691845    4645 cpu_manager.go:414] "containerMap: removing container" podUID="d0a6fdf8-881f-4993-9515-dfc51f685500" containerName="cleanup"
Oct 31 20:00:31.692035 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.691850    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0a6fdf8-881f-4993-9515-dfc51f685500" containerName="cleanup"
Oct 31 20:00:31.692035 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.691856    4645 cpu_manager.go:414] "containerMap: removing container" podUID="674684f6-15f9-4e2e-a5d6-136098c534d3" containerName="cleanup"
Oct 31 20:00:31.692035 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.691860    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="674684f6-15f9-4e2e-a5d6-136098c534d3" containerName="cleanup"
Oct 31 20:00:31.692035 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.691870    4645 cpu_manager.go:414] "containerMap: removing container" podUID="f1c79dcb-63c1-4844-9527-5ef43f45eb95" containerName="cleanup"
Oct 31 20:00:31.692035 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.691875    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1c79dcb-63c1-4844-9527-5ef43f45eb95" containerName="cleanup"
Oct 31 20:00:31.692035 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.691884    4645 cpu_manager.go:414] "containerMap: removing container" podUID="4360bbcf-669f-4d6e-933c-0dcdfa76071a" containerName="cleanup"
Oct 31 20:00:31.692035 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.691888    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4360bbcf-669f-4d6e-933c-0dcdfa76071a" containerName="cleanup"
Oct 31 20:00:31.692035 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.691895    4645 cpu_manager.go:414] "containerMap: removing container" podUID="d0a6fdf8-881f-4993-9515-dfc51f685500" containerName="cleanup"
Oct 31 20:00:31.692035 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.691900    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0a6fdf8-881f-4993-9515-dfc51f685500" containerName="cleanup"
Oct 31 20:00:31.692035 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.691908    4645 cpu_manager.go:414] "containerMap: removing container" podUID="4360bbcf-669f-4d6e-933c-0dcdfa76071a" containerName="cleanup"
Oct 31 20:00:31.692035 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.691912    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4360bbcf-669f-4d6e-933c-0dcdfa76071a" containerName="cleanup"
Oct 31 20:00:31.692035 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.691918    4645 cpu_manager.go:414] "containerMap: removing container" podUID="4f88b769-7931-40c2-8cf6-ef92c2a85327" containerName="cleanup"
Oct 31 20:00:31.692035 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.691922    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f88b769-7931-40c2-8cf6-ef92c2a85327" containerName="cleanup"
Oct 31 20:00:31.692035 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.691932    4645 cpu_manager.go:414] "containerMap: removing container" podUID="2665b4f7-db95-40bf-8027-63b27e28afa3" containerName="main"
Oct 31 20:00:31.692035 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.691937    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2665b4f7-db95-40bf-8027-63b27e28afa3" containerName="main"
Oct 31 20:00:31.692035 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.691944    4645 cpu_manager.go:414] "containerMap: removing container" podUID="48ab68ad-2abf-44c8-98dc-c3ec72d36362" containerName="istio-proxy"
Oct 31 20:00:31.692035 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.691948    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48ab68ad-2abf-44c8-98dc-c3ec72d36362" containerName="istio-proxy"
Oct 31 20:00:31.692035 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.691955    4645 cpu_manager.go:414] "containerMap: removing container" podUID="3b3e85d8-1b99-40b3-87ed-7e0d9c78bb86" containerName="cleanup"
Oct 31 20:00:31.692035 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.691959    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b3e85d8-1b99-40b3-87ed-7e0d9c78bb86" containerName="cleanup"
Oct 31 20:00:31.692035 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.691968    4645 cpu_manager.go:414] "containerMap: removing container" podUID="2a9b3262-5fdf-4375-8411-a0640050c34e" containerName="cleanup"
Oct 31 20:00:31.692035 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.691972    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a9b3262-5fdf-4375-8411-a0640050c34e" containerName="cleanup"
Oct 31 20:00:31.692035 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.691980    4645 cpu_manager.go:414] "containerMap: removing container" podUID="0b226e15-ab1f-4f6c-9b08-1ce863756e43" containerName="cleanup"
Oct 31 20:00:31.692035 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.691984    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b226e15-ab1f-4f6c-9b08-1ce863756e43" containerName="cleanup"
Oct 31 20:00:31.692035 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.691991    4645 cpu_manager.go:414] "containerMap: removing container" podUID="3b3e85d8-1b99-40b3-87ed-7e0d9c78bb86" containerName="cleanup"
Oct 31 20:00:31.692035 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.691995    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b3e85d8-1b99-40b3-87ed-7e0d9c78bb86" containerName="cleanup"
Oct 31 20:00:31.692035 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.692010    4645 cpu_manager.go:414] "containerMap: removing container" podUID="3b3e85d8-1b99-40b3-87ed-7e0d9c78bb86" containerName="cleanup"
Oct 31 20:00:31.692035 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.692015    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b3e85d8-1b99-40b3-87ed-7e0d9c78bb86" containerName="cleanup"
Oct 31 20:00:31.692035 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.692028    4645 cpu_manager.go:414] "containerMap: removing container" podUID="1c8e3489-4fca-4756-a7a0-0c494eab1ba5" containerName="cleanup"
Oct 31 20:00:31.692035 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.692032    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c8e3489-4fca-4756-a7a0-0c494eab1ba5" containerName="cleanup"
Oct 31 20:00:31.692035 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.692043    4645 cpu_manager.go:414] "containerMap: removing container" podUID="46fe6436-5347-408b-b173-aa153f96236e" containerName="cleanup"
Oct 31 20:00:31.692035 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.692047    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46fe6436-5347-408b-b173-aa153f96236e" containerName="cleanup"
Oct 31 20:00:31.694004 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.692057    4645 cpu_manager.go:414] "containerMap: removing container" podUID="4360bbcf-669f-4d6e-933c-0dcdfa76071a" containerName="cleanup"
Oct 31 20:00:31.694004 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.692062    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4360bbcf-669f-4d6e-933c-0dcdfa76071a" containerName="cleanup"
Oct 31 20:00:31.694004 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.692077    4645 cpu_manager.go:414] "containerMap: removing container" podUID="0b226e15-ab1f-4f6c-9b08-1ce863756e43" containerName="cleanup"
Oct 31 20:00:31.694004 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.692082    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b226e15-ab1f-4f6c-9b08-1ce863756e43" containerName="cleanup"
Oct 31 20:00:31.694004 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.692092    4645 cpu_manager.go:414] "containerMap: removing container" podUID="674684f6-15f9-4e2e-a5d6-136098c534d3" containerName="cleanup"
Oct 31 20:00:31.694004 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.692099    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="674684f6-15f9-4e2e-a5d6-136098c534d3" containerName="cleanup"
Oct 31 20:00:31.694004 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.692212    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b3e85d8-1b99-40b3-87ed-7e0d9c78bb86" containerName="cleanup"
Oct 31 20:00:31.694004 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.692223    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b226e15-ab1f-4f6c-9b08-1ce863756e43" containerName="cleanup"
Oct 31 20:00:31.694004 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.692229    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a9b3262-5fdf-4375-8411-a0640050c34e" containerName="cleanup"
Oct 31 20:00:31.694004 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.692235    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b3e85d8-1b99-40b3-87ed-7e0d9c78bb86" containerName="cleanup"
Oct 31 20:00:31.694004 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.692241    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1c79dcb-63c1-4844-9527-5ef43f45eb95" containerName="cleanup"
Oct 31 20:00:31.694004 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.692248    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0a6fdf8-881f-4993-9515-dfc51f685500" containerName="cleanup"
Oct 31 20:00:31.694004 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.692253    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b226e15-ab1f-4f6c-9b08-1ce863756e43" containerName="cleanup"
Oct 31 20:00:31.694004 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.692259    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b3e85d8-1b99-40b3-87ed-7e0d9c78bb86" containerName="cleanup"
Oct 31 20:00:31.694004 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.692267    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="48ab68ad-2abf-44c8-98dc-c3ec72d36362" containerName="istio-proxy"
Oct 31 20:00:31.694004 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.692272    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="4360bbcf-669f-4d6e-933c-0dcdfa76071a" containerName="cleanup"
Oct 31 20:00:31.694004 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.692278    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="674684f6-15f9-4e2e-a5d6-136098c534d3" containerName="cleanup"
Oct 31 20:00:31.694004 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.692284    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="674684f6-15f9-4e2e-a5d6-136098c534d3" containerName="cleanup"
Oct 31 20:00:31.694004 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.692290    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f876b87-1980-4834-89d7-47b4db24a2c1" containerName="main"
Oct 31 20:00:31.694004 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.692297    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f88b769-7931-40c2-8cf6-ef92c2a85327" containerName="cleanup"
Oct 31 20:00:31.694004 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.692302    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0a6fdf8-881f-4993-9515-dfc51f685500" containerName="cleanup"
Oct 31 20:00:31.694004 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.692308    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f88b769-7931-40c2-8cf6-ef92c2a85327" containerName="cleanup"
Oct 31 20:00:31.694004 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.692313    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="674684f6-15f9-4e2e-a5d6-136098c534d3" containerName="cleanup"
Oct 31 20:00:31.694004 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.692320    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="4360bbcf-669f-4d6e-933c-0dcdfa76071a" containerName="cleanup"
Oct 31 20:00:31.694004 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.692340    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0a6fdf8-881f-4993-9515-dfc51f685500" containerName="cleanup"
Oct 31 20:00:31.694004 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.692350    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f88b769-7931-40c2-8cf6-ef92c2a85327" containerName="cleanup"
Oct 31 20:00:31.694004 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.692357    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="2665b4f7-db95-40bf-8027-63b27e28afa3" containerName="main"
Oct 31 20:00:31.694004 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.692366    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="48ab68ad-2abf-44c8-98dc-c3ec72d36362" containerName="istio-proxy"
Oct 31 20:00:31.694004 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.692373    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c8e3489-4fca-4756-a7a0-0c494eab1ba5" containerName="cleanup"
Oct 31 20:00:31.694004 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.692380    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="46fe6436-5347-408b-b173-aa153f96236e" containerName="cleanup"
Oct 31 20:00:31.698252 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.698233    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-e1d060/rate-test-model-e1d060-kserve-5969df469-4mfjn"
Oct 31 20:00:31.715370 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.711011    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w6-e1d060\"/\"kube-root-ca.crt\""
Oct 31 20:00:31.715370 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.711130    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w6-e1d060\"/\"default-dockercfg-ckzqx\""
Oct 31 20:00:31.715370 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.712396    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w6-e1d060\"/\"openshift-service-ca.crt\""
Oct 31 20:00:31.715370 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.712479    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w6-e1d060\"/\"rate-test-model-e1d060-kserve-self-signed-certs\""
Oct 31 20:00:31.715370 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.714309    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w6-e1d060/rate-test-model-e1d060-kserve-5969df469-4mfjn"]
Oct 31 20:00:31.723095 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.723052    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:31.767101 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.767073    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b0c4b8a-bf5b-4af7-b272-aeefb1149b03-tls-certs\") pod \"rate-test-model-e1d060-kserve-5969df469-4mfjn\" (UID: \"8b0c4b8a-bf5b-4af7-b272-aeefb1149b03\") " pod="ai-tenant-e2e-shared-b-w6-e1d060/rate-test-model-e1d060-kserve-5969df469-4mfjn"
Oct 31 20:00:31.767376 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.767283    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b0c4b8a-bf5b-4af7-b272-aeefb1149b03-home\") pod \"rate-test-model-e1d060-kserve-5969df469-4mfjn\" (UID: \"8b0c4b8a-bf5b-4af7-b272-aeefb1149b03\") " pod="ai-tenant-e2e-shared-b-w6-e1d060/rate-test-model-e1d060-kserve-5969df469-4mfjn"
Oct 31 20:00:31.767647 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.767618    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b0c4b8a-bf5b-4af7-b272-aeefb1149b03-model-cache\") pod \"rate-test-model-e1d060-kserve-5969df469-4mfjn\" (UID: \"8b0c4b8a-bf5b-4af7-b272-aeefb1149b03\") " pod="ai-tenant-e2e-shared-b-w6-e1d060/rate-test-model-e1d060-kserve-5969df469-4mfjn"
Oct 31 20:00:31.767747 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.767724    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b0c4b8a-bf5b-4af7-b272-aeefb1149b03-tmp-dir\") pod \"rate-test-model-e1d060-kserve-5969df469-4mfjn\" (UID: \"8b0c4b8a-bf5b-4af7-b272-aeefb1149b03\") " pod="ai-tenant-e2e-shared-b-w6-e1d060/rate-test-model-e1d060-kserve-5969df469-4mfjn"
Oct 31 20:00:31.767814 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.767763    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b0c4b8a-bf5b-4af7-b272-aeefb1149b03-dshm\") pod \"rate-test-model-e1d060-kserve-5969df469-4mfjn\" (UID: \"8b0c4b8a-bf5b-4af7-b272-aeefb1149b03\") " pod="ai-tenant-e2e-shared-b-w6-e1d060/rate-test-model-e1d060-kserve-5969df469-4mfjn"
Oct 31 20:00:31.767871 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.767848    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qw8v\" (UniqueName: \"kubernetes.io/projected/8b0c4b8a-bf5b-4af7-b272-aeefb1149b03-kube-api-access-9qw8v\") pod \"rate-test-model-e1d060-kserve-5969df469-4mfjn\" (UID: \"8b0c4b8a-bf5b-4af7-b272-aeefb1149b03\") " pod="ai-tenant-e2e-shared-b-w6-e1d060/rate-test-model-e1d060-kserve-5969df469-4mfjn"
Oct 31 20:00:31.844262 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.844236    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:31.869593 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.869560    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b0c4b8a-bf5b-4af7-b272-aeefb1149b03-model-cache\") pod \"rate-test-model-e1d060-kserve-5969df469-4mfjn\" (UID: \"8b0c4b8a-bf5b-4af7-b272-aeefb1149b03\") " pod="ai-tenant-e2e-shared-b-w6-e1d060/rate-test-model-e1d060-kserve-5969df469-4mfjn"
Oct 31 20:00:31.869691 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.869602    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b0c4b8a-bf5b-4af7-b272-aeefb1149b03-tmp-dir\") pod \"rate-test-model-e1d060-kserve-5969df469-4mfjn\" (UID: \"8b0c4b8a-bf5b-4af7-b272-aeefb1149b03\") " pod="ai-tenant-e2e-shared-b-w6-e1d060/rate-test-model-e1d060-kserve-5969df469-4mfjn"
Oct 31 20:00:31.869691 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.869619    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b0c4b8a-bf5b-4af7-b272-aeefb1149b03-dshm\") pod \"rate-test-model-e1d060-kserve-5969df469-4mfjn\" (UID: \"8b0c4b8a-bf5b-4af7-b272-aeefb1149b03\") " pod="ai-tenant-e2e-shared-b-w6-e1d060/rate-test-model-e1d060-kserve-5969df469-4mfjn"
Oct 31 20:00:31.869691 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.869642    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qw8v\" (UniqueName: \"kubernetes.io/projected/8b0c4b8a-bf5b-4af7-b272-aeefb1149b03-kube-api-access-9qw8v\") pod \"rate-test-model-e1d060-kserve-5969df469-4mfjn\" (UID: \"8b0c4b8a-bf5b-4af7-b272-aeefb1149b03\") " pod="ai-tenant-e2e-shared-b-w6-e1d060/rate-test-model-e1d060-kserve-5969df469-4mfjn"
Oct 31 20:00:31.869884 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.869711    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b0c4b8a-bf5b-4af7-b272-aeefb1149b03-tls-certs\") pod \"rate-test-model-e1d060-kserve-5969df469-4mfjn\" (UID: \"8b0c4b8a-bf5b-4af7-b272-aeefb1149b03\") " pod="ai-tenant-e2e-shared-b-w6-e1d060/rate-test-model-e1d060-kserve-5969df469-4mfjn"
Oct 31 20:00:31.869884 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.869791    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b0c4b8a-bf5b-4af7-b272-aeefb1149b03-home\") pod \"rate-test-model-e1d060-kserve-5969df469-4mfjn\" (UID: \"8b0c4b8a-bf5b-4af7-b272-aeefb1149b03\") " pod="ai-tenant-e2e-shared-b-w6-e1d060/rate-test-model-e1d060-kserve-5969df469-4mfjn"
Oct 31 20:00:31.870021 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.870000    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b0c4b8a-bf5b-4af7-b272-aeefb1149b03-model-cache\") pod \"rate-test-model-e1d060-kserve-5969df469-4mfjn\" (UID: \"8b0c4b8a-bf5b-4af7-b272-aeefb1149b03\") " pod="ai-tenant-e2e-shared-b-w6-e1d060/rate-test-model-e1d060-kserve-5969df469-4mfjn"
Oct 31 20:00:31.870073 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.870043    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b0c4b8a-bf5b-4af7-b272-aeefb1149b03-tmp-dir\") pod \"rate-test-model-e1d060-kserve-5969df469-4mfjn\" (UID: \"8b0c4b8a-bf5b-4af7-b272-aeefb1149b03\") " pod="ai-tenant-e2e-shared-b-w6-e1d060/rate-test-model-e1d060-kserve-5969df469-4mfjn"
Oct 31 20:00:31.870176 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.870156    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b0c4b8a-bf5b-4af7-b272-aeefb1149b03-home\") pod \"rate-test-model-e1d060-kserve-5969df469-4mfjn\" (UID: \"8b0c4b8a-bf5b-4af7-b272-aeefb1149b03\") " pod="ai-tenant-e2e-shared-b-w6-e1d060/rate-test-model-e1d060-kserve-5969df469-4mfjn"
Oct 31 20:00:31.871976 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.871954    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b0c4b8a-bf5b-4af7-b272-aeefb1149b03-dshm\") pod \"rate-test-model-e1d060-kserve-5969df469-4mfjn\" (UID: \"8b0c4b8a-bf5b-4af7-b272-aeefb1149b03\") " pod="ai-tenant-e2e-shared-b-w6-e1d060/rate-test-model-e1d060-kserve-5969df469-4mfjn"
Oct 31 20:00:31.872183 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.872157    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b0c4b8a-bf5b-4af7-b272-aeefb1149b03-tls-certs\") pod \"rate-test-model-e1d060-kserve-5969df469-4mfjn\" (UID: \"8b0c4b8a-bf5b-4af7-b272-aeefb1149b03\") " pod="ai-tenant-e2e-shared-b-w6-e1d060/rate-test-model-e1d060-kserve-5969df469-4mfjn"
Oct 31 20:00:31.944350 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:31.916139    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qw8v\" (UniqueName: \"kubernetes.io/projected/8b0c4b8a-bf5b-4af7-b272-aeefb1149b03-kube-api-access-9qw8v\") pod \"rate-test-model-e1d060-kserve-5969df469-4mfjn\" (UID: \"8b0c4b8a-bf5b-4af7-b272-aeefb1149b03\") " pod="ai-tenant-e2e-shared-b-w6-e1d060/rate-test-model-e1d060-kserve-5969df469-4mfjn"
Oct 31 20:00:32.013415 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:32.013086    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-e1d060/rate-test-model-e1d060-kserve-5969df469-4mfjn"
Oct 31 20:00:32.317517 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:32.317486    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w6-e1d060/rate-test-model-e1d060-kserve-5969df469-4mfjn"]
Oct 31 20:00:32.317930 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:00:32.317907    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8b0c4b8a_bf5b_4af7_b272_aeefb1149b03.slice/crio-393715bd38586726f501e893d1276f50ce6a3c79e30216f8f1634051eed56a43 WatchSource:0}: Error finding container 393715bd38586726f501e893d1276f50ce6a3c79e30216f8f1634051eed56a43: Status 404 returned error can't find the container with id 393715bd38586726f501e893d1276f50ce6a3c79e30216f8f1634051eed56a43
Oct 31 20:00:32.346904 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:32.346875    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-e1d060/rate-test-model-e1d060-kserve-5969df469-4mfjn" event={"ID":"8b0c4b8a-bf5b-4af7-b272-aeefb1149b03","Type":"ContainerStarted","Data":"393715bd38586726f501e893d1276f50ce6a3c79e30216f8f1634051eed56a43"}
Oct 31 20:00:33.156389 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:33.156357    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:33.357033 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:33.356994    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-e1d060/rate-test-model-e1d060-kserve-5969df469-4mfjn" event={"ID":"8b0c4b8a-bf5b-4af7-b272-aeefb1149b03","Type":"ContainerStarted","Data":"8d8ef7c5b454d39dfbe4433c26006191b791f35bf1fb59e88d73007f15de6d05"}
Oct 31 20:00:33.381573 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:33.381527    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w6-e1d060/rate-test-model-e1d060-kserve-5969df469-4mfjn" podStartSLOduration=2.381514802 podStartE2EDuration="2.381514802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:00:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:00:33.376743257 +0000 UTC m=+2463.949977386" watchObservedRunningTime="2026-10-31 20:00:33.381514802 +0000 UTC m=+2463.954748931"
Oct 31 20:00:34.014002 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:34.013972    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w6-e1d060/rate-test-model-e1d060-kserve-5969df469-4mfjn"
Oct 31 20:00:34.143311 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:34.143260    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:34.425801 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:34.425767    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5f3eff-7655b4885c-k6dn9"]
Oct 31 20:00:34.430226 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:34.430198    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5f3eff-6f85d65cf9-2c4l7"]
Oct 31 20:00:34.544868 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:34.544820    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:34.654278 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:34.654249    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:34.791405 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:34.790800    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:35.149640 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:35.149553    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:35.285004 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:35.284978    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c734f5-699f7df769-c5swh"
Oct 31 20:00:36.134466 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:36.134427    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:36.259538 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:00:36.259494    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 20:00:36.664975 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:36.664939    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-ae2e60/rate-test-model-ae2e60-kserve-744475655c-xk2vp"]
Oct 31 20:00:36.706183 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:36.706143    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:37.149204 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:37.149119    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:37.420152 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:37.420120    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:37.837407 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:37.834517    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6"]
Oct 31 20:00:37.837407 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:37.835657    4645 cpu_manager.go:414] "containerMap: removing container" podUID="0b226e15-ab1f-4f6c-9b08-1ce863756e43" containerName="cleanup"
Oct 31 20:00:37.837407 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:37.835681    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b226e15-ab1f-4f6c-9b08-1ce863756e43" containerName="cleanup"
Oct 31 20:00:37.837407 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:37.835713    4645 cpu_manager.go:414] "containerMap: removing container" podUID="674684f6-15f9-4e2e-a5d6-136098c534d3" containerName="cleanup"
Oct 31 20:00:37.837407 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:37.835722    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="674684f6-15f9-4e2e-a5d6-136098c534d3" containerName="cleanup"
Oct 31 20:00:37.837407 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:37.835742    4645 cpu_manager.go:414] "containerMap: removing container" podUID="d0a6fdf8-881f-4993-9515-dfc51f685500" containerName="cleanup"
Oct 31 20:00:37.837407 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:37.835751    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0a6fdf8-881f-4993-9515-dfc51f685500" containerName="cleanup"
Oct 31 20:00:37.837407 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:37.835773    4645 cpu_manager.go:414] "containerMap: removing container" podUID="48ab68ad-2abf-44c8-98dc-c3ec72d36362" containerName="istio-proxy"
Oct 31 20:00:37.837407 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:37.835782    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48ab68ad-2abf-44c8-98dc-c3ec72d36362" containerName="istio-proxy"
Oct 31 20:00:37.837407 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:37.835988    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="4360bbcf-669f-4d6e-933c-0dcdfa76071a" containerName="cleanup"
Oct 31 20:00:37.837407 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:37.836006    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b226e15-ab1f-4f6c-9b08-1ce863756e43" containerName="cleanup"
Oct 31 20:00:37.938810 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:37.938769    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:38.032638 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:38.032605    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6"]
Oct 31 20:00:38.032964 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:38.032946    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6"
Oct 31 20:00:38.042426 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:38.042384    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-bd1c9f-openshift-default-dockercfg-jh8fx\""
Oct 31 20:00:38.121041 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:38.120968    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/679339df-b722-4990-b395-b14fea7f6458-workload-socket\") pod \"e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6\" (UID: \"679339df-b722-4990-b395-b14fea7f6458\") " pod="openshift-ingress/e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6"
Oct 31 20:00:38.121201 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:38.121108    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/679339df-b722-4990-b395-b14fea7f6458-istio-token\") pod \"e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6\" (UID: \"679339df-b722-4990-b395-b14fea7f6458\") " pod="openshift-ingress/e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6"
Oct 31 20:00:38.121201 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:38.121188    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/679339df-b722-4990-b395-b14fea7f6458-istiod-ca-cert\") pod \"e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6\" (UID: \"679339df-b722-4990-b395-b14fea7f6458\") " pod="openshift-ingress/e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6"
Oct 31 20:00:38.121283 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:38.121223    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/679339df-b722-4990-b395-b14fea7f6458-credential-socket\") pod \"e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6\" (UID: \"679339df-b722-4990-b395-b14fea7f6458\") " pod="openshift-ingress/e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6"
Oct 31 20:00:38.121283 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:38.121277    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/679339df-b722-4990-b395-b14fea7f6458-istio-envoy\") pod \"e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6\" (UID: \"679339df-b722-4990-b395-b14fea7f6458\") " pod="openshift-ingress/e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6"
Oct 31 20:00:38.121380 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:38.121320    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/679339df-b722-4990-b395-b14fea7f6458-workload-certs\") pod \"e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6\" (UID: \"679339df-b722-4990-b395-b14fea7f6458\") " pod="openshift-ingress/e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6"
Oct 31 20:00:38.121430 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:38.121401    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltmpk\" (UniqueName: \"kubernetes.io/projected/679339df-b722-4990-b395-b14fea7f6458-kube-api-access-ltmpk\") pod \"e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6\" (UID: \"679339df-b722-4990-b395-b14fea7f6458\") " pod="openshift-ingress/e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6"
Oct 31 20:00:38.121520 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:38.121505    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/679339df-b722-4990-b395-b14fea7f6458-istio-data\") pod \"e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6\" (UID: \"679339df-b722-4990-b395-b14fea7f6458\") " pod="openshift-ingress/e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6"
Oct 31 20:00:38.121555 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:38.121530    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/679339df-b722-4990-b395-b14fea7f6458-istio-podinfo\") pod \"e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6\" (UID: \"679339df-b722-4990-b395-b14fea7f6458\") " pod="openshift-ingress/e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6"
Oct 31 20:00:38.223079 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:38.223043    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/679339df-b722-4990-b395-b14fea7f6458-istio-data\") pod \"e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6\" (UID: \"679339df-b722-4990-b395-b14fea7f6458\") " pod="openshift-ingress/e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6"
Oct 31 20:00:38.223555 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:38.223089    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/679339df-b722-4990-b395-b14fea7f6458-istio-podinfo\") pod \"e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6\" (UID: \"679339df-b722-4990-b395-b14fea7f6458\") " pod="openshift-ingress/e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6"
Oct 31 20:00:38.223555 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:38.223229    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/679339df-b722-4990-b395-b14fea7f6458-workload-socket\") pod \"e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6\" (UID: \"679339df-b722-4990-b395-b14fea7f6458\") " pod="openshift-ingress/e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6"
Oct 31 20:00:38.223555 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:38.223315    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/679339df-b722-4990-b395-b14fea7f6458-istio-token\") pod \"e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6\" (UID: \"679339df-b722-4990-b395-b14fea7f6458\") " pod="openshift-ingress/e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6"
Oct 31 20:00:38.223555 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:38.223382    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/679339df-b722-4990-b395-b14fea7f6458-istiod-ca-cert\") pod \"e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6\" (UID: \"679339df-b722-4990-b395-b14fea7f6458\") " pod="openshift-ingress/e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6"
Oct 31 20:00:38.223555 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:38.223411    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/679339df-b722-4990-b395-b14fea7f6458-credential-socket\") pod \"e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6\" (UID: \"679339df-b722-4990-b395-b14fea7f6458\") " pod="openshift-ingress/e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6"
Oct 31 20:00:38.223555 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:38.223462    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/679339df-b722-4990-b395-b14fea7f6458-istio-data\") pod \"e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6\" (UID: \"679339df-b722-4990-b395-b14fea7f6458\") " pod="openshift-ingress/e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6"
Oct 31 20:00:38.223555 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:38.223467    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/679339df-b722-4990-b395-b14fea7f6458-istio-envoy\") pod \"e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6\" (UID: \"679339df-b722-4990-b395-b14fea7f6458\") " pod="openshift-ingress/e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6"
Oct 31 20:00:38.224000 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:38.223572    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/679339df-b722-4990-b395-b14fea7f6458-workload-certs\") pod \"e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6\" (UID: \"679339df-b722-4990-b395-b14fea7f6458\") " pod="openshift-ingress/e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6"
Oct 31 20:00:38.224000 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:38.223618    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltmpk\" (UniqueName: \"kubernetes.io/projected/679339df-b722-4990-b395-b14fea7f6458-kube-api-access-ltmpk\") pod \"e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6\" (UID: \"679339df-b722-4990-b395-b14fea7f6458\") " pod="openshift-ingress/e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6"
Oct 31 20:00:38.224000 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:38.223785    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/679339df-b722-4990-b395-b14fea7f6458-credential-socket\") pod \"e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6\" (UID: \"679339df-b722-4990-b395-b14fea7f6458\") " pod="openshift-ingress/e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6"
Oct 31 20:00:38.224172 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:38.224026    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/679339df-b722-4990-b395-b14fea7f6458-workload-certs\") pod \"e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6\" (UID: \"679339df-b722-4990-b395-b14fea7f6458\") " pod="openshift-ingress/e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6"
Oct 31 20:00:38.224172 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:38.224115    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/679339df-b722-4990-b395-b14fea7f6458-istiod-ca-cert\") pod \"e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6\" (UID: \"679339df-b722-4990-b395-b14fea7f6458\") " pod="openshift-ingress/e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6"
Oct 31 20:00:38.224350 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:38.224307    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/679339df-b722-4990-b395-b14fea7f6458-workload-socket\") pod \"e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6\" (UID: \"679339df-b722-4990-b395-b14fea7f6458\") " pod="openshift-ingress/e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6"
Oct 31 20:00:38.225800 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:38.225778    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/679339df-b722-4990-b395-b14fea7f6458-istio-envoy\") pod \"e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6\" (UID: \"679339df-b722-4990-b395-b14fea7f6458\") " pod="openshift-ingress/e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6"
Oct 31 20:00:38.226452 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:38.226426    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/679339df-b722-4990-b395-b14fea7f6458-istio-podinfo\") pod \"e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6\" (UID: \"679339df-b722-4990-b395-b14fea7f6458\") " pod="openshift-ingress/e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6"
Oct 31 20:00:38.237535 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:38.237517    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltmpk\" (UniqueName: \"kubernetes.io/projected/679339df-b722-4990-b395-b14fea7f6458-kube-api-access-ltmpk\") pod \"e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6\" (UID: \"679339df-b722-4990-b395-b14fea7f6458\") " pod="openshift-ingress/e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6"
Oct 31 20:00:38.237929 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:38.237911    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/679339df-b722-4990-b395-b14fea7f6458-istio-token\") pod \"e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6\" (UID: \"679339df-b722-4990-b395-b14fea7f6458\") " pod="openshift-ingress/e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6"
Oct 31 20:00:38.347440 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:38.347406    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:38.390361 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:38.390244    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6"
Oct 31 20:00:38.458443 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:38.456078    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:38.717099 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:00:38.717066    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod679339df_b722_4990_b395_b14fea7f6458.slice/crio-b88c7fb6aefcc24f7ce4b491dd43d4f4e81d9dd76867d164159c77045b303fc2 WatchSource:0}: Error finding container b88c7fb6aefcc24f7ce4b491dd43d4f4e81d9dd76867d164159c77045b303fc2: Status 404 returned error can't find the container with id b88c7fb6aefcc24f7ce4b491dd43d4f4e81d9dd76867d164159c77045b303fc2
Oct 31 20:00:38.717880 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:38.717856    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6"]
Oct 31 20:00:38.721554 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:38.721518    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:00:38.721684 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:38.721584    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:00:38.721684 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:38.721613    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:00:39.157402 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:39.157311    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:39.409939 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:39.409906    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6" event={"ID":"679339df-b722-4990-b395-b14fea7f6458","Type":"ContainerStarted","Data":"56fe3396ecf1c21d316da758dac8576fc1887b1d90379d62b9cb2b8754cb241b"}
Oct 31 20:00:39.410287 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:39.409961    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6" event={"ID":"679339df-b722-4990-b395-b14fea7f6458","Type":"ContainerStarted","Data":"b88c7fb6aefcc24f7ce4b491dd43d4f4e81d9dd76867d164159c77045b303fc2"}
Oct 31 20:00:39.426119 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:39.426086    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-5f3eff-7655b4885c-k6dn9" podUID="71f14650-d480-4e47-90c3-2ca4afac56d9" containerName="payload-processing" containerID="cri-o://8b995ebe402bf29905f5a735fa29d3d12377326b2dacf40b683150b80dfdec79" gracePeriod=25
Oct 31 20:00:39.430589 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:39.430550    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5f3eff-6f85d65cf9-2c4l7" podUID="7b4ca125-b910-4fe9-91ed-0a8e1dfa6e3e" containerName="payload-pre-processing" containerID="cri-o://39eed9102bca75571d7403dfd57f986de07d4b85ca7f4be5f42d6c3eb346a3c0" gracePeriod=25
Oct 31 20:00:39.439610 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:39.439570    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6" podStartSLOduration=2.439556165 podStartE2EDuration="2.439556165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:00:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:00:39.431785673 +0000 UTC m=+2470.005019803" watchObservedRunningTime="2026-10-31 20:00:39.439556165 +0000 UTC m=+2470.012790295"
Oct 31 20:00:39.804121 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:39.804094    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5f3eff-7655b4885c-k6dn9"
Oct 31 20:00:39.837934 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:39.837909    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5f3eff-6f85d65cf9-2c4l7"
Oct 31 20:00:39.868373 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:39.862393    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71f14650-d480-4e47-90c3-2ca4afac56d9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71f14650-d480-4e47-90c3-2ca4afac56d9-plugins-config-volume\") pod \"71f14650-d480-4e47-90c3-2ca4afac56d9\" (UID: \"71f14650-d480-4e47-90c3-2ca4afac56d9\") "
Oct 31 20:00:39.868373 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:39.862554    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71f14650-d480-4e47-90c3-2ca4afac56d9-kube-api-access-jgpcg\" (UniqueName: \"kubernetes.io/projected/71f14650-d480-4e47-90c3-2ca4afac56d9-kube-api-access-jgpcg\") pod \"71f14650-d480-4e47-90c3-2ca4afac56d9\" (UID: \"71f14650-d480-4e47-90c3-2ca4afac56d9\") "
Oct 31 20:00:39.868373 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:39.862819    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71f14650-d480-4e47-90c3-2ca4afac56d9-plugins-config-volume" pod "71f14650-d480-4e47-90c3-2ca4afac56d9" (UID: "71f14650-d480-4e47-90c3-2ca4afac56d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:00:39.868373 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:39.863372    4645 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71f14650-d480-4e47-90c3-2ca4afac56d9-plugins-config-volume\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:39.868373 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:39.865468    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71f14650-d480-4e47-90c3-2ca4afac56d9-kube-api-access-jgpcg" pod "71f14650-d480-4e47-90c3-2ca4afac56d9" (UID: "71f14650-d480-4e47-90c3-2ca4afac56d9"). InnerVolumeSpecName "kube-api-access-jgpcg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:00:39.964256 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:39.964161    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b4ca125-b910-4fe9-91ed-0a8e1dfa6e3e-kube-api-access-49zt2\" (UniqueName: \"kubernetes.io/projected/7b4ca125-b910-4fe9-91ed-0a8e1dfa6e3e-kube-api-access-49zt2\") pod \"7b4ca125-b910-4fe9-91ed-0a8e1dfa6e3e\" (UID: \"7b4ca125-b910-4fe9-91ed-0a8e1dfa6e3e\") "
Oct 31 20:00:39.964419 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:39.964266    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b4ca125-b910-4fe9-91ed-0a8e1dfa6e3e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b4ca125-b910-4fe9-91ed-0a8e1dfa6e3e-plugins-config-volume\") pod \"7b4ca125-b910-4fe9-91ed-0a8e1dfa6e3e\" (UID: \"7b4ca125-b910-4fe9-91ed-0a8e1dfa6e3e\") "
Oct 31 20:00:39.964695 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:39.964666    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b4ca125-b910-4fe9-91ed-0a8e1dfa6e3e-plugins-config-volume" pod "7b4ca125-b910-4fe9-91ed-0a8e1dfa6e3e" (UID: "7b4ca125-b910-4fe9-91ed-0a8e1dfa6e3e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:00:39.965028 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:39.965008    4645 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b4ca125-b910-4fe9-91ed-0a8e1dfa6e3e-plugins-config-volume\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:39.965099 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:39.965030    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgpcg\" (UniqueName: \"kubernetes.io/projected/71f14650-d480-4e47-90c3-2ca4afac56d9-kube-api-access-jgpcg\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:39.966322 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:39.966294    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b4ca125-b910-4fe9-91ed-0a8e1dfa6e3e-kube-api-access-49zt2" pod "7b4ca125-b910-4fe9-91ed-0a8e1dfa6e3e" (UID: "7b4ca125-b910-4fe9-91ed-0a8e1dfa6e3e"). InnerVolumeSpecName "kube-api-access-49zt2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:00:40.066749 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:40.066708    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49zt2\" (UniqueName: \"kubernetes.io/projected/7b4ca125-b910-4fe9-91ed-0a8e1dfa6e3e-kube-api-access-49zt2\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:40.131051 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:40.131021    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:40.392107 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:40.392075    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6"
Oct 31 20:00:40.397933 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:40.397906    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6"
Oct 31 20:00:40.418192 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:40.418162    4645 generic.go:356] "Generic (PLEG): container finished" podID="7b4ca125-b910-4fe9-91ed-0a8e1dfa6e3e" containerID="39eed9102bca75571d7403dfd57f986de07d4b85ca7f4be5f42d6c3eb346a3c0" exitCode=0
Oct 31 20:00:40.418599 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:40.418209    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5f3eff-6f85d65cf9-2c4l7"
Oct 31 20:00:40.418599 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:40.418248    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5f3eff-6f85d65cf9-2c4l7" event={"ID":"7b4ca125-b910-4fe9-91ed-0a8e1dfa6e3e","Type":"ContainerDied","Data":"39eed9102bca75571d7403dfd57f986de07d4b85ca7f4be5f42d6c3eb346a3c0"}
Oct 31 20:00:40.418599 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:40.418300    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5f3eff-6f85d65cf9-2c4l7" event={"ID":"7b4ca125-b910-4fe9-91ed-0a8e1dfa6e3e","Type":"ContainerDied","Data":"fb4355d6bf25a03a442260e2dd4a346484bab67555d2ee1c63c23544e8d8083a"}
Oct 31 20:00:40.418599 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:40.418354    4645 scope.go:122] "RemoveContainer" containerID="39eed9102bca75571d7403dfd57f986de07d4b85ca7f4be5f42d6c3eb346a3c0"
Oct 31 20:00:40.419825 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:40.419797    4645 generic.go:356] "Generic (PLEG): container finished" podID="71f14650-d480-4e47-90c3-2ca4afac56d9" containerID="8b995ebe402bf29905f5a735fa29d3d12377326b2dacf40b683150b80dfdec79" exitCode=0
Oct 31 20:00:40.419951 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:40.419842    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5f3eff-7655b4885c-k6dn9"
Oct 31 20:00:40.419951 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:40.419859    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5f3eff-7655b4885c-k6dn9" event={"ID":"71f14650-d480-4e47-90c3-2ca4afac56d9","Type":"ContainerDied","Data":"8b995ebe402bf29905f5a735fa29d3d12377326b2dacf40b683150b80dfdec79"}
Oct 31 20:00:40.419951 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:40.419900    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5f3eff-7655b4885c-k6dn9" event={"ID":"71f14650-d480-4e47-90c3-2ca4afac56d9","Type":"ContainerDied","Data":"69a6a3c4cd4b1114574ade0bcd60a9a1b2cfe87c9be1b3d2632ca8349b5bdfe8"}
Oct 31 20:00:40.420185 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:40.420160    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6"
Oct 31 20:00:40.423627 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:40.423567    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6"
Oct 31 20:00:40.432112 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:40.431870    4645 scope.go:122] "RemoveContainer" containerID="39eed9102bca75571d7403dfd57f986de07d4b85ca7f4be5f42d6c3eb346a3c0"
Oct 31 20:00:40.432520 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:00:40.432499    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39eed9102bca75571d7403dfd57f986de07d4b85ca7f4be5f42d6c3eb346a3c0\": container with ID starting with 39eed9102bca75571d7403dfd57f986de07d4b85ca7f4be5f42d6c3eb346a3c0 not found: ID does not exist" containerID="39eed9102bca75571d7403dfd57f986de07d4b85ca7f4be5f42d6c3eb346a3c0"
Oct 31 20:00:40.432597 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:40.432527    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39eed9102bca75571d7403dfd57f986de07d4b85ca7f4be5f42d6c3eb346a3c0"} err="failed to get container status \"39eed9102bca75571d7403dfd57f986de07d4b85ca7f4be5f42d6c3eb346a3c0\": rpc error: code = NotFound desc = could not find container \"39eed9102bca75571d7403dfd57f986de07d4b85ca7f4be5f42d6c3eb346a3c0\": container with ID starting with 39eed9102bca75571d7403dfd57f986de07d4b85ca7f4be5f42d6c3eb346a3c0 not found: ID does not exist"
Oct 31 20:00:40.432597 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:40.432542    4645 scope.go:122] "RemoveContainer" containerID="8b995ebe402bf29905f5a735fa29d3d12377326b2dacf40b683150b80dfdec79"
Oct 31 20:00:40.442780 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:40.442762    4645 scope.go:122] "RemoveContainer" containerID="8b995ebe402bf29905f5a735fa29d3d12377326b2dacf40b683150b80dfdec79"
Oct 31 20:00:40.443023 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:00:40.443004    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b995ebe402bf29905f5a735fa29d3d12377326b2dacf40b683150b80dfdec79\": container with ID starting with 8b995ebe402bf29905f5a735fa29d3d12377326b2dacf40b683150b80dfdec79 not found: ID does not exist" containerID="8b995ebe402bf29905f5a735fa29d3d12377326b2dacf40b683150b80dfdec79"
Oct 31 20:00:40.443066 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:40.443030    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b995ebe402bf29905f5a735fa29d3d12377326b2dacf40b683150b80dfdec79"} err="failed to get container status \"8b995ebe402bf29905f5a735fa29d3d12377326b2dacf40b683150b80dfdec79\": rpc error: code = NotFound desc = could not find container \"8b995ebe402bf29905f5a735fa29d3d12377326b2dacf40b683150b80dfdec79\": container with ID starting with 8b995ebe402bf29905f5a735fa29d3d12377326b2dacf40b683150b80dfdec79 not found: ID does not exist"
Oct 31 20:00:40.473032 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:40.472674    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5f3eff-6f85d65cf9-2c4l7"]
Oct 31 20:00:40.477854 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:40.477824    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5f3eff-6f85d65cf9-2c4l7"]
Oct 31 20:00:40.499449 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:40.494499    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5f3eff-7655b4885c-k6dn9"]
Oct 31 20:00:40.499449 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:40.494547    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5f3eff-7655b4885c-k6dn9"]
Oct 31 20:00:40.838082 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:40.837998    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:40.850095 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:40.850059    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-01afb5-db5b465fd-jkvfg"]
Oct 31 20:00:40.850652 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:40.850346    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-01afb5-db5b465fd-jkvfg" podUID="4bc15cf8-897f-4501-98f8-1a7abbea383c" containerName="payload-processing" containerID="cri-o://7363d9ba9ba6afe3327cce6e8961e051eeea2ed963419cf591fdee568ee7a333" gracePeriod=30
Oct 31 20:00:40.874578 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:40.874551    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-01afb5-6fdd5b76b7-qggmv"]
Oct 31 20:00:40.874806 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:40.874779    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01afb5-6fdd5b76b7-qggmv" podUID="2f35e911-c0c8-45a1-862e-f6a55fb88b77" containerName="payload-pre-processing" containerID="cri-o://11e56de629a037edadaa266a1362d815c4775dbb2090fe98079008197cc0b13d" gracePeriod=30
Oct 31 20:00:40.949018 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:40.948986    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:41.142527 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:41.140499    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:41.340358 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:41.305784    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:41.431316 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:41.415535    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:41.433009 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:41.432974    4645 generic.go:356] "Generic (PLEG): container finished" podID="2f35e911-c0c8-45a1-862e-f6a55fb88b77" containerID="11e56de629a037edadaa266a1362d815c4775dbb2090fe98079008197cc0b13d" exitCode=0
Oct 31 20:00:41.433229 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:41.433201    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01afb5-6fdd5b76b7-qggmv" event={"ID":"2f35e911-c0c8-45a1-862e-f6a55fb88b77","Type":"ContainerDied","Data":"11e56de629a037edadaa266a1362d815c4775dbb2090fe98079008197cc0b13d"}
Oct 31 20:00:41.444355 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:41.435571    4645 generic.go:356] "Generic (PLEG): container finished" podID="4bc15cf8-897f-4501-98f8-1a7abbea383c" containerID="7363d9ba9ba6afe3327cce6e8961e051eeea2ed963419cf591fdee568ee7a333" exitCode=0
Oct 31 20:00:41.444355 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:41.435659    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-01afb5-db5b465fd-jkvfg" event={"ID":"4bc15cf8-897f-4501-98f8-1a7abbea383c","Type":"ContainerDied","Data":"7363d9ba9ba6afe3327cce6e8961e051eeea2ed963419cf591fdee568ee7a333"}
Oct 31 20:00:41.511038 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:41.511014    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-01afb5-db5b465fd-jkvfg"
Oct 31 20:00:41.602885 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:41.602854    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01afb5-6fdd5b76b7-qggmv"
Oct 31 20:00:41.621533 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:41.621503    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71f14650-d480-4e47-90c3-2ca4afac56d9" path="/var/lib/kubelet/pods/71f14650-d480-4e47-90c3-2ca4afac56d9/volumes"
Oct 31 20:00:41.622073 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:41.622050    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b4ca125-b910-4fe9-91ed-0a8e1dfa6e3e" path="/var/lib/kubelet/pods/7b4ca125-b910-4fe9-91ed-0a8e1dfa6e3e/volumes"
Oct 31 20:00:41.663424 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:41.663382    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bc15cf8-897f-4501-98f8-1a7abbea383c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bc15cf8-897f-4501-98f8-1a7abbea383c-plugins-config-volume\") pod \"4bc15cf8-897f-4501-98f8-1a7abbea383c\" (UID: \"4bc15cf8-897f-4501-98f8-1a7abbea383c\") "
Oct 31 20:00:41.666074 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:41.663443    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bc15cf8-897f-4501-98f8-1a7abbea383c-kube-api-access-jfgkl\" (UniqueName: \"kubernetes.io/projected/4bc15cf8-897f-4501-98f8-1a7abbea383c-kube-api-access-jfgkl\") pod \"4bc15cf8-897f-4501-98f8-1a7abbea383c\" (UID: \"4bc15cf8-897f-4501-98f8-1a7abbea383c\") "
Oct 31 20:00:41.666074 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:41.664692    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bc15cf8-897f-4501-98f8-1a7abbea383c-plugins-config-volume" pod "4bc15cf8-897f-4501-98f8-1a7abbea383c" (UID: "4bc15cf8-897f-4501-98f8-1a7abbea383c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:00:41.666494 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:41.666467    4645 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bc15cf8-897f-4501-98f8-1a7abbea383c-plugins-config-volume\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:41.669667 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:41.669637    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bc15cf8-897f-4501-98f8-1a7abbea383c-kube-api-access-jfgkl" pod "4bc15cf8-897f-4501-98f8-1a7abbea383c" (UID: "4bc15cf8-897f-4501-98f8-1a7abbea383c"). InnerVolumeSpecName "kube-api-access-jfgkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:00:41.747623 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:41.747588    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:41.767895 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:41.767870    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f35e911-c0c8-45a1-862e-f6a55fb88b77-kube-api-access-4zw56\" (UniqueName: \"kubernetes.io/projected/2f35e911-c0c8-45a1-862e-f6a55fb88b77-kube-api-access-4zw56\") pod \"2f35e911-c0c8-45a1-862e-f6a55fb88b77\" (UID: \"2f35e911-c0c8-45a1-862e-f6a55fb88b77\") "
Oct 31 20:00:41.768027 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:41.767987    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f35e911-c0c8-45a1-862e-f6a55fb88b77-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f35e911-c0c8-45a1-862e-f6a55fb88b77-plugins-config-volume\") pod \"2f35e911-c0c8-45a1-862e-f6a55fb88b77\" (UID: \"2f35e911-c0c8-45a1-862e-f6a55fb88b77\") "
Oct 31 20:00:41.768384 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:41.768347    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f35e911-c0c8-45a1-862e-f6a55fb88b77-plugins-config-volume" pod "2f35e911-c0c8-45a1-862e-f6a55fb88b77" (UID: "2f35e911-c0c8-45a1-862e-f6a55fb88b77"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:00:41.768615 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:41.768596    4645 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f35e911-c0c8-45a1-862e-f6a55fb88b77-plugins-config-volume\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:41.768694 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:41.768621    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfgkl\" (UniqueName: \"kubernetes.io/projected/4bc15cf8-897f-4501-98f8-1a7abbea383c-kube-api-access-jfgkl\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:41.770033 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:41.770012    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f35e911-c0c8-45a1-862e-f6a55fb88b77-kube-api-access-4zw56" pod "2f35e911-c0c8-45a1-862e-f6a55fb88b77" (UID: "2f35e911-c0c8-45a1-862e-f6a55fb88b77"). InnerVolumeSpecName "kube-api-access-4zw56". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:00:41.870083 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:41.870052    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zw56\" (UniqueName: \"kubernetes.io/projected/2f35e911-c0c8-45a1-862e-f6a55fb88b77-kube-api-access-4zw56\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:41.879239 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:41.879215    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:42.014856 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:42.014772    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w6-e1d060/rate-test-model-e1d060-kserve-5969df469-4mfjn"
Oct 31 20:00:42.027466 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:42.027444    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w6-e1d060/rate-test-model-e1d060-kserve-5969df469-4mfjn"
Oct 31 20:00:42.431627 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:42.431596    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:42.448243 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:42.448219    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01afb5-6fdd5b76b7-qggmv"
Oct 31 20:00:42.448440 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:42.448228    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01afb5-6fdd5b76b7-qggmv" event={"ID":"2f35e911-c0c8-45a1-862e-f6a55fb88b77","Type":"ContainerDied","Data":"15c2cd42fb65139779f6e36f608bd5363ebdf410c24abfc0079c7bfbed2765f0"}
Oct 31 20:00:42.448440 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:42.448384    4645 scope.go:122] "RemoveContainer" containerID="11e56de629a037edadaa266a1362d815c4775dbb2090fe98079008197cc0b13d"
Oct 31 20:00:42.449875 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:42.449857    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-01afb5-db5b465fd-jkvfg"
Oct 31 20:00:42.450002 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:42.449890    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-01afb5-db5b465fd-jkvfg" event={"ID":"4bc15cf8-897f-4501-98f8-1a7abbea383c","Type":"ContainerDied","Data":"6f60e313d1c34394d2a01631f56fe3281e5157df4cfcec18449c2bc6ed7b62a1"}
Oct 31 20:00:42.465298 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:42.465272    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w6-e1d060/rate-test-model-e1d060-kserve-5969df469-4mfjn"
Oct 31 20:00:42.549556 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:42.527799    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-4bac5a/rate-test-model-4bac5a-kserve-564dc4dfdf-vv676" podUID="f9b4e54a-b150-415f-b266-caa45004d860" containerName="main" containerID="cri-o://58cadced6ea856f6d2d2ecd0660748c2cd605d10f7a4bd7c7504b26f000ad45d" gracePeriod=45
Oct 31 20:00:42.549556 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:42.528890    4645 scope.go:122] "RemoveContainer" containerID="7363d9ba9ba6afe3327cce6e8961e051eeea2ed963419cf591fdee568ee7a333"
Oct 31 20:00:42.549556 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:42.547069    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-01afb5-6fdd5b76b7-qggmv"]
Oct 31 20:00:42.561376 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:42.556637    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-01afb5-6fdd5b76b7-qggmv"]
Oct 31 20:00:42.574436 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:42.574406    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-01afb5-db5b465fd-jkvfg"]
Oct 31 20:00:42.576963 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:42.576942    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-01afb5-db5b465fd-jkvfg"]
Oct 31 20:00:42.605224 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:42.598402    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl"]
Oct 31 20:00:42.610155 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:42.610115    4645 cpu_manager.go:414] "containerMap: removing container" podUID="7b4ca125-b910-4fe9-91ed-0a8e1dfa6e3e" containerName="payload-pre-processing"
Oct 31 20:00:42.610155 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:42.610142    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b4ca125-b910-4fe9-91ed-0a8e1dfa6e3e" containerName="payload-pre-processing"
Oct 31 20:00:42.610405 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:42.610178    4645 cpu_manager.go:414] "containerMap: removing container" podUID="71f14650-d480-4e47-90c3-2ca4afac56d9" containerName="payload-processing"
Oct 31 20:00:42.610405 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:42.610184    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71f14650-d480-4e47-90c3-2ca4afac56d9" containerName="payload-processing"
Oct 31 20:00:42.610405 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:42.610202    4645 cpu_manager.go:414] "containerMap: removing container" podUID="2f35e911-c0c8-45a1-862e-f6a55fb88b77" containerName="payload-pre-processing"
Oct 31 20:00:42.610405 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:42.610210    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f35e911-c0c8-45a1-862e-f6a55fb88b77" containerName="payload-pre-processing"
Oct 31 20:00:42.610405 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:42.610239    4645 cpu_manager.go:414] "containerMap: removing container" podUID="4bc15cf8-897f-4501-98f8-1a7abbea383c" containerName="payload-processing"
Oct 31 20:00:42.610405 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:42.610244    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bc15cf8-897f-4501-98f8-1a7abbea383c" containerName="payload-processing"
Oct 31 20:00:42.610769 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:42.610411    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="71f14650-d480-4e47-90c3-2ca4afac56d9" containerName="payload-processing"
Oct 31 20:00:42.610769 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:42.610426    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f35e911-c0c8-45a1-862e-f6a55fb88b77" containerName="payload-pre-processing"
Oct 31 20:00:42.610769 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:42.610437    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bc15cf8-897f-4501-98f8-1a7abbea383c" containerName="payload-processing"
Oct 31 20:00:42.610769 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:42.610448    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b4ca125-b910-4fe9-91ed-0a8e1dfa6e3e" containerName="payload-pre-processing"
Oct 31 20:00:42.616587 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:42.616562    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl"
Oct 31 20:00:42.626709 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:42.626413    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9deed825-0c5f-4950-96e4-098a01b928f4-istio-token\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl\" (UID: \"9deed825-0c5f-4950-96e4-098a01b928f4\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl"
Oct 31 20:00:42.626709 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:42.626466    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9deed825-0c5f-4950-96e4-098a01b928f4-istio-podinfo\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl\" (UID: \"9deed825-0c5f-4950-96e4-098a01b928f4\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl"
Oct 31 20:00:42.626709 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:42.626578    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrnk2\" (UniqueName: \"kubernetes.io/projected/9deed825-0c5f-4950-96e4-098a01b928f4-kube-api-access-hrnk2\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl\" (UID: \"9deed825-0c5f-4950-96e4-098a01b928f4\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl"
Oct 31 20:00:42.626709 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:42.626633    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9deed825-0c5f-4950-96e4-098a01b928f4-istio-data\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl\" (UID: \"9deed825-0c5f-4950-96e4-098a01b928f4\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl"
Oct 31 20:00:42.626709 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:42.626671    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9deed825-0c5f-4950-96e4-098a01b928f4-istiod-ca-cert\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl\" (UID: \"9deed825-0c5f-4950-96e4-098a01b928f4\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl"
Oct 31 20:00:42.627157 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:42.626720    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9deed825-0c5f-4950-96e4-098a01b928f4-workload-certs\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl\" (UID: \"9deed825-0c5f-4950-96e4-098a01b928f4\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl"
Oct 31 20:00:42.627157 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:42.626909    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9deed825-0c5f-4950-96e4-098a01b928f4-istio-envoy\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl\" (UID: \"9deed825-0c5f-4950-96e4-098a01b928f4\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl"
Oct 31 20:00:42.627157 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:42.626985    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9deed825-0c5f-4950-96e4-098a01b928f4-credential-socket\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl\" (UID: \"9deed825-0c5f-4950-96e4-098a01b928f4\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl"
Oct 31 20:00:42.627157 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:42.627103    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9deed825-0c5f-4950-96e4-098a01b928f4-workload-socket\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl\" (UID: \"9deed825-0c5f-4950-96e4-098a01b928f4\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl"
Oct 31 20:00:42.627420 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:42.627400    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl"]
Oct 31 20:00:42.728727 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:42.728613    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9deed825-0c5f-4950-96e4-098a01b928f4-workload-socket\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl\" (UID: \"9deed825-0c5f-4950-96e4-098a01b928f4\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl"
Oct 31 20:00:42.729199 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:42.729092    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9deed825-0c5f-4950-96e4-098a01b928f4-workload-socket\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl\" (UID: \"9deed825-0c5f-4950-96e4-098a01b928f4\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl"
Oct 31 20:00:42.729348 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:42.729243    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9deed825-0c5f-4950-96e4-098a01b928f4-istio-token\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl\" (UID: \"9deed825-0c5f-4950-96e4-098a01b928f4\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl"
Oct 31 20:00:42.729348 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:42.729299    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9deed825-0c5f-4950-96e4-098a01b928f4-istio-podinfo\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl\" (UID: \"9deed825-0c5f-4950-96e4-098a01b928f4\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl"
Oct 31 20:00:42.729475 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:42.729422    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrnk2\" (UniqueName: \"kubernetes.io/projected/9deed825-0c5f-4950-96e4-098a01b928f4-kube-api-access-hrnk2\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl\" (UID: \"9deed825-0c5f-4950-96e4-098a01b928f4\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl"
Oct 31 20:00:42.729536 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:42.729472    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9deed825-0c5f-4950-96e4-098a01b928f4-istio-data\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl\" (UID: \"9deed825-0c5f-4950-96e4-098a01b928f4\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl"
Oct 31 20:00:42.729536 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:42.729519    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9deed825-0c5f-4950-96e4-098a01b928f4-istiod-ca-cert\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl\" (UID: \"9deed825-0c5f-4950-96e4-098a01b928f4\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl"
Oct 31 20:00:42.729643 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:42.729582    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9deed825-0c5f-4950-96e4-098a01b928f4-workload-certs\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl\" (UID: \"9deed825-0c5f-4950-96e4-098a01b928f4\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl"
Oct 31 20:00:42.729709 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:42.729665    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9deed825-0c5f-4950-96e4-098a01b928f4-istio-envoy\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl\" (UID: \"9deed825-0c5f-4950-96e4-098a01b928f4\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl"
Oct 31 20:00:42.729808 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:42.729702    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9deed825-0c5f-4950-96e4-098a01b928f4-credential-socket\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl\" (UID: \"9deed825-0c5f-4950-96e4-098a01b928f4\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl"
Oct 31 20:00:42.730095 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:42.730074    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9deed825-0c5f-4950-96e4-098a01b928f4-credential-socket\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl\" (UID: \"9deed825-0c5f-4950-96e4-098a01b928f4\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl"
Oct 31 20:00:42.730292 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:42.730273    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9deed825-0c5f-4950-96e4-098a01b928f4-workload-certs\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl\" (UID: \"9deed825-0c5f-4950-96e4-098a01b928f4\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl"
Oct 31 20:00:42.730726 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:42.730699    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9deed825-0c5f-4950-96e4-098a01b928f4-istio-data\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl\" (UID: \"9deed825-0c5f-4950-96e4-098a01b928f4\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl"
Oct 31 20:00:42.731072 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:42.731051    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9deed825-0c5f-4950-96e4-098a01b928f4-istiod-ca-cert\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl\" (UID: \"9deed825-0c5f-4950-96e4-098a01b928f4\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl"
Oct 31 20:00:42.747365 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:42.739103    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9deed825-0c5f-4950-96e4-098a01b928f4-istio-envoy\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl\" (UID: \"9deed825-0c5f-4950-96e4-098a01b928f4\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl"
Oct 31 20:00:42.747365 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:42.739345    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9deed825-0c5f-4950-96e4-098a01b928f4-istio-podinfo\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl\" (UID: \"9deed825-0c5f-4950-96e4-098a01b928f4\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl"
Oct 31 20:00:42.750667 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:42.750644    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrnk2\" (UniqueName: \"kubernetes.io/projected/9deed825-0c5f-4950-96e4-098a01b928f4-kube-api-access-hrnk2\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl\" (UID: \"9deed825-0c5f-4950-96e4-098a01b928f4\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl"
Oct 31 20:00:42.752284 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:42.752262    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9deed825-0c5f-4950-96e4-098a01b928f4-istio-token\") pod \"e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl\" (UID: \"9deed825-0c5f-4950-96e4-098a01b928f4\") " pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl"
Oct 31 20:00:42.815172 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:42.815149    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-4bac5a/rate-test-model-4bac5a-kserve-564dc4dfdf-vv676"
Oct 31 20:00:42.830921 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:42.830897    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9b4e54a-b150-415f-b266-caa45004d860-home\" (UniqueName: \"kubernetes.io/empty-dir/f9b4e54a-b150-415f-b266-caa45004d860-home\") pod \"f9b4e54a-b150-415f-b266-caa45004d860\" (UID: \"f9b4e54a-b150-415f-b266-caa45004d860\") "
Oct 31 20:00:42.831040 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:42.831006    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9b4e54a-b150-415f-b266-caa45004d860-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9b4e54a-b150-415f-b266-caa45004d860-model-cache\") pod \"f9b4e54a-b150-415f-b266-caa45004d860\" (UID: \"f9b4e54a-b150-415f-b266-caa45004d860\") "
Oct 31 20:00:42.831102 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:42.831089    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9b4e54a-b150-415f-b266-caa45004d860-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9b4e54a-b150-415f-b266-caa45004d860-tmp-dir\") pod \"f9b4e54a-b150-415f-b266-caa45004d860\" (UID: \"f9b4e54a-b150-415f-b266-caa45004d860\") "
Oct 31 20:00:42.831158 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:42.831139    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9b4e54a-b150-415f-b266-caa45004d860-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9b4e54a-b150-415f-b266-caa45004d860-dshm\") pod \"f9b4e54a-b150-415f-b266-caa45004d860\" (UID: \"f9b4e54a-b150-415f-b266-caa45004d860\") "
Oct 31 20:00:42.831210 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:42.831162    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9b4e54a-b150-415f-b266-caa45004d860-home" pod "f9b4e54a-b150-415f-b266-caa45004d860" (UID: "f9b4e54a-b150-415f-b266-caa45004d860"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:00:42.831210 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:42.831188    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9b4e54a-b150-415f-b266-caa45004d860-tls-certs\" (UniqueName: \"kubernetes.io/secret/f9b4e54a-b150-415f-b266-caa45004d860-tls-certs\") pod \"f9b4e54a-b150-415f-b266-caa45004d860\" (UID: \"f9b4e54a-b150-415f-b266-caa45004d860\") "
Oct 31 20:00:42.831344 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:42.831241    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9b4e54a-b150-415f-b266-caa45004d860-model-cache" pod "f9b4e54a-b150-415f-b266-caa45004d860" (UID: "f9b4e54a-b150-415f-b266-caa45004d860"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:00:42.831405 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:42.831316    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9b4e54a-b150-415f-b266-caa45004d860-tmp-dir" pod "f9b4e54a-b150-415f-b266-caa45004d860" (UID: "f9b4e54a-b150-415f-b266-caa45004d860"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:00:42.831470 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:42.831408    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9b4e54a-b150-415f-b266-caa45004d860-kube-api-access-5hr84\" (UniqueName: \"kubernetes.io/projected/f9b4e54a-b150-415f-b266-caa45004d860-kube-api-access-5hr84\") pod \"f9b4e54a-b150-415f-b266-caa45004d860\" (UID: \"f9b4e54a-b150-415f-b266-caa45004d860\") "
Oct 31 20:00:42.832016 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:42.831995    4645 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9b4e54a-b150-415f-b266-caa45004d860-home\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:42.832016 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:42.832017    4645 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9b4e54a-b150-415f-b266-caa45004d860-model-cache\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:42.832191 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:42.832026    4645 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9b4e54a-b150-415f-b266-caa45004d860-tmp-dir\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:42.833234 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:42.833208    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9b4e54a-b150-415f-b266-caa45004d860-dshm" pod "f9b4e54a-b150-415f-b266-caa45004d860" (UID: "f9b4e54a-b150-415f-b266-caa45004d860"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:00:42.833367 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:42.833347    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9b4e54a-b150-415f-b266-caa45004d860-kube-api-access-5hr84" pod "f9b4e54a-b150-415f-b266-caa45004d860" (UID: "f9b4e54a-b150-415f-b266-caa45004d860"). InnerVolumeSpecName "kube-api-access-5hr84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:00:42.833426 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:42.833365    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9b4e54a-b150-415f-b266-caa45004d860-tls-certs" pod "f9b4e54a-b150-415f-b266-caa45004d860" (UID: "f9b4e54a-b150-415f-b266-caa45004d860"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:00:42.934157 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:42.934129    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hr84\" (UniqueName: \"kubernetes.io/projected/f9b4e54a-b150-415f-b266-caa45004d860-kube-api-access-5hr84\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:42.934306 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:42.934161    4645 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9b4e54a-b150-415f-b266-caa45004d860-dshm\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:42.934306 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:42.934177    4645 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9b4e54a-b150-415f-b266-caa45004d860-tls-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:42.934306 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:42.934251    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl"
Oct 31 20:00:43.146641 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:43.146612    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:43.206164 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:43.206137    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bd1c9f-6dbbc4dc79-v5wmk"]
Oct 31 20:00:43.206958 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:43.206939    4645 cpu_manager.go:414] "containerMap: removing container" podUID="f9b4e54a-b150-415f-b266-caa45004d860" containerName="main"
Oct 31 20:00:43.207030 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:43.206959    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9b4e54a-b150-415f-b266-caa45004d860" containerName="main"
Oct 31 20:00:43.207068 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:43.207059    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9b4e54a-b150-415f-b266-caa45004d860" containerName="main"
Oct 31 20:00:43.211930 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:43.211910    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd1c9f-6dbbc4dc79-v5wmk"
Oct 31 20:00:43.226602 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:43.222105    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bd1c9f-66bb876fcf-qnnw8"]
Oct 31 20:00:43.234460 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:43.234422    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-bd1c9f\""
Oct 31 20:00:43.240132 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:43.240104    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spvd6\" (UniqueName: \"kubernetes.io/projected/263d2ffb-ab33-432a-83cd-dc73f244bc82-kube-api-access-spvd6\") pod \"payload-pre-processing-e2e-ipp-a-bd1c9f-6dbbc4dc79-v5wmk\" (UID: \"263d2ffb-ab33-432a-83cd-dc73f244bc82\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd1c9f-6dbbc4dc79-v5wmk"
Oct 31 20:00:43.240272 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:43.240196    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/263d2ffb-ab33-432a-83cd-dc73f244bc82-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bd1c9f-6dbbc4dc79-v5wmk\" (UID: \"263d2ffb-ab33-432a-83cd-dc73f244bc82\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd1c9f-6dbbc4dc79-v5wmk"
Oct 31 20:00:43.251643 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:43.244479    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bd1c9f-6dbbc4dc79-v5wmk"]
Oct 31 20:00:43.251643 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:43.244587    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bd1c9f-66bb876fcf-qnnw8"
Oct 31 20:00:43.259963 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:43.259936    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bd1c9f-66bb876fcf-qnnw8"]
Oct 31 20:00:43.266665 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:43.266643    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-bd1c9f-dockercfg-wdbx8\""
Oct 31 20:00:43.284236 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:00:43.284211    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9deed825_0c5f_4950_96e4_098a01b928f4.slice/crio-e1d1970e38322b81a328aa1bfa7c8115d30bd19b5303dfa7fe63400d6faa690d WatchSource:0}: Error finding container e1d1970e38322b81a328aa1bfa7c8115d30bd19b5303dfa7fe63400d6faa690d: Status 404 returned error can't find the container with id e1d1970e38322b81a328aa1bfa7c8115d30bd19b5303dfa7fe63400d6faa690d
Oct 31 20:00:43.287575 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:43.287536    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:00:43.287667 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:43.287614    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:00:43.287667 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:43.287645    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:00:43.290108 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:43.290080    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl"]
Oct 31 20:00:43.402369 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:43.398239    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:43.402369 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:43.399402    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spvd6\" (UniqueName: \"kubernetes.io/projected/263d2ffb-ab33-432a-83cd-dc73f244bc82-kube-api-access-spvd6\") pod \"payload-pre-processing-e2e-ipp-a-bd1c9f-6dbbc4dc79-v5wmk\" (UID: \"263d2ffb-ab33-432a-83cd-dc73f244bc82\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd1c9f-6dbbc4dc79-v5wmk"
Oct 31 20:00:43.402369 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:43.399488    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/263d2ffb-ab33-432a-83cd-dc73f244bc82-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bd1c9f-6dbbc4dc79-v5wmk\" (UID: \"263d2ffb-ab33-432a-83cd-dc73f244bc82\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd1c9f-6dbbc4dc79-v5wmk"
Oct 31 20:00:43.402369 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:43.399597    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1474c24c-f01e-4b1d-93a0-c5e2d4b1f929-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-bd1c9f-66bb876fcf-qnnw8\" (UID: \"1474c24c-f01e-4b1d-93a0-c5e2d4b1f929\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bd1c9f-66bb876fcf-qnnw8"
Oct 31 20:00:43.402369 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:43.399668    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngnlk\" (UniqueName: \"kubernetes.io/projected/1474c24c-f01e-4b1d-93a0-c5e2d4b1f929-kube-api-access-ngnlk\") pod \"payload-processing-e2e-ipp-a-bd1c9f-66bb876fcf-qnnw8\" (UID: \"1474c24c-f01e-4b1d-93a0-c5e2d4b1f929\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bd1c9f-66bb876fcf-qnnw8"
Oct 31 20:00:43.402369 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:43.401012    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/263d2ffb-ab33-432a-83cd-dc73f244bc82-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bd1c9f-6dbbc4dc79-v5wmk\" (UID: \"263d2ffb-ab33-432a-83cd-dc73f244bc82\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd1c9f-6dbbc4dc79-v5wmk"
Oct 31 20:00:43.442687 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:43.440787    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spvd6\" (UniqueName: \"kubernetes.io/projected/263d2ffb-ab33-432a-83cd-dc73f244bc82-kube-api-access-spvd6\") pod \"payload-pre-processing-e2e-ipp-a-bd1c9f-6dbbc4dc79-v5wmk\" (UID: \"263d2ffb-ab33-432a-83cd-dc73f244bc82\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd1c9f-6dbbc4dc79-v5wmk"
Oct 31 20:00:43.506431 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:43.503490    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngnlk\" (UniqueName: \"kubernetes.io/projected/1474c24c-f01e-4b1d-93a0-c5e2d4b1f929-kube-api-access-ngnlk\") pod \"payload-processing-e2e-ipp-a-bd1c9f-66bb876fcf-qnnw8\" (UID: \"1474c24c-f01e-4b1d-93a0-c5e2d4b1f929\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bd1c9f-66bb876fcf-qnnw8"
Oct 31 20:00:43.506431 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:43.503967    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1474c24c-f01e-4b1d-93a0-c5e2d4b1f929-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-bd1c9f-66bb876fcf-qnnw8\" (UID: \"1474c24c-f01e-4b1d-93a0-c5e2d4b1f929\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bd1c9f-66bb876fcf-qnnw8"
Oct 31 20:00:43.506431 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:43.504988    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1474c24c-f01e-4b1d-93a0-c5e2d4b1f929-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-bd1c9f-66bb876fcf-qnnw8\" (UID: \"1474c24c-f01e-4b1d-93a0-c5e2d4b1f929\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bd1c9f-66bb876fcf-qnnw8"
Oct 31 20:00:43.545352 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:43.535613    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd1c9f-6dbbc4dc79-v5wmk"
Oct 31 20:00:43.545352 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:43.541120    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngnlk\" (UniqueName: \"kubernetes.io/projected/1474c24c-f01e-4b1d-93a0-c5e2d4b1f929-kube-api-access-ngnlk\") pod \"payload-processing-e2e-ipp-a-bd1c9f-66bb876fcf-qnnw8\" (UID: \"1474c24c-f01e-4b1d-93a0-c5e2d4b1f929\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bd1c9f-66bb876fcf-qnnw8"
Oct 31 20:00:43.569392 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:43.568644    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bd1c9f-66bb876fcf-qnnw8"
Oct 31 20:00:43.573380 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:43.573230    4645 generic.go:356] "Generic (PLEG): container finished" podID="f9b4e54a-b150-415f-b266-caa45004d860" containerID="58cadced6ea856f6d2d2ecd0660748c2cd605d10f7a4bd7c7504b26f000ad45d" exitCode=0
Oct 31 20:00:43.577348 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:43.573611    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-4bac5a/rate-test-model-4bac5a-kserve-564dc4dfdf-vv676" event={"ID":"f9b4e54a-b150-415f-b266-caa45004d860","Type":"ContainerDied","Data":"58cadced6ea856f6d2d2ecd0660748c2cd605d10f7a4bd7c7504b26f000ad45d"}
Oct 31 20:00:43.577348 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:43.573663    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-4bac5a/rate-test-model-4bac5a-kserve-564dc4dfdf-vv676" event={"ID":"f9b4e54a-b150-415f-b266-caa45004d860","Type":"ContainerDied","Data":"4768c937071a8140f6903bdc8373b5043059a0b580c8e26805bb54364960702c"}
Oct 31 20:00:43.577348 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:43.573689    4645 scope.go:122] "RemoveContainer" containerID="58cadced6ea856f6d2d2ecd0660748c2cd605d10f7a4bd7c7504b26f000ad45d"
Oct 31 20:00:43.577348 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:43.573876    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-4bac5a/rate-test-model-4bac5a-kserve-564dc4dfdf-vv676"
Oct 31 20:00:43.591343 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:43.591291    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl" event={"ID":"9deed825-0c5f-4950-96e4-098a01b928f4","Type":"ContainerStarted","Data":"e1d1970e38322b81a328aa1bfa7c8115d30bd19b5303dfa7fe63400d6faa690d"}
Oct 31 20:00:43.650622 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:43.650593    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f35e911-c0c8-45a1-862e-f6a55fb88b77" path="/var/lib/kubelet/pods/2f35e911-c0c8-45a1-862e-f6a55fb88b77/volumes"
Oct 31 20:00:43.651494 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:43.651470    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bc15cf8-897f-4501-98f8-1a7abbea383c" path="/var/lib/kubelet/pods/4bc15cf8-897f-4501-98f8-1a7abbea383c/volumes"
Oct 31 20:00:43.674515 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:43.674490    4645 scope.go:122] "RemoveContainer" containerID="58cadced6ea856f6d2d2ecd0660748c2cd605d10f7a4bd7c7504b26f000ad45d"
Oct 31 20:00:43.674903 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:00:43.674879    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58cadced6ea856f6d2d2ecd0660748c2cd605d10f7a4bd7c7504b26f000ad45d\": container with ID starting with 58cadced6ea856f6d2d2ecd0660748c2cd605d10f7a4bd7c7504b26f000ad45d not found: ID does not exist" containerID="58cadced6ea856f6d2d2ecd0660748c2cd605d10f7a4bd7c7504b26f000ad45d"
Oct 31 20:00:43.675022 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:43.674926    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58cadced6ea856f6d2d2ecd0660748c2cd605d10f7a4bd7c7504b26f000ad45d"} err="failed to get container status \"58cadced6ea856f6d2d2ecd0660748c2cd605d10f7a4bd7c7504b26f000ad45d\": rpc error: code = NotFound desc = could not find container \"58cadced6ea856f6d2d2ecd0660748c2cd605d10f7a4bd7c7504b26f000ad45d\": container with ID starting with 58cadced6ea856f6d2d2ecd0660748c2cd605d10f7a4bd7c7504b26f000ad45d not found: ID does not exist"
Oct 31 20:00:43.717864 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:43.717839    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-4bac5a/rate-test-model-4bac5a-kserve-564dc4dfdf-vv676"]
Oct 31 20:00:43.752428 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:43.752401    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-4bac5a/rate-test-model-4bac5a-kserve-564dc4dfdf-vv676"]
Oct 31 20:00:43.768741 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:43.768609    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:43.882385 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:43.882228    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:44.001742 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:43.999081    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bd1c9f-66bb876fcf-qnnw8"]
Oct 31 20:00:44.028513 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:44.028485    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bd1c9f-6dbbc4dc79-v5wmk"]
Oct 31 20:00:44.036814 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:00:44.036784    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod263d2ffb_ab33_432a_83cd_dc73f244bc82.slice/crio-93a95f6c14b54016d9e366c9baee6a992d05ac8ee5ad63629e43732dd7b5d03e WatchSource:0}: Error finding container 93a95f6c14b54016d9e366c9baee6a992d05ac8ee5ad63629e43732dd7b5d03e: Status 404 returned error can't find the container with id 93a95f6c14b54016d9e366c9baee6a992d05ac8ee5ad63629e43732dd7b5d03e
Oct 31 20:00:44.683942 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:44.683888    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bd1c9f-66bb876fcf-qnnw8" event={"ID":"1474c24c-f01e-4b1d-93a0-c5e2d4b1f929","Type":"ContainerStarted","Data":"5fa51893c657bd2c2700aa6c5e6ba19827c6736d440c0d852de467b67936b238"}
Oct 31 20:00:44.684517 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:44.683998    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bd1c9f-66bb876fcf-qnnw8" event={"ID":"1474c24c-f01e-4b1d-93a0-c5e2d4b1f929","Type":"ContainerStarted","Data":"2dfa62eab31978818bc82184f896621f209830728844f8681580554f47374476"}
Oct 31 20:00:44.684517 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:44.684316    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-bd1c9f-66bb876fcf-qnnw8"
Oct 31 20:00:44.686430 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:44.686389    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd1c9f-6dbbc4dc79-v5wmk" event={"ID":"263d2ffb-ab33-432a-83cd-dc73f244bc82","Type":"ContainerStarted","Data":"5c9d41c0c25e9551c365d3a92514a6448b4f837bad42d886ba13b720c164c8fc"}
Oct 31 20:00:44.686570 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:44.686446    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd1c9f-6dbbc4dc79-v5wmk"
Oct 31 20:00:44.686570 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:44.686471    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd1c9f-6dbbc4dc79-v5wmk" event={"ID":"263d2ffb-ab33-432a-83cd-dc73f244bc82","Type":"ContainerStarted","Data":"93a95f6c14b54016d9e366c9baee6a992d05ac8ee5ad63629e43732dd7b5d03e"}
Oct 31 20:00:44.689510 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:44.689485    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl" event={"ID":"9deed825-0c5f-4950-96e4-098a01b928f4","Type":"ContainerStarted","Data":"d61f9ae3975309d54fda8264fda0f42f298e67f4b14db44a3365d1ad9257cf4b"}
Oct 31 20:00:44.707588 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:44.707548    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-bd1c9f-66bb876fcf-qnnw8" podStartSLOduration=1.707536854 podStartE2EDuration="1.707536854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:00:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:00:44.704915777 +0000 UTC m=+2475.278149906" watchObservedRunningTime="2026-10-31 20:00:44.707536854 +0000 UTC m=+2475.280770980"
Oct 31 20:00:44.728181 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:44.728138    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd1c9f-6dbbc4dc79-v5wmk" podStartSLOduration=1.7281274309999999 podStartE2EDuration="1.728127431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:00:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:00:44.723727536 +0000 UTC m=+2475.296961665" watchObservedRunningTime="2026-10-31 20:00:44.728127431 +0000 UTC m=+2475.301361593"
Oct 31 20:00:44.739726 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:44.739701    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:44.749315 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:44.749269    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl" podStartSLOduration=2.749256822 podStartE2EDuration="2.749256822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:00:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:00:44.747681985 +0000 UTC m=+2475.320916114" watchObservedRunningTime="2026-10-31 20:00:44.749256822 +0000 UTC m=+2475.322490951"
Oct 31 20:00:44.934545 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:44.934460    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl"
Oct 31 20:00:44.937081 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:44.937045    4645 patch_prober.go:29] interesting pod/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" start-of-body=
Oct 31 20:00:44.937217 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:44.937108    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl" podUID="9deed825-0c5f-4950-96e4-098a01b928f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused"
Oct 31 20:00:45.621046 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:45.621012    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9b4e54a-b150-415f-b266-caa45004d860" path="/var/lib/kubelet/pods/f9b4e54a-b150-415f-b266-caa45004d860/volumes"
Oct 31 20:00:45.681822 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:45.681793    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:45.935200 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:45.935169    4645 patch_prober.go:29] interesting pod/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" start-of-body=
Oct 31 20:00:45.935698 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:45.935219    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl" podUID="9deed825-0c5f-4950-96e4-098a01b928f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused"
Oct 31 20:00:46.754727 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:46.754693    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:46.935573 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:46.935537    4645 patch_prober.go:29] interesting pod/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" start-of-body=
Oct 31 20:00:46.935948 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:46.935616    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl" podUID="9deed825-0c5f-4950-96e4-098a01b928f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused"
Oct 31 20:00:47.586787 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:47.586749    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:47.935323 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:47.935289    4645 patch_prober.go:29] interesting pod/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" start-of-body=
Oct 31 20:00:47.935513 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:47.935350    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl" podUID="9deed825-0c5f-4950-96e4-098a01b928f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused"
Oct 31 20:00:48.521059 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:48.521022    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:48.934846 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:48.934817    4645 patch_prober.go:29] interesting pod/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" start-of-body=
Oct 31 20:00:48.935048 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:48.934892    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl" podUID="9deed825-0c5f-4950-96e4-098a01b928f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused"
Oct 31 20:00:49.291103 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:49.290683    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c173d8-8c8c9bf69-d6xbz"]
Oct 31 20:00:49.297503 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:49.297470    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c173d8-56dc894d7f-pgf8r"]
Oct 31 20:00:49.382945 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:49.382908    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:49.513358 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:49.513310    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:49.666407 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:49.662419    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:49.885354 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:49.885285    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:49.935146 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:49.935058    4645 patch_prober.go:29] interesting pod/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" start-of-body=
Oct 31 20:00:49.935482 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:49.935449    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl" podUID="9deed825-0c5f-4950-96e4-098a01b928f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused"
Oct 31 20:00:49.993095 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:49.993049    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:50.211175 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:50.211081    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n"]
Oct 31 20:00:50.211390 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:50.211354    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n" podUID="f54f6ce5-712d-430a-be69-8ec4a29e44d0" containerName="istio-proxy" containerID="cri-o://4ba94447b82efb89836e1cc98145b9fb1404c061978e32f68989cef941847038" gracePeriod=30
Oct 31 20:00:50.244627 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:50.244580    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:50.688062 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:50.688027    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:50.804963 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:50.804922    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:50.986982 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:50.986897    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl"
Oct 31 20:00:50.987390 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:50.987362    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl"
Oct 31 20:00:50.988401 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:50.988374    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl"
Oct 31 20:00:51.687039 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:51.686984    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-ae2e60/rate-test-model-ae2e60-kserve-744475655c-xk2vp" podUID="44f86c77-4686-48fe-8cf3-cc0e688af7a4" containerName="main" containerID="cri-o://84b053c3f262360af868dd7a162b1266fc3c7d2d4f764dcd68807583fbe1ab35" gracePeriod=45
Oct 31 20:00:51.980967 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:51.980944    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-ae2e60/rate-test-model-ae2e60-kserve-744475655c-xk2vp"
Oct 31 20:00:52.124339 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:52.124295    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44f86c77-4686-48fe-8cf3-cc0e688af7a4-home\" (UniqueName: \"kubernetes.io/empty-dir/44f86c77-4686-48fe-8cf3-cc0e688af7a4-home\") pod \"44f86c77-4686-48fe-8cf3-cc0e688af7a4\" (UID: \"44f86c77-4686-48fe-8cf3-cc0e688af7a4\") "
Oct 31 20:00:52.124535 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:52.124362    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44f86c77-4686-48fe-8cf3-cc0e688af7a4-tls-certs\" (UniqueName: \"kubernetes.io/secret/44f86c77-4686-48fe-8cf3-cc0e688af7a4-tls-certs\") pod \"44f86c77-4686-48fe-8cf3-cc0e688af7a4\" (UID: \"44f86c77-4686-48fe-8cf3-cc0e688af7a4\") "
Oct 31 20:00:52.124535 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:52.124498    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44f86c77-4686-48fe-8cf3-cc0e688af7a4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44f86c77-4686-48fe-8cf3-cc0e688af7a4-model-cache\") pod \"44f86c77-4686-48fe-8cf3-cc0e688af7a4\" (UID: \"44f86c77-4686-48fe-8cf3-cc0e688af7a4\") "
Oct 31 20:00:52.124535 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:52.124520    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44f86c77-4686-48fe-8cf3-cc0e688af7a4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/44f86c77-4686-48fe-8cf3-cc0e688af7a4-dshm\") pod \"44f86c77-4686-48fe-8cf3-cc0e688af7a4\" (UID: \"44f86c77-4686-48fe-8cf3-cc0e688af7a4\") "
Oct 31 20:00:52.124986 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:52.124575    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44f86c77-4686-48fe-8cf3-cc0e688af7a4-home" pod "44f86c77-4686-48fe-8cf3-cc0e688af7a4" (UID: "44f86c77-4686-48fe-8cf3-cc0e688af7a4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:00:52.124986 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:52.124656    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44f86c77-4686-48fe-8cf3-cc0e688af7a4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44f86c77-4686-48fe-8cf3-cc0e688af7a4-tmp-dir\") pod \"44f86c77-4686-48fe-8cf3-cc0e688af7a4\" (UID: \"44f86c77-4686-48fe-8cf3-cc0e688af7a4\") "
Oct 31 20:00:52.124986 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:52.124709    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44f86c77-4686-48fe-8cf3-cc0e688af7a4-kube-api-access-gstzb\" (UniqueName: \"kubernetes.io/projected/44f86c77-4686-48fe-8cf3-cc0e688af7a4-kube-api-access-gstzb\") pod \"44f86c77-4686-48fe-8cf3-cc0e688af7a4\" (UID: \"44f86c77-4686-48fe-8cf3-cc0e688af7a4\") "
Oct 31 20:00:52.124986 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:52.124758    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44f86c77-4686-48fe-8cf3-cc0e688af7a4-model-cache" pod "44f86c77-4686-48fe-8cf3-cc0e688af7a4" (UID: "44f86c77-4686-48fe-8cf3-cc0e688af7a4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:00:52.124986 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:52.124907    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44f86c77-4686-48fe-8cf3-cc0e688af7a4-tmp-dir" pod "44f86c77-4686-48fe-8cf3-cc0e688af7a4" (UID: "44f86c77-4686-48fe-8cf3-cc0e688af7a4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:00:52.125249 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:52.125225    4645 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44f86c77-4686-48fe-8cf3-cc0e688af7a4-tmp-dir\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:52.125249 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:52.125252    4645 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/44f86c77-4686-48fe-8cf3-cc0e688af7a4-home\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:52.125460 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:52.125264    4645 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/44f86c77-4686-48fe-8cf3-cc0e688af7a4-model-cache\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:52.127006 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:52.126981    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44f86c77-4686-48fe-8cf3-cc0e688af7a4-tls-certs" pod "44f86c77-4686-48fe-8cf3-cc0e688af7a4" (UID: "44f86c77-4686-48fe-8cf3-cc0e688af7a4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:00:52.127113 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:52.127070    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44f86c77-4686-48fe-8cf3-cc0e688af7a4-dshm" pod "44f86c77-4686-48fe-8cf3-cc0e688af7a4" (UID: "44f86c77-4686-48fe-8cf3-cc0e688af7a4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:00:52.127187 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:52.127124    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44f86c77-4686-48fe-8cf3-cc0e688af7a4-kube-api-access-gstzb" pod "44f86c77-4686-48fe-8cf3-cc0e688af7a4" (UID: "44f86c77-4686-48fe-8cf3-cc0e688af7a4"). InnerVolumeSpecName "kube-api-access-gstzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:00:52.219699 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:52.219641    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:52.230130 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:52.226768    4645 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/44f86c77-4686-48fe-8cf3-cc0e688af7a4-tls-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:52.230130 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:52.226817    4645 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/44f86c77-4686-48fe-8cf3-cc0e688af7a4-dshm\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:52.230130 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:52.226832    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gstzb\" (UniqueName: \"kubernetes.io/projected/44f86c77-4686-48fe-8cf3-cc0e688af7a4-kube-api-access-gstzb\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:52.768300 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:52.768262    4645 generic.go:356] "Generic (PLEG): container finished" podID="44f86c77-4686-48fe-8cf3-cc0e688af7a4" containerID="84b053c3f262360af868dd7a162b1266fc3c7d2d4f764dcd68807583fbe1ab35" exitCode=0
Oct 31 20:00:52.768540 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:52.768301    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-ae2e60/rate-test-model-ae2e60-kserve-744475655c-xk2vp"
Oct 31 20:00:52.768540 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:52.768358    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-ae2e60/rate-test-model-ae2e60-kserve-744475655c-xk2vp" event={"ID":"44f86c77-4686-48fe-8cf3-cc0e688af7a4","Type":"ContainerDied","Data":"84b053c3f262360af868dd7a162b1266fc3c7d2d4f764dcd68807583fbe1ab35"}
Oct 31 20:00:52.768540 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:52.768404    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-ae2e60/rate-test-model-ae2e60-kserve-744475655c-xk2vp" event={"ID":"44f86c77-4686-48fe-8cf3-cc0e688af7a4","Type":"ContainerDied","Data":"9c4602ce6a49471da8257b4e3472e0c0932e552af560c466d513b57478d8d924"}
Oct 31 20:00:52.768540 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:52.768421    4645 scope.go:122] "RemoveContainer" containerID="84b053c3f262360af868dd7a162b1266fc3c7d2d4f764dcd68807583fbe1ab35"
Oct 31 20:00:52.780665 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:52.780645    4645 scope.go:122] "RemoveContainer" containerID="84b053c3f262360af868dd7a162b1266fc3c7d2d4f764dcd68807583fbe1ab35"
Oct 31 20:00:52.781021 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:00:52.780948    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84b053c3f262360af868dd7a162b1266fc3c7d2d4f764dcd68807583fbe1ab35\": container with ID starting with 84b053c3f262360af868dd7a162b1266fc3c7d2d4f764dcd68807583fbe1ab35 not found: ID does not exist" containerID="84b053c3f262360af868dd7a162b1266fc3c7d2d4f764dcd68807583fbe1ab35"
Oct 31 20:00:52.781021 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:52.780976    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84b053c3f262360af868dd7a162b1266fc3c7d2d4f764dcd68807583fbe1ab35"} err="failed to get container status \"84b053c3f262360af868dd7a162b1266fc3c7d2d4f764dcd68807583fbe1ab35\": rpc error: code = NotFound desc = could not find container \"84b053c3f262360af868dd7a162b1266fc3c7d2d4f764dcd68807583fbe1ab35\": container with ID starting with 84b053c3f262360af868dd7a162b1266fc3c7d2d4f764dcd68807583fbe1ab35 not found: ID does not exist"
Oct 31 20:00:52.801192 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:52.801170    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-ae2e60/rate-test-model-ae2e60-kserve-744475655c-xk2vp"]
Oct 31 20:00:52.805085 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:52.805065    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-ae2e60/rate-test-model-ae2e60-kserve-744475655c-xk2vp"]
Oct 31 20:00:53.243154 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:53.243102    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:53.619607 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:53.619529    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44f86c77-4686-48fe-8cf3-cc0e688af7a4" path="/var/lib/kubelet/pods/44f86c77-4686-48fe-8cf3-cc0e688af7a4/volumes"
Oct 31 20:00:53.959897 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:53.959865    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:54.291457 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:54.291377    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-c173d8-8c8c9bf69-d6xbz" podUID="0c35e9d4-0b05-42c6-b07b-d76a793639cb" containerName="payload-processing" containerID="cri-o://edb36725084bb94870c72378333e38f2da88ab6fe953881380b7dabeb25bc227" gracePeriod=25
Oct 31 20:00:54.298957 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:54.298916    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c173d8-56dc894d7f-pgf8r" podUID="4a5d740e-e1fd-4461-976e-a3a49ab9232f" containerName="payload-pre-processing" containerID="cri-o://b792558802b97a658d3b5871c37fed245d99b065ddcbbffc0ec5977396e42389" gracePeriod=25
Oct 31 20:00:54.386845 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:00:54.386823    4645 logging.go:55] [core] [Channel #5431 SubChannel #5432]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.248:50052", ServerName: "10.133.0.248:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.248:50052: connect: connection refused"
Oct 31 20:00:54.580120 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:54.580094    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c173d8-8c8c9bf69-d6xbz"
Oct 31 20:00:54.656685 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:54.656658    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c173d8-56dc894d7f-pgf8r"
Oct 31 20:00:54.659680 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:54.659655    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c35e9d4-0b05-42c6-b07b-d76a793639cb-kube-api-access-2ckrk\" (UniqueName: \"kubernetes.io/projected/0c35e9d4-0b05-42c6-b07b-d76a793639cb-kube-api-access-2ckrk\") pod \"0c35e9d4-0b05-42c6-b07b-d76a793639cb\" (UID: \"0c35e9d4-0b05-42c6-b07b-d76a793639cb\") "
Oct 31 20:00:54.659830 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:54.659812    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c35e9d4-0b05-42c6-b07b-d76a793639cb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c35e9d4-0b05-42c6-b07b-d76a793639cb-plugins-config-volume\") pod \"0c35e9d4-0b05-42c6-b07b-d76a793639cb\" (UID: \"0c35e9d4-0b05-42c6-b07b-d76a793639cb\") "
Oct 31 20:00:54.659891 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:54.659863    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a5d740e-e1fd-4461-976e-a3a49ab9232f-kube-api-access-q7ckh\" (UniqueName: \"kubernetes.io/projected/4a5d740e-e1fd-4461-976e-a3a49ab9232f-kube-api-access-q7ckh\") pod \"4a5d740e-e1fd-4461-976e-a3a49ab9232f\" (UID: \"4a5d740e-e1fd-4461-976e-a3a49ab9232f\") "
Oct 31 20:00:54.659960 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:54.659945    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a5d740e-e1fd-4461-976e-a3a49ab9232f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a5d740e-e1fd-4461-976e-a3a49ab9232f-plugins-config-volume\") pod \"4a5d740e-e1fd-4461-976e-a3a49ab9232f\" (UID: \"4a5d740e-e1fd-4461-976e-a3a49ab9232f\") "
Oct 31 20:00:54.660271 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:54.660244    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c35e9d4-0b05-42c6-b07b-d76a793639cb-plugins-config-volume" pod "0c35e9d4-0b05-42c6-b07b-d76a793639cb" (UID: "0c35e9d4-0b05-42c6-b07b-d76a793639cb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:00:54.660741 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:54.660713    4645 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c35e9d4-0b05-42c6-b07b-d76a793639cb-plugins-config-volume\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:54.662524 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:54.662492    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c35e9d4-0b05-42c6-b07b-d76a793639cb-kube-api-access-2ckrk" pod "0c35e9d4-0b05-42c6-b07b-d76a793639cb" (UID: "0c35e9d4-0b05-42c6-b07b-d76a793639cb"). InnerVolumeSpecName "kube-api-access-2ckrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:00:54.662724 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:54.662703    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a5d740e-e1fd-4461-976e-a3a49ab9232f-plugins-config-volume" pod "4a5d740e-e1fd-4461-976e-a3a49ab9232f" (UID: "4a5d740e-e1fd-4461-976e-a3a49ab9232f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:00:54.662868 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:54.662848    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a5d740e-e1fd-4461-976e-a3a49ab9232f-kube-api-access-q7ckh" pod "4a5d740e-e1fd-4461-976e-a3a49ab9232f" (UID: "4a5d740e-e1fd-4461-976e-a3a49ab9232f"). InnerVolumeSpecName "kube-api-access-q7ckh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:00:54.724508 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:54.723048    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:54.771585 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:54.767296    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ckrk\" (UniqueName: \"kubernetes.io/projected/0c35e9d4-0b05-42c6-b07b-d76a793639cb-kube-api-access-2ckrk\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:54.771585 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:54.767361    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7ckh\" (UniqueName: \"kubernetes.io/projected/4a5d740e-e1fd-4461-976e-a3a49ab9232f-kube-api-access-q7ckh\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:54.771585 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:54.767377    4645 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a5d740e-e1fd-4461-976e-a3a49ab9232f-plugins-config-volume\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:54.808734 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:54.808605    4645 generic.go:356] "Generic (PLEG): container finished" podID="0c35e9d4-0b05-42c6-b07b-d76a793639cb" containerID="edb36725084bb94870c72378333e38f2da88ab6fe953881380b7dabeb25bc227" exitCode=0
Oct 31 20:00:54.809099 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:54.808667    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c173d8-8c8c9bf69-d6xbz" event={"ID":"0c35e9d4-0b05-42c6-b07b-d76a793639cb","Type":"ContainerDied","Data":"edb36725084bb94870c72378333e38f2da88ab6fe953881380b7dabeb25bc227"}
Oct 31 20:00:54.809228 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:54.809145    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c173d8-8c8c9bf69-d6xbz" event={"ID":"0c35e9d4-0b05-42c6-b07b-d76a793639cb","Type":"ContainerDied","Data":"0bd0d31befe08b8f394fa54b9121fcac173921a83ff3de016055f8694e874979"}
Oct 31 20:00:54.809228 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:54.808691    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c173d8-8c8c9bf69-d6xbz"
Oct 31 20:00:54.809228 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:54.809184    4645 scope.go:122] "RemoveContainer" containerID="edb36725084bb94870c72378333e38f2da88ab6fe953881380b7dabeb25bc227"
Oct 31 20:00:54.811178 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:54.811138    4645 generic.go:356] "Generic (PLEG): container finished" podID="4a5d740e-e1fd-4461-976e-a3a49ab9232f" containerID="b792558802b97a658d3b5871c37fed245d99b065ddcbbffc0ec5977396e42389" exitCode=0
Oct 31 20:00:54.811300 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:54.811174    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c173d8-56dc894d7f-pgf8r" event={"ID":"4a5d740e-e1fd-4461-976e-a3a49ab9232f","Type":"ContainerDied","Data":"b792558802b97a658d3b5871c37fed245d99b065ddcbbffc0ec5977396e42389"}
Oct 31 20:00:54.811300 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:54.811239    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c173d8-56dc894d7f-pgf8r" event={"ID":"4a5d740e-e1fd-4461-976e-a3a49ab9232f","Type":"ContainerDied","Data":"e4c01f105f75d5bf352b812edb6bf86db37327d4ff067e719d2bce52b3ce6a3d"}
Oct 31 20:00:54.811300 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:54.811184    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c173d8-56dc894d7f-pgf8r"
Oct 31 20:00:54.822025 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:54.822007    4645 scope.go:122] "RemoveContainer" containerID="edb36725084bb94870c72378333e38f2da88ab6fe953881380b7dabeb25bc227"
Oct 31 20:00:54.822293 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:00:54.822276    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edb36725084bb94870c72378333e38f2da88ab6fe953881380b7dabeb25bc227\": container with ID starting with edb36725084bb94870c72378333e38f2da88ab6fe953881380b7dabeb25bc227 not found: ID does not exist" containerID="edb36725084bb94870c72378333e38f2da88ab6fe953881380b7dabeb25bc227"
Oct 31 20:00:54.822372 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:54.822303    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edb36725084bb94870c72378333e38f2da88ab6fe953881380b7dabeb25bc227"} err="failed to get container status \"edb36725084bb94870c72378333e38f2da88ab6fe953881380b7dabeb25bc227\": rpc error: code = NotFound desc = could not find container \"edb36725084bb94870c72378333e38f2da88ab6fe953881380b7dabeb25bc227\": container with ID starting with edb36725084bb94870c72378333e38f2da88ab6fe953881380b7dabeb25bc227 not found: ID does not exist"
Oct 31 20:00:54.822431 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:54.822324    4645 scope.go:122] "RemoveContainer" containerID="b792558802b97a658d3b5871c37fed245d99b065ddcbbffc0ec5977396e42389"
Oct 31 20:00:54.832134 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:54.832117    4645 scope.go:122] "RemoveContainer" containerID="b792558802b97a658d3b5871c37fed245d99b065ddcbbffc0ec5977396e42389"
Oct 31 20:00:54.832490 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:00:54.832471    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b792558802b97a658d3b5871c37fed245d99b065ddcbbffc0ec5977396e42389\": container with ID starting with b792558802b97a658d3b5871c37fed245d99b065ddcbbffc0ec5977396e42389 not found: ID does not exist" containerID="b792558802b97a658d3b5871c37fed245d99b065ddcbbffc0ec5977396e42389"
Oct 31 20:00:54.832575 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:54.832494    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b792558802b97a658d3b5871c37fed245d99b065ddcbbffc0ec5977396e42389"} err="failed to get container status \"b792558802b97a658d3b5871c37fed245d99b065ddcbbffc0ec5977396e42389\": rpc error: code = NotFound desc = could not find container \"b792558802b97a658d3b5871c37fed245d99b065ddcbbffc0ec5977396e42389\": container with ID starting with b792558802b97a658d3b5871c37fed245d99b065ddcbbffc0ec5977396e42389 not found: ID does not exist"
Oct 31 20:00:54.844348 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:54.839709    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c173d8-56dc894d7f-pgf8r"]
Oct 31 20:00:54.849523 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:54.849494    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c173d8-56dc894d7f-pgf8r"]
Oct 31 20:00:54.864971 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:54.864944    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c173d8-8c8c9bf69-d6xbz"]
Oct 31 20:00:54.868758 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:54.868737    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c173d8-8c8c9bf69-d6xbz"]
Oct 31 20:00:55.386586 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:55.386544    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-a-c173d8-8c8c9bf69-d6xbz" podUID="0c35e9d4-0b05-42c6-b07b-d76a793639cb" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.248:50052\" within 1s: context deadline exceeded"
Oct 31 20:00:55.443709 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:55.443609    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:55.568478 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:55.568452    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n"
Oct 31 20:00:55.592716 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:55.592687    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f54f6ce5-712d-430a-be69-8ec4a29e44d0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f54f6ce5-712d-430a-be69-8ec4a29e44d0-credential-socket\") pod \"f54f6ce5-712d-430a-be69-8ec4a29e44d0\" (UID: \"f54f6ce5-712d-430a-be69-8ec4a29e44d0\") "
Oct 31 20:00:55.592919 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:55.592782    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f54f6ce5-712d-430a-be69-8ec4a29e44d0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f54f6ce5-712d-430a-be69-8ec4a29e44d0-istio-podinfo\") pod \"f54f6ce5-712d-430a-be69-8ec4a29e44d0\" (UID: \"f54f6ce5-712d-430a-be69-8ec4a29e44d0\") "
Oct 31 20:00:55.592919 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:55.592818    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f54f6ce5-712d-430a-be69-8ec4a29e44d0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f54f6ce5-712d-430a-be69-8ec4a29e44d0-istio-data\") pod \"f54f6ce5-712d-430a-be69-8ec4a29e44d0\" (UID: \"f54f6ce5-712d-430a-be69-8ec4a29e44d0\") "
Oct 31 20:00:55.592919 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:55.592896    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f54f6ce5-712d-430a-be69-8ec4a29e44d0-kube-api-access-hfwzg\" (UniqueName: \"kubernetes.io/projected/f54f6ce5-712d-430a-be69-8ec4a29e44d0-kube-api-access-hfwzg\") pod \"f54f6ce5-712d-430a-be69-8ec4a29e44d0\" (UID: \"f54f6ce5-712d-430a-be69-8ec4a29e44d0\") "
Oct 31 20:00:55.593160 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:55.592959    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f54f6ce5-712d-430a-be69-8ec4a29e44d0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f54f6ce5-712d-430a-be69-8ec4a29e44d0-workload-socket\") pod \"f54f6ce5-712d-430a-be69-8ec4a29e44d0\" (UID: \"f54f6ce5-712d-430a-be69-8ec4a29e44d0\") "
Oct 31 20:00:55.593160 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:55.593021    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f54f6ce5-712d-430a-be69-8ec4a29e44d0-credential-socket" pod "f54f6ce5-712d-430a-be69-8ec4a29e44d0" (UID: "f54f6ce5-712d-430a-be69-8ec4a29e44d0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:00:55.593160 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:55.593054    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f54f6ce5-712d-430a-be69-8ec4a29e44d0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f54f6ce5-712d-430a-be69-8ec4a29e44d0-istiod-ca-cert\") pod \"f54f6ce5-712d-430a-be69-8ec4a29e44d0\" (UID: \"f54f6ce5-712d-430a-be69-8ec4a29e44d0\") "
Oct 31 20:00:55.593381 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:55.593259    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f54f6ce5-712d-430a-be69-8ec4a29e44d0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f54f6ce5-712d-430a-be69-8ec4a29e44d0-workload-certs\") pod \"f54f6ce5-712d-430a-be69-8ec4a29e44d0\" (UID: \"f54f6ce5-712d-430a-be69-8ec4a29e44d0\") "
Oct 31 20:00:55.593381 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:55.593315    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f54f6ce5-712d-430a-be69-8ec4a29e44d0-istio-token\" (UniqueName: \"kubernetes.io/projected/f54f6ce5-712d-430a-be69-8ec4a29e44d0-istio-token\") pod \"f54f6ce5-712d-430a-be69-8ec4a29e44d0\" (UID: \"f54f6ce5-712d-430a-be69-8ec4a29e44d0\") "
Oct 31 20:00:55.593554 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:55.593400    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f54f6ce5-712d-430a-be69-8ec4a29e44d0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f54f6ce5-712d-430a-be69-8ec4a29e44d0-istio-envoy\") pod \"f54f6ce5-712d-430a-be69-8ec4a29e44d0\" (UID: \"f54f6ce5-712d-430a-be69-8ec4a29e44d0\") "
Oct 31 20:00:55.594447 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:55.594168    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f54f6ce5-712d-430a-be69-8ec4a29e44d0-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:55.594447 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:55.593257    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f54f6ce5-712d-430a-be69-8ec4a29e44d0-istio-data" pod "f54f6ce5-712d-430a-be69-8ec4a29e44d0" (UID: "f54f6ce5-712d-430a-be69-8ec4a29e44d0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:00:55.594447 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:55.593994    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f54f6ce5-712d-430a-be69-8ec4a29e44d0-istiod-ca-cert" pod "f54f6ce5-712d-430a-be69-8ec4a29e44d0" (UID: "f54f6ce5-712d-430a-be69-8ec4a29e44d0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:00:55.594686 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:55.594542    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f54f6ce5-712d-430a-be69-8ec4a29e44d0-workload-socket" pod "f54f6ce5-712d-430a-be69-8ec4a29e44d0" (UID: "f54f6ce5-712d-430a-be69-8ec4a29e44d0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:00:55.594686 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:55.594615    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f54f6ce5-712d-430a-be69-8ec4a29e44d0-workload-certs" pod "f54f6ce5-712d-430a-be69-8ec4a29e44d0" (UID: "f54f6ce5-712d-430a-be69-8ec4a29e44d0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:00:55.597143 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:55.597106    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f54f6ce5-712d-430a-be69-8ec4a29e44d0-istio-podinfo" pod "f54f6ce5-712d-430a-be69-8ec4a29e44d0" (UID: "f54f6ce5-712d-430a-be69-8ec4a29e44d0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 20:00:55.597269 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:55.597249    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f54f6ce5-712d-430a-be69-8ec4a29e44d0-kube-api-access-hfwzg" pod "f54f6ce5-712d-430a-be69-8ec4a29e44d0" (UID: "f54f6ce5-712d-430a-be69-8ec4a29e44d0"). InnerVolumeSpecName "kube-api-access-hfwzg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:00:55.600921 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:55.598722    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f54f6ce5-712d-430a-be69-8ec4a29e44d0-istio-token" pod "f54f6ce5-712d-430a-be69-8ec4a29e44d0" (UID: "f54f6ce5-712d-430a-be69-8ec4a29e44d0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:00:55.600921 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:55.599081    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f54f6ce5-712d-430a-be69-8ec4a29e44d0-istio-envoy" pod "f54f6ce5-712d-430a-be69-8ec4a29e44d0" (UID: "f54f6ce5-712d-430a-be69-8ec4a29e44d0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:00:55.623293 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:55.623262    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c35e9d4-0b05-42c6-b07b-d76a793639cb" path="/var/lib/kubelet/pods/0c35e9d4-0b05-42c6-b07b-d76a793639cb/volumes"
Oct 31 20:00:55.624145 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:55.624120    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a5d740e-e1fd-4461-976e-a3a49ab9232f" path="/var/lib/kubelet/pods/4a5d740e-e1fd-4461-976e-a3a49ab9232f/volumes"
Oct 31 20:00:55.659851 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:55.659814    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669"]
Oct 31 20:00:55.666950 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:55.666926    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4"]
Oct 31 20:00:55.690156 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:55.690106    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x"]
Oct 31 20:00:55.696644 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:55.696614    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f54f6ce5-712d-430a-be69-8ec4a29e44d0-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:55.696797 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:55.696647    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f54f6ce5-712d-430a-be69-8ec4a29e44d0-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:55.696797 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:55.696662    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f54f6ce5-712d-430a-be69-8ec4a29e44d0-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:55.696797 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:55.696677    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f54f6ce5-712d-430a-be69-8ec4a29e44d0-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:55.696797 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:55.696691    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f54f6ce5-712d-430a-be69-8ec4a29e44d0-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:55.696797 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:55.696705    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfwzg\" (UniqueName: \"kubernetes.io/projected/f54f6ce5-712d-430a-be69-8ec4a29e44d0-kube-api-access-hfwzg\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:55.696797 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:55.696721    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f54f6ce5-712d-430a-be69-8ec4a29e44d0-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:55.696797 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:55.696734    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f54f6ce5-712d-430a-be69-8ec4a29e44d0-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:00:55.713844 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:55.713823    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-bd1c9f-66bb876fcf-qnnw8"
Oct 31 20:00:55.718055 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:55.718031    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd1c9f-6dbbc4dc79-v5wmk"
Oct 31 20:00:55.774812 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:55.774782    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:55.829376 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:55.829282    4645 generic.go:356] "Generic (PLEG): container finished" podID="f54f6ce5-712d-430a-be69-8ec4a29e44d0" containerID="4ba94447b82efb89836e1cc98145b9fb1404c061978e32f68989cef941847038" exitCode=0
Oct 31 20:00:55.829376 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:55.829354    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n"
Oct 31 20:00:55.829716 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:55.829409    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n" event={"ID":"f54f6ce5-712d-430a-be69-8ec4a29e44d0","Type":"ContainerDied","Data":"4ba94447b82efb89836e1cc98145b9fb1404c061978e32f68989cef941847038"}
Oct 31 20:00:55.829716 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:55.829494    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n" event={"ID":"f54f6ce5-712d-430a-be69-8ec4a29e44d0","Type":"ContainerDied","Data":"a6be4e00341227506d05c169f9b766396d20c99da89adf05141a4ecee7aaf4d1"}
Oct 31 20:00:55.829716 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:55.829530    4645 scope.go:122] "RemoveContainer" containerID="4ba94447b82efb89836e1cc98145b9fb1404c061978e32f68989cef941847038"
Oct 31 20:00:55.843981 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:55.843962    4645 scope.go:122] "RemoveContainer" containerID="f7a48a62a3970b16f21af40bb7b30de5f7d063f23813c6ccd68033d4c6e884c9"
Oct 31 20:00:55.854736 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:55.854718    4645 scope.go:122] "RemoveContainer" containerID="4ba94447b82efb89836e1cc98145b9fb1404c061978e32f68989cef941847038"
Oct 31 20:00:55.855022 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:00:55.855003    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ba94447b82efb89836e1cc98145b9fb1404c061978e32f68989cef941847038\": container with ID starting with 4ba94447b82efb89836e1cc98145b9fb1404c061978e32f68989cef941847038 not found: ID does not exist" containerID="4ba94447b82efb89836e1cc98145b9fb1404c061978e32f68989cef941847038"
Oct 31 20:00:55.855098 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:55.855028    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ba94447b82efb89836e1cc98145b9fb1404c061978e32f68989cef941847038"} err="failed to get container status \"4ba94447b82efb89836e1cc98145b9fb1404c061978e32f68989cef941847038\": rpc error: code = NotFound desc = could not find container \"4ba94447b82efb89836e1cc98145b9fb1404c061978e32f68989cef941847038\": container with ID starting with 4ba94447b82efb89836e1cc98145b9fb1404c061978e32f68989cef941847038 not found: ID does not exist"
Oct 31 20:00:55.855098 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:55.855045    4645 scope.go:122] "RemoveContainer" containerID="f7a48a62a3970b16f21af40bb7b30de5f7d063f23813c6ccd68033d4c6e884c9"
Oct 31 20:00:55.855310 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:00:55.855290    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7a48a62a3970b16f21af40bb7b30de5f7d063f23813c6ccd68033d4c6e884c9\": container with ID starting with f7a48a62a3970b16f21af40bb7b30de5f7d063f23813c6ccd68033d4c6e884c9 not found: ID does not exist" containerID="f7a48a62a3970b16f21af40bb7b30de5f7d063f23813c6ccd68033d4c6e884c9"
Oct 31 20:00:55.855437 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:55.855316    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7a48a62a3970b16f21af40bb7b30de5f7d063f23813c6ccd68033d4c6e884c9"} err="failed to get container status \"f7a48a62a3970b16f21af40bb7b30de5f7d063f23813c6ccd68033d4c6e884c9\": rpc error: code = NotFound desc = could not find container \"f7a48a62a3970b16f21af40bb7b30de5f7d063f23813c6ccd68033d4c6e884c9\": container with ID starting with f7a48a62a3970b16f21af40bb7b30de5f7d063f23813c6ccd68033d4c6e884c9 not found: ID does not exist"
Oct 31 20:00:55.874337 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:55.874311    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n"]
Oct 31 20:00:55.883547 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:55.883528    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-01afb5-openshift-default-5464b8fc9-qbt9n"]
Oct 31 20:00:56.490246 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:56.490200    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:57.047970 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:57.047939    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:57.148961 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:57.148930    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5f3eff-65c86b7b8b-mc8nv"]
Oct 31 20:00:57.149626 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:57.149610    4645 cpu_manager.go:414] "containerMap: removing container" podUID="44f86c77-4686-48fe-8cf3-cc0e688af7a4" containerName="main"
Oct 31 20:00:57.149685 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:57.149627    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44f86c77-4686-48fe-8cf3-cc0e688af7a4" containerName="main"
Oct 31 20:00:57.149685 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:57.149640    4645 cpu_manager.go:414] "containerMap: removing container" podUID="4a5d740e-e1fd-4461-976e-a3a49ab9232f" containerName="payload-pre-processing"
Oct 31 20:00:57.149685 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:57.149645    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a5d740e-e1fd-4461-976e-a3a49ab9232f" containerName="payload-pre-processing"
Oct 31 20:00:57.149685 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:57.149673    4645 cpu_manager.go:414] "containerMap: removing container" podUID="f54f6ce5-712d-430a-be69-8ec4a29e44d0" containerName="istio-proxy"
Oct 31 20:00:57.149685 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:57.149678    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f54f6ce5-712d-430a-be69-8ec4a29e44d0" containerName="istio-proxy"
Oct 31 20:00:57.149853 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:57.149692    4645 cpu_manager.go:414] "containerMap: removing container" podUID="0c35e9d4-0b05-42c6-b07b-d76a793639cb" containerName="payload-processing"
Oct 31 20:00:57.149853 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:57.149697    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c35e9d4-0b05-42c6-b07b-d76a793639cb" containerName="payload-processing"
Oct 31 20:00:57.149853 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:57.149707    4645 cpu_manager.go:414] "containerMap: removing container" podUID="f54f6ce5-712d-430a-be69-8ec4a29e44d0" containerName="istio-proxy"
Oct 31 20:00:57.149853 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:57.149711    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f54f6ce5-712d-430a-be69-8ec4a29e44d0" containerName="istio-proxy"
Oct 31 20:00:57.149853 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:57.149798    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="f54f6ce5-712d-430a-be69-8ec4a29e44d0" containerName="istio-proxy"
Oct 31 20:00:57.149853 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:57.149809    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="44f86c77-4686-48fe-8cf3-cc0e688af7a4" containerName="main"
Oct 31 20:00:57.149853 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:57.149815    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a5d740e-e1fd-4461-976e-a3a49ab9232f" containerName="payload-pre-processing"
Oct 31 20:00:57.149853 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:57.149826    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="f54f6ce5-712d-430a-be69-8ec4a29e44d0" containerName="istio-proxy"
Oct 31 20:00:57.149853 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:57.149834    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c35e9d4-0b05-42c6-b07b-d76a793639cb" containerName="payload-processing"
Oct 31 20:00:57.155247 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:57.155230    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5f3eff-65c86b7b8b-mc8nv"
Oct 31 20:00:57.163652 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:57.163632    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-5f3eff\""
Oct 31 20:00:57.164244 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:57.164139    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-5f3eff-dockercfg-9hdjb\""
Oct 31 20:00:57.166908 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:57.166878    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:57.170014 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:57.169991    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5f3eff-65c86b7b8b-mc8nv"]
Oct 31 20:00:57.221212 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:57.221173    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5f3eff-7db97588f5-ld47d"]
Oct 31 20:00:57.227558 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:57.227025    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ec291f9-5a7f-4d42-946e-668aeb30a976-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5f3eff-65c86b7b8b-mc8nv\" (UID: \"0ec291f9-5a7f-4d42-946e-668aeb30a976\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5f3eff-65c86b7b8b-mc8nv"
Oct 31 20:00:57.227558 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:57.227234    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmz78\" (UniqueName: \"kubernetes.io/projected/0ec291f9-5a7f-4d42-946e-668aeb30a976-kube-api-access-zmz78\") pod \"payload-processing-e2e-ipp-b-5f3eff-65c86b7b8b-mc8nv\" (UID: \"0ec291f9-5a7f-4d42-946e-668aeb30a976\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5f3eff-65c86b7b8b-mc8nv"
Oct 31 20:00:57.231863 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:57.231838    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5f3eff-7db97588f5-ld47d"
Oct 31 20:00:57.240014 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:57.239987    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5f3eff-7db97588f5-ld47d"]
Oct 31 20:00:57.342403 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:57.329403    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svx27\" (UniqueName: \"kubernetes.io/projected/ca2e9a7f-b3d9-4f2f-90e6-414543715ada-kube-api-access-svx27\") pod \"payload-pre-processing-e2e-ipp-b-5f3eff-7db97588f5-ld47d\" (UID: \"ca2e9a7f-b3d9-4f2f-90e6-414543715ada\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5f3eff-7db97588f5-ld47d"
Oct 31 20:00:57.342403 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:57.329480    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ec291f9-5a7f-4d42-946e-668aeb30a976-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5f3eff-65c86b7b8b-mc8nv\" (UID: \"0ec291f9-5a7f-4d42-946e-668aeb30a976\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5f3eff-65c86b7b8b-mc8nv"
Oct 31 20:00:57.342403 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:57.329567    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca2e9a7f-b3d9-4f2f-90e6-414543715ada-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5f3eff-7db97588f5-ld47d\" (UID: \"ca2e9a7f-b3d9-4f2f-90e6-414543715ada\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5f3eff-7db97588f5-ld47d"
Oct 31 20:00:57.342403 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:57.341299    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ec291f9-5a7f-4d42-946e-668aeb30a976-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5f3eff-65c86b7b8b-mc8nv\" (UID: \"0ec291f9-5a7f-4d42-946e-668aeb30a976\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5f3eff-65c86b7b8b-mc8nv"
Oct 31 20:00:57.371352 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:57.370785    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmz78\" (UniqueName: \"kubernetes.io/projected/0ec291f9-5a7f-4d42-946e-668aeb30a976-kube-api-access-zmz78\") pod \"payload-processing-e2e-ipp-b-5f3eff-65c86b7b8b-mc8nv\" (UID: \"0ec291f9-5a7f-4d42-946e-668aeb30a976\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5f3eff-65c86b7b8b-mc8nv"
Oct 31 20:00:57.476353 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:57.475550    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svx27\" (UniqueName: \"kubernetes.io/projected/ca2e9a7f-b3d9-4f2f-90e6-414543715ada-kube-api-access-svx27\") pod \"payload-pre-processing-e2e-ipp-b-5f3eff-7db97588f5-ld47d\" (UID: \"ca2e9a7f-b3d9-4f2f-90e6-414543715ada\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5f3eff-7db97588f5-ld47d"
Oct 31 20:00:57.476353 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:57.475682    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca2e9a7f-b3d9-4f2f-90e6-414543715ada-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5f3eff-7db97588f5-ld47d\" (UID: \"ca2e9a7f-b3d9-4f2f-90e6-414543715ada\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5f3eff-7db97588f5-ld47d"
Oct 31 20:00:57.476610 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:57.476531    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmz78\" (UniqueName: \"kubernetes.io/projected/0ec291f9-5a7f-4d42-946e-668aeb30a976-kube-api-access-zmz78\") pod \"payload-processing-e2e-ipp-b-5f3eff-65c86b7b8b-mc8nv\" (UID: \"0ec291f9-5a7f-4d42-946e-668aeb30a976\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5f3eff-65c86b7b8b-mc8nv"
Oct 31 20:00:57.500350 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:57.487078    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5f3eff-65c86b7b8b-mc8nv"
Oct 31 20:00:57.534072 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:57.534013    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca2e9a7f-b3d9-4f2f-90e6-414543715ada-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5f3eff-7db97588f5-ld47d\" (UID: \"ca2e9a7f-b3d9-4f2f-90e6-414543715ada\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5f3eff-7db97588f5-ld47d"
Oct 31 20:00:57.587964 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:57.587898    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svx27\" (UniqueName: \"kubernetes.io/projected/ca2e9a7f-b3d9-4f2f-90e6-414543715ada-kube-api-access-svx27\") pod \"payload-pre-processing-e2e-ipp-b-5f3eff-7db97588f5-ld47d\" (UID: \"ca2e9a7f-b3d9-4f2f-90e6-414543715ada\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5f3eff-7db97588f5-ld47d"
Oct 31 20:00:57.622227 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:57.622146    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f54f6ce5-712d-430a-be69-8ec4a29e44d0" path="/var/lib/kubelet/pods/f54f6ce5-712d-430a-be69-8ec4a29e44d0/volumes"
Oct 31 20:00:57.683573 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:57.683537    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:57.855237 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:57.855200    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5f3eff-7db97588f5-ld47d"
Oct 31 20:00:57.859194 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:57.859164    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5f3eff-65c86b7b8b-mc8nv"]
Oct 31 20:00:57.859715 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:00:57.859687    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ec291f9_5a7f_4d42_946e_668aeb30a976.slice/crio-c012ff8201df29026ce2ef9999d410f170d6cb087fd1682390e78d596993aae7 WatchSource:0}: Error finding container c012ff8201df29026ce2ef9999d410f170d6cb087fd1682390e78d596993aae7: Status 404 returned error can't find the container with id c012ff8201df29026ce2ef9999d410f170d6cb087fd1682390e78d596993aae7
Oct 31 20:00:58.145968 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:00:58.145940    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca2e9a7f_b3d9_4f2f_90e6_414543715ada.slice/crio-ee0a195bf6132d05fb0a84f5fd123dd7111959ee62cf8d4f9a15eec663fc2b22 WatchSource:0}: Error finding container ee0a195bf6132d05fb0a84f5fd123dd7111959ee62cf8d4f9a15eec663fc2b22: Status 404 returned error can't find the container with id ee0a195bf6132d05fb0a84f5fd123dd7111959ee62cf8d4f9a15eec663fc2b22
Oct 31 20:00:58.148526 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:58.148497    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5f3eff-7db97588f5-ld47d"]
Oct 31 20:00:58.558156 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:58.558122    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:00:58.856016 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:58.855926    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5f3eff-65c86b7b8b-mc8nv" event={"ID":"0ec291f9-5a7f-4d42-946e-668aeb30a976","Type":"ContainerStarted","Data":"66f6315668c4415b82937e276c2880660cff888a5e7aeb0f98f40a480ae2905f"}
Oct 31 20:00:58.856276 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:58.856251    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5f3eff-65c86b7b8b-mc8nv" event={"ID":"0ec291f9-5a7f-4d42-946e-668aeb30a976","Type":"ContainerStarted","Data":"c012ff8201df29026ce2ef9999d410f170d6cb087fd1682390e78d596993aae7"}
Oct 31 20:00:58.856479 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:58.856458    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-5f3eff-65c86b7b8b-mc8nv"
Oct 31 20:00:58.857851 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:58.857825    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5f3eff-7db97588f5-ld47d" event={"ID":"ca2e9a7f-b3d9-4f2f-90e6-414543715ada","Type":"ContainerStarted","Data":"844bab7531935759b5cc1424a31d94811b7640793d67ef5f00dfe4116c183637"}
Oct 31 20:00:58.857968 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:58.857863    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5f3eff-7db97588f5-ld47d" event={"ID":"ca2e9a7f-b3d9-4f2f-90e6-414543715ada","Type":"ContainerStarted","Data":"ee0a195bf6132d05fb0a84f5fd123dd7111959ee62cf8d4f9a15eec663fc2b22"}
Oct 31 20:00:58.857968 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:58.857952    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5f3eff-7db97588f5-ld47d"
Oct 31 20:00:58.878533 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:58.878488    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-5f3eff-65c86b7b8b-mc8nv" podStartSLOduration=1.8784760459999998 podStartE2EDuration="1.878476046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:00:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:00:58.873359272 +0000 UTC m=+2489.446593394" watchObservedRunningTime="2026-10-31 20:00:58.878476046 +0000 UTC m=+2489.451710175"
Oct 31 20:00:58.891995 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:58.891949    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5f3eff-7db97588f5-ld47d" podStartSLOduration=1.89194043 podStartE2EDuration="1.89194043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:00:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:00:58.890901576 +0000 UTC m=+2489.464135704" watchObservedRunningTime="2026-10-31 20:00:58.89194043 +0000 UTC m=+2489.465174558"
Oct 31 20:00:59.362933 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:00:59.362892    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:00.072375 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:00.072324    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:00.963600 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:00.963561    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:01.252628 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:01.252540    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:01.360690 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:01.360657    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:01.863996 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:01.863961    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:02.778712 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:02.778663    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:03.554357 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:03.552924    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:04.592049 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:04.592005    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:04.712547 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:04.712515    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:04.852836 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:04.852734    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:05.649376 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:05.649308    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:06.247049 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:01:06.247011    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 20:01:06.543548 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:06.543462    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:07.257952 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:07.257915    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:08.166961 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:08.166900    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:08.275071 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:08.275030    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:09.258346 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:09.258300    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:09.868375 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:09.868342    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5f3eff-7db97588f5-ld47d"
Oct 31 20:01:09.868809 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:09.868492    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-5f3eff-65c86b7b8b-mc8nv"
Oct 31 20:01:10.070795 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:10.070763    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:10.144854 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:10.144763    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c734f5-699f7df769-c5swh"]
Oct 31 20:01:10.254376 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:10.254305    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:10.354164 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:10.354130    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:10.488877 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:10.488849    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:10.696317 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:10.696278    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669" podUID="9285bee0-bf7c-4fd4-9da7-a1db021cae05" containerName="main" containerID="cri-o://cd72eae84adda09d32a83ce40769c2177eda422d27bd920117ae51111a429827" gracePeriod=45
Oct 31 20:01:10.725311 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:10.722795    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4" podUID="6032a997-b795-4561-805c-110cf67a9d5d" containerName="main" containerID="cri-o://304d3fda0c7387e793e8eef7d4cb8217c921aae2a0ae0b86994b47f2f32bdf70" gracePeriod=45
Oct 31 20:01:10.737347 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:10.733624    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x" podUID="a774e286-dcb8-4a2f-8541-26bf231eb08d" containerName="main" containerID="cri-o://37915b42aa2185f229e626755de139318b29fc6f797c16185768157c18805912" gracePeriod=45
Oct 31 20:01:11.043403 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.039689    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:11.097521 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.097483    4645 generic.go:356] "Generic (PLEG): container finished" podID="a774e286-dcb8-4a2f-8541-26bf231eb08d" containerID="37915b42aa2185f229e626755de139318b29fc6f797c16185768157c18805912" exitCode=0
Oct 31 20:01:11.097687 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.097663    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x" event={"ID":"a774e286-dcb8-4a2f-8541-26bf231eb08d","Type":"ContainerDied","Data":"37915b42aa2185f229e626755de139318b29fc6f797c16185768157c18805912"}
Oct 31 20:01:11.099980 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.099941    4645 generic.go:356] "Generic (PLEG): container finished" podID="9285bee0-bf7c-4fd4-9da7-a1db021cae05" containerID="cd72eae84adda09d32a83ce40769c2177eda422d27bd920117ae51111a429827" exitCode=0
Oct 31 20:01:11.100162 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.100136    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669" event={"ID":"9285bee0-bf7c-4fd4-9da7-a1db021cae05","Type":"ContainerDied","Data":"cd72eae84adda09d32a83ce40769c2177eda422d27bd920117ae51111a429827"}
Oct 31 20:01:11.102099 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.102072    4645 generic.go:356] "Generic (PLEG): container finished" podID="6032a997-b795-4561-805c-110cf67a9d5d" containerID="304d3fda0c7387e793e8eef7d4cb8217c921aae2a0ae0b86994b47f2f32bdf70" exitCode=0
Oct 31 20:01:11.102207 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.102170    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4" event={"ID":"6032a997-b795-4561-805c-110cf67a9d5d","Type":"ContainerDied","Data":"304d3fda0c7387e793e8eef7d4cb8217c921aae2a0ae0b86994b47f2f32bdf70"}
Oct 31 20:01:11.154808 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.154782    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4"
Oct 31 20:01:11.159053 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.158983    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669"
Oct 31 20:01:11.169162 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.169137    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x"
Oct 31 20:01:11.248953 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.248916    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a774e286-dcb8-4a2f-8541-26bf231eb08d-home\" (UniqueName: \"kubernetes.io/empty-dir/a774e286-dcb8-4a2f-8541-26bf231eb08d-home\") pod \"a774e286-dcb8-4a2f-8541-26bf231eb08d\" (UID: \"a774e286-dcb8-4a2f-8541-26bf231eb08d\") "
Oct 31 20:01:11.249151 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.248985    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a774e286-dcb8-4a2f-8541-26bf231eb08d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a774e286-dcb8-4a2f-8541-26bf231eb08d-dshm\") pod \"a774e286-dcb8-4a2f-8541-26bf231eb08d\" (UID: \"a774e286-dcb8-4a2f-8541-26bf231eb08d\") "
Oct 31 20:01:11.249151 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.249048    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9285bee0-bf7c-4fd4-9da7-a1db021cae05-kube-api-access-rghn4\" (UniqueName: \"kubernetes.io/projected/9285bee0-bf7c-4fd4-9da7-a1db021cae05-kube-api-access-rghn4\") pod \"9285bee0-bf7c-4fd4-9da7-a1db021cae05\" (UID: \"9285bee0-bf7c-4fd4-9da7-a1db021cae05\") "
Oct 31 20:01:11.249151 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.249071    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9285bee0-bf7c-4fd4-9da7-a1db021cae05-tls-certs\" (UniqueName: \"kubernetes.io/secret/9285bee0-bf7c-4fd4-9da7-a1db021cae05-tls-certs\") pod \"9285bee0-bf7c-4fd4-9da7-a1db021cae05\" (UID: \"9285bee0-bf7c-4fd4-9da7-a1db021cae05\") "
Oct 31 20:01:11.249151 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.249095    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6032a997-b795-4561-805c-110cf67a9d5d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6032a997-b795-4561-805c-110cf67a9d5d-tmp-dir\") pod \"6032a997-b795-4561-805c-110cf67a9d5d\" (UID: \"6032a997-b795-4561-805c-110cf67a9d5d\") "
Oct 31 20:01:11.249151 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.249120    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6032a997-b795-4561-805c-110cf67a9d5d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6032a997-b795-4561-805c-110cf67a9d5d-model-cache\") pod \"6032a997-b795-4561-805c-110cf67a9d5d\" (UID: \"6032a997-b795-4561-805c-110cf67a9d5d\") "
Oct 31 20:01:11.249489 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.249162    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6032a997-b795-4561-805c-110cf67a9d5d-tls-certs\" (UniqueName: \"kubernetes.io/secret/6032a997-b795-4561-805c-110cf67a9d5d-tls-certs\") pod \"6032a997-b795-4561-805c-110cf67a9d5d\" (UID: \"6032a997-b795-4561-805c-110cf67a9d5d\") "
Oct 31 20:01:11.249489 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.249200    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6032a997-b795-4561-805c-110cf67a9d5d-kube-api-access-9nxdb\" (UniqueName: \"kubernetes.io/projected/6032a997-b795-4561-805c-110cf67a9d5d-kube-api-access-9nxdb\") pod \"6032a997-b795-4561-805c-110cf67a9d5d\" (UID: \"6032a997-b795-4561-805c-110cf67a9d5d\") "
Oct 31 20:01:11.249489 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.249219    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a774e286-dcb8-4a2f-8541-26bf231eb08d-tls-certs\" (UniqueName: \"kubernetes.io/secret/a774e286-dcb8-4a2f-8541-26bf231eb08d-tls-certs\") pod \"a774e286-dcb8-4a2f-8541-26bf231eb08d\" (UID: \"a774e286-dcb8-4a2f-8541-26bf231eb08d\") "
Oct 31 20:01:11.249489 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.249243    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9285bee0-bf7c-4fd4-9da7-a1db021cae05-home\" (UniqueName: \"kubernetes.io/empty-dir/9285bee0-bf7c-4fd4-9da7-a1db021cae05-home\") pod \"9285bee0-bf7c-4fd4-9da7-a1db021cae05\" (UID: \"9285bee0-bf7c-4fd4-9da7-a1db021cae05\") "
Oct 31 20:01:11.249489 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.249266    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9285bee0-bf7c-4fd4-9da7-a1db021cae05-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9285bee0-bf7c-4fd4-9da7-a1db021cae05-model-cache\") pod \"9285bee0-bf7c-4fd4-9da7-a1db021cae05\" (UID: \"9285bee0-bf7c-4fd4-9da7-a1db021cae05\") "
Oct 31 20:01:11.249489 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.249299    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a774e286-dcb8-4a2f-8541-26bf231eb08d-kube-api-access-89x8m\" (UniqueName: \"kubernetes.io/projected/a774e286-dcb8-4a2f-8541-26bf231eb08d-kube-api-access-89x8m\") pod \"a774e286-dcb8-4a2f-8541-26bf231eb08d\" (UID: \"a774e286-dcb8-4a2f-8541-26bf231eb08d\") "
Oct 31 20:01:11.249489 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.249385    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a774e286-dcb8-4a2f-8541-26bf231eb08d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a774e286-dcb8-4a2f-8541-26bf231eb08d-tmp-dir\") pod \"a774e286-dcb8-4a2f-8541-26bf231eb08d\" (UID: \"a774e286-dcb8-4a2f-8541-26bf231eb08d\") "
Oct 31 20:01:11.249489 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.249421    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a774e286-dcb8-4a2f-8541-26bf231eb08d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a774e286-dcb8-4a2f-8541-26bf231eb08d-model-cache\") pod \"a774e286-dcb8-4a2f-8541-26bf231eb08d\" (UID: \"a774e286-dcb8-4a2f-8541-26bf231eb08d\") "
Oct 31 20:01:11.249489 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.249474    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9285bee0-bf7c-4fd4-9da7-a1db021cae05-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9285bee0-bf7c-4fd4-9da7-a1db021cae05-tmp-dir\") pod \"9285bee0-bf7c-4fd4-9da7-a1db021cae05\" (UID: \"9285bee0-bf7c-4fd4-9da7-a1db021cae05\") "
Oct 31 20:01:11.249991 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.249531    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6032a997-b795-4561-805c-110cf67a9d5d-model-cache" pod "6032a997-b795-4561-805c-110cf67a9d5d" (UID: "6032a997-b795-4561-805c-110cf67a9d5d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:01:11.249991 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.249657    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6032a997-b795-4561-805c-110cf67a9d5d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6032a997-b795-4561-805c-110cf67a9d5d-dshm\") pod \"6032a997-b795-4561-805c-110cf67a9d5d\" (UID: \"6032a997-b795-4561-805c-110cf67a9d5d\") "
Oct 31 20:01:11.249991 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.249717    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9285bee0-bf7c-4fd4-9da7-a1db021cae05-tmp-dir" pod "9285bee0-bf7c-4fd4-9da7-a1db021cae05" (UID: "9285bee0-bf7c-4fd4-9da7-a1db021cae05"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:01:11.249991 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.249730    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6032a997-b795-4561-805c-110cf67a9d5d-home\" (UniqueName: \"kubernetes.io/empty-dir/6032a997-b795-4561-805c-110cf67a9d5d-home\") pod \"6032a997-b795-4561-805c-110cf67a9d5d\" (UID: \"6032a997-b795-4561-805c-110cf67a9d5d\") "
Oct 31 20:01:11.249991 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.249764    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9285bee0-bf7c-4fd4-9da7-a1db021cae05-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9285bee0-bf7c-4fd4-9da7-a1db021cae05-dshm\") pod \"9285bee0-bf7c-4fd4-9da7-a1db021cae05\" (UID: \"9285bee0-bf7c-4fd4-9da7-a1db021cae05\") "
Oct 31 20:01:11.249991 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.249925    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9285bee0-bf7c-4fd4-9da7-a1db021cae05-model-cache" pod "9285bee0-bf7c-4fd4-9da7-a1db021cae05" (UID: "9285bee0-bf7c-4fd4-9da7-a1db021cae05"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:01:11.249991 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.249919    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9285bee0-bf7c-4fd4-9da7-a1db021cae05-home" pod "9285bee0-bf7c-4fd4-9da7-a1db021cae05" (UID: "9285bee0-bf7c-4fd4-9da7-a1db021cae05"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:01:11.250479 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.250215    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6032a997-b795-4561-805c-110cf67a9d5d-tmp-dir" pod "6032a997-b795-4561-805c-110cf67a9d5d" (UID: "6032a997-b795-4561-805c-110cf67a9d5d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:01:11.251045 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.250576    4645 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6032a997-b795-4561-805c-110cf67a9d5d-tmp-dir\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:01:11.251045 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.250606    4645 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6032a997-b795-4561-805c-110cf67a9d5d-model-cache\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:01:11.251045 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.250623    4645 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9285bee0-bf7c-4fd4-9da7-a1db021cae05-home\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:01:11.251045 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.250636    4645 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9285bee0-bf7c-4fd4-9da7-a1db021cae05-model-cache\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:01:11.251045 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.250656    4645 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9285bee0-bf7c-4fd4-9da7-a1db021cae05-tmp-dir\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:01:11.251842 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.251672    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a774e286-dcb8-4a2f-8541-26bf231eb08d-home" pod "a774e286-dcb8-4a2f-8541-26bf231eb08d" (UID: "a774e286-dcb8-4a2f-8541-26bf231eb08d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:01:11.252149 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.251980    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a774e286-dcb8-4a2f-8541-26bf231eb08d-tmp-dir" pod "a774e286-dcb8-4a2f-8541-26bf231eb08d" (UID: "a774e286-dcb8-4a2f-8541-26bf231eb08d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:01:11.252585 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.252290    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a774e286-dcb8-4a2f-8541-26bf231eb08d-model-cache" pod "a774e286-dcb8-4a2f-8541-26bf231eb08d" (UID: "a774e286-dcb8-4a2f-8541-26bf231eb08d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:01:11.253418 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.253393    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6032a997-b795-4561-805c-110cf67a9d5d-home" pod "6032a997-b795-4561-805c-110cf67a9d5d" (UID: "6032a997-b795-4561-805c-110cf67a9d5d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:01:11.254506 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.254476    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6032a997-b795-4561-805c-110cf67a9d5d-dshm" pod "6032a997-b795-4561-805c-110cf67a9d5d" (UID: "6032a997-b795-4561-805c-110cf67a9d5d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:01:11.256107 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.256080    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a774e286-dcb8-4a2f-8541-26bf231eb08d-kube-api-access-89x8m" pod "a774e286-dcb8-4a2f-8541-26bf231eb08d" (UID: "a774e286-dcb8-4a2f-8541-26bf231eb08d"). InnerVolumeSpecName "kube-api-access-89x8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:01:11.257308 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.257281    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6032a997-b795-4561-805c-110cf67a9d5d-kube-api-access-9nxdb" pod "6032a997-b795-4561-805c-110cf67a9d5d" (UID: "6032a997-b795-4561-805c-110cf67a9d5d"). InnerVolumeSpecName "kube-api-access-9nxdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:01:11.257308 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.257285    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a774e286-dcb8-4a2f-8541-26bf231eb08d-tls-certs" pod "a774e286-dcb8-4a2f-8541-26bf231eb08d" (UID: "a774e286-dcb8-4a2f-8541-26bf231eb08d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:01:11.259677 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.259644    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9285bee0-bf7c-4fd4-9da7-a1db021cae05-dshm" pod "9285bee0-bf7c-4fd4-9da7-a1db021cae05" (UID: "9285bee0-bf7c-4fd4-9da7-a1db021cae05"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:01:11.259677 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.259664    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6032a997-b795-4561-805c-110cf67a9d5d-tls-certs" pod "6032a997-b795-4561-805c-110cf67a9d5d" (UID: "6032a997-b795-4561-805c-110cf67a9d5d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:01:11.259830 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.259689    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9285bee0-bf7c-4fd4-9da7-a1db021cae05-kube-api-access-rghn4" pod "9285bee0-bf7c-4fd4-9da7-a1db021cae05" (UID: "9285bee0-bf7c-4fd4-9da7-a1db021cae05"). InnerVolumeSpecName "kube-api-access-rghn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:01:11.259830 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.259794    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a774e286-dcb8-4a2f-8541-26bf231eb08d-dshm" pod "a774e286-dcb8-4a2f-8541-26bf231eb08d" (UID: "a774e286-dcb8-4a2f-8541-26bf231eb08d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:01:11.261071 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.261047    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9285bee0-bf7c-4fd4-9da7-a1db021cae05-tls-certs" pod "9285bee0-bf7c-4fd4-9da7-a1db021cae05" (UID: "9285bee0-bf7c-4fd4-9da7-a1db021cae05"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:01:11.352108 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.352034    4645 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6032a997-b795-4561-805c-110cf67a9d5d-tls-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:01:11.352108 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.352064    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nxdb\" (UniqueName: \"kubernetes.io/projected/6032a997-b795-4561-805c-110cf67a9d5d-kube-api-access-9nxdb\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:01:11.352108 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.352078    4645 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a774e286-dcb8-4a2f-8541-26bf231eb08d-tls-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:01:11.352108 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.352092    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89x8m\" (UniqueName: \"kubernetes.io/projected/a774e286-dcb8-4a2f-8541-26bf231eb08d-kube-api-access-89x8m\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:01:11.352108 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.352104    4645 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a774e286-dcb8-4a2f-8541-26bf231eb08d-tmp-dir\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:01:11.352430 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.352118    4645 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a774e286-dcb8-4a2f-8541-26bf231eb08d-model-cache\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:01:11.352430 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.352132    4645 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6032a997-b795-4561-805c-110cf67a9d5d-dshm\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:01:11.352430 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.352143    4645 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6032a997-b795-4561-805c-110cf67a9d5d-home\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:01:11.352430 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.352154    4645 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9285bee0-bf7c-4fd4-9da7-a1db021cae05-dshm\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:01:11.352430 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.352167    4645 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a774e286-dcb8-4a2f-8541-26bf231eb08d-home\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:01:11.352430 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.352185    4645 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a774e286-dcb8-4a2f-8541-26bf231eb08d-dshm\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:01:11.352430 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.352198    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rghn4\" (UniqueName: \"kubernetes.io/projected/9285bee0-bf7c-4fd4-9da7-a1db021cae05-kube-api-access-rghn4\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:01:11.352430 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.352211    4645 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9285bee0-bf7c-4fd4-9da7-a1db021cae05-tls-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:01:11.646654 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.646627    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:11.742801 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.742775    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c173d8-6d47cf995d-2bjft"]
Oct 31 20:01:11.743544 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.743525    4645 cpu_manager.go:414] "containerMap: removing container" podUID="a774e286-dcb8-4a2f-8541-26bf231eb08d" containerName="main"
Oct 31 20:01:11.743544 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.743544    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a774e286-dcb8-4a2f-8541-26bf231eb08d" containerName="main"
Oct 31 20:01:11.743679 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.743557    4645 cpu_manager.go:414] "containerMap: removing container" podUID="9285bee0-bf7c-4fd4-9da7-a1db021cae05" containerName="main"
Oct 31 20:01:11.743679 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.743562    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9285bee0-bf7c-4fd4-9da7-a1db021cae05" containerName="main"
Oct 31 20:01:11.743679 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.743582    4645 cpu_manager.go:414] "containerMap: removing container" podUID="6032a997-b795-4561-805c-110cf67a9d5d" containerName="main"
Oct 31 20:01:11.743679 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.743587    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6032a997-b795-4561-805c-110cf67a9d5d" containerName="main"
Oct 31 20:01:11.743804 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.743728    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="a774e286-dcb8-4a2f-8541-26bf231eb08d" containerName="main"
Oct 31 20:01:11.743804 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.743738    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="9285bee0-bf7c-4fd4-9da7-a1db021cae05" containerName="main"
Oct 31 20:01:11.743804 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.743748    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="6032a997-b795-4561-805c-110cf67a9d5d" containerName="main"
Oct 31 20:01:11.748489 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.748473    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c173d8-6d47cf995d-2bjft"
Oct 31 20:01:11.756009 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.755991    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-c173d8\""
Oct 31 20:01:11.765288 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.765264    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c173d8-6d47cf995d-2bjft"]
Oct 31 20:01:11.853682 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.853653    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:11.854223 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.854200    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c173d8-7dbb76c547-zwgd7"]
Oct 31 20:01:11.858618 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.858599    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c173d8-7dbb76c547-zwgd7"
Oct 31 20:01:11.859384 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.859359    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bjm6\" (UniqueName: \"kubernetes.io/projected/c08e5b4b-a7da-4cfe-a48a-140363b68362-kube-api-access-9bjm6\") pod \"payload-processing-e2e-ipp-a-c173d8-6d47cf995d-2bjft\" (UID: \"c08e5b4b-a7da-4cfe-a48a-140363b68362\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c173d8-6d47cf995d-2bjft"
Oct 31 20:01:11.859513 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.859491    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c08e5b4b-a7da-4cfe-a48a-140363b68362-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c173d8-6d47cf995d-2bjft\" (UID: \"c08e5b4b-a7da-4cfe-a48a-140363b68362\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c173d8-6d47cf995d-2bjft"
Oct 31 20:01:11.985352 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.945491    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-c173d8-dockercfg-nknw8\""
Oct 31 20:01:11.985352 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.970453    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c08e5b4b-a7da-4cfe-a48a-140363b68362-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c173d8-6d47cf995d-2bjft\" (UID: \"c08e5b4b-a7da-4cfe-a48a-140363b68362\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c173d8-6d47cf995d-2bjft"
Oct 31 20:01:11.985352 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.970599    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfgzh\" (UniqueName: \"kubernetes.io/projected/0121438a-b3ff-4d68-91c9-ab91f97fa181-kube-api-access-mfgzh\") pod \"payload-pre-processing-e2e-ipp-a-c173d8-7dbb76c547-zwgd7\" (UID: \"0121438a-b3ff-4d68-91c9-ab91f97fa181\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c173d8-7dbb76c547-zwgd7"
Oct 31 20:01:11.985352 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.970643    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0121438a-b3ff-4d68-91c9-ab91f97fa181-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c173d8-7dbb76c547-zwgd7\" (UID: \"0121438a-b3ff-4d68-91c9-ab91f97fa181\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c173d8-7dbb76c547-zwgd7"
Oct 31 20:01:11.985352 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.970838    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bjm6\" (UniqueName: \"kubernetes.io/projected/c08e5b4b-a7da-4cfe-a48a-140363b68362-kube-api-access-9bjm6\") pod \"payload-processing-e2e-ipp-a-c173d8-6d47cf995d-2bjft\" (UID: \"c08e5b4b-a7da-4cfe-a48a-140363b68362\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c173d8-6d47cf995d-2bjft"
Oct 31 20:01:12.008349 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:11.996870    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c173d8-7dbb76c547-zwgd7"]
Oct 31 20:01:12.030351 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:12.010775    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c08e5b4b-a7da-4cfe-a48a-140363b68362-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c173d8-6d47cf995d-2bjft\" (UID: \"c08e5b4b-a7da-4cfe-a48a-140363b68362\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c173d8-6d47cf995d-2bjft"
Oct 31 20:01:12.096353 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:12.072016    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfgzh\" (UniqueName: \"kubernetes.io/projected/0121438a-b3ff-4d68-91c9-ab91f97fa181-kube-api-access-mfgzh\") pod \"payload-pre-processing-e2e-ipp-a-c173d8-7dbb76c547-zwgd7\" (UID: \"0121438a-b3ff-4d68-91c9-ab91f97fa181\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c173d8-7dbb76c547-zwgd7"
Oct 31 20:01:12.096353 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:12.072077    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0121438a-b3ff-4d68-91c9-ab91f97fa181-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c173d8-7dbb76c547-zwgd7\" (UID: \"0121438a-b3ff-4d68-91c9-ab91f97fa181\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c173d8-7dbb76c547-zwgd7"
Oct 31 20:01:12.096353 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:12.073358    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0121438a-b3ff-4d68-91c9-ab91f97fa181-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c173d8-7dbb76c547-zwgd7\" (UID: \"0121438a-b3ff-4d68-91c9-ab91f97fa181\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c173d8-7dbb76c547-zwgd7"
Oct 31 20:01:12.193793 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:12.191423    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfgzh\" (UniqueName: \"kubernetes.io/projected/0121438a-b3ff-4d68-91c9-ab91f97fa181-kube-api-access-mfgzh\") pod \"payload-pre-processing-e2e-ipp-a-c173d8-7dbb76c547-zwgd7\" (UID: \"0121438a-b3ff-4d68-91c9-ab91f97fa181\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c173d8-7dbb76c547-zwgd7"
Oct 31 20:01:12.193793 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:12.192055    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bjm6\" (UniqueName: \"kubernetes.io/projected/c08e5b4b-a7da-4cfe-a48a-140363b68362-kube-api-access-9bjm6\") pod \"payload-processing-e2e-ipp-a-c173d8-6d47cf995d-2bjft\" (UID: \"c08e5b4b-a7da-4cfe-a48a-140363b68362\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c173d8-6d47cf995d-2bjft"
Oct 31 20:01:12.203384 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:12.202755    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4" event={"ID":"6032a997-b795-4561-805c-110cf67a9d5d","Type":"ContainerDied","Data":"335f8a49cc9e45e22fef97a198408dd433032c770bf87600d25dac647139a2b7"}
Oct 31 20:01:12.203384 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:12.202800    4645 scope.go:122] "RemoveContainer" containerID="304d3fda0c7387e793e8eef7d4cb8217c921aae2a0ae0b86994b47f2f32bdf70"
Oct 31 20:01:12.203384 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:12.202907    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4"
Oct 31 20:01:12.207764 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:12.207723    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x"
Oct 31 20:01:12.208113 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:12.208083    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c173d8-7dbb76c547-zwgd7"
Oct 31 20:01:12.208959 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:12.208929    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x" event={"ID":"a774e286-dcb8-4a2f-8541-26bf231eb08d","Type":"ContainerDied","Data":"1a1ddc08963e669aa5adfedffc62ae41eac6520e9080fc4069c6945dfed6e560"}
Oct 31 20:01:12.213687 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:12.213672    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669"
Oct 31 20:01:12.214195 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:12.213717    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669" event={"ID":"9285bee0-bf7c-4fd4-9da7-a1db021cae05","Type":"ContainerDied","Data":"e108f5ab597b3f8176df1527f7997be4cc61674360ecb579fba7593f9d0e77e2"}
Oct 31 20:01:12.229302 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:12.229286    4645 scope.go:122] "RemoveContainer" containerID="37915b42aa2185f229e626755de139318b29fc6f797c16185768157c18805912"
Oct 31 20:01:12.235870 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:12.234368    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x"]
Oct 31 20:01:12.240186 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:12.240159    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-5bbf75/simulator-w5-5bbf75-kserve-55ccdfccc-cnp7x"]
Oct 31 20:01:12.251683 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:12.251660    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4"]
Oct 31 20:01:12.259498 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:12.259475    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-5bbf75/premium-w5-5bbf75-kserve-79df6f5f68-8tgl4"]
Oct 31 20:01:12.266371 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:12.266352    4645 scope.go:122] "RemoveContainer" containerID="cd72eae84adda09d32a83ce40769c2177eda422d27bd920117ae51111a429827"
Oct 31 20:01:12.278712 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:12.278690    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669"]
Oct 31 20:01:12.287948 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:12.287920    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-5bbf75/distinct-w5-5bbf75-kserve-5cb5875f47-wp669"]
Oct 31 20:01:12.364085 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:12.364052    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c173d8-6d47cf995d-2bjft"
Oct 31 20:01:12.400023 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:12.399995    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:12.531840 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:12.531812    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c173d8-7dbb76c547-zwgd7"]
Oct 31 20:01:12.532884 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:01:12.532856    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0121438a_b3ff_4d68_91c9_ab91f97fa181.slice/crio-e37e2f4376bea81108df8326f99699a41f288ee64927e40157f85abece1ab742 WatchSource:0}: Error finding container e37e2f4376bea81108df8326f99699a41f288ee64927e40157f85abece1ab742: Status 404 returned error can't find the container with id e37e2f4376bea81108df8326f99699a41f288ee64927e40157f85abece1ab742
Oct 31 20:01:12.708455 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:12.708428    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c173d8-6d47cf995d-2bjft"]
Oct 31 20:01:12.712261 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:01:12.712233    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc08e5b4b_a7da_4cfe_a48a_140363b68362.slice/crio-b4d0225aa84b54b39d6fe7460fa9a37e50e24ebb743ee72add2ad4ea82c45cdf WatchSource:0}: Error finding container b4d0225aa84b54b39d6fe7460fa9a37e50e24ebb743ee72add2ad4ea82c45cdf: Status 404 returned error can't find the container with id b4d0225aa84b54b39d6fe7460fa9a37e50e24ebb743ee72add2ad4ea82c45cdf
Oct 31 20:01:13.226367 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:13.226310    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c173d8-6d47cf995d-2bjft" event={"ID":"c08e5b4b-a7da-4cfe-a48a-140363b68362","Type":"ContainerStarted","Data":"9f9c5c0ed0aef45da1b4adcdf968a16c2e934079bd18c95b77ac8cd8bd2602b7"}
Oct 31 20:01:13.226833 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:13.226548    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c173d8-6d47cf995d-2bjft"
Oct 31 20:01:13.226833 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:13.226585    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c173d8-6d47cf995d-2bjft" event={"ID":"c08e5b4b-a7da-4cfe-a48a-140363b68362","Type":"ContainerStarted","Data":"b4d0225aa84b54b39d6fe7460fa9a37e50e24ebb743ee72add2ad4ea82c45cdf"}
Oct 31 20:01:13.228130 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:13.228102    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c173d8-7dbb76c547-zwgd7" event={"ID":"0121438a-b3ff-4d68-91c9-ab91f97fa181","Type":"ContainerStarted","Data":"ef809c72a50a92bd9176e11c1ec8dafa95845f170ea12a9e75902ab1a8906e73"}
Oct 31 20:01:13.228245 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:13.228149    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c173d8-7dbb76c547-zwgd7" event={"ID":"0121438a-b3ff-4d68-91c9-ab91f97fa181","Type":"ContainerStarted","Data":"e37e2f4376bea81108df8326f99699a41f288ee64927e40157f85abece1ab742"}
Oct 31 20:01:13.228245 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:13.228207    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c173d8-7dbb76c547-zwgd7"
Oct 31 20:01:13.246786 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:13.246747    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-c173d8-6d47cf995d-2bjft" podStartSLOduration=2.24673589 podStartE2EDuration="2.24673589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:01:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:01:13.243589374 +0000 UTC m=+2503.816823502" watchObservedRunningTime="2026-10-31 20:01:13.24673589 +0000 UTC m=+2503.819970018"
Oct 31 20:01:13.267131 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:13.267084    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c173d8-7dbb76c547-zwgd7" podStartSLOduration=2.2670686079999998 podStartE2EDuration="2.267068608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:01:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:01:13.261820609 +0000 UTC m=+2503.835054739" watchObservedRunningTime="2026-10-31 20:01:13.267068608 +0000 UTC m=+2503.840302741"
Oct 31 20:01:13.495286 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:13.495195    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:13.620268 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:13.620237    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6032a997-b795-4561-805c-110cf67a9d5d" path="/var/lib/kubelet/pods/6032a997-b795-4561-805c-110cf67a9d5d/volumes"
Oct 31 20:01:13.620839 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:13.620820    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9285bee0-bf7c-4fd4-9da7-a1db021cae05" path="/var/lib/kubelet/pods/9285bee0-bf7c-4fd4-9da7-a1db021cae05/volumes"
Oct 31 20:01:13.621303 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:13.621289    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a774e286-dcb8-4a2f-8541-26bf231eb08d" path="/var/lib/kubelet/pods/a774e286-dcb8-4a2f-8541-26bf231eb08d/volumes"
Oct 31 20:01:14.452050 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:14.451998    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:15.146217 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:15.146175    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-c734f5-699f7df769-c5swh" podUID="7a2fa9c5-2e84-4531-b72c-183909b077a7" containerName="payload-processing" containerID="cri-o://c2865b8413714dbde4532183ec36e0ff878923494dc50387570ff56c433c89c0" gracePeriod=25
Oct 31 20:01:15.262722 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:15.262689    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:15.283149 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:01:15.283125    4645 logging.go:55] [core] [Channel #5502 SubChannel #5503]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.1.5:50052", ServerName: "10.133.1.5:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.1.5:50052: connect: connection refused"
Oct 31 20:01:16.050352 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:16.049420    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c734f5-699f7df769-c5swh"
Oct 31 20:01:16.111789 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:16.111752    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a2fa9c5-2e84-4531-b72c-183909b077a7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a2fa9c5-2e84-4531-b72c-183909b077a7-plugins-config-volume\") pod \"7a2fa9c5-2e84-4531-b72c-183909b077a7\" (UID: \"7a2fa9c5-2e84-4531-b72c-183909b077a7\") "
Oct 31 20:01:16.112260 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:16.112231    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a2fa9c5-2e84-4531-b72c-183909b077a7-kube-api-access-s5tdl\" (UniqueName: \"kubernetes.io/projected/7a2fa9c5-2e84-4531-b72c-183909b077a7-kube-api-access-s5tdl\") pod \"7a2fa9c5-2e84-4531-b72c-183909b077a7\" (UID: \"7a2fa9c5-2e84-4531-b72c-183909b077a7\") "
Oct 31 20:01:16.146431 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:16.126321    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a2fa9c5-2e84-4531-b72c-183909b077a7-plugins-config-volume" pod "7a2fa9c5-2e84-4531-b72c-183909b077a7" (UID: "7a2fa9c5-2e84-4531-b72c-183909b077a7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:01:16.239457 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:16.237678    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a2fa9c5-2e84-4531-b72c-183909b077a7-kube-api-access-s5tdl" pod "7a2fa9c5-2e84-4531-b72c-183909b077a7" (UID: "7a2fa9c5-2e84-4531-b72c-183909b077a7"). InnerVolumeSpecName "kube-api-access-s5tdl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:01:16.285379 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:16.284029    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-a-c734f5-699f7df769-c5swh" podUID="7a2fa9c5-2e84-4531-b72c-183909b077a7" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.1.5:50052\" within 1s: context deadline exceeded"
Oct 31 20:01:16.357241 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:16.341869    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c53dcfe0-kxlzm"]
Oct 31 20:01:16.357241 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:16.348565    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a2fa9c5-2e84-4531-b72c-183909b077a7-kube-api-access-s5tdl\" (UniqueName: \"kubernetes.io/projected/7a2fa9c5-2e84-4531-b72c-183909b077a7-kube-api-access-s5tdl\") pod \"7a2fa9c5-2e84-4531-b72c-183909b077a7\" (UID: \"7a2fa9c5-2e84-4531-b72c-183909b077a7\") "
Oct 31 20:01:16.357241 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:16.349439    4645 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a2fa9c5-2e84-4531-b72c-183909b077a7-plugins-config-volume\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:01:16.357241 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:01:16.349548    4645 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/7a2fa9c5-2e84-4531-b72c-183909b077a7/volumes/kubernetes.io~projected/kube-api-access-s5tdl
Oct 31 20:01:16.357241 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:16.349618    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a2fa9c5-2e84-4531-b72c-183909b077a7-kube-api-access-s5tdl" pod "7a2fa9c5-2e84-4531-b72c-183909b077a7" (UID: "7a2fa9c5-2e84-4531-b72c-183909b077a7"). InnerVolumeSpecName "kube-api-access-s5tdl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:01:16.401489 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:16.384098    4645 cpu_manager.go:414] "containerMap: removing container" podUID="7a2fa9c5-2e84-4531-b72c-183909b077a7" containerName="payload-processing"
Oct 31 20:01:16.407424 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:16.407384    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a2fa9c5-2e84-4531-b72c-183909b077a7" containerName="payload-processing"
Oct 31 20:01:16.422354 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:16.411650    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a2fa9c5-2e84-4531-b72c-183909b077a7" containerName="payload-processing"
Oct 31 20:01:16.427593 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:16.427565    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c53dcfe0-kxlzm"
Oct 31 20:01:16.460361 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:16.457809    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5tdl\" (UniqueName: \"kubernetes.io/projected/7a2fa9c5-2e84-4531-b72c-183909b077a7-kube-api-access-s5tdl\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:01:16.492559 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:16.492528    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:01:16.530559 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:16.530487    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w6-e1d060/rate-test-model-e1d060-kserve-5969df469-4mfjn" podUID="8b0c4b8a-bf5b-4af7-b272-aeefb1149b03" containerName="main" probeResult="failure" output="Get \"https://10.133.1.6:8000/ready\": context deadline exceeded"
Oct 31 20:01:16.950675 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:16.950635    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 20:01:16.999490 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:16.984397    4645 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n" podUID="dbb97168-db93-4757-9a1b-f9c5eb704fc5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": context deadline exceeded"
Oct 31 20:01:17.188405 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:17.187714    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:17.196366 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:17.191061    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c53dcfe0-kxlzm"]
Oct 31 20:01:17.471742 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:17.471702    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n" podUID="dbb97168-db93-4757-9a1b-f9c5eb704fc5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 20:01:17.533385 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:17.482888    4645 generic.go:356] "Generic (PLEG): container finished" podID="7a2fa9c5-2e84-4531-b72c-183909b077a7" containerID="c2865b8413714dbde4532183ec36e0ff878923494dc50387570ff56c433c89c0" exitCode=0
Oct 31 20:01:17.533385 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:17.482916    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c734f5-699f7df769-c5swh" event={"ID":"7a2fa9c5-2e84-4531-b72c-183909b077a7","Type":"ContainerDied","Data":"c2865b8413714dbde4532183ec36e0ff878923494dc50387570ff56c433c89c0"}
Oct 31 20:01:17.533633 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:17.533419    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c734f5-699f7df769-c5swh" event={"ID":"7a2fa9c5-2e84-4531-b72c-183909b077a7","Type":"ContainerDied","Data":"ab3830e1b609ad1234b79611a422832e9c85338a2a18cd156d374fb6da038a81"}
Oct 31 20:01:17.533633 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:17.490765    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw" podUID="1287db54-a587-4c11-9eb6-1606de555894" containerName="main" probeResult="failure" output="Get \"https://10.133.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 20:01:17.533761 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:17.483026    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c734f5-699f7df769-c5swh"
Oct 31 20:01:17.534150 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:17.534125    4645 scope.go:122] "RemoveContainer" containerID="c2865b8413714dbde4532183ec36e0ff878923494dc50387570ff56c433c89c0"
Oct 31 20:01:17.534297 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:17.493968    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hrds\" (UniqueName: \"kubernetes.io/projected/647f9975-219b-4b00-aa2a-93adf2ef16ab-kube-api-access-4hrds\") pod \"maas-api-revoke-sub-c53dcfe0-kxlzm\" (UID: \"647f9975-219b-4b00-aa2a-93adf2ef16ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c53dcfe0-kxlzm"
Oct 31 20:01:17.563352 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:17.548658    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w6-e1d060/rate-test-model-e1d060-kserve-5969df469-4mfjn" podUID="8b0c4b8a-bf5b-4af7-b272-aeefb1149b03" containerName="main" probeResult="failure" output="Get \"https://10.133.1.6:8000/ready\": context deadline exceeded"
Oct 31 20:01:17.563352 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:17.549269    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w6-e1d060/rate-test-model-e1d060-kserve-5969df469-4mfjn"
Oct 31 20:01:17.563352 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:17.562757    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/647f9975-219b-4b00-aa2a-93adf2ef16ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c53dcfe0-kxlzm\" (UID: \"647f9975-219b-4b00-aa2a-93adf2ef16ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c53dcfe0-kxlzm"
Oct 31 20:01:17.563352 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:17.562830    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/647f9975-219b-4b00-aa2a-93adf2ef16ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c53dcfe0-kxlzm\" (UID: \"647f9975-219b-4b00-aa2a-93adf2ef16ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c53dcfe0-kxlzm"
Oct 31 20:01:17.731404 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:17.670896    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/647f9975-219b-4b00-aa2a-93adf2ef16ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c53dcfe0-kxlzm\" (UID: \"647f9975-219b-4b00-aa2a-93adf2ef16ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c53dcfe0-kxlzm"
Oct 31 20:01:17.731404 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:17.670976    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/647f9975-219b-4b00-aa2a-93adf2ef16ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c53dcfe0-kxlzm\" (UID: \"647f9975-219b-4b00-aa2a-93adf2ef16ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c53dcfe0-kxlzm"
Oct 31 20:01:17.731404 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:17.671095    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hrds\" (UniqueName: \"kubernetes.io/projected/647f9975-219b-4b00-aa2a-93adf2ef16ab-kube-api-access-4hrds\") pod \"maas-api-revoke-sub-c53dcfe0-kxlzm\" (UID: \"647f9975-219b-4b00-aa2a-93adf2ef16ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c53dcfe0-kxlzm"
Oct 31 20:01:17.731404 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:17.672407    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/647f9975-219b-4b00-aa2a-93adf2ef16ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c53dcfe0-kxlzm\" (UID: \"647f9975-219b-4b00-aa2a-93adf2ef16ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c53dcfe0-kxlzm"
Oct 31 20:01:18.099454 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:18.087672    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w6-e1d060/rate-test-model-e1d060-kserve-5969df469-4mfjn"
Oct 31 20:01:18.118355 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:18.100484    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hrds\" (UniqueName: \"kubernetes.io/projected/647f9975-219b-4b00-aa2a-93adf2ef16ab-kube-api-access-4hrds\") pod \"maas-api-revoke-sub-c53dcfe0-kxlzm\" (UID: \"647f9975-219b-4b00-aa2a-93adf2ef16ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c53dcfe0-kxlzm"
Oct 31 20:01:18.144579 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:18.126776    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:18.382527 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:18.382490    4645 scope.go:122] "RemoveContainer" containerID="c2865b8413714dbde4532183ec36e0ff878923494dc50387570ff56c433c89c0"
Oct 31 20:01:18.385652 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:18.385621    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/647f9975-219b-4b00-aa2a-93adf2ef16ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c53dcfe0-kxlzm\" (UID: \"647f9975-219b-4b00-aa2a-93adf2ef16ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c53dcfe0-kxlzm"
Oct 31 20:01:18.458350 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:01:18.433562    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2865b8413714dbde4532183ec36e0ff878923494dc50387570ff56c433c89c0\": container with ID starting with c2865b8413714dbde4532183ec36e0ff878923494dc50387570ff56c433c89c0 not found: ID does not exist" containerID="c2865b8413714dbde4532183ec36e0ff878923494dc50387570ff56c433c89c0"
Oct 31 20:01:18.487482 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:18.458575    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2865b8413714dbde4532183ec36e0ff878923494dc50387570ff56c433c89c0"} err="failed to get container status \"c2865b8413714dbde4532183ec36e0ff878923494dc50387570ff56c433c89c0\": rpc error: code = NotFound desc = could not find container \"c2865b8413714dbde4532183ec36e0ff878923494dc50387570ff56c433c89c0\": container with ID starting with c2865b8413714dbde4532183ec36e0ff878923494dc50387570ff56c433c89c0 not found: ID does not exist"
Oct 31 20:01:18.550794 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:18.473948    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c53dcfe0-kxlzm"
Oct 31 20:01:18.585662 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:18.585624    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:18.645031 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:18.642208    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c734f5-699f7df769-c5swh"]
Oct 31 20:01:18.726635 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:18.725542    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c734f5-699f7df769-c5swh"]
Oct 31 20:01:19.048030 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:19.043259    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:19.203297 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:19.202702    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:19.633973 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:01:19.633361    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod647f9975_219b_4b00_aa2a_93adf2ef16ab.slice/crio-db606244da324c7bbb50e0074d9b081ec00e8afa6560cabdbe9699e3b76b5bbe WatchSource:0}: Error finding container db606244da324c7bbb50e0074d9b081ec00e8afa6560cabdbe9699e3b76b5bbe: Status 404 returned error can't find the container with id db606244da324c7bbb50e0074d9b081ec00e8afa6560cabdbe9699e3b76b5bbe
Oct 31 20:01:19.644264 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:19.643553    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a2fa9c5-2e84-4531-b72c-183909b077a7" path="/var/lib/kubelet/pods/7a2fa9c5-2e84-4531-b72c-183909b077a7/volumes"
Oct 31 20:01:19.644264 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:19.643889    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c53dcfe0-kxlzm"]
Oct 31 20:01:19.670576 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:19.670544    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c53dcfe0-kxlzm" event={"ID":"647f9975-219b-4b00-aa2a-93adf2ef16ab","Type":"ContainerStarted","Data":"db606244da324c7bbb50e0074d9b081ec00e8afa6560cabdbe9699e3b76b5bbe"}
Oct 31 20:01:19.899196 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:19.892716    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:20.403047 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:20.403014    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:20.721403 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:20.721360    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c53dcfe0-kxlzm" event={"ID":"647f9975-219b-4b00-aa2a-93adf2ef16ab","Type":"ContainerStarted","Data":"da58cf3b3eff3e47fed59ff37bade58a3b7232285727b4c4e92f99e620fc1fa2"}
Oct 31 20:01:20.747005 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:20.746955    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c53dcfe0-kxlzm" podStartSLOduration=4.746940929 podStartE2EDuration="4.746940929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:01:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:01:20.742515934 +0000 UTC m=+2511.315750063" watchObservedRunningTime="2026-10-31 20:01:20.746940929 +0000 UTC m=+2511.320175058"
Oct 31 20:01:21.504922 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:21.504883    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:21.741573 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:21.741535    4645 generic.go:356] "Generic (PLEG): container finished" podID="647f9975-219b-4b00-aa2a-93adf2ef16ab" containerID="da58cf3b3eff3e47fed59ff37bade58a3b7232285727b4c4e92f99e620fc1fa2" exitCode=0
Oct 31 20:01:21.741986 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:21.741607    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c53dcfe0-kxlzm" event={"ID":"647f9975-219b-4b00-aa2a-93adf2ef16ab","Type":"ContainerDied","Data":"da58cf3b3eff3e47fed59ff37bade58a3b7232285727b4c4e92f99e620fc1fa2"}
Oct 31 20:01:22.448937 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:22.448897    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:23.125265 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:23.125234    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c53dcfe0-kxlzm"
Oct 31 20:01:23.220368 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:23.220290    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/647f9975-219b-4b00-aa2a-93adf2ef16ab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/647f9975-219b-4b00-aa2a-93adf2ef16ab-maas-api-xks-ca\") pod \"647f9975-219b-4b00-aa2a-93adf2ef16ab\" (UID: \"647f9975-219b-4b00-aa2a-93adf2ef16ab\") "
Oct 31 20:01:23.220719 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:23.220680    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/647f9975-219b-4b00-aa2a-93adf2ef16ab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/647f9975-219b-4b00-aa2a-93adf2ef16ab-maas-api-openshift-ca\") pod \"647f9975-219b-4b00-aa2a-93adf2ef16ab\" (UID: \"647f9975-219b-4b00-aa2a-93adf2ef16ab\") "
Oct 31 20:01:23.221054 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:23.221030    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/647f9975-219b-4b00-aa2a-93adf2ef16ab-kube-api-access-4hrds\" (UniqueName: \"kubernetes.io/projected/647f9975-219b-4b00-aa2a-93adf2ef16ab-kube-api-access-4hrds\") pod \"647f9975-219b-4b00-aa2a-93adf2ef16ab\" (UID: \"647f9975-219b-4b00-aa2a-93adf2ef16ab\") "
Oct 31 20:01:23.221685 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:23.221656    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/647f9975-219b-4b00-aa2a-93adf2ef16ab-maas-api-openshift-ca" pod "647f9975-219b-4b00-aa2a-93adf2ef16ab" (UID: "647f9975-219b-4b00-aa2a-93adf2ef16ab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:01:23.221977 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:23.221955    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/647f9975-219b-4b00-aa2a-93adf2ef16ab-maas-api-openshift-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:01:23.224811 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:23.224783    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/647f9975-219b-4b00-aa2a-93adf2ef16ab-kube-api-access-4hrds" pod "647f9975-219b-4b00-aa2a-93adf2ef16ab" (UID: "647f9975-219b-4b00-aa2a-93adf2ef16ab"). InnerVolumeSpecName "kube-api-access-4hrds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:01:23.235585 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:23.235560    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/647f9975-219b-4b00-aa2a-93adf2ef16ab-maas-api-xks-ca" pod "647f9975-219b-4b00-aa2a-93adf2ef16ab" (UID: "647f9975-219b-4b00-aa2a-93adf2ef16ab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:01:23.323299 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:23.323220    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/647f9975-219b-4b00-aa2a-93adf2ef16ab-maas-api-xks-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:01:23.323544 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:23.323521    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hrds\" (UniqueName: \"kubernetes.io/projected/647f9975-219b-4b00-aa2a-93adf2ef16ab-kube-api-access-4hrds\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:01:23.464901 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:23.464862    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:23.811641 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:23.811593    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c53dcfe0-kxlzm" event={"ID":"647f9975-219b-4b00-aa2a-93adf2ef16ab","Type":"ContainerDied","Data":"db606244da324c7bbb50e0074d9b081ec00e8afa6560cabdbe9699e3b76b5bbe"}
Oct 31 20:01:23.811840 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:23.811663    4645 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db606244da324c7bbb50e0074d9b081ec00e8afa6560cabdbe9699e3b76b5bbe"
Oct 31 20:01:23.811840 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:23.811734    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c53dcfe0-kxlzm"
Oct 31 20:01:24.239830 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:24.239656    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c173d8-6d47cf995d-2bjft"
Oct 31 20:01:24.240316 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:24.240099    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c173d8-7dbb76c547-zwgd7"
Oct 31 20:01:24.499991 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:24.499948    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:24.541544 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:24.541194    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-26cd59-84899b8c48-bgz8c"]
Oct 31 20:01:24.543545 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:24.543347    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-26cd59-84899b8c48-bgz8c" podUID="35f95598-1a16-43a3-84c0-f85af4894aeb" containerName="payload-processing" containerID="cri-o://999aa10dcd792ff4c7ac1d5dbd7e89e3df8b5a72e5c3070c7202f8ca18796336" gracePeriod=30
Oct 31 20:01:24.584507 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:24.584468    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-26cd59-585984b9f5-z4pwr"]
Oct 31 20:01:24.584746 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:24.584691    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-26cd59-585984b9f5-z4pwr" podUID="1aa2c43f-d011-4aaf-9f2a-63c63479f3c4" containerName="payload-pre-processing" containerID="cri-o://0f6ed8be296db4fe8f82ed8ba6688325b258d0b07978a2d2ea2fb6c1396d003c" gracePeriod=30
Oct 31 20:01:24.627176 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:24.621131    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:24.733994 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:24.733960    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:24.976833 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:24.976791    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:24.979148 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:24.979096    4645 generic.go:356] "Generic (PLEG): container finished" podID="35f95598-1a16-43a3-84c0-f85af4894aeb" containerID="999aa10dcd792ff4c7ac1d5dbd7e89e3df8b5a72e5c3070c7202f8ca18796336" exitCode=0
Oct 31 20:01:24.979318 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:24.979224    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-26cd59-84899b8c48-bgz8c" event={"ID":"35f95598-1a16-43a3-84c0-f85af4894aeb","Type":"ContainerDied","Data":"999aa10dcd792ff4c7ac1d5dbd7e89e3df8b5a72e5c3070c7202f8ca18796336"}
Oct 31 20:01:24.981895 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:24.981539    4645 generic.go:356] "Generic (PLEG): container finished" podID="1aa2c43f-d011-4aaf-9f2a-63c63479f3c4" containerID="0f6ed8be296db4fe8f82ed8ba6688325b258d0b07978a2d2ea2fb6c1396d003c" exitCode=0
Oct 31 20:01:24.981895 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:24.981670    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-26cd59-585984b9f5-z4pwr" event={"ID":"1aa2c43f-d011-4aaf-9f2a-63c63479f3c4","Type":"ContainerDied","Data":"0f6ed8be296db4fe8f82ed8ba6688325b258d0b07978a2d2ea2fb6c1396d003c"}
Oct 31 20:01:25.087378 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:25.086750    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:25.221977 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:25.221938    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-26cd59-84899b8c48-bgz8c"
Oct 31 20:01:25.264293 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:25.264241    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-26cd59-585984b9f5-z4pwr"
Oct 31 20:01:25.319261 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:25.319223    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1aa2c43f-d011-4aaf-9f2a-63c63479f3c4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1aa2c43f-d011-4aaf-9f2a-63c63479f3c4-plugins-config-volume\") pod \"1aa2c43f-d011-4aaf-9f2a-63c63479f3c4\" (UID: \"1aa2c43f-d011-4aaf-9f2a-63c63479f3c4\") "
Oct 31 20:01:25.319567 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:25.319543    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35f95598-1a16-43a3-84c0-f85af4894aeb-kube-api-access-fm56h\" (UniqueName: \"kubernetes.io/projected/35f95598-1a16-43a3-84c0-f85af4894aeb-kube-api-access-fm56h\") pod \"35f95598-1a16-43a3-84c0-f85af4894aeb\" (UID: \"35f95598-1a16-43a3-84c0-f85af4894aeb\") "
Oct 31 20:01:25.319785 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:25.319766    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1aa2c43f-d011-4aaf-9f2a-63c63479f3c4-kube-api-access-xc5j8\" (UniqueName: \"kubernetes.io/projected/1aa2c43f-d011-4aaf-9f2a-63c63479f3c4-kube-api-access-xc5j8\") pod \"1aa2c43f-d011-4aaf-9f2a-63c63479f3c4\" (UID: \"1aa2c43f-d011-4aaf-9f2a-63c63479f3c4\") "
Oct 31 20:01:25.320162 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:25.320137    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35f95598-1a16-43a3-84c0-f85af4894aeb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35f95598-1a16-43a3-84c0-f85af4894aeb-plugins-config-volume\") pod \"35f95598-1a16-43a3-84c0-f85af4894aeb\" (UID: \"35f95598-1a16-43a3-84c0-f85af4894aeb\") "
Oct 31 20:01:25.321870 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:25.321844    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35f95598-1a16-43a3-84c0-f85af4894aeb-plugins-config-volume" pod "35f95598-1a16-43a3-84c0-f85af4894aeb" (UID: "35f95598-1a16-43a3-84c0-f85af4894aeb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:01:25.322477 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:25.322451    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1aa2c43f-d011-4aaf-9f2a-63c63479f3c4-plugins-config-volume" pod "1aa2c43f-d011-4aaf-9f2a-63c63479f3c4" (UID: "1aa2c43f-d011-4aaf-9f2a-63c63479f3c4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:01:25.332201 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:25.332166    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35f95598-1a16-43a3-84c0-f85af4894aeb-kube-api-access-fm56h" pod "35f95598-1a16-43a3-84c0-f85af4894aeb" (UID: "35f95598-1a16-43a3-84c0-f85af4894aeb"). InnerVolumeSpecName "kube-api-access-fm56h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:01:25.333482 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:25.333451    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aa2c43f-d011-4aaf-9f2a-63c63479f3c4-kube-api-access-xc5j8" pod "1aa2c43f-d011-4aaf-9f2a-63c63479f3c4" (UID: "1aa2c43f-d011-4aaf-9f2a-63c63479f3c4"). InnerVolumeSpecName "kube-api-access-xc5j8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:01:25.390771 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:25.390744    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:25.422823 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:25.422794    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xc5j8\" (UniqueName: \"kubernetes.io/projected/1aa2c43f-d011-4aaf-9f2a-63c63479f3c4-kube-api-access-xc5j8\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:01:25.422947 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:25.422828    4645 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/35f95598-1a16-43a3-84c0-f85af4894aeb-plugins-config-volume\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:01:25.422947 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:25.422843    4645 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1aa2c43f-d011-4aaf-9f2a-63c63479f3c4-plugins-config-volume\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:01:25.422947 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:25.422857    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fm56h\" (UniqueName: \"kubernetes.io/projected/35f95598-1a16-43a3-84c0-f85af4894aeb-kube-api-access-fm56h\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:01:25.997269 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:25.997224    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-26cd59-84899b8c48-bgz8c" event={"ID":"35f95598-1a16-43a3-84c0-f85af4894aeb","Type":"ContainerDied","Data":"e7ec891feee59d75fc0ed13e42eacd6a06114d5f0c55fe3ada2d7464a72cc97e"}
Oct 31 20:01:25.997486 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:25.997308    4645 scope.go:122] "RemoveContainer" containerID="999aa10dcd792ff4c7ac1d5dbd7e89e3df8b5a72e5c3070c7202f8ca18796336"
Oct 31 20:01:25.997645 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:25.997623    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-26cd59-84899b8c48-bgz8c"
Oct 31 20:01:26.001415 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:26.001382    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-26cd59-585984b9f5-z4pwr" event={"ID":"1aa2c43f-d011-4aaf-9f2a-63c63479f3c4","Type":"ContainerDied","Data":"13e60e2339ec7db90b1066d419d7f8d23ecacc1e27054afdc268d540a556aa64"}
Oct 31 20:01:26.001585 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:26.001501    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-26cd59-585984b9f5-z4pwr"
Oct 31 20:01:26.019565 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:26.019538    4645 scope.go:122] "RemoveContainer" containerID="0f6ed8be296db4fe8f82ed8ba6688325b258d0b07978a2d2ea2fb6c1396d003c"
Oct 31 20:01:26.034128 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:26.034097    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-26cd59-84899b8c48-bgz8c"]
Oct 31 20:01:26.041624 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:26.040366    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-26cd59-84899b8c48-bgz8c"]
Oct 31 20:01:26.050384 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:26.050354    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-26cd59-585984b9f5-z4pwr"]
Oct 31 20:01:26.058806 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:26.058771    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-26cd59-585984b9f5-z4pwr"]
Oct 31 20:01:26.095666 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:26.095628    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:26.553620 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:26.553574    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:26.984805 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:26.984774    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8"]
Oct 31 20:01:26.985578 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:26.985550    4645 cpu_manager.go:414] "containerMap: removing container" podUID="1aa2c43f-d011-4aaf-9f2a-63c63479f3c4" containerName="payload-pre-processing"
Oct 31 20:01:26.985578 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:26.985571    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1aa2c43f-d011-4aaf-9f2a-63c63479f3c4" containerName="payload-pre-processing"
Oct 31 20:01:26.985777 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:26.985621    4645 cpu_manager.go:414] "containerMap: removing container" podUID="647f9975-219b-4b00-aa2a-93adf2ef16ab" containerName="revoke-keys"
Oct 31 20:01:26.985777 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:26.985630    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="647f9975-219b-4b00-aa2a-93adf2ef16ab" containerName="revoke-keys"
Oct 31 20:01:26.985777 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:26.985659    4645 cpu_manager.go:414] "containerMap: removing container" podUID="35f95598-1a16-43a3-84c0-f85af4894aeb" containerName="payload-processing"
Oct 31 20:01:26.985777 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:26.985667    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35f95598-1a16-43a3-84c0-f85af4894aeb" containerName="payload-processing"
Oct 31 20:01:26.986003 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:26.985824    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="647f9975-219b-4b00-aa2a-93adf2ef16ab" containerName="revoke-keys"
Oct 31 20:01:26.986003 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:26.985841    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="35f95598-1a16-43a3-84c0-f85af4894aeb" containerName="payload-processing"
Oct 31 20:01:26.986003 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:26.985859    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="1aa2c43f-d011-4aaf-9f2a-63c63479f3c4" containerName="payload-pre-processing"
Oct 31 20:01:26.995145 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:26.995114    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:27.067400 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:27.067317    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8"]
Oct 31 20:01:27.067602 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:27.067438    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8"
Oct 31 20:01:27.073901 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:27.073865    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-c9b9a1-openshift-default-dockercfg-mjrg4\""
Oct 31 20:01:27.177407 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:27.164567    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af802480-762e-446b-95d5-7754f4bad230-istio-data\") pod \"e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8\" (UID: \"af802480-762e-446b-95d5-7754f4bad230\") " pod="openshift-ingress/e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8"
Oct 31 20:01:27.177407 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:27.164619    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw44d\" (UniqueName: \"kubernetes.io/projected/af802480-762e-446b-95d5-7754f4bad230-kube-api-access-fw44d\") pod \"e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8\" (UID: \"af802480-762e-446b-95d5-7754f4bad230\") " pod="openshift-ingress/e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8"
Oct 31 20:01:27.177407 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:27.164685    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af802480-762e-446b-95d5-7754f4bad230-workload-certs\") pod \"e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8\" (UID: \"af802480-762e-446b-95d5-7754f4bad230\") " pod="openshift-ingress/e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8"
Oct 31 20:01:27.177407 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:27.164720    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af802480-762e-446b-95d5-7754f4bad230-istio-envoy\") pod \"e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8\" (UID: \"af802480-762e-446b-95d5-7754f4bad230\") " pod="openshift-ingress/e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8"
Oct 31 20:01:27.177407 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:27.164817    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af802480-762e-446b-95d5-7754f4bad230-istio-token\") pod \"e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8\" (UID: \"af802480-762e-446b-95d5-7754f4bad230\") " pod="openshift-ingress/e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8"
Oct 31 20:01:27.177407 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:27.164861    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af802480-762e-446b-95d5-7754f4bad230-istiod-ca-cert\") pod \"e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8\" (UID: \"af802480-762e-446b-95d5-7754f4bad230\") " pod="openshift-ingress/e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8"
Oct 31 20:01:27.177407 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:27.164915    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af802480-762e-446b-95d5-7754f4bad230-workload-socket\") pod \"e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8\" (UID: \"af802480-762e-446b-95d5-7754f4bad230\") " pod="openshift-ingress/e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8"
Oct 31 20:01:27.177407 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:27.164999    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af802480-762e-446b-95d5-7754f4bad230-credential-socket\") pod \"e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8\" (UID: \"af802480-762e-446b-95d5-7754f4bad230\") " pod="openshift-ingress/e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8"
Oct 31 20:01:27.177407 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:27.165203    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af802480-762e-446b-95d5-7754f4bad230-istio-podinfo\") pod \"e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8\" (UID: \"af802480-762e-446b-95d5-7754f4bad230\") " pod="openshift-ingress/e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8"
Oct 31 20:01:27.182564 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:27.182533    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:27.271401 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:27.270398    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af802480-762e-446b-95d5-7754f4bad230-istio-podinfo\") pod \"e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8\" (UID: \"af802480-762e-446b-95d5-7754f4bad230\") " pod="openshift-ingress/e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8"
Oct 31 20:01:27.271401 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:27.270546    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af802480-762e-446b-95d5-7754f4bad230-istio-data\") pod \"e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8\" (UID: \"af802480-762e-446b-95d5-7754f4bad230\") " pod="openshift-ingress/e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8"
Oct 31 20:01:27.271401 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:27.270577    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw44d\" (UniqueName: \"kubernetes.io/projected/af802480-762e-446b-95d5-7754f4bad230-kube-api-access-fw44d\") pod \"e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8\" (UID: \"af802480-762e-446b-95d5-7754f4bad230\") " pod="openshift-ingress/e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8"
Oct 31 20:01:27.271401 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:27.270631    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af802480-762e-446b-95d5-7754f4bad230-workload-certs\") pod \"e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8\" (UID: \"af802480-762e-446b-95d5-7754f4bad230\") " pod="openshift-ingress/e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8"
Oct 31 20:01:27.271401 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:27.270663    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af802480-762e-446b-95d5-7754f4bad230-istio-envoy\") pod \"e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8\" (UID: \"af802480-762e-446b-95d5-7754f4bad230\") " pod="openshift-ingress/e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8"
Oct 31 20:01:27.271401 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:27.270739    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af802480-762e-446b-95d5-7754f4bad230-istio-token\") pod \"e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8\" (UID: \"af802480-762e-446b-95d5-7754f4bad230\") " pod="openshift-ingress/e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8"
Oct 31 20:01:27.271401 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:27.270775    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af802480-762e-446b-95d5-7754f4bad230-istiod-ca-cert\") pod \"e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8\" (UID: \"af802480-762e-446b-95d5-7754f4bad230\") " pod="openshift-ingress/e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8"
Oct 31 20:01:27.271401 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:27.270821    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af802480-762e-446b-95d5-7754f4bad230-workload-socket\") pod \"e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8\" (UID: \"af802480-762e-446b-95d5-7754f4bad230\") " pod="openshift-ingress/e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8"
Oct 31 20:01:27.271401 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:27.270884    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af802480-762e-446b-95d5-7754f4bad230-credential-socket\") pod \"e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8\" (UID: \"af802480-762e-446b-95d5-7754f4bad230\") " pod="openshift-ingress/e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8"
Oct 31 20:01:27.272065 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:27.271523    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af802480-762e-446b-95d5-7754f4bad230-credential-socket\") pod \"e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8\" (UID: \"af802480-762e-446b-95d5-7754f4bad230\") " pod="openshift-ingress/e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8"
Oct 31 20:01:27.290351 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:27.273020    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af802480-762e-446b-95d5-7754f4bad230-istiod-ca-cert\") pod \"e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8\" (UID: \"af802480-762e-446b-95d5-7754f4bad230\") " pod="openshift-ingress/e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8"
Oct 31 20:01:27.290351 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:27.273939    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af802480-762e-446b-95d5-7754f4bad230-istio-data\") pod \"e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8\" (UID: \"af802480-762e-446b-95d5-7754f4bad230\") " pod="openshift-ingress/e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8"
Oct 31 20:01:27.290351 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:27.274181    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af802480-762e-446b-95d5-7754f4bad230-workload-socket\") pod \"e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8\" (UID: \"af802480-762e-446b-95d5-7754f4bad230\") " pod="openshift-ingress/e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8"
Oct 31 20:01:27.290351 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:27.276758    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af802480-762e-446b-95d5-7754f4bad230-workload-certs\") pod \"e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8\" (UID: \"af802480-762e-446b-95d5-7754f4bad230\") " pod="openshift-ingress/e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8"
Oct 31 20:01:27.316350 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:27.297138    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af802480-762e-446b-95d5-7754f4bad230-istio-podinfo\") pod \"e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8\" (UID: \"af802480-762e-446b-95d5-7754f4bad230\") " pod="openshift-ingress/e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8"
Oct 31 20:01:27.316350 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:27.311912    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af802480-762e-446b-95d5-7754f4bad230-istio-envoy\") pod \"e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8\" (UID: \"af802480-762e-446b-95d5-7754f4bad230\") " pod="openshift-ingress/e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8"
Oct 31 20:01:27.330410 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:27.326716    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af802480-762e-446b-95d5-7754f4bad230-istio-token\") pod \"e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8\" (UID: \"af802480-762e-446b-95d5-7754f4bad230\") " pod="openshift-ingress/e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8"
Oct 31 20:01:27.330410 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:27.327312    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw44d\" (UniqueName: \"kubernetes.io/projected/af802480-762e-446b-95d5-7754f4bad230-kube-api-access-fw44d\") pod \"e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8\" (UID: \"af802480-762e-446b-95d5-7754f4bad230\") " pod="openshift-ingress/e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8"
Oct 31 20:01:27.453082 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:27.453047    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8"
Oct 31 20:01:27.625084 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:27.625047    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1aa2c43f-d011-4aaf-9f2a-63c63479f3c4" path="/var/lib/kubelet/pods/1aa2c43f-d011-4aaf-9f2a-63c63479f3c4/volumes"
Oct 31 20:01:27.625735 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:27.625711    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35f95598-1a16-43a3-84c0-f85af4894aeb" path="/var/lib/kubelet/pods/35f95598-1a16-43a3-84c0-f85af4894aeb/volumes"
Oct 31 20:01:27.786921 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:27.786891    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8"]
Oct 31 20:01:27.787096 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:01:27.786892    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf802480_762e_446b_95d5_7754f4bad230.slice/crio-82320e1b9e68cf114bcc959d6692b7a5b9394a9f9f6c370ea205c077d6991210 WatchSource:0}: Error finding container 82320e1b9e68cf114bcc959d6692b7a5b9394a9f9f6c370ea205c077d6991210: Status 404 returned error can't find the container with id 82320e1b9e68cf114bcc959d6692b7a5b9394a9f9f6c370ea205c077d6991210
Oct 31 20:01:27.789626 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:27.789593    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:01:27.789693 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:27.789657    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:01:27.789693 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:27.789684    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:01:27.848692 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:27.848661    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:27.897872 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:27.897438    4645 patch_prober.go:29] interesting pod/e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 20:01:27.897872 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:27.897508    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b" podUID="710e3436-3828-45fb-b090-e28d1147cbfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 20:01:28.022222 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:28.022186    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8" event={"ID":"af802480-762e-446b-95d5-7754f4bad230","Type":"ContainerStarted","Data":"56b8550b6909ce6e6c49e17a78b45605af8219a8eb57b4f2cb4aacb62527450e"}
Oct 31 20:01:28.022437 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:28.022242    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8" event={"ID":"af802480-762e-446b-95d5-7754f4bad230","Type":"ContainerStarted","Data":"82320e1b9e68cf114bcc959d6692b7a5b9394a9f9f6c370ea205c077d6991210"}
Oct 31 20:01:28.049635 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:28.049577    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8" podStartSLOduration=2.049558667 podStartE2EDuration="2.049558667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:01:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:01:28.045656901 +0000 UTC m=+2518.618891028" watchObservedRunningTime="2026-10-31 20:01:28.049558667 +0000 UTC m=+2518.622792796"
Oct 31 20:01:28.112526 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:28.112450    4645 patch_prober.go:29] interesting pod/e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 20:01:28.112526 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:28.112500    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v" podUID="59f3365e-9975-44b4-b7c5-02e91e891f8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 20:01:28.453710 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:28.453678    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8"
Oct 31 20:01:28.459120 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:28.459098    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8"
Oct 31 20:01:28.646178 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:28.646146    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:29.031636 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:29.031598    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8"
Oct 31 20:01:29.032646 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:29.032623    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8"
Oct 31 20:01:29.765816 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:29.765782    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:30.444152 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:30.444102    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:30.552496 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:30.552452    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:30.972472 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:30.972385    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:31.660366 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:31.660321    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:31.743005 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:31.742980    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c9b9a1-79688949cf-7krhv"]
Oct 31 20:01:31.786851 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:31.785981    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c9b9a1-79688949cf-7krhv"]
Oct 31 20:01:31.786851 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:31.786036    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c9b9a1-5c968dc557-lrnmf"]
Oct 31 20:01:31.786851 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:31.786528    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c9b9a1-79688949cf-7krhv"
Oct 31 20:01:31.788905 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:31.788664    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:31.803782 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:31.801589    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-c9b9a1\""
Oct 31 20:01:31.803782 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:31.801608    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-c9b9a1-dockercfg-452h5\""
Oct 31 20:01:31.839136 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:31.838916    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c9b9a1-5c968dc557-lrnmf"]
Oct 31 20:01:31.839369 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:31.839064    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c9b9a1-5c968dc557-lrnmf"
Oct 31 20:01:31.878793 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:31.878618    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48jr9\" (UniqueName: \"kubernetes.io/projected/14316266-1376-4285-8df1-a57cab31ab04-kube-api-access-48jr9\") pod \"payload-pre-processing-e2e-ipp-b-c9b9a1-79688949cf-7krhv\" (UID: \"14316266-1376-4285-8df1-a57cab31ab04\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c9b9a1-79688949cf-7krhv"
Oct 31 20:01:31.879168 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:31.879143    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14316266-1376-4285-8df1-a57cab31ab04-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c9b9a1-79688949cf-7krhv\" (UID: \"14316266-1376-4285-8df1-a57cab31ab04\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c9b9a1-79688949cf-7krhv"
Oct 31 20:01:31.879490 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:31.879462    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfntt\" (UniqueName: \"kubernetes.io/projected/b29a6202-a717-4880-8c43-03967211c64c-kube-api-access-wfntt\") pod \"payload-processing-e2e-ipp-b-c9b9a1-5c968dc557-lrnmf\" (UID: \"b29a6202-a717-4880-8c43-03967211c64c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c9b9a1-5c968dc557-lrnmf"
Oct 31 20:01:31.879752 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:31.879733    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b29a6202-a717-4880-8c43-03967211c64c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c9b9a1-5c968dc557-lrnmf\" (UID: \"b29a6202-a717-4880-8c43-03967211c64c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c9b9a1-5c968dc557-lrnmf"
Oct 31 20:01:31.972130 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:31.971699    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:32.005351 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:32.002716    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfntt\" (UniqueName: \"kubernetes.io/projected/b29a6202-a717-4880-8c43-03967211c64c-kube-api-access-wfntt\") pod \"payload-processing-e2e-ipp-b-c9b9a1-5c968dc557-lrnmf\" (UID: \"b29a6202-a717-4880-8c43-03967211c64c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c9b9a1-5c968dc557-lrnmf"
Oct 31 20:01:32.005351 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:32.002906    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b29a6202-a717-4880-8c43-03967211c64c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c9b9a1-5c968dc557-lrnmf\" (UID: \"b29a6202-a717-4880-8c43-03967211c64c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c9b9a1-5c968dc557-lrnmf"
Oct 31 20:01:32.005351 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:32.003015    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48jr9\" (UniqueName: \"kubernetes.io/projected/14316266-1376-4285-8df1-a57cab31ab04-kube-api-access-48jr9\") pod \"payload-pre-processing-e2e-ipp-b-c9b9a1-79688949cf-7krhv\" (UID: \"14316266-1376-4285-8df1-a57cab31ab04\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c9b9a1-79688949cf-7krhv"
Oct 31 20:01:32.005351 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:32.003151    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14316266-1376-4285-8df1-a57cab31ab04-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c9b9a1-79688949cf-7krhv\" (UID: \"14316266-1376-4285-8df1-a57cab31ab04\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c9b9a1-79688949cf-7krhv"
Oct 31 20:01:32.005351 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:32.004283    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14316266-1376-4285-8df1-a57cab31ab04-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c9b9a1-79688949cf-7krhv\" (UID: \"14316266-1376-4285-8df1-a57cab31ab04\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c9b9a1-79688949cf-7krhv"
Oct 31 20:01:32.016490 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:32.005493    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b29a6202-a717-4880-8c43-03967211c64c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c9b9a1-5c968dc557-lrnmf\" (UID: \"b29a6202-a717-4880-8c43-03967211c64c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c9b9a1-5c968dc557-lrnmf"
Oct 31 20:01:32.086747 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:32.076623    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48jr9\" (UniqueName: \"kubernetes.io/projected/14316266-1376-4285-8df1-a57cab31ab04-kube-api-access-48jr9\") pod \"payload-pre-processing-e2e-ipp-b-c9b9a1-79688949cf-7krhv\" (UID: \"14316266-1376-4285-8df1-a57cab31ab04\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c9b9a1-79688949cf-7krhv"
Oct 31 20:01:32.131398 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:32.107835    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfntt\" (UniqueName: \"kubernetes.io/projected/b29a6202-a717-4880-8c43-03967211c64c-kube-api-access-wfntt\") pod \"payload-processing-e2e-ipp-b-c9b9a1-5c968dc557-lrnmf\" (UID: \"b29a6202-a717-4880-8c43-03967211c64c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c9b9a1-5c968dc557-lrnmf"
Oct 31 20:01:32.152427 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:32.140919    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c9b9a1-79688949cf-7krhv"
Oct 31 20:01:32.240572 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:32.238713    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c9b9a1-5c968dc557-lrnmf"
Oct 31 20:01:32.555009 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:32.554959    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:32.669587 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:32.669470    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:32.674382 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:32.674350    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bd1c9f-66bb876fcf-qnnw8"]
Oct 31 20:01:32.698409 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:32.696153    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bd1c9f-6dbbc4dc79-v5wmk"]
Oct 31 20:01:32.710847 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:32.710816    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c734f5-786f585bf5-bs8jm"]
Oct 31 20:01:32.769518 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:01:32.769484    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14316266_1376_4285_8df1_a57cab31ab04.slice/crio-239ebeb918eff856b4a5baf2304099bfbd1efcb094f59c36ee2b36a18f9aff3c WatchSource:0}: Error finding container 239ebeb918eff856b4a5baf2304099bfbd1efcb094f59c36ee2b36a18f9aff3c: Status 404 returned error can't find the container with id 239ebeb918eff856b4a5baf2304099bfbd1efcb094f59c36ee2b36a18f9aff3c
Oct 31 20:01:32.801456 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:32.801295    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:32.838108 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:32.836312    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c734f5-786f585bf5-bs8jm"]
Oct 31 20:01:32.838108 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:32.836401    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c9b9a1-79688949cf-7krhv"]
Oct 31 20:01:32.838108 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:32.836434    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c9b9a1-5c968dc557-lrnmf"]
Oct 31 20:01:32.838108 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:32.836480    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c734f5-684c4f6c9d-xdjpr"]
Oct 31 20:01:32.838108 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:32.837618    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c734f5-786f585bf5-bs8jm"
Oct 31 20:01:32.850013 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:32.849990    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-c734f5\""
Oct 31 20:01:32.862616 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:32.862589    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c734f5-684c4f6c9d-xdjpr"]
Oct 31 20:01:32.862752 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:32.862723    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c734f5-684c4f6c9d-xdjpr"
Oct 31 20:01:32.882693 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:32.882667    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-c734f5-dockercfg-4bcbr\""
Oct 31 20:01:33.132415 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:33.051261    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:33.132415 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:33.117520    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8e321a4-db7b-4db9-ab08-d70604893cef-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c734f5-684c4f6c9d-xdjpr\" (UID: \"b8e321a4-db7b-4db9-ab08-d70604893cef\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c734f5-684c4f6c9d-xdjpr"
Oct 31 20:01:33.132415 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:33.117657    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c58165b3-72d7-4195-b703-17b3104d7207-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c734f5-786f585bf5-bs8jm\" (UID: \"c58165b3-72d7-4195-b703-17b3104d7207\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c734f5-786f585bf5-bs8jm"
Oct 31 20:01:33.132415 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:33.117689    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7lzl\" (UniqueName: \"kubernetes.io/projected/b8e321a4-db7b-4db9-ab08-d70604893cef-kube-api-access-x7lzl\") pod \"payload-processing-e2e-ipp-a-c734f5-684c4f6c9d-xdjpr\" (UID: \"b8e321a4-db7b-4db9-ab08-d70604893cef\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c734f5-684c4f6c9d-xdjpr"
Oct 31 20:01:33.132415 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:33.117888    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl64n\" (UniqueName: \"kubernetes.io/projected/c58165b3-72d7-4195-b703-17b3104d7207-kube-api-access-wl64n\") pod \"payload-pre-processing-e2e-ipp-a-c734f5-786f585bf5-bs8jm\" (UID: \"c58165b3-72d7-4195-b703-17b3104d7207\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c734f5-786f585bf5-bs8jm"
Oct 31 20:01:33.176361 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:33.167636    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c9b9a1-5c968dc557-lrnmf" event={"ID":"b29a6202-a717-4880-8c43-03967211c64c","Type":"ContainerStarted","Data":"ad2c45fe415be4240ede033c59630fe5ab5c2fdc6f969b7b06d0ba0baf4f3812"}
Oct 31 20:01:33.176361 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:33.167703    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c9b9a1-5c968dc557-lrnmf" event={"ID":"b29a6202-a717-4880-8c43-03967211c64c","Type":"ContainerStarted","Data":"f016d9baf0ce85d306eaa15469e5a3c24067f661aec368ea4f2700f96c4c216e"}
Oct 31 20:01:33.176361 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:33.169597    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c9b9a1-5c968dc557-lrnmf"
Oct 31 20:01:33.230369 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:33.228505    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:33.230369 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:33.230026    4645 generic.go:356] "Generic (PLEG): container finished" podID="710e3436-3828-45fb-b090-e28d1147cbfe" containerID="0af901541f97b5801a6cbce660bb1a1c5085864c764964cb4ce45babf30fafa0" exitCode=0
Oct 31 20:01:33.230369 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:33.230265    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b" event={"ID":"710e3436-3828-45fb-b090-e28d1147cbfe","Type":"ContainerDied","Data":"0af901541f97b5801a6cbce660bb1a1c5085864c764964cb4ce45babf30fafa0"}
Oct 31 20:01:33.235356 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:33.230779    4645 scope.go:122] "RemoveContainer" containerID="0af901541f97b5801a6cbce660bb1a1c5085864c764964cb4ce45babf30fafa0"
Oct 31 20:01:33.235356 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:33.233032    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-c9b9a1-5c968dc557-lrnmf" podStartSLOduration=2.233015419 podStartE2EDuration="2.233015419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:01:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:01:33.225559456 +0000 UTC m=+2523.798793586" watchObservedRunningTime="2026-10-31 20:01:33.233015419 +0000 UTC m=+2523.806249551"
Oct 31 20:01:33.235356 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:33.234253    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:01:33.235356 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:33.234420    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:01:33.235356 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:33.234519    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:01:33.246508 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:33.246476    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c9b9a1-79688949cf-7krhv" event={"ID":"14316266-1376-4285-8df1-a57cab31ab04","Type":"ContainerStarted","Data":"ac455eac69b35afda7482c90af276773fc60378cac4df99d961f61ce8e259bd8"}
Oct 31 20:01:33.247478 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:33.247458    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c9b9a1-79688949cf-7krhv"
Oct 31 20:01:33.247575 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:33.247484    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c9b9a1-79688949cf-7krhv" event={"ID":"14316266-1376-4285-8df1-a57cab31ab04","Type":"ContainerStarted","Data":"239ebeb918eff856b4a5baf2304099bfbd1efcb094f59c36ee2b36a18f9aff3c"}
Oct 31 20:01:33.288997 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:33.286954    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8e321a4-db7b-4db9-ab08-d70604893cef-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c734f5-684c4f6c9d-xdjpr\" (UID: \"b8e321a4-db7b-4db9-ab08-d70604893cef\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c734f5-684c4f6c9d-xdjpr"
Oct 31 20:01:33.288997 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:33.287041    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c58165b3-72d7-4195-b703-17b3104d7207-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c734f5-786f585bf5-bs8jm\" (UID: \"c58165b3-72d7-4195-b703-17b3104d7207\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c734f5-786f585bf5-bs8jm"
Oct 31 20:01:33.288997 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:33.287067    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7lzl\" (UniqueName: \"kubernetes.io/projected/b8e321a4-db7b-4db9-ab08-d70604893cef-kube-api-access-x7lzl\") pod \"payload-processing-e2e-ipp-a-c734f5-684c4f6c9d-xdjpr\" (UID: \"b8e321a4-db7b-4db9-ab08-d70604893cef\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c734f5-684c4f6c9d-xdjpr"
Oct 31 20:01:33.288997 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:33.287236    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl64n\" (UniqueName: \"kubernetes.io/projected/c58165b3-72d7-4195-b703-17b3104d7207-kube-api-access-wl64n\") pod \"payload-pre-processing-e2e-ipp-a-c734f5-786f585bf5-bs8jm\" (UID: \"c58165b3-72d7-4195-b703-17b3104d7207\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c734f5-786f585bf5-bs8jm"
Oct 31 20:01:33.291359 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:33.289493    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c58165b3-72d7-4195-b703-17b3104d7207-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c734f5-786f585bf5-bs8jm\" (UID: \"c58165b3-72d7-4195-b703-17b3104d7207\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c734f5-786f585bf5-bs8jm"
Oct 31 20:01:33.291359 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:33.290230    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8e321a4-db7b-4db9-ab08-d70604893cef-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c734f5-684c4f6c9d-xdjpr\" (UID: \"b8e321a4-db7b-4db9-ab08-d70604893cef\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c734f5-684c4f6c9d-xdjpr"
Oct 31 20:01:33.291359 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:33.291256    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c9b9a1-79688949cf-7krhv" podStartSLOduration=2.291242919 podStartE2EDuration="2.291242919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:01:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:01:33.283441152 +0000 UTC m=+2523.856675278" watchObservedRunningTime="2026-10-31 20:01:33.291242919 +0000 UTC m=+2523.864477046"
Oct 31 20:01:33.320069 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:33.310858    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7lzl\" (UniqueName: \"kubernetes.io/projected/b8e321a4-db7b-4db9-ab08-d70604893cef-kube-api-access-x7lzl\") pod \"payload-processing-e2e-ipp-a-c734f5-684c4f6c9d-xdjpr\" (UID: \"b8e321a4-db7b-4db9-ab08-d70604893cef\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c734f5-684c4f6c9d-xdjpr"
Oct 31 20:01:33.320069 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:33.317880    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl64n\" (UniqueName: \"kubernetes.io/projected/c58165b3-72d7-4195-b703-17b3104d7207-kube-api-access-wl64n\") pod \"payload-pre-processing-e2e-ipp-a-c734f5-786f585bf5-bs8jm\" (UID: \"c58165b3-72d7-4195-b703-17b3104d7207\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c734f5-786f585bf5-bs8jm"
Oct 31 20:01:33.401658 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:33.401632    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:33.492743 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:33.492709    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c734f5-786f585bf5-bs8jm"
Oct 31 20:01:33.500799 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:33.500768    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c734f5-684c4f6c9d-xdjpr"
Oct 31 20:01:34.115978 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:01:34.115938    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc58165b3_72d7_4195_b703_17b3104d7207.slice/crio-f6936057dec68641affce77bba6c7799332cdb212eace4e20ca364d5ba493500 WatchSource:0}: Error finding container f6936057dec68641affce77bba6c7799332cdb212eace4e20ca364d5ba493500: Status 404 returned error can't find the container with id f6936057dec68641affce77bba6c7799332cdb212eace4e20ca364d5ba493500
Oct 31 20:01:34.116440 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:34.116050    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c734f5-786f585bf5-bs8jm"]
Oct 31 20:01:34.146964 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:34.146348    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c734f5-684c4f6c9d-xdjpr"]
Oct 31 20:01:34.146964 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:01:34.146846    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8e321a4_db7b_4db9_ab08_d70604893cef.slice/crio-1df7e2f662ebe37d05b2955a077ebdde3c352f23fae8d9108c0c7ea7d971d3eb WatchSource:0}: Error finding container 1df7e2f662ebe37d05b2955a077ebdde3c352f23fae8d9108c0c7ea7d971d3eb: Status 404 returned error can't find the container with id 1df7e2f662ebe37d05b2955a077ebdde3c352f23fae8d9108c0c7ea7d971d3eb
Oct 31 20:01:34.255690 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:34.255654    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c734f5-786f585bf5-bs8jm" event={"ID":"c58165b3-72d7-4195-b703-17b3104d7207","Type":"ContainerStarted","Data":"f6936057dec68641affce77bba6c7799332cdb212eace4e20ca364d5ba493500"}
Oct 31 20:01:34.257118 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:34.257091    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c734f5-684c4f6c9d-xdjpr" event={"ID":"b8e321a4-db7b-4db9-ab08-d70604893cef","Type":"ContainerStarted","Data":"1df7e2f662ebe37d05b2955a077ebdde3c352f23fae8d9108c0c7ea7d971d3eb"}
Oct 31 20:01:34.260147 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:34.260122    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b" event={"ID":"710e3436-3828-45fb-b090-e28d1147cbfe","Type":"ContainerStarted","Data":"02bcde84b1170dd9c98e7e4e3557752013a43e5911a91514dbbd3ac1076bf67f"}
Oct 31 20:01:34.391063 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:34.390980    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:34.731111 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:34.730180    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b"
Oct 31 20:01:34.731111 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:34.730285    4645 patch_prober.go:29] interesting pod/e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body=
Oct 31 20:01:34.731111 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:34.730341    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b" podUID="710e3436-3828-45fb-b090-e28d1147cbfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused"
Oct 31 20:01:34.788368 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:34.787660    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:35.019756 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:35.019519    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:35.235656 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:35.235498    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:35.336682 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:35.336525    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c734f5-786f585bf5-bs8jm" event={"ID":"c58165b3-72d7-4195-b703-17b3104d7207","Type":"ContainerStarted","Data":"4217234607ca6c91f1e0a1834bad73da3bd005576bfefcb1c6809b3cdea67264"}
Oct 31 20:01:35.338231 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:35.338001    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c734f5-786f585bf5-bs8jm"
Oct 31 20:01:35.342073 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:35.341570    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c734f5-684c4f6c9d-xdjpr" event={"ID":"b8e321a4-db7b-4db9-ab08-d70604893cef","Type":"ContainerStarted","Data":"f61a0e09d9fffc1217432055e1d7383a9fb3ea5c9354f1f41328c59f1136d3ff"}
Oct 31 20:01:35.342073 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:35.342020    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c734f5-684c4f6c9d-xdjpr"
Oct 31 20:01:35.416366 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:35.413093    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:35.489367 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:35.436201    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c734f5-786f585bf5-bs8jm" podStartSLOduration=3.436172046 podStartE2EDuration="3.436172046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:01:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:01:35.375479073 +0000 UTC m=+2525.948713200" watchObservedRunningTime="2026-10-31 20:01:35.436172046 +0000 UTC m=+2526.009406176"
Oct 31 20:01:35.489367 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:35.438733    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-c734f5-684c4f6c9d-xdjpr" podStartSLOduration=3.438715544 podStartE2EDuration="3.438715544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:01:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:01:35.414903318 +0000 UTC m=+2525.988137446" watchObservedRunningTime="2026-10-31 20:01:35.438715544 +0000 UTC m=+2526.011949675"
Oct 31 20:01:35.653137 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:35.653048    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:35.748422 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:35.737725    4645 patch_prober.go:29] interesting pod/e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body=
Oct 31 20:01:35.748422 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:35.737789    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b" podUID="710e3436-3828-45fb-b090-e28d1147cbfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused"
Oct 31 20:01:36.304726 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:01:36.304676    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 20:01:36.318799 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:36.318766    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:36.728365 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:36.728316    4645 patch_prober.go:29] interesting pod/e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body=
Oct 31 20:01:36.728538 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:36.728391    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b" podUID="710e3436-3828-45fb-b090-e28d1147cbfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused"
Oct 31 20:01:36.790013 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:36.789913    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:37.476353 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:37.472710    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:37.682349 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:37.675701    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-bd1c9f-66bb876fcf-qnnw8" podUID="1474c24c-f01e-4b1d-93a0-c5e2d4b1f929" containerName="payload-processing" containerID="cri-o://5fa51893c657bd2c2700aa6c5e6ba19827c6736d440c0d852de467b67936b238" gracePeriod=25
Oct 31 20:01:37.697222 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:37.697184    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd1c9f-6dbbc4dc79-v5wmk" podUID="263d2ffb-ab33-432a-83cd-dc73f244bc82" containerName="payload-pre-processing" containerID="cri-o://5c9d41c0c25e9551c365d3a92514a6448b4f837bad42d886ba13b720c164c8fc" gracePeriod=25
Oct 31 20:01:37.729253 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:37.729206    4645 patch_prober.go:29] interesting pod/e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body=
Oct 31 20:01:37.729406 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:37.729268    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b" podUID="710e3436-3828-45fb-b090-e28d1147cbfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused"
Oct 31 20:01:38.050790 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:38.050758    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bd1c9f-66bb876fcf-qnnw8"
Oct 31 20:01:38.104608 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:38.103988    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd1c9f-6dbbc4dc79-v5wmk"
Oct 31 20:01:38.132049 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:38.132012    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:38.137436 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:38.137409    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1474c24c-f01e-4b1d-93a0-c5e2d4b1f929-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1474c24c-f01e-4b1d-93a0-c5e2d4b1f929-plugins-config-volume\") pod \"1474c24c-f01e-4b1d-93a0-c5e2d4b1f929\" (UID: \"1474c24c-f01e-4b1d-93a0-c5e2d4b1f929\") "
Oct 31 20:01:38.137684 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:38.137663    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1474c24c-f01e-4b1d-93a0-c5e2d4b1f929-kube-api-access-ngnlk\" (UniqueName: \"kubernetes.io/projected/1474c24c-f01e-4b1d-93a0-c5e2d4b1f929-kube-api-access-ngnlk\") pod \"1474c24c-f01e-4b1d-93a0-c5e2d4b1f929\" (UID: \"1474c24c-f01e-4b1d-93a0-c5e2d4b1f929\") "
Oct 31 20:01:38.138746 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:38.138716    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1474c24c-f01e-4b1d-93a0-c5e2d4b1f929-plugins-config-volume" pod "1474c24c-f01e-4b1d-93a0-c5e2d4b1f929" (UID: "1474c24c-f01e-4b1d-93a0-c5e2d4b1f929"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:01:38.140223 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:38.140195    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1474c24c-f01e-4b1d-93a0-c5e2d4b1f929-kube-api-access-ngnlk" pod "1474c24c-f01e-4b1d-93a0-c5e2d4b1f929" (UID: "1474c24c-f01e-4b1d-93a0-c5e2d4b1f929"). InnerVolumeSpecName "kube-api-access-ngnlk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:01:38.250403 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:38.246686    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngnlk\" (UniqueName: \"kubernetes.io/projected/1474c24c-f01e-4b1d-93a0-c5e2d4b1f929-kube-api-access-ngnlk\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:01:38.250403 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:38.246724    4645 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1474c24c-f01e-4b1d-93a0-c5e2d4b1f929-plugins-config-volume\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:01:38.347649 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:38.347615    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/263d2ffb-ab33-432a-83cd-dc73f244bc82-kube-api-access-spvd6\" (UniqueName: \"kubernetes.io/projected/263d2ffb-ab33-432a-83cd-dc73f244bc82-kube-api-access-spvd6\") pod \"263d2ffb-ab33-432a-83cd-dc73f244bc82\" (UID: \"263d2ffb-ab33-432a-83cd-dc73f244bc82\") "
Oct 31 20:01:38.347829 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:38.347721    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/263d2ffb-ab33-432a-83cd-dc73f244bc82-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/263d2ffb-ab33-432a-83cd-dc73f244bc82-plugins-config-volume\") pod \"263d2ffb-ab33-432a-83cd-dc73f244bc82\" (UID: \"263d2ffb-ab33-432a-83cd-dc73f244bc82\") "
Oct 31 20:01:38.348166 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:38.348138    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/263d2ffb-ab33-432a-83cd-dc73f244bc82-plugins-config-volume" pod "263d2ffb-ab33-432a-83cd-dc73f244bc82" (UID: "263d2ffb-ab33-432a-83cd-dc73f244bc82"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:01:38.348489 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:38.348462    4645 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/263d2ffb-ab33-432a-83cd-dc73f244bc82-plugins-config-volume\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:01:38.350128 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:38.350102    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/263d2ffb-ab33-432a-83cd-dc73f244bc82-kube-api-access-spvd6" pod "263d2ffb-ab33-432a-83cd-dc73f244bc82" (UID: "263d2ffb-ab33-432a-83cd-dc73f244bc82"). InnerVolumeSpecName "kube-api-access-spvd6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:01:38.410178 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:38.410151    4645 generic.go:356] "Generic (PLEG): container finished" podID="59f3365e-9975-44b4-b7c5-02e91e891f8d" containerID="3ed7967e7fa87c094576a3a4207749c0f93597da2f202741498966e30ff1a490" exitCode=0
Oct 31 20:01:38.410293 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:38.410224    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v" event={"ID":"59f3365e-9975-44b4-b7c5-02e91e891f8d","Type":"ContainerDied","Data":"3ed7967e7fa87c094576a3a4207749c0f93597da2f202741498966e30ff1a490"}
Oct 31 20:01:38.410729 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:38.410700    4645 scope.go:122] "RemoveContainer" containerID="3ed7967e7fa87c094576a3a4207749c0f93597da2f202741498966e30ff1a490"
Oct 31 20:01:38.411829 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:38.411806    4645 generic.go:356] "Generic (PLEG): container finished" podID="1474c24c-f01e-4b1d-93a0-c5e2d4b1f929" containerID="5fa51893c657bd2c2700aa6c5e6ba19827c6736d440c0d852de467b67936b238" exitCode=0
Oct 31 20:01:38.411948 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:38.411898    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bd1c9f-66bb876fcf-qnnw8" event={"ID":"1474c24c-f01e-4b1d-93a0-c5e2d4b1f929","Type":"ContainerDied","Data":"5fa51893c657bd2c2700aa6c5e6ba19827c6736d440c0d852de467b67936b238"}
Oct 31 20:01:38.412015 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:38.411948    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bd1c9f-66bb876fcf-qnnw8" event={"ID":"1474c24c-f01e-4b1d-93a0-c5e2d4b1f929","Type":"ContainerDied","Data":"2dfa62eab31978818bc82184f896621f209830728844f8681580554f47374476"}
Oct 31 20:01:38.412015 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:38.411982    4645 scope.go:122] "RemoveContainer" containerID="5fa51893c657bd2c2700aa6c5e6ba19827c6736d440c0d852de467b67936b238"
Oct 31 20:01:38.412402 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:38.412385    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bd1c9f-66bb876fcf-qnnw8"
Oct 31 20:01:38.412791 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:38.412736    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:01:38.412894 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:38.412843    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:01:38.412956 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:38.412891    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:01:38.414133 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:38.414112    4645 generic.go:356] "Generic (PLEG): container finished" podID="263d2ffb-ab33-432a-83cd-dc73f244bc82" containerID="5c9d41c0c25e9551c365d3a92514a6448b4f837bad42d886ba13b720c164c8fc" exitCode=0
Oct 31 20:01:38.414261 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:38.414241    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd1c9f-6dbbc4dc79-v5wmk" event={"ID":"263d2ffb-ab33-432a-83cd-dc73f244bc82","Type":"ContainerDied","Data":"5c9d41c0c25e9551c365d3a92514a6448b4f837bad42d886ba13b720c164c8fc"}
Oct 31 20:01:38.414360 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:38.414289    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd1c9f-6dbbc4dc79-v5wmk" event={"ID":"263d2ffb-ab33-432a-83cd-dc73f244bc82","Type":"ContainerDied","Data":"93a95f6c14b54016d9e366c9baee6a992d05ac8ee5ad63629e43732dd7b5d03e"}
Oct 31 20:01:38.414431 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:38.414226    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd1c9f-6dbbc4dc79-v5wmk"
Oct 31 20:01:38.430274 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:38.429497    4645 scope.go:122] "RemoveContainer" containerID="5fa51893c657bd2c2700aa6c5e6ba19827c6736d440c0d852de467b67936b238"
Oct 31 20:01:38.430274 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:01:38.429761    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fa51893c657bd2c2700aa6c5e6ba19827c6736d440c0d852de467b67936b238\": container with ID starting with 5fa51893c657bd2c2700aa6c5e6ba19827c6736d440c0d852de467b67936b238 not found: ID does not exist" containerID="5fa51893c657bd2c2700aa6c5e6ba19827c6736d440c0d852de467b67936b238"
Oct 31 20:01:38.430274 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:38.429781    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fa51893c657bd2c2700aa6c5e6ba19827c6736d440c0d852de467b67936b238"} err="failed to get container status \"5fa51893c657bd2c2700aa6c5e6ba19827c6736d440c0d852de467b67936b238\": rpc error: code = NotFound desc = could not find container \"5fa51893c657bd2c2700aa6c5e6ba19827c6736d440c0d852de467b67936b238\": container with ID starting with 5fa51893c657bd2c2700aa6c5e6ba19827c6736d440c0d852de467b67936b238 not found: ID does not exist"
Oct 31 20:01:38.430274 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:38.429799    4645 scope.go:122] "RemoveContainer" containerID="5c9d41c0c25e9551c365d3a92514a6448b4f837bad42d886ba13b720c164c8fc"
Oct 31 20:01:38.452352 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:38.450114    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spvd6\" (UniqueName: \"kubernetes.io/projected/263d2ffb-ab33-432a-83cd-dc73f244bc82-kube-api-access-spvd6\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:01:38.454432 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:38.454408    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bd1c9f-6dbbc4dc79-v5wmk"]
Oct 31 20:01:38.493722 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:38.493168    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bd1c9f-6dbbc4dc79-v5wmk"]
Oct 31 20:01:38.504667 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:38.503482    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bd1c9f-66bb876fcf-qnnw8"]
Oct 31 20:01:38.509371 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:38.509308    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bd1c9f-66bb876fcf-qnnw8"]
Oct 31 20:01:38.515225 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:38.515199    4645 scope.go:122] "RemoveContainer" containerID="5c9d41c0c25e9551c365d3a92514a6448b4f837bad42d886ba13b720c164c8fc"
Oct 31 20:01:38.516052 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:01:38.515707    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c9d41c0c25e9551c365d3a92514a6448b4f837bad42d886ba13b720c164c8fc\": container with ID starting with 5c9d41c0c25e9551c365d3a92514a6448b4f837bad42d886ba13b720c164c8fc not found: ID does not exist" containerID="5c9d41c0c25e9551c365d3a92514a6448b4f837bad42d886ba13b720c164c8fc"
Oct 31 20:01:38.516052 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:38.515747    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c9d41c0c25e9551c365d3a92514a6448b4f837bad42d886ba13b720c164c8fc"} err="failed to get container status \"5c9d41c0c25e9551c365d3a92514a6448b4f837bad42d886ba13b720c164c8fc\": rpc error: code = NotFound desc = could not find container \"5c9d41c0c25e9551c365d3a92514a6448b4f837bad42d886ba13b720c164c8fc\": container with ID starting with 5c9d41c0c25e9551c365d3a92514a6448b4f837bad42d886ba13b720c164c8fc not found: ID does not exist"
Oct 31 20:01:38.728920 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:38.728882    4645 patch_prober.go:29] interesting pod/e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body=
Oct 31 20:01:38.729100 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:38.728935    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b" podUID="710e3436-3828-45fb-b090-e28d1147cbfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused"
Oct 31 20:01:39.055821 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:39.055495    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:39.427257 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:39.427225    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v" event={"ID":"59f3365e-9975-44b4-b7c5-02e91e891f8d","Type":"ContainerStarted","Data":"beaf92083c6876ba81650f71806ab23dd0f58cddfc2050c7f2db0cde32cf5005"}
Oct 31 20:01:39.623544 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:39.623507    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1474c24c-f01e-4b1d-93a0-c5e2d4b1f929" path="/var/lib/kubelet/pods/1474c24c-f01e-4b1d-93a0-c5e2d4b1f929/volumes"
Oct 31 20:01:39.624075 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:39.624059    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="263d2ffb-ab33-432a-83cd-dc73f244bc82" path="/var/lib/kubelet/pods/263d2ffb-ab33-432a-83cd-dc73f244bc82/volumes"
Oct 31 20:01:39.729388 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:39.729288    4645 patch_prober.go:29] interesting pod/e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body=
Oct 31 20:01:39.729388 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:39.729372    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b" podUID="710e3436-3828-45fb-b090-e28d1147cbfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused"
Oct 31 20:01:39.914278 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:39.914151    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:40.197248 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:40.197213    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v"
Oct 31 20:01:40.198507 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:40.198380    4645 patch_prober.go:29] interesting pod/e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body=
Oct 31 20:01:40.198739 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:40.198715    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v" podUID="59f3365e-9975-44b4-b7c5-02e91e891f8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused"
Oct 31 20:01:40.685706 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:40.685265    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:41.018114 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:41.018023    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b"
Oct 31 20:01:41.019677 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:41.019650    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b"
Oct 31 20:01:41.020552 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:41.020520    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b"
Oct 31 20:01:41.203278 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:41.203240    4645 patch_prober.go:29] interesting pod/e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body=
Oct 31 20:01:41.203521 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:41.203300    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v" podUID="59f3365e-9975-44b4-b7c5-02e91e891f8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused"
Oct 31 20:01:41.481291 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:41.481081    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:42.073353 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:42.073251    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v"
Oct 31 20:01:42.189736 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:42.189700    4645 patch_prober.go:29] interesting pod/e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body=
Oct 31 20:01:42.189889 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:42.189775    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v" podUID="59f3365e-9975-44b4-b7c5-02e91e891f8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused"
Oct 31 20:01:42.281701 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:42.281651    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:43.051768 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:43.051720    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:43.189657 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:43.189619    4645 patch_prober.go:29] interesting pod/e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body=
Oct 31 20:01:43.190138 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:43.189692    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v" podUID="59f3365e-9975-44b4-b7c5-02e91e891f8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused"
Oct 31 20:01:43.638733 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:43.638698    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:43.772351 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:43.766269    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:43.913121 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:43.913086    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-5bbf75-6c987cff47-d4kgj"]
Oct 31 20:01:43.913402 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:43.913365    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-5bbf75-6c987cff47-d4kgj" podUID="8274db35-2b32-4fae-9f7d-e3fbb51bf4ed" containerName="maas-api" containerID="cri-o://62687938032ac4db04b1747f9d42568acd00b1862feb0254780d2da59ce8f5e6" gracePeriod=30
Oct 31 20:01:43.991073 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:43.990998    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:44.229406 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:44.214979    4645 patch_prober.go:29] interesting pod/e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body=
Oct 31 20:01:44.229406 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:44.215036    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v" podUID="59f3365e-9975-44b4-b7c5-02e91e891f8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused"
Oct 31 20:01:44.231092 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:44.230907    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:44.522435 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:44.508937    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:44.543766 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:44.543738    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c9b9a1-79688949cf-7krhv"
Oct 31 20:01:44.610216 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:44.609968    4645 generic.go:356] "Generic (PLEG): container finished" podID="8274db35-2b32-4fae-9f7d-e3fbb51bf4ed" containerID="62687938032ac4db04b1747f9d42568acd00b1862feb0254780d2da59ce8f5e6" exitCode=0
Oct 31 20:01:44.610216 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:44.610120    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-5bbf75-6c987cff47-d4kgj" event={"ID":"8274db35-2b32-4fae-9f7d-e3fbb51bf4ed","Type":"ContainerDied","Data":"62687938032ac4db04b1747f9d42568acd00b1862feb0254780d2da59ce8f5e6"}
Oct 31 20:01:44.652869 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:44.652841    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-5bbf75-6c987cff47-d4kgj"
Oct 31 20:01:44.812610 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:44.812528    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8274db35-2b32-4fae-9f7d-e3fbb51bf4ed-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8274db35-2b32-4fae-9f7d-e3fbb51bf4ed-maas-api-tls\") pod \"8274db35-2b32-4fae-9f7d-e3fbb51bf4ed\" (UID: \"8274db35-2b32-4fae-9f7d-e3fbb51bf4ed\") "
Oct 31 20:01:44.812767 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:44.812692    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8274db35-2b32-4fae-9f7d-e3fbb51bf4ed-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8274db35-2b32-4fae-9f7d-e3fbb51bf4ed-metrics-certs\") pod \"8274db35-2b32-4fae-9f7d-e3fbb51bf4ed\" (UID: \"8274db35-2b32-4fae-9f7d-e3fbb51bf4ed\") "
Oct 31 20:01:44.812811 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:44.812778    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8274db35-2b32-4fae-9f7d-e3fbb51bf4ed-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8274db35-2b32-4fae-9f7d-e3fbb51bf4ed-openshift-service-ca\") pod \"8274db35-2b32-4fae-9f7d-e3fbb51bf4ed\" (UID: \"8274db35-2b32-4fae-9f7d-e3fbb51bf4ed\") "
Oct 31 20:01:44.812940 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:44.812923    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8274db35-2b32-4fae-9f7d-e3fbb51bf4ed-kube-api-access-zv6bv\" (UniqueName: \"kubernetes.io/projected/8274db35-2b32-4fae-9f7d-e3fbb51bf4ed-kube-api-access-zv6bv\") pod \"8274db35-2b32-4fae-9f7d-e3fbb51bf4ed\" (UID: \"8274db35-2b32-4fae-9f7d-e3fbb51bf4ed\") "
Oct 31 20:01:44.813136 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:44.813106    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8274db35-2b32-4fae-9f7d-e3fbb51bf4ed-openshift-service-ca" pod "8274db35-2b32-4fae-9f7d-e3fbb51bf4ed" (UID: "8274db35-2b32-4fae-9f7d-e3fbb51bf4ed"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:01:44.813689 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:44.813666    4645 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8274db35-2b32-4fae-9f7d-e3fbb51bf4ed-openshift-service-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:01:44.814763 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:44.814737    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8274db35-2b32-4fae-9f7d-e3fbb51bf4ed-maas-api-tls" pod "8274db35-2b32-4fae-9f7d-e3fbb51bf4ed" (UID: "8274db35-2b32-4fae-9f7d-e3fbb51bf4ed"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:01:44.814904 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:44.814883    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8274db35-2b32-4fae-9f7d-e3fbb51bf4ed-metrics-certs" pod "8274db35-2b32-4fae-9f7d-e3fbb51bf4ed" (UID: "8274db35-2b32-4fae-9f7d-e3fbb51bf4ed"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:01:44.815014 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:44.814935    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8274db35-2b32-4fae-9f7d-e3fbb51bf4ed-kube-api-access-zv6bv" pod "8274db35-2b32-4fae-9f7d-e3fbb51bf4ed" (UID: "8274db35-2b32-4fae-9f7d-e3fbb51bf4ed"). InnerVolumeSpecName "kube-api-access-zv6bv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:01:44.915197 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:44.915153    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zv6bv\" (UniqueName: \"kubernetes.io/projected/8274db35-2b32-4fae-9f7d-e3fbb51bf4ed-kube-api-access-zv6bv\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:01:44.915492 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:44.915455    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8274db35-2b32-4fae-9f7d-e3fbb51bf4ed-maas-api-tls\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:01:44.915492 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:44.915483    4645 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8274db35-2b32-4fae-9f7d-e3fbb51bf4ed-metrics-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:01:44.961292 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:44.961252    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:45.013065 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:45.013024    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w"]
Oct 31 20:01:45.013802 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:45.013781    4645 cpu_manager.go:414] "containerMap: removing container" podUID="1474c24c-f01e-4b1d-93a0-c5e2d4b1f929" containerName="payload-processing"
Oct 31 20:01:45.013802 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:45.013798    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1474c24c-f01e-4b1d-93a0-c5e2d4b1f929" containerName="payload-processing"
Oct 31 20:01:45.014012 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:45.013810    4645 cpu_manager.go:414] "containerMap: removing container" podUID="263d2ffb-ab33-432a-83cd-dc73f244bc82" containerName="payload-pre-processing"
Oct 31 20:01:45.014012 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:45.013815    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="263d2ffb-ab33-432a-83cd-dc73f244bc82" containerName="payload-pre-processing"
Oct 31 20:01:45.014012 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:45.013856    4645 cpu_manager.go:414] "containerMap: removing container" podUID="8274db35-2b32-4fae-9f7d-e3fbb51bf4ed" containerName="maas-api"
Oct 31 20:01:45.014012 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:45.013861    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8274db35-2b32-4fae-9f7d-e3fbb51bf4ed" containerName="maas-api"
Oct 31 20:01:45.014012 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:45.013949    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="8274db35-2b32-4fae-9f7d-e3fbb51bf4ed" containerName="maas-api"
Oct 31 20:01:45.014012 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:45.013959    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="1474c24c-f01e-4b1d-93a0-c5e2d4b1f929" containerName="payload-processing"
Oct 31 20:01:45.014012 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:45.013966    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="263d2ffb-ab33-432a-83cd-dc73f244bc82" containerName="payload-pre-processing"
Oct 31 20:01:45.017794 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:45.017770    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w"
Oct 31 20:01:45.030940 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:45.030920    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-c734f5-openshift-default-dockercfg-jg24l\""
Oct 31 20:01:45.047874 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:45.047846    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w"]
Oct 31 20:01:45.119505 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:45.119409    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f655e4a8-4347-4492-b5c5-aa5e74e690ac-istio-token\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w\" (UID: \"f655e4a8-4347-4492-b5c5-aa5e74e690ac\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w"
Oct 31 20:01:45.119505 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:45.119473    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f655e4a8-4347-4492-b5c5-aa5e74e690ac-istio-data\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w\" (UID: \"f655e4a8-4347-4492-b5c5-aa5e74e690ac\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w"
Oct 31 20:01:45.119733 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:45.119503    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f655e4a8-4347-4492-b5c5-aa5e74e690ac-istiod-ca-cert\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w\" (UID: \"f655e4a8-4347-4492-b5c5-aa5e74e690ac\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w"
Oct 31 20:01:45.119733 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:45.119536    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvdq6\" (UniqueName: \"kubernetes.io/projected/f655e4a8-4347-4492-b5c5-aa5e74e690ac-kube-api-access-vvdq6\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w\" (UID: \"f655e4a8-4347-4492-b5c5-aa5e74e690ac\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w"
Oct 31 20:01:45.119733 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:45.119604    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f655e4a8-4347-4492-b5c5-aa5e74e690ac-workload-certs\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w\" (UID: \"f655e4a8-4347-4492-b5c5-aa5e74e690ac\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w"
Oct 31 20:01:45.119733 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:45.119640    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f655e4a8-4347-4492-b5c5-aa5e74e690ac-credential-socket\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w\" (UID: \"f655e4a8-4347-4492-b5c5-aa5e74e690ac\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w"
Oct 31 20:01:45.119956 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:45.119853    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f655e4a8-4347-4492-b5c5-aa5e74e690ac-istio-envoy\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w\" (UID: \"f655e4a8-4347-4492-b5c5-aa5e74e690ac\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w"
Oct 31 20:01:45.119956 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:45.119918    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f655e4a8-4347-4492-b5c5-aa5e74e690ac-workload-socket\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w\" (UID: \"f655e4a8-4347-4492-b5c5-aa5e74e690ac\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w"
Oct 31 20:01:45.120064 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:45.119956    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f655e4a8-4347-4492-b5c5-aa5e74e690ac-istio-podinfo\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w\" (UID: \"f655e4a8-4347-4492-b5c5-aa5e74e690ac\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w"
Oct 31 20:01:45.194822 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:45.194791    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v"
Oct 31 20:01:45.195994 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:45.195968    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v"
Oct 31 20:01:45.221288 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:45.221256    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f655e4a8-4347-4492-b5c5-aa5e74e690ac-istio-envoy\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w\" (UID: \"f655e4a8-4347-4492-b5c5-aa5e74e690ac\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w"
Oct 31 20:01:45.221436 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:45.221311    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f655e4a8-4347-4492-b5c5-aa5e74e690ac-workload-socket\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w\" (UID: \"f655e4a8-4347-4492-b5c5-aa5e74e690ac\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w"
Oct 31 20:01:45.221436 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:45.221358    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f655e4a8-4347-4492-b5c5-aa5e74e690ac-istio-podinfo\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w\" (UID: \"f655e4a8-4347-4492-b5c5-aa5e74e690ac\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w"
Oct 31 20:01:45.221588 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:45.221562    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f655e4a8-4347-4492-b5c5-aa5e74e690ac-istio-token\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w\" (UID: \"f655e4a8-4347-4492-b5c5-aa5e74e690ac\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w"
Oct 31 20:01:45.221645 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:45.221612    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f655e4a8-4347-4492-b5c5-aa5e74e690ac-istio-data\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w\" (UID: \"f655e4a8-4347-4492-b5c5-aa5e74e690ac\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w"
Oct 31 20:01:45.221702 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:45.221642    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f655e4a8-4347-4492-b5c5-aa5e74e690ac-istiod-ca-cert\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w\" (UID: \"f655e4a8-4347-4492-b5c5-aa5e74e690ac\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w"
Oct 31 20:01:45.221702 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:45.221678    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvdq6\" (UniqueName: \"kubernetes.io/projected/f655e4a8-4347-4492-b5c5-aa5e74e690ac-kube-api-access-vvdq6\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w\" (UID: \"f655e4a8-4347-4492-b5c5-aa5e74e690ac\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w"
Oct 31 20:01:45.221812 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:45.221753    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f655e4a8-4347-4492-b5c5-aa5e74e690ac-workload-certs\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w\" (UID: \"f655e4a8-4347-4492-b5c5-aa5e74e690ac\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w"
Oct 31 20:01:45.221812 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:45.221789    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f655e4a8-4347-4492-b5c5-aa5e74e690ac-credential-socket\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w\" (UID: \"f655e4a8-4347-4492-b5c5-aa5e74e690ac\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w"
Oct 31 20:01:45.221919 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:45.221820    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f655e4a8-4347-4492-b5c5-aa5e74e690ac-workload-socket\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w\" (UID: \"f655e4a8-4347-4492-b5c5-aa5e74e690ac\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w"
Oct 31 20:01:45.221979 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:45.221939    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f655e4a8-4347-4492-b5c5-aa5e74e690ac-istio-data\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w\" (UID: \"f655e4a8-4347-4492-b5c5-aa5e74e690ac\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w"
Oct 31 20:01:45.222073 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:45.222048    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f655e4a8-4347-4492-b5c5-aa5e74e690ac-workload-certs\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w\" (UID: \"f655e4a8-4347-4492-b5c5-aa5e74e690ac\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w"
Oct 31 20:01:45.222162 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:45.222143    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f655e4a8-4347-4492-b5c5-aa5e74e690ac-credential-socket\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w\" (UID: \"f655e4a8-4347-4492-b5c5-aa5e74e690ac\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w"
Oct 31 20:01:45.222474 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:45.222457    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f655e4a8-4347-4492-b5c5-aa5e74e690ac-istiod-ca-cert\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w\" (UID: \"f655e4a8-4347-4492-b5c5-aa5e74e690ac\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w"
Oct 31 20:01:45.223750 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:45.223725    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f655e4a8-4347-4492-b5c5-aa5e74e690ac-istio-envoy\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w\" (UID: \"f655e4a8-4347-4492-b5c5-aa5e74e690ac\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w"
Oct 31 20:01:45.224442 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:45.224419    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f655e4a8-4347-4492-b5c5-aa5e74e690ac-istio-podinfo\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w\" (UID: \"f655e4a8-4347-4492-b5c5-aa5e74e690ac\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w"
Oct 31 20:01:45.239341 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:45.239304    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvdq6\" (UniqueName: \"kubernetes.io/projected/f655e4a8-4347-4492-b5c5-aa5e74e690ac-kube-api-access-vvdq6\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w\" (UID: \"f655e4a8-4347-4492-b5c5-aa5e74e690ac\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w"
Oct 31 20:01:45.240269 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:45.240249    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f655e4a8-4347-4492-b5c5-aa5e74e690ac-istio-token\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w\" (UID: \"f655e4a8-4347-4492-b5c5-aa5e74e690ac\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w"
Oct 31 20:01:45.336810 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:45.336778    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w"
Oct 31 20:01:45.345360 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:45.345302    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c9b9a1-5c968dc557-lrnmf"
Oct 31 20:01:45.621972 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:45.621941    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-5bbf75-6c987cff47-d4kgj"
Oct 31 20:01:45.622166 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:45.621972    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-5bbf75-6c987cff47-d4kgj" event={"ID":"8274db35-2b32-4fae-9f7d-e3fbb51bf4ed","Type":"ContainerDied","Data":"979f319f84be73d13dc617b9278c1dd13494e14f69759d819fbd65b6228c0162"}
Oct 31 20:01:45.622166 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:45.622034    4645 scope.go:122] "RemoveContainer" containerID="62687938032ac4db04b1747f9d42568acd00b1862feb0254780d2da59ce8f5e6"
Oct 31 20:01:45.640441 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:45.640413    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w"]
Oct 31 20:01:45.641935 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:01:45.641911    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf655e4a8_4347_4492_b5c5_aa5e74e690ac.slice/crio-4125fb93d7c6bae961f06d0efb606531ddc1f18d605c56c865f3943a691a03b7 WatchSource:0}: Error finding container 4125fb93d7c6bae961f06d0efb606531ddc1f18d605c56c865f3943a691a03b7: Status 404 returned error can't find the container with id 4125fb93d7c6bae961f06d0efb606531ddc1f18d605c56c865f3943a691a03b7
Oct 31 20:01:45.644547 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:45.644520    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:01:45.644618 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:45.644577    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:01:45.644618 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:45.644605    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:01:45.653741 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:45.653718    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-5bbf75-6c987cff47-d4kgj"]
Oct 31 20:01:45.660876 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:45.660855    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-5bbf75-6c987cff47-d4kgj"]
Oct 31 20:01:45.664376 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:45.664354    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:46.452360 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:46.452292    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:46.632340 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:46.632283    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w" event={"ID":"f655e4a8-4347-4492-b5c5-aa5e74e690ac","Type":"ContainerStarted","Data":"261c4acad2435340f349218a91ab931436d74fb224f2404d2a281cc939cfd258"}
Oct 31 20:01:46.632499 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:46.632362    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w" event={"ID":"f655e4a8-4347-4492-b5c5-aa5e74e690ac","Type":"ContainerStarted","Data":"4125fb93d7c6bae961f06d0efb606531ddc1f18d605c56c865f3943a691a03b7"}
Oct 31 20:01:46.678354 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:46.666424    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w" podStartSLOduration=2.666402073 podStartE2EDuration="2.666402073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:01:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:01:46.654414482 +0000 UTC m=+2537.227648612" watchObservedRunningTime="2026-10-31 20:01:46.666402073 +0000 UTC m=+2537.239636203"
Oct 31 20:01:47.286129 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:47.286085    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-5bbf75-6c987cff47-d4kgj" podUID="8274db35-2b32-4fae-9f7d-e3fbb51bf4ed" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.52:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 20:01:47.337887 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:47.337859    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w"
Oct 31 20:01:47.339114 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:47.339090    4645 patch_prober.go:29] interesting pod/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused" start-of-body=
Oct 31 20:01:47.339223 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:47.339131    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w" podUID="f655e4a8-4347-4492-b5c5-aa5e74e690ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused"
Oct 31 20:01:47.342211 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:47.342188    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:47.411941 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:47.408374    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c734f5-786f585bf5-bs8jm"
Oct 31 20:01:47.412833 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:47.412806    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c734f5-684c4f6c9d-xdjpr"
Oct 31 20:01:47.620390 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:47.620299    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8274db35-2b32-4fae-9f7d-e3fbb51bf4ed" path="/var/lib/kubelet/pods/8274db35-2b32-4fae-9f7d-e3fbb51bf4ed/volumes"
Oct 31 20:01:48.180354 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:48.179991    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:48.337863 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:48.337687    4645 patch_prober.go:29] interesting pod/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused" start-of-body=
Oct 31 20:01:48.338116 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:48.338077    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w" podUID="f655e4a8-4347-4492-b5c5-aa5e74e690ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused"
Oct 31 20:01:48.980956 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:48.980924    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:49.337240 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:49.337167    4645 patch_prober.go:29] interesting pod/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused" start-of-body=
Oct 31 20:01:49.337240 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:49.337216    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w" podUID="f655e4a8-4347-4492-b5c5-aa5e74e690ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused"
Oct 31 20:01:49.873196 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:49.873163    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:50.338635 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:50.338570    4645 patch_prober.go:29] interesting pod/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused" start-of-body=
Oct 31 20:01:50.338635 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:50.338622    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w" podUID="f655e4a8-4347-4492-b5c5-aa5e74e690ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused"
Oct 31 20:01:50.875354 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:50.869127    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:51.337273 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:51.337239    4645 patch_prober.go:29] interesting pod/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused" start-of-body=
Oct 31 20:01:51.337489 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:51.337297    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w" podUID="f655e4a8-4347-4492-b5c5-aa5e74e690ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused"
Oct 31 20:01:51.442603 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:51.442537    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:51.553428 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:51.553388    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:51.620564 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:51.620489    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bd1c9f-745586fb75-mzfnm"]
Oct 31 20:01:51.626138 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:51.626118    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd1c9f-745586fb75-mzfnm"
Oct 31 20:01:51.629686 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:51.629663    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-bd1c9f-dockercfg-t66h7\""
Oct 31 20:01:51.629844 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:51.629667    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-bd1c9f\""
Oct 31 20:01:51.632271 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:51.632248    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bd1c9f-745586fb75-mzfnm"]
Oct 31 20:01:51.948322 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:51.935081    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/140d8a5f-7b8a-409b-9fcc-5a050fdeff9c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bd1c9f-745586fb75-mzfnm\" (UID: \"140d8a5f-7b8a-409b-9fcc-5a050fdeff9c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd1c9f-745586fb75-mzfnm"
Oct 31 20:01:51.948322 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:51.935185    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bflwg\" (UniqueName: \"kubernetes.io/projected/140d8a5f-7b8a-409b-9fcc-5a050fdeff9c-kube-api-access-bflwg\") pod \"payload-pre-processing-e2e-ipp-a-bd1c9f-745586fb75-mzfnm\" (UID: \"140d8a5f-7b8a-409b-9fcc-5a050fdeff9c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd1c9f-745586fb75-mzfnm"
Oct 31 20:01:51.948322 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:51.943146    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bd1c9f-65b6b696bd-2bqtm"]
Oct 31 20:01:52.018359 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:52.016279    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bd1c9f-65b6b696bd-2bqtm"]
Oct 31 20:01:52.018359 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:52.016464    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bd1c9f-65b6b696bd-2bqtm"
Oct 31 20:01:52.030361 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:52.030300    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:52.037843 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:52.037815    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bflwg\" (UniqueName: \"kubernetes.io/projected/140d8a5f-7b8a-409b-9fcc-5a050fdeff9c-kube-api-access-bflwg\") pod \"payload-pre-processing-e2e-ipp-a-bd1c9f-745586fb75-mzfnm\" (UID: \"140d8a5f-7b8a-409b-9fcc-5a050fdeff9c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd1c9f-745586fb75-mzfnm"
Oct 31 20:01:52.038183 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:52.038155    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/140d8a5f-7b8a-409b-9fcc-5a050fdeff9c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bd1c9f-745586fb75-mzfnm\" (UID: \"140d8a5f-7b8a-409b-9fcc-5a050fdeff9c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd1c9f-745586fb75-mzfnm"
Oct 31 20:01:52.039212 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:52.039186    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/140d8a5f-7b8a-409b-9fcc-5a050fdeff9c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bd1c9f-745586fb75-mzfnm\" (UID: \"140d8a5f-7b8a-409b-9fcc-5a050fdeff9c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd1c9f-745586fb75-mzfnm"
Oct 31 20:01:52.059564 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:52.059533    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bflwg\" (UniqueName: \"kubernetes.io/projected/140d8a5f-7b8a-409b-9fcc-5a050fdeff9c-kube-api-access-bflwg\") pod \"payload-pre-processing-e2e-ipp-a-bd1c9f-745586fb75-mzfnm\" (UID: \"140d8a5f-7b8a-409b-9fcc-5a050fdeff9c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd1c9f-745586fb75-mzfnm"
Oct 31 20:01:52.140063 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:52.140011    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c653fdd-5545-415b-b7f9-cf5dcadff326-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-bd1c9f-65b6b696bd-2bqtm\" (UID: \"9c653fdd-5545-415b-b7f9-cf5dcadff326\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bd1c9f-65b6b696bd-2bqtm"
Oct 31 20:01:52.140525 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:52.140492    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgrpf\" (UniqueName: \"kubernetes.io/projected/9c653fdd-5545-415b-b7f9-cf5dcadff326-kube-api-access-rgrpf\") pod \"payload-processing-e2e-ipp-a-bd1c9f-65b6b696bd-2bqtm\" (UID: \"9c653fdd-5545-415b-b7f9-cf5dcadff326\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bd1c9f-65b6b696bd-2bqtm"
Oct 31 20:01:52.198660 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:52.188121    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:52.242631 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:52.242584    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgrpf\" (UniqueName: \"kubernetes.io/projected/9c653fdd-5545-415b-b7f9-cf5dcadff326-kube-api-access-rgrpf\") pod \"payload-processing-e2e-ipp-a-bd1c9f-65b6b696bd-2bqtm\" (UID: \"9c653fdd-5545-415b-b7f9-cf5dcadff326\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bd1c9f-65b6b696bd-2bqtm"
Oct 31 20:01:52.242846 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:52.242725    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c653fdd-5545-415b-b7f9-cf5dcadff326-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-bd1c9f-65b6b696bd-2bqtm\" (UID: \"9c653fdd-5545-415b-b7f9-cf5dcadff326\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bd1c9f-65b6b696bd-2bqtm"
Oct 31 20:01:52.243170 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:52.243146    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd1c9f-745586fb75-mzfnm"
Oct 31 20:01:52.243763 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:52.243738    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c653fdd-5545-415b-b7f9-cf5dcadff326-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-bd1c9f-65b6b696bd-2bqtm\" (UID: \"9c653fdd-5545-415b-b7f9-cf5dcadff326\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bd1c9f-65b6b696bd-2bqtm"
Oct 31 20:01:52.256771 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:52.256746    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgrpf\" (UniqueName: \"kubernetes.io/projected/9c653fdd-5545-415b-b7f9-cf5dcadff326-kube-api-access-rgrpf\") pod \"payload-processing-e2e-ipp-a-bd1c9f-65b6b696bd-2bqtm\" (UID: \"9c653fdd-5545-415b-b7f9-cf5dcadff326\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bd1c9f-65b6b696bd-2bqtm"
Oct 31 20:01:52.352504 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:52.339192    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:52.353625 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:52.353375    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bd1c9f-65b6b696bd-2bqtm"
Oct 31 20:01:52.375847 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:52.374107    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w"
Oct 31 20:01:52.378061 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:52.377982    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w"
Oct 31 20:01:52.382667 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:52.382525    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w"
Oct 31 20:01:52.452752 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:52.452661    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:52.587839 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:52.587803    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bd1c9f-745586fb75-mzfnm"]
Oct 31 20:01:52.590103 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:01:52.590080    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod140d8a5f_7b8a_409b_9fcc_5a050fdeff9c.slice/crio-a5c33810c2aa344e7056b56c2ff0a340838c46ab8b385b42950f24e6e40cc5f6 WatchSource:0}: Error finding container a5c33810c2aa344e7056b56c2ff0a340838c46ab8b385b42950f24e6e40cc5f6: Status 404 returned error can't find the container with id a5c33810c2aa344e7056b56c2ff0a340838c46ab8b385b42950f24e6e40cc5f6
Oct 31 20:01:52.735836 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:52.735470    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:52.946234 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:52.946178    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bd1c9f-65b6b696bd-2bqtm"]
Oct 31 20:01:53.005226 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:53.005199    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bd1c9f-65b6b696bd-2bqtm" event={"ID":"9c653fdd-5545-415b-b7f9-cf5dcadff326","Type":"ContainerStarted","Data":"f743761bff9418ee54f2602a121bf0d6b79b00e03241a4ece114a3cfa97970b3"}
Oct 31 20:01:53.011768 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:53.010853    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd1c9f-745586fb75-mzfnm" event={"ID":"140d8a5f-7b8a-409b-9fcc-5a050fdeff9c","Type":"ContainerStarted","Data":"d21b66ba02111d09f39482fad8118cba23c585fda475f6e7878adc46fb0f7775"}
Oct 31 20:01:53.011768 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:53.010910    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd1c9f-745586fb75-mzfnm"
Oct 31 20:01:53.011768 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:53.010931    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd1c9f-745586fb75-mzfnm" event={"ID":"140d8a5f-7b8a-409b-9fcc-5a050fdeff9c","Type":"ContainerStarted","Data":"a5c33810c2aa344e7056b56c2ff0a340838c46ab8b385b42950f24e6e40cc5f6"}
Oct 31 20:01:53.034476 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:53.034421    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd1c9f-745586fb75-mzfnm" podStartSLOduration=2.034403099 podStartE2EDuration="2.034403099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:01:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:01:53.034211687 +0000 UTC m=+2543.607445817" watchObservedRunningTime="2026-10-31 20:01:53.034403099 +0000 UTC m=+2543.607637230"
Oct 31 20:01:53.595522 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:53.595484    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:53.728679 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:53.728576    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:53.995551 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:53.995518    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:54.054383 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:54.054274    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bd1c9f-65b6b696bd-2bqtm" event={"ID":"9c653fdd-5545-415b-b7f9-cf5dcadff326","Type":"ContainerStarted","Data":"eb74d2f583a1da1d6ad93983665726fcfda433c433fef3aaa9ec5119606415ae"}
Oct 31 20:01:54.054635 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:54.054396    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-bd1c9f-65b6b696bd-2bqtm"
Oct 31 20:01:54.086303 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:54.085274    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-bd1c9f-65b6b696bd-2bqtm" podStartSLOduration=3.0852558 podStartE2EDuration="3.0852558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:01:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:01:54.078595979 +0000 UTC m=+2544.651830109" watchObservedRunningTime="2026-10-31 20:01:54.0852558 +0000 UTC m=+2544.658489933"
Oct 31 20:01:54.874140 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:54.874103    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:55.653467 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:55.653437    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:57.156987 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:57.156952    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:57.311356 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:57.311268    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:58.380633 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:58.380597    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:58.498774 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:58.498737    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:58.659406 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:58.658929    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:01:59.546877 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:01:59.546839    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:00.351497 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:00.351461    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:01.287836 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:01.287802    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:02.245848 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:02.245808    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:02.949002 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:02.948962    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:03.749523 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:03.749485    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:04.056542 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:04.056455    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd1c9f-745586fb75-mzfnm"
Oct 31 20:02:04.546419 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:04.546378    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:05.066235 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:05.066201    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-bd1c9f-65b6b696bd-2bqtm"
Oct 31 20:02:05.444350 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:05.438263    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:06.254074 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:02:06.254034    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 20:02:06.517406 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:06.516944    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:07.305699 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:07.305660    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:07.349956 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:07.349918    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5f3eff-65c86b7b8b-mc8nv"]
Oct 31 20:02:07.350232 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:07.350170    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-5f3eff-65c86b7b8b-mc8nv" podUID="0ec291f9-5a7f-4d42-946e-668aeb30a976" containerName="payload-processing" containerID="cri-o://66f6315668c4415b82937e276c2880660cff888a5e7aeb0f98f40a480ae2905f" gracePeriod=30
Oct 31 20:02:07.395600 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:07.395563    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5f3eff-7db97588f5-ld47d"]
Oct 31 20:02:07.395834 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:07.395805    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5f3eff-7db97588f5-ld47d" podUID="ca2e9a7f-b3d9-4f2f-90e6-414543715ada" containerName="payload-pre-processing" containerID="cri-o://844bab7531935759b5cc1424a31d94811b7640793d67ef5f00dfe4116c183637" gracePeriod=30
Oct 31 20:02:07.438713 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:07.438685    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:07.554696 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:07.554658    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:07.981583 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:07.933666    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:08.153378 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:08.147314    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:08.205743 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:08.205719    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5f3eff-65c86b7b8b-mc8nv"
Oct 31 20:02:08.212271 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:08.212236    4645 generic.go:356] "Generic (PLEG): container finished" podID="ca2e9a7f-b3d9-4f2f-90e6-414543715ada" containerID="844bab7531935759b5cc1424a31d94811b7640793d67ef5f00dfe4116c183637" exitCode=0
Oct 31 20:02:08.212492 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:08.212384    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5f3eff-7db97588f5-ld47d" event={"ID":"ca2e9a7f-b3d9-4f2f-90e6-414543715ada","Type":"ContainerDied","Data":"844bab7531935759b5cc1424a31d94811b7640793d67ef5f00dfe4116c183637"}
Oct 31 20:02:08.216294 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:08.216261    4645 generic.go:356] "Generic (PLEG): container finished" podID="0ec291f9-5a7f-4d42-946e-668aeb30a976" containerID="66f6315668c4415b82937e276c2880660cff888a5e7aeb0f98f40a480ae2905f" exitCode=0
Oct 31 20:02:08.216454 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:08.216404    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5f3eff-65c86b7b8b-mc8nv" event={"ID":"0ec291f9-5a7f-4d42-946e-668aeb30a976","Type":"ContainerDied","Data":"66f6315668c4415b82937e276c2880660cff888a5e7aeb0f98f40a480ae2905f"}
Oct 31 20:02:08.216454 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:08.216449    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5f3eff-65c86b7b8b-mc8nv" event={"ID":"0ec291f9-5a7f-4d42-946e-668aeb30a976","Type":"ContainerDied","Data":"c012ff8201df29026ce2ef9999d410f170d6cb087fd1682390e78d596993aae7"}
Oct 31 20:02:08.216585 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:08.216476    4645 scope.go:122] "RemoveContainer" containerID="66f6315668c4415b82937e276c2880660cff888a5e7aeb0f98f40a480ae2905f"
Oct 31 20:02:08.216646 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:08.216635    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5f3eff-65c86b7b8b-mc8nv"
Oct 31 20:02:08.231343 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:08.231242    4645 scope.go:122] "RemoveContainer" containerID="66f6315668c4415b82937e276c2880660cff888a5e7aeb0f98f40a480ae2905f"
Oct 31 20:02:08.231746 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:02:08.231722    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66f6315668c4415b82937e276c2880660cff888a5e7aeb0f98f40a480ae2905f\": container with ID starting with 66f6315668c4415b82937e276c2880660cff888a5e7aeb0f98f40a480ae2905f not found: ID does not exist" containerID="66f6315668c4415b82937e276c2880660cff888a5e7aeb0f98f40a480ae2905f"
Oct 31 20:02:08.231858 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:08.231756    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66f6315668c4415b82937e276c2880660cff888a5e7aeb0f98f40a480ae2905f"} err="failed to get container status \"66f6315668c4415b82937e276c2880660cff888a5e7aeb0f98f40a480ae2905f\": rpc error: code = NotFound desc = could not find container \"66f6315668c4415b82937e276c2880660cff888a5e7aeb0f98f40a480ae2905f\": container with ID starting with 66f6315668c4415b82937e276c2880660cff888a5e7aeb0f98f40a480ae2905f not found: ID does not exist"
Oct 31 20:02:08.249579 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:08.249548    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ec291f9-5a7f-4d42-946e-668aeb30a976-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ec291f9-5a7f-4d42-946e-668aeb30a976-plugins-config-volume\") pod \"0ec291f9-5a7f-4d42-946e-668aeb30a976\" (UID: \"0ec291f9-5a7f-4d42-946e-668aeb30a976\") "
Oct 31 20:02:08.249796 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:08.249780    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ec291f9-5a7f-4d42-946e-668aeb30a976-kube-api-access-zmz78\" (UniqueName: \"kubernetes.io/projected/0ec291f9-5a7f-4d42-946e-668aeb30a976-kube-api-access-zmz78\") pod \"0ec291f9-5a7f-4d42-946e-668aeb30a976\" (UID: \"0ec291f9-5a7f-4d42-946e-668aeb30a976\") "
Oct 31 20:02:08.250012 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:08.249983    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ec291f9-5a7f-4d42-946e-668aeb30a976-plugins-config-volume" pod "0ec291f9-5a7f-4d42-946e-668aeb30a976" (UID: "0ec291f9-5a7f-4d42-946e-668aeb30a976"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:02:08.250516 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:08.250491    4645 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ec291f9-5a7f-4d42-946e-668aeb30a976-plugins-config-volume\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:08.252912 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:08.252885    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ec291f9-5a7f-4d42-946e-668aeb30a976-kube-api-access-zmz78" pod "0ec291f9-5a7f-4d42-946e-668aeb30a976" (UID: "0ec291f9-5a7f-4d42-946e-668aeb30a976"). InnerVolumeSpecName "kube-api-access-zmz78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:02:08.309693 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:08.309663    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5f3eff-7db97588f5-ld47d"
Oct 31 20:02:08.351847 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:08.351817    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca2e9a7f-b3d9-4f2f-90e6-414543715ada-kube-api-access-svx27\" (UniqueName: \"kubernetes.io/projected/ca2e9a7f-b3d9-4f2f-90e6-414543715ada-kube-api-access-svx27\") pod \"ca2e9a7f-b3d9-4f2f-90e6-414543715ada\" (UID: \"ca2e9a7f-b3d9-4f2f-90e6-414543715ada\") "
Oct 31 20:02:08.352034 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:08.351961    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca2e9a7f-b3d9-4f2f-90e6-414543715ada-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca2e9a7f-b3d9-4f2f-90e6-414543715ada-plugins-config-volume\") pod \"ca2e9a7f-b3d9-4f2f-90e6-414543715ada\" (UID: \"ca2e9a7f-b3d9-4f2f-90e6-414543715ada\") "
Oct 31 20:02:08.352414 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:08.352391    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmz78\" (UniqueName: \"kubernetes.io/projected/0ec291f9-5a7f-4d42-946e-668aeb30a976-kube-api-access-zmz78\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:08.352639 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:08.352619    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca2e9a7f-b3d9-4f2f-90e6-414543715ada-plugins-config-volume" pod "ca2e9a7f-b3d9-4f2f-90e6-414543715ada" (UID: "ca2e9a7f-b3d9-4f2f-90e6-414543715ada"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:02:08.354069 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:08.354044    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca2e9a7f-b3d9-4f2f-90e6-414543715ada-kube-api-access-svx27" pod "ca2e9a7f-b3d9-4f2f-90e6-414543715ada" (UID: "ca2e9a7f-b3d9-4f2f-90e6-414543715ada"). InnerVolumeSpecName "kube-api-access-svx27". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:02:08.454109 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:08.454075    4645 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca2e9a7f-b3d9-4f2f-90e6-414543715ada-plugins-config-volume\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:08.454109 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:08.454106    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svx27\" (UniqueName: \"kubernetes.io/projected/ca2e9a7f-b3d9-4f2f-90e6-414543715ada-kube-api-access-svx27\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:08.478352 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:08.477391    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:08.549938 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:08.549895    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5f3eff-65c86b7b8b-mc8nv"]
Oct 31 20:02:08.554778 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:08.554745    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5f3eff-65c86b7b8b-mc8nv"]
Oct 31 20:02:08.981151 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:08.981097    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:09.228388 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:09.228357    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5f3eff-7db97588f5-ld47d"
Oct 31 20:02:09.228388 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:09.228374    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5f3eff-7db97588f5-ld47d" event={"ID":"ca2e9a7f-b3d9-4f2f-90e6-414543715ada","Type":"ContainerDied","Data":"ee0a195bf6132d05fb0a84f5fd123dd7111959ee62cf8d4f9a15eec663fc2b22"}
Oct 31 20:02:09.228668 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:09.228449    4645 scope.go:122] "RemoveContainer" containerID="844bab7531935759b5cc1424a31d94811b7640793d67ef5f00dfe4116c183637"
Oct 31 20:02:09.283531 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:09.283491    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5f3eff-7db97588f5-ld47d"]
Oct 31 20:02:09.288463 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:09.288431    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5f3eff-7db97588f5-ld47d"]
Oct 31 20:02:09.621033 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:09.620956    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ec291f9-5a7f-4d42-946e-668aeb30a976" path="/var/lib/kubelet/pods/0ec291f9-5a7f-4d42-946e-668aeb30a976/volumes"
Oct 31 20:02:09.621448 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:09.621417    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca2e9a7f-b3d9-4f2f-90e6-414543715ada" path="/var/lib/kubelet/pods/ca2e9a7f-b3d9-4f2f-90e6-414543715ada/volumes"
Oct 31 20:02:09.945737 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:09.945699    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:10.081982 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:10.081938    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:10.496196 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:10.495798    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb"]
Oct 31 20:02:10.496876 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:10.496847    4645 cpu_manager.go:414] "containerMap: removing container" podUID="ca2e9a7f-b3d9-4f2f-90e6-414543715ada" containerName="payload-pre-processing"
Oct 31 20:02:10.496876 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:10.496873    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca2e9a7f-b3d9-4f2f-90e6-414543715ada" containerName="payload-pre-processing"
Oct 31 20:02:10.497081 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:10.496902    4645 cpu_manager.go:414] "containerMap: removing container" podUID="0ec291f9-5a7f-4d42-946e-668aeb30a976" containerName="payload-processing"
Oct 31 20:02:10.497081 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:10.496911    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ec291f9-5a7f-4d42-946e-668aeb30a976" containerName="payload-processing"
Oct 31 20:02:10.497222 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:10.497116    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca2e9a7f-b3d9-4f2f-90e6-414543715ada" containerName="payload-pre-processing"
Oct 31 20:02:10.497222 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:10.497130    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ec291f9-5a7f-4d42-946e-668aeb30a976" containerName="payload-processing"
Oct 31 20:02:10.502640 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:10.502615    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb"
Oct 31 20:02:10.523515 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:10.523480    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb"]
Oct 31 20:02:10.543254 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:10.543225    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-a84fcb-openshift-default-dockercfg-5hjpj\""
Oct 31 20:02:10.551942 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:10.551911    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:10.611435 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:10.611403    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba05261e-eadf-44c3-abda-cb561b4ea6a5-istio-podinfo\") pod \"e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb\" (UID: \"ba05261e-eadf-44c3-abda-cb561b4ea6a5\") " pod="openshift-ingress/e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb"
Oct 31 20:02:10.611617 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:10.611451    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba05261e-eadf-44c3-abda-cb561b4ea6a5-istiod-ca-cert\") pod \"e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb\" (UID: \"ba05261e-eadf-44c3-abda-cb561b4ea6a5\") " pod="openshift-ingress/e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb"
Oct 31 20:02:10.611617 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:10.611519    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba05261e-eadf-44c3-abda-cb561b4ea6a5-istio-token\") pod \"e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb\" (UID: \"ba05261e-eadf-44c3-abda-cb561b4ea6a5\") " pod="openshift-ingress/e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb"
Oct 31 20:02:10.611617 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:10.611569    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tm7pw\" (UniqueName: \"kubernetes.io/projected/ba05261e-eadf-44c3-abda-cb561b4ea6a5-kube-api-access-tm7pw\") pod \"e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb\" (UID: \"ba05261e-eadf-44c3-abda-cb561b4ea6a5\") " pod="openshift-ingress/e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb"
Oct 31 20:02:10.611741 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:10.611724    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba05261e-eadf-44c3-abda-cb561b4ea6a5-credential-socket\") pod \"e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb\" (UID: \"ba05261e-eadf-44c3-abda-cb561b4ea6a5\") " pod="openshift-ingress/e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb"
Oct 31 20:02:10.611783 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:10.611769    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba05261e-eadf-44c3-abda-cb561b4ea6a5-istio-data\") pod \"e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb\" (UID: \"ba05261e-eadf-44c3-abda-cb561b4ea6a5\") " pod="openshift-ingress/e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb"
Oct 31 20:02:10.611836 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:10.611822    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba05261e-eadf-44c3-abda-cb561b4ea6a5-workload-socket\") pod \"e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb\" (UID: \"ba05261e-eadf-44c3-abda-cb561b4ea6a5\") " pod="openshift-ingress/e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb"
Oct 31 20:02:10.611940 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:10.611929    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba05261e-eadf-44c3-abda-cb561b4ea6a5-workload-certs\") pod \"e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb\" (UID: \"ba05261e-eadf-44c3-abda-cb561b4ea6a5\") " pod="openshift-ingress/e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb"
Oct 31 20:02:10.611978 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:10.611948    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba05261e-eadf-44c3-abda-cb561b4ea6a5-istio-envoy\") pod \"e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb\" (UID: \"ba05261e-eadf-44c3-abda-cb561b4ea6a5\") " pod="openshift-ingress/e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb"
Oct 31 20:02:10.712351 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:10.708096    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:10.716396 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:10.714843    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba05261e-eadf-44c3-abda-cb561b4ea6a5-istio-token\") pod \"e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb\" (UID: \"ba05261e-eadf-44c3-abda-cb561b4ea6a5\") " pod="openshift-ingress/e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb"
Oct 31 20:02:10.736361 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:10.736269    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm7pw\" (UniqueName: \"kubernetes.io/projected/ba05261e-eadf-44c3-abda-cb561b4ea6a5-kube-api-access-tm7pw\") pod \"e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb\" (UID: \"ba05261e-eadf-44c3-abda-cb561b4ea6a5\") " pod="openshift-ingress/e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb"
Oct 31 20:02:10.749397 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:10.736540    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba05261e-eadf-44c3-abda-cb561b4ea6a5-credential-socket\") pod \"e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb\" (UID: \"ba05261e-eadf-44c3-abda-cb561b4ea6a5\") " pod="openshift-ingress/e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb"
Oct 31 20:02:10.749397 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:10.736663    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba05261e-eadf-44c3-abda-cb561b4ea6a5-istio-data\") pod \"e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb\" (UID: \"ba05261e-eadf-44c3-abda-cb561b4ea6a5\") " pod="openshift-ingress/e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb"
Oct 31 20:02:10.749397 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:10.736717    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba05261e-eadf-44c3-abda-cb561b4ea6a5-workload-socket\") pod \"e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb\" (UID: \"ba05261e-eadf-44c3-abda-cb561b4ea6a5\") " pod="openshift-ingress/e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb"
Oct 31 20:02:10.749397 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:10.736899    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba05261e-eadf-44c3-abda-cb561b4ea6a5-workload-certs\") pod \"e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb\" (UID: \"ba05261e-eadf-44c3-abda-cb561b4ea6a5\") " pod="openshift-ingress/e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb"
Oct 31 20:02:10.749397 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:10.736925    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba05261e-eadf-44c3-abda-cb561b4ea6a5-istio-envoy\") pod \"e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb\" (UID: \"ba05261e-eadf-44c3-abda-cb561b4ea6a5\") " pod="openshift-ingress/e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb"
Oct 31 20:02:10.749397 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:10.737052    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba05261e-eadf-44c3-abda-cb561b4ea6a5-istio-podinfo\") pod \"e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb\" (UID: \"ba05261e-eadf-44c3-abda-cb561b4ea6a5\") " pod="openshift-ingress/e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb"
Oct 31 20:02:10.749397 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:10.737127    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba05261e-eadf-44c3-abda-cb561b4ea6a5-istiod-ca-cert\") pod \"e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb\" (UID: \"ba05261e-eadf-44c3-abda-cb561b4ea6a5\") " pod="openshift-ingress/e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb"
Oct 31 20:02:10.749397 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:10.738400    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba05261e-eadf-44c3-abda-cb561b4ea6a5-istiod-ca-cert\") pod \"e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb\" (UID: \"ba05261e-eadf-44c3-abda-cb561b4ea6a5\") " pod="openshift-ingress/e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb"
Oct 31 20:02:10.749397 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:10.739063    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba05261e-eadf-44c3-abda-cb561b4ea6a5-credential-socket\") pod \"e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb\" (UID: \"ba05261e-eadf-44c3-abda-cb561b4ea6a5\") " pod="openshift-ingress/e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb"
Oct 31 20:02:10.749397 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:10.745816    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba05261e-eadf-44c3-abda-cb561b4ea6a5-workload-certs\") pod \"e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb\" (UID: \"ba05261e-eadf-44c3-abda-cb561b4ea6a5\") " pod="openshift-ingress/e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb"
Oct 31 20:02:10.749397 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:10.746077    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba05261e-eadf-44c3-abda-cb561b4ea6a5-istio-data\") pod \"e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb\" (UID: \"ba05261e-eadf-44c3-abda-cb561b4ea6a5\") " pod="openshift-ingress/e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb"
Oct 31 20:02:10.749397 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:10.746318    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba05261e-eadf-44c3-abda-cb561b4ea6a5-workload-socket\") pod \"e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb\" (UID: \"ba05261e-eadf-44c3-abda-cb561b4ea6a5\") " pod="openshift-ingress/e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb"
Oct 31 20:02:10.770407 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:10.764731    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba05261e-eadf-44c3-abda-cb561b4ea6a5-istio-envoy\") pod \"e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb\" (UID: \"ba05261e-eadf-44c3-abda-cb561b4ea6a5\") " pod="openshift-ingress/e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb"
Oct 31 20:02:10.770925 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:10.770889    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba05261e-eadf-44c3-abda-cb561b4ea6a5-istio-podinfo\") pod \"e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb\" (UID: \"ba05261e-eadf-44c3-abda-cb561b4ea6a5\") " pod="openshift-ingress/e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb"
Oct 31 20:02:10.791350 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:10.773729    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba05261e-eadf-44c3-abda-cb561b4ea6a5-istio-token\") pod \"e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb\" (UID: \"ba05261e-eadf-44c3-abda-cb561b4ea6a5\") " pod="openshift-ingress/e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb"
Oct 31 20:02:10.802755 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:10.793627    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm7pw\" (UniqueName: \"kubernetes.io/projected/ba05261e-eadf-44c3-abda-cb561b4ea6a5-kube-api-access-tm7pw\") pod \"e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb\" (UID: \"ba05261e-eadf-44c3-abda-cb561b4ea6a5\") " pod="openshift-ingress/e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb"
Oct 31 20:02:10.834349 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:10.833651    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb"
Oct 31 20:02:11.030137 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:11.028166    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:11.399207 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:11.399177    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb"]
Oct 31 20:02:11.399511 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:02:11.399487    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba05261e_eadf_44c3_abda_cb561b4ea6a5.slice/crio-10a9f2f047601abbb0068e9ee3ddbac9f74f9b69bc8c0997b269cf901ad8dca9 WatchSource:0}: Error finding container 10a9f2f047601abbb0068e9ee3ddbac9f74f9b69bc8c0997b269cf901ad8dca9: Status 404 returned error can't find the container with id 10a9f2f047601abbb0068e9ee3ddbac9f74f9b69bc8c0997b269cf901ad8dca9
Oct 31 20:02:11.402557 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:11.402521    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:02:11.402647 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:11.402606    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:02:11.402647 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:11.402637    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:02:11.736590 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:11.736555    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:12.283885 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:12.283840    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb" event={"ID":"ba05261e-eadf-44c3-abda-cb561b4ea6a5","Type":"ContainerStarted","Data":"28340e41fc47b32fbd85c43ddd7a92a8220d26df04ef0518263bdd162b889516"}
Oct 31 20:02:12.284123 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:12.283909    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb" event={"ID":"ba05261e-eadf-44c3-abda-cb561b4ea6a5","Type":"ContainerStarted","Data":"10a9f2f047601abbb0068e9ee3ddbac9f74f9b69bc8c0997b269cf901ad8dca9"}
Oct 31 20:02:12.310919 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:12.310860    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb" podStartSLOduration=2.310845662 podStartE2EDuration="2.310845662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:02:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:02:12.307311588 +0000 UTC m=+2562.880545716" watchObservedRunningTime="2026-10-31 20:02:12.310845662 +0000 UTC m=+2562.884079791"
Oct 31 20:02:12.580449 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:12.580346    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:12.840165 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:12.840078    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb"
Oct 31 20:02:12.845812 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:12.845786    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb"
Oct 31 20:02:13.293584 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:13.293540    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb"
Oct 31 20:02:13.294479 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:13.294458    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb"
Oct 31 20:02:13.480413 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:13.469542    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:14.159304 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:14.159268    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:14.615814 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:14.615770    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:14.735090 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:14.735051    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:15.144351 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:15.141451    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w"]
Oct 31 20:02:15.183353 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:15.181738    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w"]
Oct 31 20:02:15.183353 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:15.181908    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w"
Oct 31 20:02:15.297513 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:15.297477    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72bb253a-434b-4921-97f2-9f23df7249fe-istio-podinfo\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w\" (UID: \"72bb253a-434b-4921-97f2-9f23df7249fe\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w"
Oct 31 20:02:15.297748 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:15.297542    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72bb253a-434b-4921-97f2-9f23df7249fe-credential-socket\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w\" (UID: \"72bb253a-434b-4921-97f2-9f23df7249fe\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w"
Oct 31 20:02:15.297748 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:15.297685    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72bb253a-434b-4921-97f2-9f23df7249fe-workload-certs\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w\" (UID: \"72bb253a-434b-4921-97f2-9f23df7249fe\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w"
Oct 31 20:02:15.297748 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:15.297730    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72bb253a-434b-4921-97f2-9f23df7249fe-istio-envoy\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w\" (UID: \"72bb253a-434b-4921-97f2-9f23df7249fe\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w"
Oct 31 20:02:15.297880 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:15.297817    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72bb253a-434b-4921-97f2-9f23df7249fe-workload-socket\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w\" (UID: \"72bb253a-434b-4921-97f2-9f23df7249fe\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w"
Oct 31 20:02:15.297880 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:15.297857    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72bb253a-434b-4921-97f2-9f23df7249fe-istio-data\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w\" (UID: \"72bb253a-434b-4921-97f2-9f23df7249fe\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w"
Oct 31 20:02:15.297959 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:15.297940    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72bb253a-434b-4921-97f2-9f23df7249fe-istio-token\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w\" (UID: \"72bb253a-434b-4921-97f2-9f23df7249fe\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w"
Oct 31 20:02:15.297998 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:15.297959    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt4hf\" (UniqueName: \"kubernetes.io/projected/72bb253a-434b-4921-97f2-9f23df7249fe-kube-api-access-kt4hf\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w\" (UID: \"72bb253a-434b-4921-97f2-9f23df7249fe\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w"
Oct 31 20:02:15.298103 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:15.298091    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72bb253a-434b-4921-97f2-9f23df7249fe-istiod-ca-cert\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w\" (UID: \"72bb253a-434b-4921-97f2-9f23df7249fe\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w"
Oct 31 20:02:15.399266 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:15.399191    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72bb253a-434b-4921-97f2-9f23df7249fe-istiod-ca-cert\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w\" (UID: \"72bb253a-434b-4921-97f2-9f23df7249fe\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w"
Oct 31 20:02:15.399266 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:15.399253    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72bb253a-434b-4921-97f2-9f23df7249fe-istio-podinfo\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w\" (UID: \"72bb253a-434b-4921-97f2-9f23df7249fe\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w"
Oct 31 20:02:15.399633 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:15.399303    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72bb253a-434b-4921-97f2-9f23df7249fe-credential-socket\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w\" (UID: \"72bb253a-434b-4921-97f2-9f23df7249fe\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w"
Oct 31 20:02:15.399633 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:15.399406    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72bb253a-434b-4921-97f2-9f23df7249fe-workload-certs\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w\" (UID: \"72bb253a-434b-4921-97f2-9f23df7249fe\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w"
Oct 31 20:02:15.399633 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:15.399478    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72bb253a-434b-4921-97f2-9f23df7249fe-istio-envoy\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w\" (UID: \"72bb253a-434b-4921-97f2-9f23df7249fe\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w"
Oct 31 20:02:15.399633 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:15.399549    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72bb253a-434b-4921-97f2-9f23df7249fe-workload-socket\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w\" (UID: \"72bb253a-434b-4921-97f2-9f23df7249fe\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w"
Oct 31 20:02:15.399633 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:15.399583    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72bb253a-434b-4921-97f2-9f23df7249fe-istio-data\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w\" (UID: \"72bb253a-434b-4921-97f2-9f23df7249fe\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w"
Oct 31 20:02:15.399633 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:15.399624    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72bb253a-434b-4921-97f2-9f23df7249fe-istio-token\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w\" (UID: \"72bb253a-434b-4921-97f2-9f23df7249fe\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w"
Oct 31 20:02:15.399977 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:15.399651    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt4hf\" (UniqueName: \"kubernetes.io/projected/72bb253a-434b-4921-97f2-9f23df7249fe-kube-api-access-kt4hf\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w\" (UID: \"72bb253a-434b-4921-97f2-9f23df7249fe\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w"
Oct 31 20:02:15.399977 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:15.399873    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72bb253a-434b-4921-97f2-9f23df7249fe-workload-certs\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w\" (UID: \"72bb253a-434b-4921-97f2-9f23df7249fe\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w"
Oct 31 20:02:15.400107 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:15.400032    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72bb253a-434b-4921-97f2-9f23df7249fe-istiod-ca-cert\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w\" (UID: \"72bb253a-434b-4921-97f2-9f23df7249fe\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w"
Oct 31 20:02:15.400107 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:15.400090    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72bb253a-434b-4921-97f2-9f23df7249fe-istio-data\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w\" (UID: \"72bb253a-434b-4921-97f2-9f23df7249fe\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w"
Oct 31 20:02:15.400247 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:15.399830    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72bb253a-434b-4921-97f2-9f23df7249fe-credential-socket\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w\" (UID: \"72bb253a-434b-4921-97f2-9f23df7249fe\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w"
Oct 31 20:02:15.400247 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:15.400150    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72bb253a-434b-4921-97f2-9f23df7249fe-workload-socket\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w\" (UID: \"72bb253a-434b-4921-97f2-9f23df7249fe\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w"
Oct 31 20:02:15.402319 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:15.402293    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72bb253a-434b-4921-97f2-9f23df7249fe-istio-envoy\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w\" (UID: \"72bb253a-434b-4921-97f2-9f23df7249fe\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w"
Oct 31 20:02:15.402750 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:15.402729    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72bb253a-434b-4921-97f2-9f23df7249fe-istio-podinfo\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w\" (UID: \"72bb253a-434b-4921-97f2-9f23df7249fe\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w"
Oct 31 20:02:15.412307 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:15.412286    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72bb253a-434b-4921-97f2-9f23df7249fe-istio-token\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w\" (UID: \"72bb253a-434b-4921-97f2-9f23df7249fe\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w"
Oct 31 20:02:15.412451 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:15.412393    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt4hf\" (UniqueName: \"kubernetes.io/projected/72bb253a-434b-4921-97f2-9f23df7249fe-kube-api-access-kt4hf\") pod \"e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w\" (UID: \"72bb253a-434b-4921-97f2-9f23df7249fe\") " pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w"
Oct 31 20:02:15.444809 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:15.444783    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:15.564832 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:15.564799    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w"
Oct 31 20:02:15.938349 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:15.938302    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:15.954256 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:15.950733    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w"]
Oct 31 20:02:15.956949 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:02:15.956920    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72bb253a_434b_4921_97f2_9f23df7249fe.slice/crio-c3fd0928bb115b5dcaac8194c6f386f53ade179f10b100a88cc8ff671d491751 WatchSource:0}: Error finding container c3fd0928bb115b5dcaac8194c6f386f53ade179f10b100a88cc8ff671d491751: Status 404 returned error can't find the container with id c3fd0928bb115b5dcaac8194c6f386f53ade179f10b100a88cc8ff671d491751
Oct 31 20:02:15.965565 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:15.961442    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:02:15.965565 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:15.961520    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:02:15.965565 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:15.961565    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:02:16.059059 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:16.058936    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:16.425795 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:16.422496    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v"]
Oct 31 20:02:16.425795 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:16.422821    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v" podUID="59f3365e-9975-44b4-b7c5-02e91e891f8d" containerName="istio-proxy" containerID="cri-o://beaf92083c6876ba81650f71806ab23dd0f58cddfc2050c7f2db0cde32cf5005" gracePeriod=30
Oct 31 20:02:16.483534 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:16.472849    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:16.528521 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:16.497676    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w" event={"ID":"72bb253a-434b-4921-97f2-9f23df7249fe","Type":"ContainerStarted","Data":"c5963ca2e4f24daee898d56ffc52efd0c9db5a767b66dccd97a6914c2df41339"}
Oct 31 20:02:16.528521 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:16.497750    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w" event={"ID":"72bb253a-434b-4921-97f2-9f23df7249fe","Type":"ContainerStarted","Data":"c3fd0928bb115b5dcaac8194c6f386f53ade179f10b100a88cc8ff671d491751"}
Oct 31 20:02:16.565028 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:16.564949    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w"
Oct 31 20:02:16.779077 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:16.778962    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w" podStartSLOduration=1.778941397 podStartE2EDuration="1.778941397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:02:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:02:16.582921489 +0000 UTC m=+2567.156155620" watchObservedRunningTime="2026-10-31 20:02:16.778941397 +0000 UTC m=+2567.352175528"
Oct 31 20:02:16.881547 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:16.876376    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:17.001024 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:17.000987    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-fd1c91-r2jtb"]
Oct 31 20:02:17.026303 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:17.026259    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-fd1c91-r2jtb"]
Oct 31 20:02:17.026544 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:17.026430    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-fd1c91-r2jtb"
Oct 31 20:02:17.037401 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:17.037025    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:02:17.040060 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:17.040031    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 20:02:17.101308 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:17.101276    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30a36e47-f53c-431f-bfac-7b4e07a11f19-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-fd1c91-r2jtb\" (UID: \"30a36e47-f53c-431f-bfac-7b4e07a11f19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-fd1c91-r2jtb"
Oct 31 20:02:17.101542 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:17.101323    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrppn\" (UniqueName: \"kubernetes.io/projected/30a36e47-f53c-431f-bfac-7b4e07a11f19-kube-api-access-qrppn\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-fd1c91-r2jtb\" (UID: \"30a36e47-f53c-431f-bfac-7b4e07a11f19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-fd1c91-r2jtb"
Oct 31 20:02:17.101665 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:17.101636    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30a36e47-f53c-431f-bfac-7b4e07a11f19-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-fd1c91-r2jtb\" (UID: \"30a36e47-f53c-431f-bfac-7b4e07a11f19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-fd1c91-r2jtb"
Oct 31 20:02:17.203822 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:17.203765    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30a36e47-f53c-431f-bfac-7b4e07a11f19-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-fd1c91-r2jtb\" (UID: \"30a36e47-f53c-431f-bfac-7b4e07a11f19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-fd1c91-r2jtb"
Oct 31 20:02:17.204031 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:17.203847    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrppn\" (UniqueName: \"kubernetes.io/projected/30a36e47-f53c-431f-bfac-7b4e07a11f19-kube-api-access-qrppn\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-fd1c91-r2jtb\" (UID: \"30a36e47-f53c-431f-bfac-7b4e07a11f19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-fd1c91-r2jtb"
Oct 31 20:02:17.204791 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:17.204763    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30a36e47-f53c-431f-bfac-7b4e07a11f19-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-fd1c91-r2jtb\" (UID: \"30a36e47-f53c-431f-bfac-7b4e07a11f19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-fd1c91-r2jtb"
Oct 31 20:02:17.208537 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:17.208502    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30a36e47-f53c-431f-bfac-7b4e07a11f19-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-fd1c91-r2jtb\" (UID: \"30a36e47-f53c-431f-bfac-7b4e07a11f19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-fd1c91-r2jtb"
Oct 31 20:02:17.208652 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:17.208510    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30a36e47-f53c-431f-bfac-7b4e07a11f19-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-fd1c91-r2jtb\" (UID: \"30a36e47-f53c-431f-bfac-7b4e07a11f19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-fd1c91-r2jtb"
Oct 31 20:02:17.285847 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:17.285664    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrppn\" (UniqueName: \"kubernetes.io/projected/30a36e47-f53c-431f-bfac-7b4e07a11f19-kube-api-access-qrppn\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-fd1c91-r2jtb\" (UID: \"30a36e47-f53c-431f-bfac-7b4e07a11f19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-fd1c91-r2jtb"
Oct 31 20:02:17.377753 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:17.376643    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-fd1c91-r2jtb"
Oct 31 20:02:17.570357 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:17.567070    4645 patch_prober.go:29] interesting pod/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused" start-of-body=
Oct 31 20:02:17.570357 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:17.567182    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w" podUID="72bb253a-434b-4921-97f2-9f23df7249fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused"
Oct 31 20:02:17.719702 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:17.719655    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-fd1c91-r2jtb"]
Oct 31 20:02:17.723767 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:02:17.723736    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30a36e47_f53c_431f_bfac_7b4e07a11f19.slice/crio-60cf469fbc8b0daefa74462e98f2ca1cb412da645905f0c6897a982ab7864411 WatchSource:0}: Error finding container 60cf469fbc8b0daefa74462e98f2ca1cb412da645905f0c6897a982ab7864411: Status 404 returned error can't find the container with id 60cf469fbc8b0daefa74462e98f2ca1cb412da645905f0c6897a982ab7864411
Oct 31 20:02:18.041558 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:18.041525    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:18.147494 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:18.147458    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:18.532435 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:18.532392    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-fd1c91-r2jtb" event={"ID":"30a36e47-f53c-431f-bfac-7b4e07a11f19","Type":"ContainerStarted","Data":"b218c166a957be7627b3663410a013d8e811514929e9c142dc8ec5073a3226b9"}
Oct 31 20:02:18.532633 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:18.532469    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-fd1c91-r2jtb" event={"ID":"30a36e47-f53c-431f-bfac-7b4e07a11f19","Type":"ContainerStarted","Data":"60cf469fbc8b0daefa74462e98f2ca1cb412da645905f0c6897a982ab7864411"}
Oct 31 20:02:18.550175 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:18.550118    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-fd1c91-r2jtb" podStartSLOduration=2.5500999760000003 podStartE2EDuration="2.550099976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:02:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:02:18.548792214 +0000 UTC m=+2569.122026342" watchObservedRunningTime="2026-10-31 20:02:18.550099976 +0000 UTC m=+2569.123334106"
Oct 31 20:02:18.573357 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:18.572236    4645 patch_prober.go:29] interesting pod/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused" start-of-body=
Oct 31 20:02:18.573357 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:18.572292    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w" podUID="72bb253a-434b-4921-97f2-9f23df7249fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused"
Oct 31 20:02:18.984860 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:18.984574    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:19.092581 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:19.091844    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:19.541806 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:19.541771    4645 generic.go:356] "Generic (PLEG): container finished" podID="30a36e47-f53c-431f-bfac-7b4e07a11f19" containerID="b218c166a957be7627b3663410a013d8e811514929e9c142dc8ec5073a3226b9" exitCode=0
Oct 31 20:02:19.542003 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:19.541856    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-fd1c91-r2jtb" event={"ID":"30a36e47-f53c-431f-bfac-7b4e07a11f19","Type":"ContainerDied","Data":"b218c166a957be7627b3663410a013d8e811514929e9c142dc8ec5073a3226b9"}
Oct 31 20:02:19.543681 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:19.543660    4645 generic.go:356] "Generic (PLEG): container finished" podID="679339df-b722-4990-b395-b14fea7f6458" containerID="56fe3396ecf1c21d316da758dac8576fc1887b1d90379d62b9cb2b8754cb241b" exitCode=0
Oct 31 20:02:19.543806 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:19.543732    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6" event={"ID":"679339df-b722-4990-b395-b14fea7f6458","Type":"ContainerDied","Data":"56fe3396ecf1c21d316da758dac8576fc1887b1d90379d62b9cb2b8754cb241b"}
Oct 31 20:02:19.544109 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:19.544095    4645 scope.go:122] "RemoveContainer" containerID="56fe3396ecf1c21d316da758dac8576fc1887b1d90379d62b9cb2b8754cb241b"
Oct 31 20:02:19.545884 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:19.545854    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:02:19.545993 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:19.545929    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:02:19.545993 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:19.545967    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:02:19.584608 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:19.582761    4645 patch_prober.go:29] interesting pod/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused" start-of-body=
Oct 31 20:02:19.584608 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:19.582812    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w" podUID="72bb253a-434b-4921-97f2-9f23df7249fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused"
Oct 31 20:02:19.850442 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:19.850321    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:20.094419 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:20.094375    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:20.236009 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:20.228255    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:20.422174 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:20.422140    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:20.592407 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:20.589917    4645 patch_prober.go:29] interesting pod/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused" start-of-body=
Oct 31 20:02:20.592407 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:20.589975    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w" podUID="72bb253a-434b-4921-97f2-9f23df7249fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused"
Oct 31 20:02:20.624342 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:20.624295    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6" event={"ID":"679339df-b722-4990-b395-b14fea7f6458","Type":"ContainerStarted","Data":"f95e143bc1d14b498d313fe8ff37414d1995008395a2383c52a2073a864a1859"}
Oct 31 20:02:21.122074 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:21.121901    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:21.249488 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:21.242893    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:21.273354 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:21.259536    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-fd1c91-r2jtb"
Oct 31 20:02:21.349665 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:21.349630    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30a36e47-f53c-431f-bfac-7b4e07a11f19-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30a36e47-f53c-431f-bfac-7b4e07a11f19-maas-api-openshift-ca\") pod \"30a36e47-f53c-431f-bfac-7b4e07a11f19\" (UID: \"30a36e47-f53c-431f-bfac-7b4e07a11f19\") "
Oct 31 20:02:21.349891 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:21.349863    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30a36e47-f53c-431f-bfac-7b4e07a11f19-kube-api-access-qrppn\" (UniqueName: \"kubernetes.io/projected/30a36e47-f53c-431f-bfac-7b4e07a11f19-kube-api-access-qrppn\") pod \"30a36e47-f53c-431f-bfac-7b4e07a11f19\" (UID: \"30a36e47-f53c-431f-bfac-7b4e07a11f19\") "
Oct 31 20:02:21.350095 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:21.350065    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30a36e47-f53c-431f-bfac-7b4e07a11f19-maas-api-openshift-ca" pod "30a36e47-f53c-431f-bfac-7b4e07a11f19" (UID: "30a36e47-f53c-431f-bfac-7b4e07a11f19"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:02:21.350176 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:21.350070    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30a36e47-f53c-431f-bfac-7b4e07a11f19-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30a36e47-f53c-431f-bfac-7b4e07a11f19-maas-api-xks-ca\") pod \"30a36e47-f53c-431f-bfac-7b4e07a11f19\" (UID: \"30a36e47-f53c-431f-bfac-7b4e07a11f19\") "
Oct 31 20:02:21.350956 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:21.350925    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30a36e47-f53c-431f-bfac-7b4e07a11f19-maas-api-openshift-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:21.353491 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:21.353458    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30a36e47-f53c-431f-bfac-7b4e07a11f19-maas-api-xks-ca" pod "30a36e47-f53c-431f-bfac-7b4e07a11f19" (UID: "30a36e47-f53c-431f-bfac-7b4e07a11f19"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:02:21.354826 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:21.354810    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30a36e47-f53c-431f-bfac-7b4e07a11f19-kube-api-access-qrppn" pod "30a36e47-f53c-431f-bfac-7b4e07a11f19" (UID: "30a36e47-f53c-431f-bfac-7b4e07a11f19"). InnerVolumeSpecName "kube-api-access-qrppn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:02:21.414306 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:21.412589    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6"
Oct 31 20:02:21.414306 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:21.413976    4645 patch_prober.go:29] interesting pod/e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused" start-of-body=
Oct 31 20:02:21.414306 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:21.414030    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6" podUID="679339df-b722-4990-b395-b14fea7f6458" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused"
Oct 31 20:02:21.452383 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:21.452343    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrppn\" (UniqueName: \"kubernetes.io/projected/30a36e47-f53c-431f-bfac-7b4e07a11f19-kube-api-access-qrppn\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:21.452383 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:21.452381    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30a36e47-f53c-431f-bfac-7b4e07a11f19-maas-api-xks-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:21.517294 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:21.499653    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c9b9a1-5c968dc557-lrnmf"]
Oct 31 20:02:21.524464 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:21.524031    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c9b9a1-79688949cf-7krhv"]
Oct 31 20:02:21.611358 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:21.581186    4645 patch_prober.go:29] interesting pod/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused" start-of-body=
Oct 31 20:02:21.611358 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:21.581304    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w" podUID="72bb253a-434b-4921-97f2-9f23df7249fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused"
Oct 31 20:02:21.624044 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:21.623647    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:21.639305 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:21.638284    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-fd1c91-r2jtb"
Oct 31 20:02:21.640424 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:21.640396    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-fd1c91-r2jtb" event={"ID":"30a36e47-f53c-431f-bfac-7b4e07a11f19","Type":"ContainerDied","Data":"60cf469fbc8b0daefa74462e98f2ca1cb412da645905f0c6897a982ab7864411"}
Oct 31 20:02:21.640496 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:21.640452    4645 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60cf469fbc8b0daefa74462e98f2ca1cb412da645905f0c6897a982ab7864411"
Oct 31 20:02:21.737555 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:21.737352    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:22.323978 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:22.317659    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:22.395802 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:22.395181    4645 patch_prober.go:29] interesting pod/e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused" start-of-body=
Oct 31 20:02:22.395802 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:22.395257    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6" podUID="679339df-b722-4990-b395-b14fea7f6458" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused"
Oct 31 20:02:22.568907 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:22.568805    4645 patch_prober.go:29] interesting pod/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused" start-of-body=
Oct 31 20:02:22.569209 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:22.569158    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w" podUID="72bb253a-434b-4921-97f2-9f23df7249fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.25:15021/healthz/ready\": dial tcp 10.133.1.25:15021: connect: connection refused"
Oct 31 20:02:22.653958 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:22.653914    4645 generic.go:356] "Generic (PLEG): container finished" podID="59f3365e-9975-44b4-b7c5-02e91e891f8d" containerID="beaf92083c6876ba81650f71806ab23dd0f58cddfc2050c7f2db0cde32cf5005" exitCode=0
Oct 31 20:02:22.654450 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:22.653981    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v" event={"ID":"59f3365e-9975-44b4-b7c5-02e91e891f8d","Type":"ContainerDied","Data":"beaf92083c6876ba81650f71806ab23dd0f58cddfc2050c7f2db0cde32cf5005"}
Oct 31 20:02:22.654450 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:22.654050    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v" event={"ID":"59f3365e-9975-44b4-b7c5-02e91e891f8d","Type":"ContainerDied","Data":"34f65d220f0917e69d90e700100b1b49a08d3573dcf70aa992cb4ed2724c8d25"}
Oct 31 20:02:22.654450 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:22.654077    4645 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34f65d220f0917e69d90e700100b1b49a08d3573dcf70aa992cb4ed2724c8d25"
Oct 31 20:02:22.654450 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:22.654097    4645 scope.go:122] "RemoveContainer" containerID="3ed7967e7fa87c094576a3a4207749c0f93597da2f202741498966e30ff1a490"
Oct 31 20:02:22.731402 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:22.731374    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v"
Oct 31 20:02:22.819142 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:22.818878    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59f3365e-9975-44b4-b7c5-02e91e891f8d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59f3365e-9975-44b4-b7c5-02e91e891f8d-istio-envoy\") pod \"59f3365e-9975-44b4-b7c5-02e91e891f8d\" (UID: \"59f3365e-9975-44b4-b7c5-02e91e891f8d\") "
Oct 31 20:02:22.820417 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:22.819462    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59f3365e-9975-44b4-b7c5-02e91e891f8d-istio-token\" (UniqueName: \"kubernetes.io/projected/59f3365e-9975-44b4-b7c5-02e91e891f8d-istio-token\") pod \"59f3365e-9975-44b4-b7c5-02e91e891f8d\" (UID: \"59f3365e-9975-44b4-b7c5-02e91e891f8d\") "
Oct 31 20:02:22.820417 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:22.819524    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59f3365e-9975-44b4-b7c5-02e91e891f8d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59f3365e-9975-44b4-b7c5-02e91e891f8d-workload-certs\") pod \"59f3365e-9975-44b4-b7c5-02e91e891f8d\" (UID: \"59f3365e-9975-44b4-b7c5-02e91e891f8d\") "
Oct 31 20:02:22.820417 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:22.819561    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/59f3365e-9975-44b4-b7c5-02e91e891f8d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59f3365e-9975-44b4-b7c5-02e91e891f8d-istio-podinfo\") pod \"59f3365e-9975-44b4-b7c5-02e91e891f8d\" (UID: \"59f3365e-9975-44b4-b7c5-02e91e891f8d\") "
Oct 31 20:02:22.821093 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:22.820712    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59f3365e-9975-44b4-b7c5-02e91e891f8d-workload-certs" pod "59f3365e-9975-44b4-b7c5-02e91e891f8d" (UID: "59f3365e-9975-44b4-b7c5-02e91e891f8d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:02:22.823753 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:22.823722    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59f3365e-9975-44b4-b7c5-02e91e891f8d-istio-token" pod "59f3365e-9975-44b4-b7c5-02e91e891f8d" (UID: "59f3365e-9975-44b4-b7c5-02e91e891f8d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:02:22.823861 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:22.823779    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/59f3365e-9975-44b4-b7c5-02e91e891f8d-istio-podinfo" pod "59f3365e-9975-44b4-b7c5-02e91e891f8d" (UID: "59f3365e-9975-44b4-b7c5-02e91e891f8d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 20:02:22.827451 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:22.827422    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59f3365e-9975-44b4-b7c5-02e91e891f8d-istio-envoy" pod "59f3365e-9975-44b4-b7c5-02e91e891f8d" (UID: "59f3365e-9975-44b4-b7c5-02e91e891f8d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:02:22.921103 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:22.921064    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59f3365e-9975-44b4-b7c5-02e91e891f8d-kube-api-access-v786v\" (UniqueName: \"kubernetes.io/projected/59f3365e-9975-44b4-b7c5-02e91e891f8d-kube-api-access-v786v\") pod \"59f3365e-9975-44b4-b7c5-02e91e891f8d\" (UID: \"59f3365e-9975-44b4-b7c5-02e91e891f8d\") "
Oct 31 20:02:22.921319 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:22.921108    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59f3365e-9975-44b4-b7c5-02e91e891f8d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f3365e-9975-44b4-b7c5-02e91e891f8d-workload-socket\") pod \"59f3365e-9975-44b4-b7c5-02e91e891f8d\" (UID: \"59f3365e-9975-44b4-b7c5-02e91e891f8d\") "
Oct 31 20:02:22.921319 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:22.921174    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59f3365e-9975-44b4-b7c5-02e91e891f8d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59f3365e-9975-44b4-b7c5-02e91e891f8d-istio-data\") pod \"59f3365e-9975-44b4-b7c5-02e91e891f8d\" (UID: \"59f3365e-9975-44b4-b7c5-02e91e891f8d\") "
Oct 31 20:02:22.921319 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:22.921225    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59f3365e-9975-44b4-b7c5-02e91e891f8d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f3365e-9975-44b4-b7c5-02e91e891f8d-credential-socket\") pod \"59f3365e-9975-44b4-b7c5-02e91e891f8d\" (UID: \"59f3365e-9975-44b4-b7c5-02e91e891f8d\") "
Oct 31 20:02:22.921319 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:22.921281    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59f3365e-9975-44b4-b7c5-02e91e891f8d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59f3365e-9975-44b4-b7c5-02e91e891f8d-istiod-ca-cert\") pod \"59f3365e-9975-44b4-b7c5-02e91e891f8d\" (UID: \"59f3365e-9975-44b4-b7c5-02e91e891f8d\") "
Oct 31 20:02:22.921629 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:22.921524    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59f3365e-9975-44b4-b7c5-02e91e891f8d-workload-socket" pod "59f3365e-9975-44b4-b7c5-02e91e891f8d" (UID: "59f3365e-9975-44b4-b7c5-02e91e891f8d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:02:22.921699 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:22.921668    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59f3365e-9975-44b4-b7c5-02e91e891f8d-istio-data" pod "59f3365e-9975-44b4-b7c5-02e91e891f8d" (UID: "59f3365e-9975-44b4-b7c5-02e91e891f8d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:02:22.921837 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:22.921795    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59f3365e-9975-44b4-b7c5-02e91e891f8d-credential-socket" pod "59f3365e-9975-44b4-b7c5-02e91e891f8d" (UID: "59f3365e-9975-44b4-b7c5-02e91e891f8d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:02:22.921948 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:22.921921    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59f3365e-9975-44b4-b7c5-02e91e891f8d-istiod-ca-cert" pod "59f3365e-9975-44b4-b7c5-02e91e891f8d" (UID: "59f3365e-9975-44b4-b7c5-02e91e891f8d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:02:22.922057 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:22.922045    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59f3365e-9975-44b4-b7c5-02e91e891f8d-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:22.922095 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:22.922060    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59f3365e-9975-44b4-b7c5-02e91e891f8d-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:22.922095 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:22.922069    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59f3365e-9975-44b4-b7c5-02e91e891f8d-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:22.922095 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:22.922081    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59f3365e-9975-44b4-b7c5-02e91e891f8d-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:22.922095 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:22.922089    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f3365e-9975-44b4-b7c5-02e91e891f8d-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:22.922231 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:22.922097    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59f3365e-9975-44b4-b7c5-02e91e891f8d-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:22.922231 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:22.922107    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f3365e-9975-44b4-b7c5-02e91e891f8d-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:22.922231 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:22.922116    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59f3365e-9975-44b4-b7c5-02e91e891f8d-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:22.923290 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:22.923272    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59f3365e-9975-44b4-b7c5-02e91e891f8d-kube-api-access-v786v" pod "59f3365e-9975-44b4-b7c5-02e91e891f8d" (UID: "59f3365e-9975-44b4-b7c5-02e91e891f8d"). InnerVolumeSpecName "kube-api-access-v786v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:02:23.023046 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:23.023005    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v786v\" (UniqueName: \"kubernetes.io/projected/59f3365e-9975-44b4-b7c5-02e91e891f8d-kube-api-access-v786v\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:23.054389 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:23.054357    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:23.390909 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:23.390872    4645 patch_prober.go:29] interesting pod/e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused" start-of-body=
Oct 31 20:02:23.391110 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:23.390929    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6" podUID="679339df-b722-4990-b395-b14fea7f6458" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused"
Oct 31 20:02:23.592672 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:23.592633    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w"
Oct 31 20:02:23.594296 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:23.594264    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w"
Oct 31 20:02:23.604368 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:23.601836    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w"
Oct 31 20:02:23.718132 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:23.715893    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v"
Oct 31 20:02:23.799072 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:23.799031    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v"]
Oct 31 20:02:23.803522 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:23.803494    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-5f3eff-openshift-default-78b5fb4875-7qq8v"]
Oct 31 20:02:23.940810 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:23.940769    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:24.390783 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:24.390749    4645 patch_prober.go:29] interesting pod/e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused" start-of-body=
Oct 31 20:02:24.390985 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:24.390806    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6" podUID="679339df-b722-4990-b395-b14fea7f6458" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused"
Oct 31 20:02:24.848779 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:24.848739    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:25.390749 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:25.390710    4645 patch_prober.go:29] interesting pod/e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused" start-of-body=
Oct 31 20:02:25.390911 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:25.390766    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6" podUID="679339df-b722-4990-b395-b14fea7f6458" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused"
Oct 31 20:02:25.420226 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:25.420200    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6"
Oct 31 20:02:25.620100 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:25.620068    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59f3365e-9975-44b4-b7c5-02e91e891f8d" path="/var/lib/kubelet/pods/59f3365e-9975-44b4-b7c5-02e91e891f8d/volumes"
Oct 31 20:02:25.819528 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:25.819494    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:26.391655 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:26.391612    4645 patch_prober.go:29] interesting pod/e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused" start-of-body=
Oct 31 20:02:26.392044 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:26.391672    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6" podUID="679339df-b722-4990-b395-b14fea7f6458" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused"
Oct 31 20:02:26.500734 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:26.500668    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-c9b9a1-5c968dc557-lrnmf" podUID="b29a6202-a717-4880-8c43-03967211c64c" containerName="payload-processing" containerID="cri-o://ad2c45fe415be4240ede033c59630fe5ab5c2fdc6f969b7b06d0ba0baf4f3812" gracePeriod=25
Oct 31 20:02:26.526533 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:26.526477    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c9b9a1-79688949cf-7krhv" podUID="14316266-1376-4285-8df1-a57cab31ab04" containerName="payload-pre-processing" containerID="cri-o://ac455eac69b35afda7482c90af276773fc60378cac4df99d961f61ce8e259bd8" gracePeriod=25
Oct 31 20:02:26.540657 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:26.540628    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:26.744018 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:26.743680    4645 generic.go:356] "Generic (PLEG): container finished" podID="b29a6202-a717-4880-8c43-03967211c64c" containerID="ad2c45fe415be4240ede033c59630fe5ab5c2fdc6f969b7b06d0ba0baf4f3812" exitCode=0
Oct 31 20:02:26.744018 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:26.743932    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c9b9a1-5c968dc557-lrnmf" event={"ID":"b29a6202-a717-4880-8c43-03967211c64c","Type":"ContainerDied","Data":"ad2c45fe415be4240ede033c59630fe5ab5c2fdc6f969b7b06d0ba0baf4f3812"}
Oct 31 20:02:26.747888 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:26.747862    4645 generic.go:356] "Generic (PLEG): container finished" podID="14316266-1376-4285-8df1-a57cab31ab04" containerID="ac455eac69b35afda7482c90af276773fc60378cac4df99d961f61ce8e259bd8" exitCode=0
Oct 31 20:02:26.748186 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:26.748159    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c9b9a1-79688949cf-7krhv" event={"ID":"14316266-1376-4285-8df1-a57cab31ab04","Type":"ContainerDied","Data":"ac455eac69b35afda7482c90af276773fc60378cac4df99d961f61ce8e259bd8"}
Oct 31 20:02:26.872564 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:26.869741    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c9b9a1-79688949cf-7krhv"
Oct 31 20:02:26.893581 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:26.893553    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c9b9a1-5c968dc557-lrnmf"
Oct 31 20:02:26.907488 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:26.907463    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14316266-1376-4285-8df1-a57cab31ab04-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14316266-1376-4285-8df1-a57cab31ab04-plugins-config-volume\") pod \"14316266-1376-4285-8df1-a57cab31ab04\" (UID: \"14316266-1376-4285-8df1-a57cab31ab04\") "
Oct 31 20:02:26.907753 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:26.907728    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b29a6202-a717-4880-8c43-03967211c64c-kube-api-access-wfntt\" (UniqueName: \"kubernetes.io/projected/b29a6202-a717-4880-8c43-03967211c64c-kube-api-access-wfntt\") pod \"b29a6202-a717-4880-8c43-03967211c64c\" (UID: \"b29a6202-a717-4880-8c43-03967211c64c\") "
Oct 31 20:02:26.907857 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:26.907807    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14316266-1376-4285-8df1-a57cab31ab04-kube-api-access-48jr9\" (UniqueName: \"kubernetes.io/projected/14316266-1376-4285-8df1-a57cab31ab04-kube-api-access-48jr9\") pod \"14316266-1376-4285-8df1-a57cab31ab04\" (UID: \"14316266-1376-4285-8df1-a57cab31ab04\") "
Oct 31 20:02:26.907857 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:26.907842    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b29a6202-a717-4880-8c43-03967211c64c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b29a6202-a717-4880-8c43-03967211c64c-plugins-config-volume\") pod \"b29a6202-a717-4880-8c43-03967211c64c\" (UID: \"b29a6202-a717-4880-8c43-03967211c64c\") "
Oct 31 20:02:26.907986 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:26.907873    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14316266-1376-4285-8df1-a57cab31ab04-plugins-config-volume" pod "14316266-1376-4285-8df1-a57cab31ab04" (UID: "14316266-1376-4285-8df1-a57cab31ab04"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:02:26.908614 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:26.908590    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b29a6202-a717-4880-8c43-03967211c64c-plugins-config-volume" pod "b29a6202-a717-4880-8c43-03967211c64c" (UID: "b29a6202-a717-4880-8c43-03967211c64c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:02:26.908769 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:26.908716    4645 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14316266-1376-4285-8df1-a57cab31ab04-plugins-config-volume\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:26.910703 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:26.910681    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b29a6202-a717-4880-8c43-03967211c64c-kube-api-access-wfntt" pod "b29a6202-a717-4880-8c43-03967211c64c" (UID: "b29a6202-a717-4880-8c43-03967211c64c"). InnerVolumeSpecName "kube-api-access-wfntt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:02:26.910814 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:26.910724    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14316266-1376-4285-8df1-a57cab31ab04-kube-api-access-48jr9" pod "14316266-1376-4285-8df1-a57cab31ab04" (UID: "14316266-1376-4285-8df1-a57cab31ab04"). InnerVolumeSpecName "kube-api-access-48jr9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:02:27.009961 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:27.009884    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfntt\" (UniqueName: \"kubernetes.io/projected/b29a6202-a717-4880-8c43-03967211c64c-kube-api-access-wfntt\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:27.009961 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:27.009912    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48jr9\" (UniqueName: \"kubernetes.io/projected/14316266-1376-4285-8df1-a57cab31ab04-kube-api-access-48jr9\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:27.009961 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:27.009922    4645 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b29a6202-a717-4880-8c43-03967211c64c-plugins-config-volume\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:27.099863 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:27.099829    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-fd1c91-84dcbc9465-5r662"]
Oct 31 20:02:27.123977 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:27.123948    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-fd1c91-5fb9c8fcf9-xvrkd"]
Oct 31 20:02:27.178820 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:27.178782    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:27.284582 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:27.284501    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:27.456350 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:27.454685    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6"
Oct 31 20:02:27.658511 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:27.658408    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:27.664381 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:27.664130    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6"
Oct 31 20:02:27.764447 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:27.764405    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c9b9a1-5c968dc557-lrnmf"
Oct 31 20:02:27.764630 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:27.764492    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c9b9a1-5c968dc557-lrnmf" event={"ID":"b29a6202-a717-4880-8c43-03967211c64c","Type":"ContainerDied","Data":"f016d9baf0ce85d306eaa15469e5a3c24067f661aec368ea4f2700f96c4c216e"}
Oct 31 20:02:27.764630 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:27.764535    4645 scope.go:122] "RemoveContainer" containerID="ad2c45fe415be4240ede033c59630fe5ab5c2fdc6f969b7b06d0ba0baf4f3812"
Oct 31 20:02:27.769833 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:27.769127    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c9b9a1-79688949cf-7krhv"
Oct 31 20:02:27.769833 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:27.769541    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c9b9a1-79688949cf-7krhv" event={"ID":"14316266-1376-4285-8df1-a57cab31ab04","Type":"ContainerDied","Data":"239ebeb918eff856b4a5baf2304099bfbd1efcb094f59c36ee2b36a18f9aff3c"}
Oct 31 20:02:27.779155 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:27.779133    4645 scope.go:122] "RemoveContainer" containerID="ac455eac69b35afda7482c90af276773fc60378cac4df99d961f61ce8e259bd8"
Oct 31 20:02:27.800006 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:27.799979    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c9b9a1-5c968dc557-lrnmf"]
Oct 31 20:02:27.804929 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:27.804901    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:27.805917 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:27.805894    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c9b9a1-5c968dc557-lrnmf"]
Oct 31 20:02:27.816149 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:27.816120    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c9b9a1-79688949cf-7krhv"]
Oct 31 20:02:27.820752 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:27.820730    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c9b9a1-79688949cf-7krhv"]
Oct 31 20:02:27.829436 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:27.829413    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-fd1c91-r2jtb"]
Oct 31 20:02:27.834778 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:27.834753    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-fd1c91-r2jtb"]
Oct 31 20:02:27.914278 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:27.914245    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:28.363062 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:28.362969    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:28.546758 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:28.546723    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:29.439431 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:29.439396    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:29.619383 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:29.619352    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14316266-1376-4285-8df1-a57cab31ab04" path="/var/lib/kubelet/pods/14316266-1376-4285-8df1-a57cab31ab04/volumes"
Oct 31 20:02:29.619858 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:29.619841    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30a36e47-f53c-431f-bfac-7b4e07a11f19" path="/var/lib/kubelet/pods/30a36e47-f53c-431f-bfac-7b4e07a11f19/volumes"
Oct 31 20:02:29.620267 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:29.620253    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b29a6202-a717-4880-8c43-03967211c64c" path="/var/lib/kubelet/pods/b29a6202-a717-4880-8c43-03967211c64c/volumes"
Oct 31 20:02:30.289352 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:30.288878    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:31.013349 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:31.013299    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:31.782519 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:31.782481    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:32.100615 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:32.100524    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-fd1c91-84dcbc9465-5r662" podUID="af2f7267-2d2f-46cf-b8e2-101e6a328b68" containerName="payload-processing" containerID="cri-o://0074c9f1f0aa3e09ff2e12f2d179a38bd914178c99b2596828a33eef2412f8c2" gracePeriod=25
Oct 31 20:02:32.125042 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:32.124980    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-fd1c91-5fb9c8fcf9-xvrkd" podUID="600fee45-c980-4705-96d9-fc084f4bea96" containerName="payload-pre-processing" containerID="cri-o://60fa6d013b43f2246d859b83c10f7cd8a182fe5e0d67b6ce03f08ddb863f7e7a" gracePeriod=25
Oct 31 20:02:32.544814 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:32.544777    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:32.811683 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:32.811595    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:33.432072 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:33.428381    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:34.170102 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:34.170059    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:34.951739 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:34.951706    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:35.940352 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:35.937427    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:36.071351 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:36.066844    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:36.185124 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:36.185088    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:36.261647 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:02:36.261565    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 20:02:36.456003 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:36.455961    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:36.703685 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:36.703351    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-fd1c91-5fb9c8fcf9-xvrkd" podUID="600fee45-c980-4705-96d9-fc084f4bea96" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:02:36.947622 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:36.947586    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f5c1ab-4n2zw"]
Oct 31 20:02:36.948322 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:36.948306    4645 cpu_manager.go:414] "containerMap: removing container" podUID="14316266-1376-4285-8df1-a57cab31ab04" containerName="payload-pre-processing"
Oct 31 20:02:36.948400 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:36.948323    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14316266-1376-4285-8df1-a57cab31ab04" containerName="payload-pre-processing"
Oct 31 20:02:36.948400 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:36.948352    4645 cpu_manager.go:414] "containerMap: removing container" podUID="30a36e47-f53c-431f-bfac-7b4e07a11f19" containerName="revoke-keys"
Oct 31 20:02:36.948400 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:36.948357    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30a36e47-f53c-431f-bfac-7b4e07a11f19" containerName="revoke-keys"
Oct 31 20:02:36.948400 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:36.948388    4645 cpu_manager.go:414] "containerMap: removing container" podUID="59f3365e-9975-44b4-b7c5-02e91e891f8d" containerName="istio-proxy"
Oct 31 20:02:36.948400 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:36.948393    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59f3365e-9975-44b4-b7c5-02e91e891f8d" containerName="istio-proxy"
Oct 31 20:02:36.948581 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:36.948409    4645 cpu_manager.go:414] "containerMap: removing container" podUID="b29a6202-a717-4880-8c43-03967211c64c" containerName="payload-processing"
Oct 31 20:02:36.948581 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:36.948415    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b29a6202-a717-4880-8c43-03967211c64c" containerName="payload-processing"
Oct 31 20:02:36.948581 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:36.948546    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="14316266-1376-4285-8df1-a57cab31ab04" containerName="payload-pre-processing"
Oct 31 20:02:36.948581 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:36.948554    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="30a36e47-f53c-431f-bfac-7b4e07a11f19" containerName="revoke-keys"
Oct 31 20:02:36.948581 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:36.948566    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="59f3365e-9975-44b4-b7c5-02e91e891f8d" containerName="istio-proxy"
Oct 31 20:02:36.948581 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:36.948574    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="59f3365e-9975-44b4-b7c5-02e91e891f8d" containerName="istio-proxy"
Oct 31 20:02:36.948581 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:36.948580    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="b29a6202-a717-4880-8c43-03967211c64c" containerName="payload-processing"
Oct 31 20:02:36.953215 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:36.953197    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f5c1ab-4n2zw"
Oct 31 20:02:36.955661 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:36.955593    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:36.961945 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:36.961919    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f5c1ab-4n2zw"]
Oct 31 20:02:37.001404 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:37.001284    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:02:37.001693 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:37.001452    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 20:02:37.102645 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:37.102612    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-947hp\" (UniqueName: \"kubernetes.io/projected/9c4eead5-086d-4c44-b2a8-c5d2597bc6e6-kube-api-access-947hp\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-f5c1ab-4n2zw\" (UID: \"9c4eead5-086d-4c44-b2a8-c5d2597bc6e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f5c1ab-4n2zw"
Oct 31 20:02:37.103032 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:37.102681    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c4eead5-086d-4c44-b2a8-c5d2597bc6e6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-f5c1ab-4n2zw\" (UID: \"9c4eead5-086d-4c44-b2a8-c5d2597bc6e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f5c1ab-4n2zw"
Oct 31 20:02:37.103032 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:37.102857    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c4eead5-086d-4c44-b2a8-c5d2597bc6e6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-f5c1ab-4n2zw\" (UID: \"9c4eead5-086d-4c44-b2a8-c5d2597bc6e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f5c1ab-4n2zw"
Oct 31 20:02:37.203965 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:37.203931    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c4eead5-086d-4c44-b2a8-c5d2597bc6e6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-f5c1ab-4n2zw\" (UID: \"9c4eead5-086d-4c44-b2a8-c5d2597bc6e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f5c1ab-4n2zw"
Oct 31 20:02:37.204135 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:37.204095    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-947hp\" (UniqueName: \"kubernetes.io/projected/9c4eead5-086d-4c44-b2a8-c5d2597bc6e6-kube-api-access-947hp\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-f5c1ab-4n2zw\" (UID: \"9c4eead5-086d-4c44-b2a8-c5d2597bc6e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f5c1ab-4n2zw"
Oct 31 20:02:37.204192 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:37.204154    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c4eead5-086d-4c44-b2a8-c5d2597bc6e6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-f5c1ab-4n2zw\" (UID: \"9c4eead5-086d-4c44-b2a8-c5d2597bc6e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f5c1ab-4n2zw"
Oct 31 20:02:37.204733 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:37.204715    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c4eead5-086d-4c44-b2a8-c5d2597bc6e6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-f5c1ab-4n2zw\" (UID: \"9c4eead5-086d-4c44-b2a8-c5d2597bc6e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f5c1ab-4n2zw"
Oct 31 20:02:37.206647 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:37.206606    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c4eead5-086d-4c44-b2a8-c5d2597bc6e6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-f5c1ab-4n2zw\" (UID: \"9c4eead5-086d-4c44-b2a8-c5d2597bc6e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f5c1ab-4n2zw"
Oct 31 20:02:37.215906 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:37.215885    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-947hp\" (UniqueName: \"kubernetes.io/projected/9c4eead5-086d-4c44-b2a8-c5d2597bc6e6-kube-api-access-947hp\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-f5c1ab-4n2zw\" (UID: \"9c4eead5-086d-4c44-b2a8-c5d2597bc6e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f5c1ab-4n2zw"
Oct 31 20:02:37.278118 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:37.278089    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f5c1ab-4n2zw"
Oct 31 20:02:37.593811 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:37.593584    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f5c1ab-4n2zw"]
Oct 31 20:02:37.594134 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:02:37.594106    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c4eead5_086d_4c44_b2a8_c5d2597bc6e6.slice/crio-a9c487b662e2d0ed272121d94f55f1622f8df1ca555d1df85ae68317d14706be WatchSource:0}: Error finding container a9c487b662e2d0ed272121d94f55f1622f8df1ca555d1df85ae68317d14706be: Status 404 returned error can't find the container with id a9c487b662e2d0ed272121d94f55f1622f8df1ca555d1df85ae68317d14706be
Oct 31 20:02:37.749628 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:37.749588    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-fd1c91-84dcbc9465-5r662" podUID="af2f7267-2d2f-46cf-b8e2-101e6a328b68" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:02:37.902920 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:37.902830    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f5c1ab-4n2zw" event={"ID":"9c4eead5-086d-4c44-b2a8-c5d2597bc6e6","Type":"ContainerStarted","Data":"a9c487b662e2d0ed272121d94f55f1622f8df1ca555d1df85ae68317d14706be"}
Oct 31 20:02:38.183569 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:38.182732    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:38.742844 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:38.742800    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:38.881263 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:38.881230    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:38.912201 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:38.912164    4645 generic.go:356] "Generic (PLEG): container finished" podID="9c4eead5-086d-4c44-b2a8-c5d2597bc6e6" containerID="1440d2ae4202389fc3b0b6b17f75b39c86eaf3a2d847875e096b3ce8d02ff764" exitCode=0
Oct 31 20:02:38.912417 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:38.912259    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f5c1ab-4n2zw" event={"ID":"9c4eead5-086d-4c44-b2a8-c5d2597bc6e6","Type":"ContainerDied","Data":"1440d2ae4202389fc3b0b6b17f75b39c86eaf3a2d847875e096b3ce8d02ff764"}
Oct 31 20:02:38.933611 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:38.933533    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c9b9a1-666b4798c-b9svq"]
Oct 31 20:02:38.934370 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:38.934349    4645 cpu_manager.go:414] "containerMap: removing container" podUID="59f3365e-9975-44b4-b7c5-02e91e891f8d" containerName="istio-proxy"
Oct 31 20:02:38.934469 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:38.934372    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59f3365e-9975-44b4-b7c5-02e91e891f8d" containerName="istio-proxy"
Oct 31 20:02:38.938077 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:38.938056    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c9b9a1-666b4798c-b9svq"
Oct 31 20:02:38.947096 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:38.947068    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-c9b9a1\""
Oct 31 20:02:38.950556 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:38.950532    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-c9b9a1-dockercfg-mgr8n\""
Oct 31 20:02:38.994354 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:38.981403    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c9b9a1-666b4798c-b9svq"]
Oct 31 20:02:39.205408 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:39.197567    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8h9x\" (UniqueName: \"kubernetes.io/projected/c9e1a3d6-8031-4e60-a4ab-c53aa827d93a-kube-api-access-l8h9x\") pod \"payload-pre-processing-e2e-ipp-b-c9b9a1-666b4798c-b9svq\" (UID: \"c9e1a3d6-8031-4e60-a4ab-c53aa827d93a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c9b9a1-666b4798c-b9svq"
Oct 31 20:02:39.205408 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:39.197740    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9e1a3d6-8031-4e60-a4ab-c53aa827d93a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c9b9a1-666b4798c-b9svq\" (UID: \"c9e1a3d6-8031-4e60-a4ab-c53aa827d93a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c9b9a1-666b4798c-b9svq"
Oct 31 20:02:39.354354 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:39.320745    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8h9x\" (UniqueName: \"kubernetes.io/projected/c9e1a3d6-8031-4e60-a4ab-c53aa827d93a-kube-api-access-l8h9x\") pod \"payload-pre-processing-e2e-ipp-b-c9b9a1-666b4798c-b9svq\" (UID: \"c9e1a3d6-8031-4e60-a4ab-c53aa827d93a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c9b9a1-666b4798c-b9svq"
Oct 31 20:02:39.354952 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:39.354912    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9e1a3d6-8031-4e60-a4ab-c53aa827d93a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c9b9a1-666b4798c-b9svq\" (UID: \"c9e1a3d6-8031-4e60-a4ab-c53aa827d93a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c9b9a1-666b4798c-b9svq"
Oct 31 20:02:39.399355 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:39.356650    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9e1a3d6-8031-4e60-a4ab-c53aa827d93a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c9b9a1-666b4798c-b9svq\" (UID: \"c9e1a3d6-8031-4e60-a4ab-c53aa827d93a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c9b9a1-666b4798c-b9svq"
Oct 31 20:02:39.399355 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:39.372914    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c9b9a1-f45479ffd-mqgll"]
Oct 31 20:02:39.402627 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:39.402595    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8h9x\" (UniqueName: \"kubernetes.io/projected/c9e1a3d6-8031-4e60-a4ab-c53aa827d93a-kube-api-access-l8h9x\") pod \"payload-pre-processing-e2e-ipp-b-c9b9a1-666b4798c-b9svq\" (UID: \"c9e1a3d6-8031-4e60-a4ab-c53aa827d93a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c9b9a1-666b4798c-b9svq"
Oct 31 20:02:39.409983 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:39.409955    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c9b9a1-f45479ffd-mqgll"
Oct 31 20:02:39.413027 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:39.412690    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c9b9a1-f45479ffd-mqgll"]
Oct 31 20:02:39.479402 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:39.464444    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb8k8\" (UniqueName: \"kubernetes.io/projected/d1e003dd-25ad-430d-bd3c-b7706d2788cd-kube-api-access-xb8k8\") pod \"payload-processing-e2e-ipp-b-c9b9a1-f45479ffd-mqgll\" (UID: \"d1e003dd-25ad-430d-bd3c-b7706d2788cd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c9b9a1-f45479ffd-mqgll"
Oct 31 20:02:39.479402 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:39.464655    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1e003dd-25ad-430d-bd3c-b7706d2788cd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c9b9a1-f45479ffd-mqgll\" (UID: \"d1e003dd-25ad-430d-bd3c-b7706d2788cd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c9b9a1-f45479ffd-mqgll"
Oct 31 20:02:39.521629 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:39.521598    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:39.567605 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:39.566376    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c9b9a1-666b4798c-b9svq"
Oct 31 20:02:39.568294 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:39.568260    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb8k8\" (UniqueName: \"kubernetes.io/projected/d1e003dd-25ad-430d-bd3c-b7706d2788cd-kube-api-access-xb8k8\") pod \"payload-processing-e2e-ipp-b-c9b9a1-f45479ffd-mqgll\" (UID: \"d1e003dd-25ad-430d-bd3c-b7706d2788cd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c9b9a1-f45479ffd-mqgll"
Oct 31 20:02:39.580352 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:39.571391    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1e003dd-25ad-430d-bd3c-b7706d2788cd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c9b9a1-f45479ffd-mqgll\" (UID: \"d1e003dd-25ad-430d-bd3c-b7706d2788cd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c9b9a1-f45479ffd-mqgll"
Oct 31 20:02:39.581580 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:39.573275    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1e003dd-25ad-430d-bd3c-b7706d2788cd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c9b9a1-f45479ffd-mqgll\" (UID: \"d1e003dd-25ad-430d-bd3c-b7706d2788cd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c9b9a1-f45479ffd-mqgll"
Oct 31 20:02:39.609536 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:39.609489    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb8k8\" (UniqueName: \"kubernetes.io/projected/d1e003dd-25ad-430d-bd3c-b7706d2788cd-kube-api-access-xb8k8\") pod \"payload-processing-e2e-ipp-b-c9b9a1-f45479ffd-mqgll\" (UID: \"d1e003dd-25ad-430d-bd3c-b7706d2788cd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c9b9a1-f45479ffd-mqgll"
Oct 31 20:02:39.726147 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:39.725771    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c9b9a1-f45479ffd-mqgll"
Oct 31 20:02:39.927940 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:02:39.927906    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9e1a3d6_8031_4e60_a4ab_c53aa827d93a.slice/crio-73a018603afcba29b2c3c70656a2557d45d382cf1e1fc694361b3dda4ac7e588 WatchSource:0}: Error finding container 73a018603afcba29b2c3c70656a2557d45d382cf1e1fc694361b3dda4ac7e588: Status 404 returned error can't find the container with id 73a018603afcba29b2c3c70656a2557d45d382cf1e1fc694361b3dda4ac7e588
Oct 31 20:02:39.930069 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:39.930038    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c9b9a1-666b4798c-b9svq"]
Oct 31 20:02:40.046942 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:40.046916    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c9b9a1-f45479ffd-mqgll"]
Oct 31 20:02:40.111062 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:40.111040    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f5c1ab-4n2zw"
Oct 31 20:02:40.198216 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:40.194960    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c4eead5-086d-4c44-b2a8-c5d2597bc6e6-kube-api-access-947hp\" (UniqueName: \"kubernetes.io/projected/9c4eead5-086d-4c44-b2a8-c5d2597bc6e6-kube-api-access-947hp\") pod \"9c4eead5-086d-4c44-b2a8-c5d2597bc6e6\" (UID: \"9c4eead5-086d-4c44-b2a8-c5d2597bc6e6\") "
Oct 31 20:02:40.198216 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:40.195040    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c4eead5-086d-4c44-b2a8-c5d2597bc6e6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c4eead5-086d-4c44-b2a8-c5d2597bc6e6-maas-api-openshift-ca\") pod \"9c4eead5-086d-4c44-b2a8-c5d2597bc6e6\" (UID: \"9c4eead5-086d-4c44-b2a8-c5d2597bc6e6\") "
Oct 31 20:02:40.198216 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:40.195252    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c4eead5-086d-4c44-b2a8-c5d2597bc6e6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c4eead5-086d-4c44-b2a8-c5d2597bc6e6-maas-api-xks-ca\") pod \"9c4eead5-086d-4c44-b2a8-c5d2597bc6e6\" (UID: \"9c4eead5-086d-4c44-b2a8-c5d2597bc6e6\") "
Oct 31 20:02:40.201683 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:40.199126    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c4eead5-086d-4c44-b2a8-c5d2597bc6e6-maas-api-openshift-ca" pod "9c4eead5-086d-4c44-b2a8-c5d2597bc6e6" (UID: "9c4eead5-086d-4c44-b2a8-c5d2597bc6e6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:02:40.203600 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:40.203569    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c4eead5-086d-4c44-b2a8-c5d2597bc6e6-kube-api-access-947hp" pod "9c4eead5-086d-4c44-b2a8-c5d2597bc6e6" (UID: "9c4eead5-086d-4c44-b2a8-c5d2597bc6e6"). InnerVolumeSpecName "kube-api-access-947hp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:02:40.206453 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:40.206425    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c4eead5-086d-4c44-b2a8-c5d2597bc6e6-maas-api-xks-ca" pod "9c4eead5-086d-4c44-b2a8-c5d2597bc6e6" (UID: "9c4eead5-086d-4c44-b2a8-c5d2597bc6e6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:02:40.278412 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:40.276142    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:40.300406 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:40.298078    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-947hp\" (UniqueName: \"kubernetes.io/projected/9c4eead5-086d-4c44-b2a8-c5d2597bc6e6-kube-api-access-947hp\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:40.300406 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:40.298117    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c4eead5-086d-4c44-b2a8-c5d2597bc6e6-maas-api-openshift-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:40.300406 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:40.298140    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c4eead5-086d-4c44-b2a8-c5d2597bc6e6-maas-api-xks-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:40.928699 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:40.928667    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f5c1ab-4n2zw"
Oct 31 20:02:40.928924 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:40.928700    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f5c1ab-4n2zw" event={"ID":"9c4eead5-086d-4c44-b2a8-c5d2597bc6e6","Type":"ContainerDied","Data":"a9c487b662e2d0ed272121d94f55f1622f8df1ca555d1df85ae68317d14706be"}
Oct 31 20:02:40.928924 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:40.928764    4645 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9c487b662e2d0ed272121d94f55f1622f8df1ca555d1df85ae68317d14706be"
Oct 31 20:02:40.931923 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:40.931897    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c9b9a1-f45479ffd-mqgll" event={"ID":"d1e003dd-25ad-430d-bd3c-b7706d2788cd","Type":"ContainerStarted","Data":"7f26d871198a41675d7b35d9e637e7b44e6536ecb8c9b3bcd8f4a8225357d31a"}
Oct 31 20:02:40.932142 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:40.931953    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c9b9a1-f45479ffd-mqgll" event={"ID":"d1e003dd-25ad-430d-bd3c-b7706d2788cd","Type":"ContainerStarted","Data":"ead99044d01033549695e63b22d1453b0750d364c1a63f9e0d16b57b06f63d44"}
Oct 31 20:02:40.932142 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:40.931980    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c9b9a1-f45479ffd-mqgll"
Oct 31 20:02:40.933340 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:40.933304    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c9b9a1-666b4798c-b9svq" event={"ID":"c9e1a3d6-8031-4e60-a4ab-c53aa827d93a","Type":"ContainerStarted","Data":"774356287b9d6786753329d8d3d7b8acb8d87fd058dd4d770845e0eff37a5472"}
Oct 31 20:02:40.933466 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:40.933359    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c9b9a1-666b4798c-b9svq" event={"ID":"c9e1a3d6-8031-4e60-a4ab-c53aa827d93a","Type":"ContainerStarted","Data":"73a018603afcba29b2c3c70656a2557d45d382cf1e1fc694361b3dda4ac7e588"}
Oct 31 20:02:40.933466 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:40.933382    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c9b9a1-666b4798c-b9svq"
Oct 31 20:02:40.954606 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:40.954563    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-c9b9a1-f45479ffd-mqgll" podStartSLOduration=1.954551755 podStartE2EDuration="1.954551755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:02:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:02:40.951055452 +0000 UTC m=+2591.524289581" watchObservedRunningTime="2026-10-31 20:02:40.954551755 +0000 UTC m=+2591.527785875"
Oct 31 20:02:40.978352 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:40.978235    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c9b9a1-666b4798c-b9svq" podStartSLOduration=2.97821701 podStartE2EDuration="2.97821701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:02:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:02:40.973694501 +0000 UTC m=+2591.546928641" watchObservedRunningTime="2026-10-31 20:02:40.97821701 +0000 UTC m=+2591.551451141"
Oct 31 20:02:41.006400 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:41.006321    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:41.936099 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:41.936061    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:42.970289 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:42.970254    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:44.035827 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:44.035783    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:44.762658 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:44.762624    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:45.073551 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:45.073463    4645 patch_prober.go:29] interesting pod/e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 20:02:45.073551 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:45.073529    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8" podUID="af802480-762e-446b-95d5-7754f4bad230" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 20:02:45.565106 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:45.565063    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:46.453733 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:46.453685    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:46.672379 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:46.672317    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-fd1c91-5fb9c8fcf9-xvrkd" podUID="600fee45-c980-4705-96d9-fc084f4bea96" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:02:47.140685 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:47.140638    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-f5c1ab-54c4d89f99-b7fr8"]
Oct 31 20:02:47.154483 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:47.154451    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f5c1ab-68b5bdf4d7-jwcll"]
Oct 31 20:02:47.269670 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:47.269632    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:47.391247 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:47.391160    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:47.751626 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:47.751578    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-fd1c91-84dcbc9465-5r662" podUID="af2f7267-2d2f-46cf-b8e2-101e6a328b68" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:02:48.190699 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:48.190656    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:48.293380 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:48.293309    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:48.483661 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:48.481290    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:48.966614 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:48.966564    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:49.694705 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:49.694670    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:50.066847 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:50.066815    4645 generic.go:356] "Generic (PLEG): container finished" podID="af802480-762e-446b-95d5-7754f4bad230" containerID="56b8550b6909ce6e6c49e17a78b45605af8219a8eb57b4f2cb4aacb62527450e" exitCode=0
Oct 31 20:02:50.067259 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:50.066854    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8" event={"ID":"af802480-762e-446b-95d5-7754f4bad230","Type":"ContainerDied","Data":"56b8550b6909ce6e6c49e17a78b45605af8219a8eb57b4f2cb4aacb62527450e"}
Oct 31 20:02:50.067303 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:50.067279    4645 scope.go:122] "RemoveContainer" containerID="56b8550b6909ce6e6c49e17a78b45605af8219a8eb57b4f2cb4aacb62527450e"
Oct 31 20:02:50.069170 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:50.069137    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:02:50.069246 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:50.069225    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:02:50.069285 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:50.069261    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:02:50.480651 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:50.480615    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:51.089092 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:51.089048    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8" event={"ID":"af802480-762e-446b-95d5-7754f4bad230","Type":"ContainerStarted","Data":"6c231135b1e2990ef65b1e6c444157573558442406d304dd10c2f9ef664c14dc"}
Oct 31 20:02:51.323979 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:51.321103    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:51.453996 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:51.453964    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8"
Oct 31 20:02:51.454173 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:51.454120    4645 patch_prober.go:29] interesting pod/e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body=
Oct 31 20:02:51.454219 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:51.454165    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8" podUID="af802480-762e-446b-95d5-7754f4bad230" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused"
Oct 31 20:02:51.944028 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:51.943993    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c9b9a1-f45479ffd-mqgll"
Oct 31 20:02:51.944307 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:51.944285    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c9b9a1-666b4798c-b9svq"
Oct 31 20:02:52.141701 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:52.141665    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f5c1ab-54c4d89f99-b7fr8" podUID="a5005342-3674-4428-ac17-a42f0dc6d8b2" containerName="payload-processing" containerID="cri-o://abd5bf92064b245b03b7801a3dd5a63c6a7f2b1eb9014bbe05b10b68f0f5ad60" gracePeriod=25
Oct 31 20:02:52.155306 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:52.155274    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f5c1ab-68b5bdf4d7-jwcll" podUID="4d05d0a8-95d5-4f44-b7d9-bae7800b7176" containerName="payload-pre-processing" containerID="cri-o://1edb64f1761ac82b92da42c3df5f7d5674bbac3840d46c187920d9ab1a8d6648" gracePeriod=25
Oct 31 20:02:52.182525 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:52.182492    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:52.453963 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:52.453879    4645 patch_prober.go:29] interesting pod/e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body=
Oct 31 20:02:52.454235 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:52.454203    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8" podUID="af802480-762e-446b-95d5-7754f4bad230" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused"
Oct 31 20:02:52.551570 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:52.551542    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-fd1c91-5fb9c8fcf9-xvrkd"
Oct 31 20:02:52.555170 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:52.555147    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-fd1c91-84dcbc9465-5r662"
Oct 31 20:02:52.625813 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:52.625776    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/600fee45-c980-4705-96d9-fc084f4bea96-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/600fee45-c980-4705-96d9-fc084f4bea96-plugins-config-volume\") pod \"600fee45-c980-4705-96d9-fc084f4bea96\" (UID: \"600fee45-c980-4705-96d9-fc084f4bea96\") "
Oct 31 20:02:52.626013 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:52.625991    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/600fee45-c980-4705-96d9-fc084f4bea96-kube-api-access-pckj7\" (UniqueName: \"kubernetes.io/projected/600fee45-c980-4705-96d9-fc084f4bea96-kube-api-access-pckj7\") pod \"600fee45-c980-4705-96d9-fc084f4bea96\" (UID: \"600fee45-c980-4705-96d9-fc084f4bea96\") "
Oct 31 20:02:52.626244 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:52.626217    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/600fee45-c980-4705-96d9-fc084f4bea96-plugins-config-volume" pod "600fee45-c980-4705-96d9-fc084f4bea96" (UID: "600fee45-c980-4705-96d9-fc084f4bea96"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:02:52.626737 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:52.626707    4645 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/600fee45-c980-4705-96d9-fc084f4bea96-plugins-config-volume\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:52.628077 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:52.628049    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/600fee45-c980-4705-96d9-fc084f4bea96-kube-api-access-pckj7" pod "600fee45-c980-4705-96d9-fc084f4bea96" (UID: "600fee45-c980-4705-96d9-fc084f4bea96"). InnerVolumeSpecName "kube-api-access-pckj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:02:52.728137 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:52.728052    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af2f7267-2d2f-46cf-b8e2-101e6a328b68-kube-api-access-x7bm6\" (UniqueName: \"kubernetes.io/projected/af2f7267-2d2f-46cf-b8e2-101e6a328b68-kube-api-access-x7bm6\") pod \"af2f7267-2d2f-46cf-b8e2-101e6a328b68\" (UID: \"af2f7267-2d2f-46cf-b8e2-101e6a328b68\") "
Oct 31 20:02:52.728302 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:52.728211    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af2f7267-2d2f-46cf-b8e2-101e6a328b68-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af2f7267-2d2f-46cf-b8e2-101e6a328b68-plugins-config-volume\") pod \"af2f7267-2d2f-46cf-b8e2-101e6a328b68\" (UID: \"af2f7267-2d2f-46cf-b8e2-101e6a328b68\") "
Oct 31 20:02:52.728631 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:52.728607    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af2f7267-2d2f-46cf-b8e2-101e6a328b68-plugins-config-volume" pod "af2f7267-2d2f-46cf-b8e2-101e6a328b68" (UID: "af2f7267-2d2f-46cf-b8e2-101e6a328b68"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:02:52.729247 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:52.729225    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pckj7\" (UniqueName: \"kubernetes.io/projected/600fee45-c980-4705-96d9-fc084f4bea96-kube-api-access-pckj7\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:52.729305 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:52.729253    4645 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af2f7267-2d2f-46cf-b8e2-101e6a328b68-plugins-config-volume\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:52.730268 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:52.730243    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af2f7267-2d2f-46cf-b8e2-101e6a328b68-kube-api-access-x7bm6" pod "af2f7267-2d2f-46cf-b8e2-101e6a328b68" (UID: "af2f7267-2d2f-46cf-b8e2-101e6a328b68"). InnerVolumeSpecName "kube-api-access-x7bm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:02:52.830958 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:52.830916    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7bm6\" (UniqueName: \"kubernetes.io/projected/af2f7267-2d2f-46cf-b8e2-101e6a328b68-kube-api-access-x7bm6\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:02:53.082263 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:53.082154    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:53.152557 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:53.152466    4645 generic.go:356] "Generic (PLEG): container finished" podID="600fee45-c980-4705-96d9-fc084f4bea96" containerID="60fa6d013b43f2246d859b83c10f7cd8a182fe5e0d67b6ce03f08ddb863f7e7a" exitCode=0
Oct 31 20:02:53.152557 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:53.152553    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-fd1c91-5fb9c8fcf9-xvrkd" event={"ID":"600fee45-c980-4705-96d9-fc084f4bea96","Type":"ContainerDied","Data":"60fa6d013b43f2246d859b83c10f7cd8a182fe5e0d67b6ce03f08ddb863f7e7a"}
Oct 31 20:02:53.153100 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:53.152618    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-fd1c91-5fb9c8fcf9-xvrkd" event={"ID":"600fee45-c980-4705-96d9-fc084f4bea96","Type":"ContainerDied","Data":"f32e481bf5131eb27288879e456bb047437e92b720e73d500241d43ccbb9dd40"}
Oct 31 20:02:53.153100 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:53.152645    4645 scope.go:122] "RemoveContainer" containerID="60fa6d013b43f2246d859b83c10f7cd8a182fe5e0d67b6ce03f08ddb863f7e7a"
Oct 31 20:02:53.153100 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:53.152516    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-fd1c91-5fb9c8fcf9-xvrkd"
Oct 31 20:02:53.180883 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:53.167019    4645 generic.go:356] "Generic (PLEG): container finished" podID="af2f7267-2d2f-46cf-b8e2-101e6a328b68" containerID="0074c9f1f0aa3e09ff2e12f2d179a38bd914178c99b2596828a33eef2412f8c2" exitCode=0
Oct 31 20:02:53.180883 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:53.167164    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-fd1c91-84dcbc9465-5r662" event={"ID":"af2f7267-2d2f-46cf-b8e2-101e6a328b68","Type":"ContainerDied","Data":"0074c9f1f0aa3e09ff2e12f2d179a38bd914178c99b2596828a33eef2412f8c2"}
Oct 31 20:02:53.180883 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:53.167210    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-fd1c91-84dcbc9465-5r662" event={"ID":"af2f7267-2d2f-46cf-b8e2-101e6a328b68","Type":"ContainerDied","Data":"39863d5ca8bf99d815b2bbfe79224ba12e1df4a1ed80b7ffe3a6a588e22b4d55"}
Oct 31 20:02:53.180883 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:53.167282    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-fd1c91-84dcbc9465-5r662"
Oct 31 20:02:53.214773 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:53.214743    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-fd1c91-84dcbc9465-5r662"]
Oct 31 20:02:53.215045 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:53.214917    4645 scope.go:122] "RemoveContainer" containerID="60fa6d013b43f2246d859b83c10f7cd8a182fe5e0d67b6ce03f08ddb863f7e7a"
Oct 31 20:02:53.216777 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:02:53.216747    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60fa6d013b43f2246d859b83c10f7cd8a182fe5e0d67b6ce03f08ddb863f7e7a\": container with ID starting with 60fa6d013b43f2246d859b83c10f7cd8a182fe5e0d67b6ce03f08ddb863f7e7a not found: ID does not exist" containerID="60fa6d013b43f2246d859b83c10f7cd8a182fe5e0d67b6ce03f08ddb863f7e7a"
Oct 31 20:02:53.216904 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:53.216787    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60fa6d013b43f2246d859b83c10f7cd8a182fe5e0d67b6ce03f08ddb863f7e7a"} err="failed to get container status \"60fa6d013b43f2246d859b83c10f7cd8a182fe5e0d67b6ce03f08ddb863f7e7a\": rpc error: code = NotFound desc = could not find container \"60fa6d013b43f2246d859b83c10f7cd8a182fe5e0d67b6ce03f08ddb863f7e7a\": container with ID starting with 60fa6d013b43f2246d859b83c10f7cd8a182fe5e0d67b6ce03f08ddb863f7e7a not found: ID does not exist"
Oct 31 20:02:53.216904 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:53.216810    4645 scope.go:122] "RemoveContainer" containerID="0074c9f1f0aa3e09ff2e12f2d179a38bd914178c99b2596828a33eef2412f8c2"
Oct 31 20:02:53.220392 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:53.220369    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-fd1c91-84dcbc9465-5r662"]
Oct 31 20:02:53.231165 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:53.231147    4645 scope.go:122] "RemoveContainer" containerID="0074c9f1f0aa3e09ff2e12f2d179a38bd914178c99b2596828a33eef2412f8c2"
Oct 31 20:02:53.231478 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:02:53.231459    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0074c9f1f0aa3e09ff2e12f2d179a38bd914178c99b2596828a33eef2412f8c2\": container with ID starting with 0074c9f1f0aa3e09ff2e12f2d179a38bd914178c99b2596828a33eef2412f8c2 not found: ID does not exist" containerID="0074c9f1f0aa3e09ff2e12f2d179a38bd914178c99b2596828a33eef2412f8c2"
Oct 31 20:02:53.231562 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:53.231489    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0074c9f1f0aa3e09ff2e12f2d179a38bd914178c99b2596828a33eef2412f8c2"} err="failed to get container status \"0074c9f1f0aa3e09ff2e12f2d179a38bd914178c99b2596828a33eef2412f8c2\": rpc error: code = NotFound desc = could not find container \"0074c9f1f0aa3e09ff2e12f2d179a38bd914178c99b2596828a33eef2412f8c2\": container with ID starting with 0074c9f1f0aa3e09ff2e12f2d179a38bd914178c99b2596828a33eef2412f8c2 not found: ID does not exist"
Oct 31 20:02:53.233997 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:53.233955    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-fd1c91-5fb9c8fcf9-xvrkd"]
Oct 31 20:02:53.238076 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:53.238051    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-fd1c91-5fb9c8fcf9-xvrkd"]
Oct 31 20:02:53.453868 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:53.453835    4645 patch_prober.go:29] interesting pod/e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body=
Oct 31 20:02:53.454052 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:53.453887    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8" podUID="af802480-762e-446b-95d5-7754f4bad230" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused"
Oct 31 20:02:53.621818 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:53.621783    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="600fee45-c980-4705-96d9-fc084f4bea96" path="/var/lib/kubelet/pods/600fee45-c980-4705-96d9-fc084f4bea96/volumes"
Oct 31 20:02:53.622297 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:53.622282    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af2f7267-2d2f-46cf-b8e2-101e6a328b68" path="/var/lib/kubelet/pods/af2f7267-2d2f-46cf-b8e2-101e6a328b68/volumes"
Oct 31 20:02:53.954037 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:53.953999    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:54.154881 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:54.154847    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-c173d8/ipp-route-c173d8-kserve-776fb486f7-zjnkc"]
Oct 31 20:02:54.185355 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:54.166800    4645 cpu_manager.go:414] "containerMap: removing container" podUID="600fee45-c980-4705-96d9-fc084f4bea96" containerName="payload-pre-processing"
Oct 31 20:02:54.185605 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:54.185571    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="600fee45-c980-4705-96d9-fc084f4bea96" containerName="payload-pre-processing"
Oct 31 20:02:54.185763 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:54.185644    4645 cpu_manager.go:414] "containerMap: removing container" podUID="9c4eead5-086d-4c44-b2a8-c5d2597bc6e6" containerName="revoke-keys"
Oct 31 20:02:54.185763 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:54.185656    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c4eead5-086d-4c44-b2a8-c5d2597bc6e6" containerName="revoke-keys"
Oct 31 20:02:54.185763 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:54.185712    4645 cpu_manager.go:414] "containerMap: removing container" podUID="af2f7267-2d2f-46cf-b8e2-101e6a328b68" containerName="payload-processing"
Oct 31 20:02:54.185763 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:54.185720    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af2f7267-2d2f-46cf-b8e2-101e6a328b68" containerName="payload-processing"
Oct 31 20:02:54.186002 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:54.185958    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c4eead5-086d-4c44-b2a8-c5d2597bc6e6" containerName="revoke-keys"
Oct 31 20:02:54.186002 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:54.185981    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="af2f7267-2d2f-46cf-b8e2-101e6a328b68" containerName="payload-processing"
Oct 31 20:02:54.186002 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:54.186000    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="600fee45-c980-4705-96d9-fc084f4bea96" containerName="payload-pre-processing"
Oct 31 20:02:54.196378 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:54.196315    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-c173d8/ipp-route-c173d8-kserve-776fb486f7-zjnkc"]
Oct 31 20:02:54.196575 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:54.196490    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-c173d8/ipp-route-c173d8-kserve-776fb486f7-zjnkc"
Oct 31 20:02:54.212583 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:54.212545    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-c173d8\"/\"ipp-route-c173d8-kserve-self-signed-certs\""
Oct 31 20:02:54.219570 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:54.219544    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-c173d8\"/\"openshift-service-ca.crt\""
Oct 31 20:02:54.219705 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:54.219585    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-c173d8\"/\"kube-root-ca.crt\""
Oct 31 20:02:54.226259 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:54.226240    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-c173d8\"/\"default-dockercfg-6sv74\""
Oct 31 20:02:54.229895 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:54.229871    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:54.254926 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:54.254899    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b4334dd-204f-4794-830d-a375db82fa39-model-cache\") pod \"ipp-route-c173d8-kserve-776fb486f7-zjnkc\" (UID: \"2b4334dd-204f-4794-830d-a375db82fa39\") " pod="ai-tenant-e2e-ipp-a-c173d8/ipp-route-c173d8-kserve-776fb486f7-zjnkc"
Oct 31 20:02:54.255027 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:54.254975    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b4334dd-204f-4794-830d-a375db82fa39-home\") pod \"ipp-route-c173d8-kserve-776fb486f7-zjnkc\" (UID: \"2b4334dd-204f-4794-830d-a375db82fa39\") " pod="ai-tenant-e2e-ipp-a-c173d8/ipp-route-c173d8-kserve-776fb486f7-zjnkc"
Oct 31 20:02:54.255071 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:54.255026    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddrxj\" (UniqueName: \"kubernetes.io/projected/2b4334dd-204f-4794-830d-a375db82fa39-kube-api-access-ddrxj\") pod \"ipp-route-c173d8-kserve-776fb486f7-zjnkc\" (UID: \"2b4334dd-204f-4794-830d-a375db82fa39\") " pod="ai-tenant-e2e-ipp-a-c173d8/ipp-route-c173d8-kserve-776fb486f7-zjnkc"
Oct 31 20:02:54.255212 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:54.255198    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b4334dd-204f-4794-830d-a375db82fa39-tls-certs\") pod \"ipp-route-c173d8-kserve-776fb486f7-zjnkc\" (UID: \"2b4334dd-204f-4794-830d-a375db82fa39\") " pod="ai-tenant-e2e-ipp-a-c173d8/ipp-route-c173d8-kserve-776fb486f7-zjnkc"
Oct 31 20:02:54.255325 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:54.255303    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b4334dd-204f-4794-830d-a375db82fa39-dshm\") pod \"ipp-route-c173d8-kserve-776fb486f7-zjnkc\" (UID: \"2b4334dd-204f-4794-830d-a375db82fa39\") " pod="ai-tenant-e2e-ipp-a-c173d8/ipp-route-c173d8-kserve-776fb486f7-zjnkc"
Oct 31 20:02:54.255432 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:54.255407    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b4334dd-204f-4794-830d-a375db82fa39-tmp-dir\") pod \"ipp-route-c173d8-kserve-776fb486f7-zjnkc\" (UID: \"2b4334dd-204f-4794-830d-a375db82fa39\") " pod="ai-tenant-e2e-ipp-a-c173d8/ipp-route-c173d8-kserve-776fb486f7-zjnkc"
Oct 31 20:02:54.367350 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:54.357529    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b4334dd-204f-4794-830d-a375db82fa39-dshm\") pod \"ipp-route-c173d8-kserve-776fb486f7-zjnkc\" (UID: \"2b4334dd-204f-4794-830d-a375db82fa39\") " pod="ai-tenant-e2e-ipp-a-c173d8/ipp-route-c173d8-kserve-776fb486f7-zjnkc"
Oct 31 20:02:54.367350 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:54.357616    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b4334dd-204f-4794-830d-a375db82fa39-tmp-dir\") pod \"ipp-route-c173d8-kserve-776fb486f7-zjnkc\" (UID: \"2b4334dd-204f-4794-830d-a375db82fa39\") " pod="ai-tenant-e2e-ipp-a-c173d8/ipp-route-c173d8-kserve-776fb486f7-zjnkc"
Oct 31 20:02:54.367350 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:54.357721    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b4334dd-204f-4794-830d-a375db82fa39-model-cache\") pod \"ipp-route-c173d8-kserve-776fb486f7-zjnkc\" (UID: \"2b4334dd-204f-4794-830d-a375db82fa39\") " pod="ai-tenant-e2e-ipp-a-c173d8/ipp-route-c173d8-kserve-776fb486f7-zjnkc"
Oct 31 20:02:54.367350 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:54.357828    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b4334dd-204f-4794-830d-a375db82fa39-home\") pod \"ipp-route-c173d8-kserve-776fb486f7-zjnkc\" (UID: \"2b4334dd-204f-4794-830d-a375db82fa39\") " pod="ai-tenant-e2e-ipp-a-c173d8/ipp-route-c173d8-kserve-776fb486f7-zjnkc"
Oct 31 20:02:54.367350 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:54.357896    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddrxj\" (UniqueName: \"kubernetes.io/projected/2b4334dd-204f-4794-830d-a375db82fa39-kube-api-access-ddrxj\") pod \"ipp-route-c173d8-kserve-776fb486f7-zjnkc\" (UID: \"2b4334dd-204f-4794-830d-a375db82fa39\") " pod="ai-tenant-e2e-ipp-a-c173d8/ipp-route-c173d8-kserve-776fb486f7-zjnkc"
Oct 31 20:02:54.367350 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:54.358062    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b4334dd-204f-4794-830d-a375db82fa39-tls-certs\") pod \"ipp-route-c173d8-kserve-776fb486f7-zjnkc\" (UID: \"2b4334dd-204f-4794-830d-a375db82fa39\") " pod="ai-tenant-e2e-ipp-a-c173d8/ipp-route-c173d8-kserve-776fb486f7-zjnkc"
Oct 31 20:02:54.367350 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:54.364839    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b4334dd-204f-4794-830d-a375db82fa39-model-cache\") pod \"ipp-route-c173d8-kserve-776fb486f7-zjnkc\" (UID: \"2b4334dd-204f-4794-830d-a375db82fa39\") " pod="ai-tenant-e2e-ipp-a-c173d8/ipp-route-c173d8-kserve-776fb486f7-zjnkc"
Oct 31 20:02:54.373636 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:54.373050    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b4334dd-204f-4794-830d-a375db82fa39-home\") pod \"ipp-route-c173d8-kserve-776fb486f7-zjnkc\" (UID: \"2b4334dd-204f-4794-830d-a375db82fa39\") " pod="ai-tenant-e2e-ipp-a-c173d8/ipp-route-c173d8-kserve-776fb486f7-zjnkc"
Oct 31 20:02:54.373636 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:54.373308    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b4334dd-204f-4794-830d-a375db82fa39-tmp-dir\") pod \"ipp-route-c173d8-kserve-776fb486f7-zjnkc\" (UID: \"2b4334dd-204f-4794-830d-a375db82fa39\") " pod="ai-tenant-e2e-ipp-a-c173d8/ipp-route-c173d8-kserve-776fb486f7-zjnkc"
Oct 31 20:02:54.414953 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:54.382052    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b4334dd-204f-4794-830d-a375db82fa39-tls-certs\") pod \"ipp-route-c173d8-kserve-776fb486f7-zjnkc\" (UID: \"2b4334dd-204f-4794-830d-a375db82fa39\") " pod="ai-tenant-e2e-ipp-a-c173d8/ipp-route-c173d8-kserve-776fb486f7-zjnkc"
Oct 31 20:02:54.414953 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:54.385595    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b4334dd-204f-4794-830d-a375db82fa39-dshm\") pod \"ipp-route-c173d8-kserve-776fb486f7-zjnkc\" (UID: \"2b4334dd-204f-4794-830d-a375db82fa39\") " pod="ai-tenant-e2e-ipp-a-c173d8/ipp-route-c173d8-kserve-776fb486f7-zjnkc"
Oct 31 20:02:54.436382 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:54.423682    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddrxj\" (UniqueName: \"kubernetes.io/projected/2b4334dd-204f-4794-830d-a375db82fa39-kube-api-access-ddrxj\") pod \"ipp-route-c173d8-kserve-776fb486f7-zjnkc\" (UID: \"2b4334dd-204f-4794-830d-a375db82fa39\") " pod="ai-tenant-e2e-ipp-a-c173d8/ipp-route-c173d8-kserve-776fb486f7-zjnkc"
Oct 31 20:02:54.478407 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:54.475746    4645 patch_prober.go:29] interesting pod/e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body=
Oct 31 20:02:54.478407 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:54.475807    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8" podUID="af802480-762e-446b-95d5-7754f4bad230" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused"
Oct 31 20:02:54.528354 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:54.522783    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-c173d8/ipp-route-c173d8-kserve-776fb486f7-zjnkc"
Oct 31 20:02:54.886056 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:54.886023    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-c173d8/ipp-route-c173d8-kserve-776fb486f7-zjnkc"]
Oct 31 20:02:54.887507 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:02:54.887480    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2b4334dd_204f_4794_830d_a375db82fa39.slice/crio-ac74accec21373d9de3cb978c2f2bbc9b42b21fac56a2d271fd0ed4b9c2fbe75 WatchSource:0}: Error finding container ac74accec21373d9de3cb978c2f2bbc9b42b21fac56a2d271fd0ed4b9c2fbe75: Status 404 returned error can't find the container with id ac74accec21373d9de3cb978c2f2bbc9b42b21fac56a2d271fd0ed4b9c2fbe75
Oct 31 20:02:55.109681 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:55.109637    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:55.210250 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:55.210208    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-c173d8/ipp-route-c173d8-kserve-776fb486f7-zjnkc" event={"ID":"2b4334dd-204f-4794-830d-a375db82fa39","Type":"ContainerStarted","Data":"f9dfe0177f43228dc26fa6c35b8657b6d3ea35ffbce66589884ee686b8d5f4b9"}
Oct 31 20:02:55.210675 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:55.210261    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-c173d8/ipp-route-c173d8-kserve-776fb486f7-zjnkc" event={"ID":"2b4334dd-204f-4794-830d-a375db82fa39","Type":"ContainerStarted","Data":"ac74accec21373d9de3cb978c2f2bbc9b42b21fac56a2d271fd0ed4b9c2fbe75"}
Oct 31 20:02:55.231196 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:55.231136    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-c173d8/ipp-route-c173d8-kserve-776fb486f7-zjnkc" podStartSLOduration=1.231120359 podStartE2EDuration="1.231120359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:02:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:02:55.23059579 +0000 UTC m=+2605.803829919" watchObservedRunningTime="2026-10-31 20:02:55.231120359 +0000 UTC m=+2605.804354489"
Oct 31 20:02:55.454394 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:55.454309    4645 patch_prober.go:29] interesting pod/e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body=
Oct 31 20:02:55.454585 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:55.454399    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8" podUID="af802480-762e-446b-95d5-7754f4bad230" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused"
Oct 31 20:02:55.523907 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:55.523827    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-c173d8/ipp-route-c173d8-kserve-776fb486f7-zjnkc"
Oct 31 20:02:55.693625 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:55.693549    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:55.975352 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:55.969018    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:56.047349 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:56.043115    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f5c1ab-4n2zw"]
Oct 31 20:02:56.084350 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:56.076747    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-f5c1ab-4n2zw"]
Oct 31 20:02:56.170420 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:56.169827    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:56.453907 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:56.453874    4645 patch_prober.go:29] interesting pod/e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body=
Oct 31 20:02:56.454300 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:56.453934    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8" podUID="af802480-762e-446b-95d5-7754f4bad230" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused"
Oct 31 20:02:56.595785 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:56.595741    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:56.958964 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:56.958929    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:57.458785 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:57.458755    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8"
Oct 31 20:02:57.459409 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:57.459170    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8"
Oct 31 20:02:57.459853 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:57.459834    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8"
Oct 31 20:02:57.631183 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:57.631148    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c4eead5-086d-4c44-b2a8-c5d2597bc6e6" path="/var/lib/kubelet/pods/9c4eead5-086d-4c44-b2a8-c5d2597bc6e6/volumes"
Oct 31 20:02:58.046519 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:58.046475    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:02:58.871010 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:58.870971    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f5c1ab-54c4d89f99-b7fr8" podUID="a5005342-3674-4428-ac17-a42f0dc6d8b2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:02:58.871010 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:58.870997    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f5c1ab-68b5bdf4d7-jwcll" podUID="4d05d0a8-95d5-4f44-b7d9-bae7800b7176" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:02:59.167699 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:02:59.167656    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:00.171282 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:00.171217    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:00.491488 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:00.491401    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:00.766060 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:00.765964    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:00.898070 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:00.898036    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:01.193053 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:01.193002    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:01.971853 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:01.971800    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:02.920041 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:02.920008    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:03.582479 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:03.582443    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:03.707201 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:03.707166    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d7647c-66bbb74874-4dsdn"]
Oct 31 20:03:03.711029 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:03.710990    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:03.731294 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:03.731259    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d7647c-66bbb74874-4dsdn"
Oct 31 20:03:03.740400 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:03.740356    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-d7647c\""
Oct 31 20:03:03.742909 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:03.742855    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d7647c-66bbb74874-4dsdn"]
Oct 31 20:03:03.778882 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:03.778842    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d7647c-85d9ccfd7f-qvj7p"]
Oct 31 20:03:03.783563 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:03.783541    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7647c-85d9ccfd7f-qvj7p"
Oct 31 20:03:03.790277 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:03.790257    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-d7647c-dockercfg-fbdkg\""
Oct 31 20:03:03.797699 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:03.797672    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d7647c-85d9ccfd7f-qvj7p"]
Oct 31 20:03:03.867271 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:03.867172    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/daf67342-fb86-4a20-96b5-c753798fa0cd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d7647c-66bbb74874-4dsdn\" (UID: \"daf67342-fb86-4a20-96b5-c753798fa0cd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d7647c-66bbb74874-4dsdn"
Oct 31 20:03:04.003846 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:04.003808    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6zp6\" (UniqueName: \"kubernetes.io/projected/daf67342-fb86-4a20-96b5-c753798fa0cd-kube-api-access-x6zp6\") pod \"payload-processing-e2e-ipp-b-d7647c-66bbb74874-4dsdn\" (UID: \"daf67342-fb86-4a20-96b5-c753798fa0cd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d7647c-66bbb74874-4dsdn"
Oct 31 20:03:04.004490 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:04.004458    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04e10b50-5ded-4487-831a-09edcb6e9f9e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d7647c-85d9ccfd7f-qvj7p\" (UID: \"04e10b50-5ded-4487-831a-09edcb6e9f9e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7647c-85d9ccfd7f-qvj7p"
Oct 31 20:03:04.019643 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:04.019606    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmwft\" (UniqueName: \"kubernetes.io/projected/04e10b50-5ded-4487-831a-09edcb6e9f9e-kube-api-access-gmwft\") pod \"payload-pre-processing-e2e-ipp-b-d7647c-85d9ccfd7f-qvj7p\" (UID: \"04e10b50-5ded-4487-831a-09edcb6e9f9e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7647c-85d9ccfd7f-qvj7p"
Oct 31 20:03:04.166294 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:04.166259    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmwft\" (UniqueName: \"kubernetes.io/projected/04e10b50-5ded-4487-831a-09edcb6e9f9e-kube-api-access-gmwft\") pod \"payload-pre-processing-e2e-ipp-b-d7647c-85d9ccfd7f-qvj7p\" (UID: \"04e10b50-5ded-4487-831a-09edcb6e9f9e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7647c-85d9ccfd7f-qvj7p"
Oct 31 20:03:04.166497 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:04.166403    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/daf67342-fb86-4a20-96b5-c753798fa0cd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d7647c-66bbb74874-4dsdn\" (UID: \"daf67342-fb86-4a20-96b5-c753798fa0cd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d7647c-66bbb74874-4dsdn"
Oct 31 20:03:04.166497 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:04.166454    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6zp6\" (UniqueName: \"kubernetes.io/projected/daf67342-fb86-4a20-96b5-c753798fa0cd-kube-api-access-x6zp6\") pod \"payload-processing-e2e-ipp-b-d7647c-66bbb74874-4dsdn\" (UID: \"daf67342-fb86-4a20-96b5-c753798fa0cd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d7647c-66bbb74874-4dsdn"
Oct 31 20:03:04.166497 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:04.166474    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04e10b50-5ded-4487-831a-09edcb6e9f9e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d7647c-85d9ccfd7f-qvj7p\" (UID: \"04e10b50-5ded-4487-831a-09edcb6e9f9e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7647c-85d9ccfd7f-qvj7p"
Oct 31 20:03:04.167211 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:04.167185    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/daf67342-fb86-4a20-96b5-c753798fa0cd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d7647c-66bbb74874-4dsdn\" (UID: \"daf67342-fb86-4a20-96b5-c753798fa0cd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d7647c-66bbb74874-4dsdn"
Oct 31 20:03:04.167323 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:04.167256    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04e10b50-5ded-4487-831a-09edcb6e9f9e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d7647c-85d9ccfd7f-qvj7p\" (UID: \"04e10b50-5ded-4487-831a-09edcb6e9f9e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7647c-85d9ccfd7f-qvj7p"
Oct 31 20:03:04.178299 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:04.178267    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmwft\" (UniqueName: \"kubernetes.io/projected/04e10b50-5ded-4487-831a-09edcb6e9f9e-kube-api-access-gmwft\") pod \"payload-pre-processing-e2e-ipp-b-d7647c-85d9ccfd7f-qvj7p\" (UID: \"04e10b50-5ded-4487-831a-09edcb6e9f9e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7647c-85d9ccfd7f-qvj7p"
Oct 31 20:03:04.178416 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:04.178323    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6zp6\" (UniqueName: \"kubernetes.io/projected/daf67342-fb86-4a20-96b5-c753798fa0cd-kube-api-access-x6zp6\") pod \"payload-processing-e2e-ipp-b-d7647c-66bbb74874-4dsdn\" (UID: \"daf67342-fb86-4a20-96b5-c753798fa0cd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d7647c-66bbb74874-4dsdn"
Oct 31 20:03:04.277637 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:04.277596    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:04.361351 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:04.357979    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d7647c-66bbb74874-4dsdn"
Oct 31 20:03:04.415351 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:04.414684    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7647c-85d9ccfd7f-qvj7p"
Oct 31 20:03:04.525352 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:04.523147    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-c173d8/ipp-route-c173d8-kserve-776fb486f7-zjnkc"
Oct 31 20:03:04.587995 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:04.587884    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:04.598498 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:04.598463    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-c173d8/ipp-route-c173d8-kserve-776fb486f7-zjnkc"
Oct 31 20:03:04.709165 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:04.702319    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:04.892760 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:04.892722    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d7647c-85d9ccfd7f-qvj7p"]
Oct 31 20:03:04.893678 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:03:04.893645    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04e10b50_5ded_4487_831a_09edcb6e9f9e.slice/crio-d9a03a9817d3f0c56504fab43129e4f349e577013065682062b3a8fffadec275 WatchSource:0}: Error finding container d9a03a9817d3f0c56504fab43129e4f349e577013065682062b3a8fffadec275: Status 404 returned error can't find the container with id d9a03a9817d3f0c56504fab43129e4f349e577013065682062b3a8fffadec275
Oct 31 20:03:04.941134 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:03:04.941088    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaf67342_fb86_4a20_96b5_c753798fa0cd.slice/crio-f392b2bcb1f3a193fa2261224fd62ab0d10fab180137dced165fce7395c31615 WatchSource:0}: Error finding container f392b2bcb1f3a193fa2261224fd62ab0d10fab180137dced165fce7395c31615: Status 404 returned error can't find the container with id f392b2bcb1f3a193fa2261224fd62ab0d10fab180137dced165fce7395c31615
Oct 31 20:03:04.941322 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:04.941215    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2"]
Oct 31 20:03:04.941527 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:04.941491    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2" podUID="813ae008-e079-4406-83f7-6eae7acc3ce1" containerName="istio-proxy" containerID="cri-o://5e7a4d56c78c54025c3ad6d7a38648b351dc701cc8fdb9b370bcbd8bd33fda4d" gracePeriod=30
Oct 31 20:03:04.948009 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:04.947980    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d7647c-66bbb74874-4dsdn"]
Oct 31 20:03:05.003604 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:05.003571    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:05.219184 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:05.219132    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:05.383435 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:05.383303    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d7647c-66bbb74874-4dsdn" event={"ID":"daf67342-fb86-4a20-96b5-c753798fa0cd","Type":"ContainerStarted","Data":"54440c3d95effcd198df4c326fdd97df16cd243c6dacd267effc83feb2d8e344"}
Oct 31 20:03:05.383738 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:05.383716    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d7647c-66bbb74874-4dsdn" event={"ID":"daf67342-fb86-4a20-96b5-c753798fa0cd","Type":"ContainerStarted","Data":"f392b2bcb1f3a193fa2261224fd62ab0d10fab180137dced165fce7395c31615"}
Oct 31 20:03:05.385794 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:05.385774    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d7647c-66bbb74874-4dsdn"
Oct 31 20:03:05.397692 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:05.397594    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7647c-85d9ccfd7f-qvj7p" event={"ID":"04e10b50-5ded-4487-831a-09edcb6e9f9e","Type":"ContainerStarted","Data":"84065f0c7da8fbbcdaa813339c3435854555b01727a616f01208b00f9e13c2cd"}
Oct 31 20:03:05.397692 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:05.397655    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7647c-85d9ccfd7f-qvj7p" event={"ID":"04e10b50-5ded-4487-831a-09edcb6e9f9e","Type":"ContainerStarted","Data":"d9a03a9817d3f0c56504fab43129e4f349e577013065682062b3a8fffadec275"}
Oct 31 20:03:05.397993 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:05.397955    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7647c-85d9ccfd7f-qvj7p"
Oct 31 20:03:05.418780 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:05.418752    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-c173d8/ipp-route-c173d8-kserve-776fb486f7-zjnkc"
Oct 31 20:03:05.428382 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:05.428293    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-d7647c-66bbb74874-4dsdn" podStartSLOduration=2.428275267 podStartE2EDuration="2.428275267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:03:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:03:05.422992146 +0000 UTC m=+2615.996226275" watchObservedRunningTime="2026-10-31 20:03:05.428275267 +0000 UTC m=+2616.001509399"
Oct 31 20:03:05.446603 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:05.446568    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:05.488663 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:05.484815    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7647c-85d9ccfd7f-qvj7p" podStartSLOduration=2.484795508 podStartE2EDuration="2.484795508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:03:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:03:05.480976869 +0000 UTC m=+2616.054211002" watchObservedRunningTime="2026-10-31 20:03:05.484795508 +0000 UTC m=+2616.058029638"
Oct 31 20:03:05.655236 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:05.655146    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-23843f-976pz"]
Oct 31 20:03:05.697997 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:05.697660    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-23843f-976pz"]
Oct 31 20:03:05.697997 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:05.697778    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-23843f-976pz"
Oct 31 20:03:05.707136 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:05.707105    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:03:05.707643 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:05.707442    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 20:03:05.847269 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:05.847231    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs9wg\" (UniqueName: \"kubernetes.io/projected/a3c4c540-6b69-4c81-bea8-6c4176cf621c-kube-api-access-hs9wg\") pod \"maas-api-revoke-keys-e2e-worker-w0-23843f-976pz\" (UID: \"a3c4c540-6b69-4c81-bea8-6c4176cf621c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-23843f-976pz"
Oct 31 20:03:05.847493 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:05.847430    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3c4c540-6b69-4c81-bea8-6c4176cf621c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-23843f-976pz\" (UID: \"a3c4c540-6b69-4c81-bea8-6c4176cf621c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-23843f-976pz"
Oct 31 20:03:05.847610 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:05.847592    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3c4c540-6b69-4c81-bea8-6c4176cf621c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-23843f-976pz\" (UID: \"a3c4c540-6b69-4c81-bea8-6c4176cf621c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-23843f-976pz"
Oct 31 20:03:05.948817 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:05.948781    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3c4c540-6b69-4c81-bea8-6c4176cf621c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-23843f-976pz\" (UID: \"a3c4c540-6b69-4c81-bea8-6c4176cf621c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-23843f-976pz"
Oct 31 20:03:05.949036 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:05.948961    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3c4c540-6b69-4c81-bea8-6c4176cf621c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-23843f-976pz\" (UID: \"a3c4c540-6b69-4c81-bea8-6c4176cf621c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-23843f-976pz"
Oct 31 20:03:05.949122 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:05.949104    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs9wg\" (UniqueName: \"kubernetes.io/projected/a3c4c540-6b69-4c81-bea8-6c4176cf621c-kube-api-access-hs9wg\") pod \"maas-api-revoke-keys-e2e-worker-w0-23843f-976pz\" (UID: \"a3c4c540-6b69-4c81-bea8-6c4176cf621c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-23843f-976pz"
Oct 31 20:03:05.949623 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:05.949602    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3c4c540-6b69-4c81-bea8-6c4176cf621c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-23843f-976pz\" (UID: \"a3c4c540-6b69-4c81-bea8-6c4176cf621c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-23843f-976pz"
Oct 31 20:03:05.951260 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:05.951241    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3c4c540-6b69-4c81-bea8-6c4176cf621c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-23843f-976pz\" (UID: \"a3c4c540-6b69-4c81-bea8-6c4176cf621c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-23843f-976pz"
Oct 31 20:03:05.960343 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:05.960302    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs9wg\" (UniqueName: \"kubernetes.io/projected/a3c4c540-6b69-4c81-bea8-6c4176cf621c-kube-api-access-hs9wg\") pod \"maas-api-revoke-keys-e2e-worker-w0-23843f-976pz\" (UID: \"a3c4c540-6b69-4c81-bea8-6c4176cf621c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-23843f-976pz"
Oct 31 20:03:06.025924 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:06.025889    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-23843f-976pz"
Oct 31 20:03:06.248365 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:03:06.248261    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 20:03:06.325967 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:06.325939    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-23843f-976pz"]
Oct 31 20:03:06.327088 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:03:06.327063    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3c4c540_6b69_4c81_bea8_6c4176cf621c.slice/crio-17f73fd4148158b6a9529f15931675ada213203de06739569d87024f3a1fe120 WatchSource:0}: Error finding container 17f73fd4148158b6a9529f15931675ada213203de06739569d87024f3a1fe120: Status 404 returned error can't find the container with id 17f73fd4148158b6a9529f15931675ada213203de06739569d87024f3a1fe120
Oct 31 20:03:06.408865 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:06.408817    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-23843f-976pz" event={"ID":"a3c4c540-6b69-4c81-bea8-6c4176cf621c","Type":"ContainerStarted","Data":"17f73fd4148158b6a9529f15931675ada213203de06739569d87024f3a1fe120"}
Oct 31 20:03:06.420694 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:06.420662    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:07.289190 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:07.289152    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:07.418794 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:07.418758    4645 generic.go:356] "Generic (PLEG): container finished" podID="a3c4c540-6b69-4c81-bea8-6c4176cf621c" containerID="e65baaa828d073b91a40188cbb32af573144a0318d023169cc6c6b5abc932c22" exitCode=0
Oct 31 20:03:07.418945 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:07.418836    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-23843f-976pz" event={"ID":"a3c4c540-6b69-4c81-bea8-6c4176cf621c","Type":"ContainerDied","Data":"e65baaa828d073b91a40188cbb32af573144a0318d023169cc6c6b5abc932c22"}
Oct 31 20:03:08.367030 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:08.366990    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:08.640363 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:08.640289    4645 patch_prober.go:29] interesting pod/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 20:03:08.640546 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:08.640386    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2" podUID="813ae008-e079-4406-83f7-6eae7acc3ce1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 20:03:08.643156 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:08.643138    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-23843f-976pz"
Oct 31 20:03:08.795906 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:08.795873    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3c4c540-6b69-4c81-bea8-6c4176cf621c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3c4c540-6b69-4c81-bea8-6c4176cf621c-maas-api-xks-ca\") pod \"a3c4c540-6b69-4c81-bea8-6c4176cf621c\" (UID: \"a3c4c540-6b69-4c81-bea8-6c4176cf621c\") "
Oct 31 20:03:08.796089 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:08.795970    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3c4c540-6b69-4c81-bea8-6c4176cf621c-kube-api-access-hs9wg\" (UniqueName: \"kubernetes.io/projected/a3c4c540-6b69-4c81-bea8-6c4176cf621c-kube-api-access-hs9wg\") pod \"a3c4c540-6b69-4c81-bea8-6c4176cf621c\" (UID: \"a3c4c540-6b69-4c81-bea8-6c4176cf621c\") "
Oct 31 20:03:08.796089 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:08.796078    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3c4c540-6b69-4c81-bea8-6c4176cf621c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3c4c540-6b69-4c81-bea8-6c4176cf621c-maas-api-openshift-ca\") pod \"a3c4c540-6b69-4c81-bea8-6c4176cf621c\" (UID: \"a3c4c540-6b69-4c81-bea8-6c4176cf621c\") "
Oct 31 20:03:08.796515 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:08.796486    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3c4c540-6b69-4c81-bea8-6c4176cf621c-maas-api-openshift-ca" pod "a3c4c540-6b69-4c81-bea8-6c4176cf621c" (UID: "a3c4c540-6b69-4c81-bea8-6c4176cf621c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:03:08.796807 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:08.796788    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3c4c540-6b69-4c81-bea8-6c4176cf621c-maas-api-openshift-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:08.798302 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:08.798276    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3c4c540-6b69-4c81-bea8-6c4176cf621c-maas-api-xks-ca" pod "a3c4c540-6b69-4c81-bea8-6c4176cf621c" (UID: "a3c4c540-6b69-4c81-bea8-6c4176cf621c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:03:08.798600 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:08.798574    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3c4c540-6b69-4c81-bea8-6c4176cf621c-kube-api-access-hs9wg" pod "a3c4c540-6b69-4c81-bea8-6c4176cf621c" (UID: "a3c4c540-6b69-4c81-bea8-6c4176cf621c"). InnerVolumeSpecName "kube-api-access-hs9wg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:03:08.871497 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:08.871421    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f5c1ab-54c4d89f99-b7fr8" podUID="a5005342-3674-4428-ac17-a42f0dc6d8b2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:03:08.871619 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:08.871425    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f5c1ab-68b5bdf4d7-jwcll" podUID="4d05d0a8-95d5-4f44-b7d9-bae7800b7176" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:03:08.898577 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:08.898551    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3c4c540-6b69-4c81-bea8-6c4176cf621c-maas-api-xks-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:08.898577 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:08.898578    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hs9wg\" (UniqueName: \"kubernetes.io/projected/a3c4c540-6b69-4c81-bea8-6c4176cf621c-kube-api-access-hs9wg\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:09.181148 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:09.181105    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:09.465551 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:09.465467    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-23843f-976pz"
Oct 31 20:03:09.465551 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:09.465491    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-23843f-976pz" event={"ID":"a3c4c540-6b69-4c81-bea8-6c4176cf621c","Type":"ContainerDied","Data":"17f73fd4148158b6a9529f15931675ada213203de06739569d87024f3a1fe120"}
Oct 31 20:03:09.465551 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:09.465536    4645 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17f73fd4148158b6a9529f15931675ada213203de06739569d87024f3a1fe120"
Oct 31 20:03:10.110315 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:10.110271    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:10.283359 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:10.283316    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2"
Oct 31 20:03:10.382445 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:10.382399    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/813ae008-e079-4406-83f7-6eae7acc3ce1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/813ae008-e079-4406-83f7-6eae7acc3ce1-istio-data\") pod \"813ae008-e079-4406-83f7-6eae7acc3ce1\" (UID: \"813ae008-e079-4406-83f7-6eae7acc3ce1\") "
Oct 31 20:03:10.382654 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:10.382519    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/813ae008-e079-4406-83f7-6eae7acc3ce1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/813ae008-e079-4406-83f7-6eae7acc3ce1-credential-socket\") pod \"813ae008-e079-4406-83f7-6eae7acc3ce1\" (UID: \"813ae008-e079-4406-83f7-6eae7acc3ce1\") "
Oct 31 20:03:10.382654 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:10.382574    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/813ae008-e079-4406-83f7-6eae7acc3ce1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/813ae008-e079-4406-83f7-6eae7acc3ce1-workload-certs\") pod \"813ae008-e079-4406-83f7-6eae7acc3ce1\" (UID: \"813ae008-e079-4406-83f7-6eae7acc3ce1\") "
Oct 31 20:03:10.382654 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:10.382615    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/813ae008-e079-4406-83f7-6eae7acc3ce1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/813ae008-e079-4406-83f7-6eae7acc3ce1-istio-envoy\") pod \"813ae008-e079-4406-83f7-6eae7acc3ce1\" (UID: \"813ae008-e079-4406-83f7-6eae7acc3ce1\") "
Oct 31 20:03:10.382843 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:10.382706    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/813ae008-e079-4406-83f7-6eae7acc3ce1-istio-token\" (UniqueName: \"kubernetes.io/projected/813ae008-e079-4406-83f7-6eae7acc3ce1-istio-token\") pod \"813ae008-e079-4406-83f7-6eae7acc3ce1\" (UID: \"813ae008-e079-4406-83f7-6eae7acc3ce1\") "
Oct 31 20:03:10.382843 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:10.382726    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/813ae008-e079-4406-83f7-6eae7acc3ce1-istio-data" pod "813ae008-e079-4406-83f7-6eae7acc3ce1" (UID: "813ae008-e079-4406-83f7-6eae7acc3ce1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:03:10.382843 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:10.382756    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/813ae008-e079-4406-83f7-6eae7acc3ce1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/813ae008-e079-4406-83f7-6eae7acc3ce1-istio-podinfo\") pod \"813ae008-e079-4406-83f7-6eae7acc3ce1\" (UID: \"813ae008-e079-4406-83f7-6eae7acc3ce1\") "
Oct 31 20:03:10.382989 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:10.382753    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/813ae008-e079-4406-83f7-6eae7acc3ce1-credential-socket" pod "813ae008-e079-4406-83f7-6eae7acc3ce1" (UID: "813ae008-e079-4406-83f7-6eae7acc3ce1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:03:10.382989 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:10.382877    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/813ae008-e079-4406-83f7-6eae7acc3ce1-kube-api-access-k9vgc\" (UniqueName: \"kubernetes.io/projected/813ae008-e079-4406-83f7-6eae7acc3ce1-kube-api-access-k9vgc\") pod \"813ae008-e079-4406-83f7-6eae7acc3ce1\" (UID: \"813ae008-e079-4406-83f7-6eae7acc3ce1\") "
Oct 31 20:03:10.382989 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:10.382912    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/813ae008-e079-4406-83f7-6eae7acc3ce1-workload-certs" pod "813ae008-e079-4406-83f7-6eae7acc3ce1" (UID: "813ae008-e079-4406-83f7-6eae7acc3ce1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:03:10.383132 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:10.382986    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/813ae008-e079-4406-83f7-6eae7acc3ce1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/813ae008-e079-4406-83f7-6eae7acc3ce1-workload-socket\") pod \"813ae008-e079-4406-83f7-6eae7acc3ce1\" (UID: \"813ae008-e079-4406-83f7-6eae7acc3ce1\") "
Oct 31 20:03:10.383132 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:10.383097    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/813ae008-e079-4406-83f7-6eae7acc3ce1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/813ae008-e079-4406-83f7-6eae7acc3ce1-istiod-ca-cert\") pod \"813ae008-e079-4406-83f7-6eae7acc3ce1\" (UID: \"813ae008-e079-4406-83f7-6eae7acc3ce1\") "
Oct 31 20:03:10.383290 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:10.383263    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/813ae008-e079-4406-83f7-6eae7acc3ce1-workload-socket" pod "813ae008-e079-4406-83f7-6eae7acc3ce1" (UID: "813ae008-e079-4406-83f7-6eae7acc3ce1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:03:10.383513 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:10.383463    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/813ae008-e079-4406-83f7-6eae7acc3ce1-istiod-ca-cert" pod "813ae008-e079-4406-83f7-6eae7acc3ce1" (UID: "813ae008-e079-4406-83f7-6eae7acc3ce1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:03:10.383892 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:10.383866    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/813ae008-e079-4406-83f7-6eae7acc3ce1-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:10.384030 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:10.383895    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/813ae008-e079-4406-83f7-6eae7acc3ce1-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:10.384030 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:10.383910    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/813ae008-e079-4406-83f7-6eae7acc3ce1-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:10.384030 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:10.383925    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/813ae008-e079-4406-83f7-6eae7acc3ce1-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:10.384030 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:10.383939    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/813ae008-e079-4406-83f7-6eae7acc3ce1-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:10.385393 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:10.385300    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/813ae008-e079-4406-83f7-6eae7acc3ce1-istio-podinfo" pod "813ae008-e079-4406-83f7-6eae7acc3ce1" (UID: "813ae008-e079-4406-83f7-6eae7acc3ce1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 20:03:10.385393 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:10.385342    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/813ae008-e079-4406-83f7-6eae7acc3ce1-istio-envoy" pod "813ae008-e079-4406-83f7-6eae7acc3ce1" (UID: "813ae008-e079-4406-83f7-6eae7acc3ce1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:03:10.385563 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:10.385485    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/813ae008-e079-4406-83f7-6eae7acc3ce1-istio-token" pod "813ae008-e079-4406-83f7-6eae7acc3ce1" (UID: "813ae008-e079-4406-83f7-6eae7acc3ce1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:03:10.385563 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:10.385511    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/813ae008-e079-4406-83f7-6eae7acc3ce1-kube-api-access-k9vgc" pod "813ae008-e079-4406-83f7-6eae7acc3ce1" (UID: "813ae008-e079-4406-83f7-6eae7acc3ce1"). InnerVolumeSpecName "kube-api-access-k9vgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:03:10.475504 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:10.475469    4645 generic.go:356] "Generic (PLEG): container finished" podID="813ae008-e079-4406-83f7-6eae7acc3ce1" containerID="5e7a4d56c78c54025c3ad6d7a38648b351dc701cc8fdb9b370bcbd8bd33fda4d" exitCode=0
Oct 31 20:03:10.475967 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:10.475523    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2"
Oct 31 20:03:10.475967 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:10.475556    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2" event={"ID":"813ae008-e079-4406-83f7-6eae7acc3ce1","Type":"ContainerDied","Data":"5e7a4d56c78c54025c3ad6d7a38648b351dc701cc8fdb9b370bcbd8bd33fda4d"}
Oct 31 20:03:10.475967 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:10.475656    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2" event={"ID":"813ae008-e079-4406-83f7-6eae7acc3ce1","Type":"ContainerDied","Data":"202d09c571de6ab9e4f9e33edbaddd0870663244742cc67b4c0d80cce9a934c5"}
Oct 31 20:03:10.475967 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:10.475688    4645 scope.go:122] "RemoveContainer" containerID="5e7a4d56c78c54025c3ad6d7a38648b351dc701cc8fdb9b370bcbd8bd33fda4d"
Oct 31 20:03:10.485306 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:10.485281    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9vgc\" (UniqueName: \"kubernetes.io/projected/813ae008-e079-4406-83f7-6eae7acc3ce1-kube-api-access-k9vgc\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:10.485444 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:10.485310    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/813ae008-e079-4406-83f7-6eae7acc3ce1-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:10.485444 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:10.485320    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/813ae008-e079-4406-83f7-6eae7acc3ce1-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:10.485444 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:10.485343    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/813ae008-e079-4406-83f7-6eae7acc3ce1-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:10.489394 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:10.489376    4645 scope.go:122] "RemoveContainer" containerID="3c0d8d30d39213108ab50f74714ca089e81860c63ad687013e351e6d86717a2c"
Oct 31 20:03:10.500836 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:10.500817    4645 scope.go:122] "RemoveContainer" containerID="5e7a4d56c78c54025c3ad6d7a38648b351dc701cc8fdb9b370bcbd8bd33fda4d"
Oct 31 20:03:10.501125 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:03:10.501097    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e7a4d56c78c54025c3ad6d7a38648b351dc701cc8fdb9b370bcbd8bd33fda4d\": container with ID starting with 5e7a4d56c78c54025c3ad6d7a38648b351dc701cc8fdb9b370bcbd8bd33fda4d not found: ID does not exist" containerID="5e7a4d56c78c54025c3ad6d7a38648b351dc701cc8fdb9b370bcbd8bd33fda4d"
Oct 31 20:03:10.501236 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:10.501130    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e7a4d56c78c54025c3ad6d7a38648b351dc701cc8fdb9b370bcbd8bd33fda4d"} err="failed to get container status \"5e7a4d56c78c54025c3ad6d7a38648b351dc701cc8fdb9b370bcbd8bd33fda4d\": rpc error: code = NotFound desc = could not find container \"5e7a4d56c78c54025c3ad6d7a38648b351dc701cc8fdb9b370bcbd8bd33fda4d\": container with ID starting with 5e7a4d56c78c54025c3ad6d7a38648b351dc701cc8fdb9b370bcbd8bd33fda4d not found: ID does not exist"
Oct 31 20:03:10.501236 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:10.501150    4645 scope.go:122] "RemoveContainer" containerID="3c0d8d30d39213108ab50f74714ca089e81860c63ad687013e351e6d86717a2c"
Oct 31 20:03:10.501554 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:03:10.501499    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c0d8d30d39213108ab50f74714ca089e81860c63ad687013e351e6d86717a2c\": container with ID starting with 3c0d8d30d39213108ab50f74714ca089e81860c63ad687013e351e6d86717a2c not found: ID does not exist" containerID="3c0d8d30d39213108ab50f74714ca089e81860c63ad687013e351e6d86717a2c"
Oct 31 20:03:10.501554 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:10.501540    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c0d8d30d39213108ab50f74714ca089e81860c63ad687013e351e6d86717a2c"} err="failed to get container status \"3c0d8d30d39213108ab50f74714ca089e81860c63ad687013e351e6d86717a2c\": rpc error: code = NotFound desc = could not find container \"3c0d8d30d39213108ab50f74714ca089e81860c63ad687013e351e6d86717a2c\": container with ID starting with 3c0d8d30d39213108ab50f74714ca089e81860c63ad687013e351e6d86717a2c not found: ID does not exist"
Oct 31 20:03:10.502597 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:10.502574    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2"]
Oct 31 20:03:10.503312 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:10.503293    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:10.508674 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:10.508652    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-f5c1ab-openshift-default-5749cfc96f-shwc2"]
Oct 31 20:03:11.029901 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:11.029866    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:11.621251 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:11.621213    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="813ae008-e079-4406-83f7-6eae7acc3ce1" path="/var/lib/kubelet/pods/813ae008-e079-4406-83f7-6eae7acc3ce1/volumes"
Oct 31 20:03:11.741045 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:11.741014    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:12.450568 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:12.450538    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:12.524945 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:12.524111    4645 generic.go:356] "Generic (PLEG): container finished" podID="a5005342-3674-4428-ac17-a42f0dc6d8b2" containerID="abd5bf92064b245b03b7801a3dd5a63c6a7f2b1eb9014bbe05b10b68f0f5ad60" exitCode=0
Oct 31 20:03:12.524945 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:12.524248    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f5c1ab-54c4d89f99-b7fr8" event={"ID":"a5005342-3674-4428-ac17-a42f0dc6d8b2","Type":"ContainerDied","Data":"abd5bf92064b245b03b7801a3dd5a63c6a7f2b1eb9014bbe05b10b68f0f5ad60"}
Oct 31 20:03:12.524945 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:12.524300    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f5c1ab-54c4d89f99-b7fr8" event={"ID":"a5005342-3674-4428-ac17-a42f0dc6d8b2","Type":"ContainerDied","Data":"78dadf43a1eecc67c317f0d4b2e6f7ed9aa13b3150cdedae0fb0e7fd2ec6e4f5"}
Oct 31 20:03:12.524945 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:12.524323    4645 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78dadf43a1eecc67c317f0d4b2e6f7ed9aa13b3150cdedae0fb0e7fd2ec6e4f5"
Oct 31 20:03:12.526001 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:12.525971    4645 generic.go:356] "Generic (PLEG): container finished" podID="4d05d0a8-95d5-4f44-b7d9-bae7800b7176" containerID="1edb64f1761ac82b92da42c3df5f7d5674bbac3840d46c187920d9ab1a8d6648" exitCode=0
Oct 31 20:03:12.526116 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:12.526099    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f5c1ab-68b5bdf4d7-jwcll" event={"ID":"4d05d0a8-95d5-4f44-b7d9-bae7800b7176","Type":"ContainerDied","Data":"1edb64f1761ac82b92da42c3df5f7d5674bbac3840d46c187920d9ab1a8d6648"}
Oct 31 20:03:12.526178 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:12.526141    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f5c1ab-68b5bdf4d7-jwcll" event={"ID":"4d05d0a8-95d5-4f44-b7d9-bae7800b7176","Type":"ContainerDied","Data":"11750d36a0ef565a9334ffc033303060193297e43684ab3e61237879f50a8cb5"}
Oct 31 20:03:12.526178 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:12.526165    4645 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11750d36a0ef565a9334ffc033303060193297e43684ab3e61237879f50a8cb5"
Oct 31 20:03:12.572264 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:12.572234    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f5c1ab-54c4d89f99-b7fr8"
Oct 31 20:03:12.578562 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:12.578537    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f5c1ab-68b5bdf4d7-jwcll"
Oct 31 20:03:12.658245 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:12.658213    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5005342-3674-4428-ac17-a42f0dc6d8b2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5005342-3674-4428-ac17-a42f0dc6d8b2-plugins-config-volume\") pod \"a5005342-3674-4428-ac17-a42f0dc6d8b2\" (UID: \"a5005342-3674-4428-ac17-a42f0dc6d8b2\") "
Oct 31 20:03:12.658774 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:12.658282    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5005342-3674-4428-ac17-a42f0dc6d8b2-kube-api-access-rptqq\" (UniqueName: \"kubernetes.io/projected/a5005342-3674-4428-ac17-a42f0dc6d8b2-kube-api-access-rptqq\") pod \"a5005342-3674-4428-ac17-a42f0dc6d8b2\" (UID: \"a5005342-3674-4428-ac17-a42f0dc6d8b2\") "
Oct 31 20:03:12.658774 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:12.658683    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5005342-3674-4428-ac17-a42f0dc6d8b2-plugins-config-volume" pod "a5005342-3674-4428-ac17-a42f0dc6d8b2" (UID: "a5005342-3674-4428-ac17-a42f0dc6d8b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:03:12.659119 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:12.659099    4645 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5005342-3674-4428-ac17-a42f0dc6d8b2-plugins-config-volume\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:12.660575 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:12.660551    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5005342-3674-4428-ac17-a42f0dc6d8b2-kube-api-access-rptqq" pod "a5005342-3674-4428-ac17-a42f0dc6d8b2" (UID: "a5005342-3674-4428-ac17-a42f0dc6d8b2"). InnerVolumeSpecName "kube-api-access-rptqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:03:12.760048 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:12.759958    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d05d0a8-95d5-4f44-b7d9-bae7800b7176-kube-api-access-zml6b\" (UniqueName: \"kubernetes.io/projected/4d05d0a8-95d5-4f44-b7d9-bae7800b7176-kube-api-access-zml6b\") pod \"4d05d0a8-95d5-4f44-b7d9-bae7800b7176\" (UID: \"4d05d0a8-95d5-4f44-b7d9-bae7800b7176\") "
Oct 31 20:03:12.760242 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:12.760226    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d05d0a8-95d5-4f44-b7d9-bae7800b7176-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d05d0a8-95d5-4f44-b7d9-bae7800b7176-plugins-config-volume\") pod \"4d05d0a8-95d5-4f44-b7d9-bae7800b7176\" (UID: \"4d05d0a8-95d5-4f44-b7d9-bae7800b7176\") "
Oct 31 20:03:12.760624 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:12.760594    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d05d0a8-95d5-4f44-b7d9-bae7800b7176-plugins-config-volume" pod "4d05d0a8-95d5-4f44-b7d9-bae7800b7176" (UID: "4d05d0a8-95d5-4f44-b7d9-bae7800b7176"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:03:12.760765 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:12.760668    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rptqq\" (UniqueName: \"kubernetes.io/projected/a5005342-3674-4428-ac17-a42f0dc6d8b2-kube-api-access-rptqq\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:12.762098 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:12.762075    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d05d0a8-95d5-4f44-b7d9-bae7800b7176-kube-api-access-zml6b" pod "4d05d0a8-95d5-4f44-b7d9-bae7800b7176" (UID: "4d05d0a8-95d5-4f44-b7d9-bae7800b7176"). InnerVolumeSpecName "kube-api-access-zml6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:03:12.862925 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:12.862891    4645 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d05d0a8-95d5-4f44-b7d9-bae7800b7176-plugins-config-volume\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:12.862925 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:12.862921    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zml6b\" (UniqueName: \"kubernetes.io/projected/4d05d0a8-95d5-4f44-b7d9-bae7800b7176-kube-api-access-zml6b\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:13.312585 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:13.312543    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:13.533741 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:13.533647    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f5c1ab-68b5bdf4d7-jwcll"
Oct 31 20:03:13.533741 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:13.533645    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-f5c1ab-54c4d89f99-b7fr8"
Oct 31 20:03:13.569386 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:13.569355    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f5c1ab-68b5bdf4d7-jwcll"]
Oct 31 20:03:13.574867 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:13.574840    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-f5c1ab-68b5bdf4d7-jwcll"]
Oct 31 20:03:13.579197 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:13.579171    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-f5c1ab-54c4d89f99-b7fr8"]
Oct 31 20:03:13.582929 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:13.582903    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-f5c1ab-54c4d89f99-b7fr8"]
Oct 31 20:03:13.618899 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:13.618876    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d05d0a8-95d5-4f44-b7d9-bae7800b7176" path="/var/lib/kubelet/pods/4d05d0a8-95d5-4f44-b7d9-bae7800b7176/volumes"
Oct 31 20:03:13.619297 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:13.619283    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5005342-3674-4428-ac17-a42f0dc6d8b2" path="/var/lib/kubelet/pods/a5005342-3674-4428-ac17-a42f0dc6d8b2/volumes"
Oct 31 20:03:14.060302 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:14.060271    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:14.842083 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:14.842049    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:15.666844 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:15.666807    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:15.981107 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:15.981023    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:16.102349 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:16.101892    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:16.427410 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:16.417802    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7647c-85d9ccfd7f-qvj7p"
Oct 31 20:03:16.587987 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:16.587947    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:17.432377 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:17.425474    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d7647c-66bbb74874-4dsdn"
Oct 31 20:03:17.683381 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:17.683270    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:18.120427 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:18.120319    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:18.604390 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:18.604357    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:19.203682 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:19.203644    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:19.333063 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:19.333021    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a84fcb-5465fc5b64-v2fc5"]
Oct 31 20:03:19.333724 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:19.333709    4645 cpu_manager.go:414] "containerMap: removing container" podUID="4d05d0a8-95d5-4f44-b7d9-bae7800b7176" containerName="payload-pre-processing"
Oct 31 20:03:19.333772 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:19.333725    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d05d0a8-95d5-4f44-b7d9-bae7800b7176" containerName="payload-pre-processing"
Oct 31 20:03:19.333772 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:19.333734    4645 cpu_manager.go:414] "containerMap: removing container" podUID="a5005342-3674-4428-ac17-a42f0dc6d8b2" containerName="payload-processing"
Oct 31 20:03:19.333772 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:19.333739    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5005342-3674-4428-ac17-a42f0dc6d8b2" containerName="payload-processing"
Oct 31 20:03:19.333772 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:19.333748    4645 cpu_manager.go:414] "containerMap: removing container" podUID="813ae008-e079-4406-83f7-6eae7acc3ce1" containerName="istio-proxy"
Oct 31 20:03:19.333772 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:19.333753    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="813ae008-e079-4406-83f7-6eae7acc3ce1" containerName="istio-proxy"
Oct 31 20:03:19.333945 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:19.333776    4645 cpu_manager.go:414] "containerMap: removing container" podUID="a3c4c540-6b69-4c81-bea8-6c4176cf621c" containerName="revoke-keys"
Oct 31 20:03:19.333945 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:19.333781    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3c4c540-6b69-4c81-bea8-6c4176cf621c" containerName="revoke-keys"
Oct 31 20:03:19.333945 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:19.333894    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="813ae008-e079-4406-83f7-6eae7acc3ce1" containerName="istio-proxy"
Oct 31 20:03:19.333945 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:19.333904    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d05d0a8-95d5-4f44-b7d9-bae7800b7176" containerName="payload-pre-processing"
Oct 31 20:03:19.333945 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:19.333912    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5005342-3674-4428-ac17-a42f0dc6d8b2" containerName="payload-processing"
Oct 31 20:03:19.333945 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:19.333920    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3c4c540-6b69-4c81-bea8-6c4176cf621c" containerName="revoke-keys"
Oct 31 20:03:19.339125 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:19.339105    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a84fcb-5465fc5b64-v2fc5"
Oct 31 20:03:19.352440 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:19.352411    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed6a7560-203e-4f9a-92ba-aa98b0564e09-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a84fcb-5465fc5b64-v2fc5\" (UID: \"ed6a7560-203e-4f9a-92ba-aa98b0564e09\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a84fcb-5465fc5b64-v2fc5"
Oct 31 20:03:19.352590 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:19.352556    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkrlt\" (UniqueName: \"kubernetes.io/projected/ed6a7560-203e-4f9a-92ba-aa98b0564e09-kube-api-access-kkrlt\") pod \"payload-processing-e2e-ipp-b-a84fcb-5465fc5b64-v2fc5\" (UID: \"ed6a7560-203e-4f9a-92ba-aa98b0564e09\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a84fcb-5465fc5b64-v2fc5"
Oct 31 20:03:19.354236 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:19.354209    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a84fcb-5465fc5b64-v2fc5"]
Oct 31 20:03:19.354882 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:19.354860    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-a84fcb\""
Oct 31 20:03:19.359788 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:19.359746    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:19.493953 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:19.486689    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed6a7560-203e-4f9a-92ba-aa98b0564e09-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a84fcb-5465fc5b64-v2fc5\" (UID: \"ed6a7560-203e-4f9a-92ba-aa98b0564e09\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a84fcb-5465fc5b64-v2fc5"
Oct 31 20:03:19.493953 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:19.486882    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkrlt\" (UniqueName: \"kubernetes.io/projected/ed6a7560-203e-4f9a-92ba-aa98b0564e09-kube-api-access-kkrlt\") pod \"payload-processing-e2e-ipp-b-a84fcb-5465fc5b64-v2fc5\" (UID: \"ed6a7560-203e-4f9a-92ba-aa98b0564e09\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a84fcb-5465fc5b64-v2fc5"
Oct 31 20:03:19.493953 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:19.488373    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed6a7560-203e-4f9a-92ba-aa98b0564e09-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a84fcb-5465fc5b64-v2fc5\" (UID: \"ed6a7560-203e-4f9a-92ba-aa98b0564e09\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a84fcb-5465fc5b64-v2fc5"
Oct 31 20:03:19.575355 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:19.575101    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkrlt\" (UniqueName: \"kubernetes.io/projected/ed6a7560-203e-4f9a-92ba-aa98b0564e09-kube-api-access-kkrlt\") pod \"payload-processing-e2e-ipp-b-a84fcb-5465fc5b64-v2fc5\" (UID: \"ed6a7560-203e-4f9a-92ba-aa98b0564e09\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a84fcb-5465fc5b64-v2fc5"
Oct 31 20:03:19.670370 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:19.658594    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a84fcb-5465fc5b64-v2fc5"
Oct 31 20:03:19.898377 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:19.894837    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:20.085254 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:03:20.085210    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded6a7560_203e_4f9a_92ba_aa98b0564e09.slice/crio-c3654a652310eff178d41e1b35393641c400ab6639db2ec6fb1bad5eb8d77c15 WatchSource:0}: Error finding container c3654a652310eff178d41e1b35393641c400ab6639db2ec6fb1bad5eb8d77c15: Status 404 returned error can't find the container with id c3654a652310eff178d41e1b35393641c400ab6639db2ec6fb1bad5eb8d77c15
Oct 31 20:03:20.086580 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:20.086553    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a84fcb-5465fc5b64-v2fc5"]
Oct 31 20:03:20.181962 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:20.181929    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:20.433933 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:20.433902    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c734f5-jf7w8"]
Oct 31 20:03:20.434643 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:20.434623    4645 cpu_manager.go:414] "containerMap: removing container" podUID="813ae008-e079-4406-83f7-6eae7acc3ce1" containerName="istio-proxy"
Oct 31 20:03:20.434643 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:20.434640    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="813ae008-e079-4406-83f7-6eae7acc3ce1" containerName="istio-proxy"
Oct 31 20:03:20.446273 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:20.446239    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="813ae008-e079-4406-83f7-6eae7acc3ce1" containerName="istio-proxy"
Oct 31 20:03:20.482396 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:20.481622    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c734f5-jf7w8"]
Oct 31 20:03:20.482396 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:20.481753    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c734f5-jf7w8"
Oct 31 20:03:20.487351 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:20.486841    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:03:20.487685 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:20.487656    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 20:03:20.589433 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:20.589395    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09052cb1-f1a8-4ccf-8583-d9b92bf9d897-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c734f5-jf7w8\" (UID: \"09052cb1-f1a8-4ccf-8583-d9b92bf9d897\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c734f5-jf7w8"
Oct 31 20:03:20.589590 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:20.589526    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bxlf\" (UniqueName: \"kubernetes.io/projected/09052cb1-f1a8-4ccf-8583-d9b92bf9d897-kube-api-access-4bxlf\") pod \"maas-api-revoke-keys-e2e-ipp-a-c734f5-jf7w8\" (UID: \"09052cb1-f1a8-4ccf-8583-d9b92bf9d897\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c734f5-jf7w8"
Oct 31 20:03:20.589692 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:20.589663    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09052cb1-f1a8-4ccf-8583-d9b92bf9d897-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c734f5-jf7w8\" (UID: \"09052cb1-f1a8-4ccf-8583-d9b92bf9d897\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c734f5-jf7w8"
Oct 31 20:03:20.691239 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:20.690794    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09052cb1-f1a8-4ccf-8583-d9b92bf9d897-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c734f5-jf7w8\" (UID: \"09052cb1-f1a8-4ccf-8583-d9b92bf9d897\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c734f5-jf7w8"
Oct 31 20:03:20.691239 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:20.690902    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bxlf\" (UniqueName: \"kubernetes.io/projected/09052cb1-f1a8-4ccf-8583-d9b92bf9d897-kube-api-access-4bxlf\") pod \"maas-api-revoke-keys-e2e-ipp-a-c734f5-jf7w8\" (UID: \"09052cb1-f1a8-4ccf-8583-d9b92bf9d897\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c734f5-jf7w8"
Oct 31 20:03:20.691239 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:20.690988    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09052cb1-f1a8-4ccf-8583-d9b92bf9d897-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c734f5-jf7w8\" (UID: \"09052cb1-f1a8-4ccf-8583-d9b92bf9d897\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c734f5-jf7w8"
Oct 31 20:03:20.692062 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:20.692028    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09052cb1-f1a8-4ccf-8583-d9b92bf9d897-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c734f5-jf7w8\" (UID: \"09052cb1-f1a8-4ccf-8583-d9b92bf9d897\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c734f5-jf7w8"
Oct 31 20:03:20.694019 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:20.693990    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09052cb1-f1a8-4ccf-8583-d9b92bf9d897-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c734f5-jf7w8\" (UID: \"09052cb1-f1a8-4ccf-8583-d9b92bf9d897\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c734f5-jf7w8"
Oct 31 20:03:20.702749 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:20.702723    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bxlf\" (UniqueName: \"kubernetes.io/projected/09052cb1-f1a8-4ccf-8583-d9b92bf9d897-kube-api-access-4bxlf\") pod \"maas-api-revoke-keys-e2e-ipp-a-c734f5-jf7w8\" (UID: \"09052cb1-f1a8-4ccf-8583-d9b92bf9d897\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c734f5-jf7w8"
Oct 31 20:03:20.750144 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:20.750117    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a84fcb-5465fc5b64-v2fc5" event={"ID":"ed6a7560-203e-4f9a-92ba-aa98b0564e09","Type":"ContainerStarted","Data":"b83f31a519e22805250dea81c103115de8a452c333e9ef32af9a054a9328a056"}
Oct 31 20:03:20.750255 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:20.750166    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a84fcb-5465fc5b64-v2fc5" event={"ID":"ed6a7560-203e-4f9a-92ba-aa98b0564e09","Type":"ContainerStarted","Data":"c3654a652310eff178d41e1b35393641c400ab6639db2ec6fb1bad5eb8d77c15"}
Oct 31 20:03:20.750385 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:20.750359    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a84fcb-5465fc5b64-v2fc5"
Oct 31 20:03:20.778221 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:20.778160    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-a84fcb-5465fc5b64-v2fc5" podStartSLOduration=1.778139882 podStartE2EDuration="1.778139882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:03:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:03:20.77571444 +0000 UTC m=+2631.348948570" watchObservedRunningTime="2026-10-31 20:03:20.778139882 +0000 UTC m=+2631.351374012"
Oct 31 20:03:20.810817 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:20.810785    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c734f5-jf7w8"
Oct 31 20:03:21.199740 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:03:21.199700    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09052cb1_f1a8_4ccf_8583_d9b92bf9d897.slice/crio-402c6b15d33c4ac498ed2ea18acf365a1342ed295b9955f112021d9583b31965 WatchSource:0}: Error finding container 402c6b15d33c4ac498ed2ea18acf365a1342ed295b9955f112021d9583b31965: Status 404 returned error can't find the container with id 402c6b15d33c4ac498ed2ea18acf365a1342ed295b9955f112021d9583b31965
Oct 31 20:03:21.205965 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:21.205908    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c734f5-jf7w8"]
Oct 31 20:03:21.206840 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:21.206818    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:21.787510 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:21.787410    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c734f5-jf7w8" event={"ID":"09052cb1-f1a8-4ccf-8583-d9b92bf9d897","Type":"ContainerStarted","Data":"e17cc4c3aeaab50d184bf6447361b1b299e4a7328d3205a677481d75c66ae785"}
Oct 31 20:03:21.787510 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:21.787479    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c734f5-jf7w8" event={"ID":"09052cb1-f1a8-4ccf-8583-d9b92bf9d897","Type":"ContainerStarted","Data":"402c6b15d33c4ac498ed2ea18acf365a1342ed295b9955f112021d9583b31965"}
Oct 31 20:03:21.807748 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:21.807698    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c734f5-jf7w8" podStartSLOduration=1.8076847310000002 podStartE2EDuration="1.807684731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:03:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:03:21.805931632 +0000 UTC m=+2632.379165760" watchObservedRunningTime="2026-10-31 20:03:21.807684731 +0000 UTC m=+2632.380918859"
Oct 31 20:03:22.346738 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:22.346702    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:22.796149 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:22.796115    4645 generic.go:356] "Generic (PLEG): container finished" podID="09052cb1-f1a8-4ccf-8583-d9b92bf9d897" containerID="e17cc4c3aeaab50d184bf6447361b1b299e4a7328d3205a677481d75c66ae785" exitCode=0
Oct 31 20:03:22.796616 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:22.796206    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c734f5-jf7w8" event={"ID":"09052cb1-f1a8-4ccf-8583-d9b92bf9d897","Type":"ContainerDied","Data":"e17cc4c3aeaab50d184bf6447361b1b299e4a7328d3205a677481d75c66ae785"}
Oct 31 20:03:23.323371 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:23.323314    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:24.020559 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:24.020533    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c734f5-jf7w8"
Oct 31 20:03:24.089072 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:24.088980    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09052cb1-f1a8-4ccf-8583-d9b92bf9d897-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09052cb1-f1a8-4ccf-8583-d9b92bf9d897-maas-api-openshift-ca\") pod \"09052cb1-f1a8-4ccf-8583-d9b92bf9d897\" (UID: \"09052cb1-f1a8-4ccf-8583-d9b92bf9d897\") "
Oct 31 20:03:24.089234 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:24.089123    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09052cb1-f1a8-4ccf-8583-d9b92bf9d897-kube-api-access-4bxlf\" (UniqueName: \"kubernetes.io/projected/09052cb1-f1a8-4ccf-8583-d9b92bf9d897-kube-api-access-4bxlf\") pod \"09052cb1-f1a8-4ccf-8583-d9b92bf9d897\" (UID: \"09052cb1-f1a8-4ccf-8583-d9b92bf9d897\") "
Oct 31 20:03:24.089234 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:24.089165    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09052cb1-f1a8-4ccf-8583-d9b92bf9d897-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09052cb1-f1a8-4ccf-8583-d9b92bf9d897-maas-api-xks-ca\") pod \"09052cb1-f1a8-4ccf-8583-d9b92bf9d897\" (UID: \"09052cb1-f1a8-4ccf-8583-d9b92bf9d897\") "
Oct 31 20:03:24.090080 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:24.090053    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09052cb1-f1a8-4ccf-8583-d9b92bf9d897-maas-api-openshift-ca" pod "09052cb1-f1a8-4ccf-8583-d9b92bf9d897" (UID: "09052cb1-f1a8-4ccf-8583-d9b92bf9d897"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:03:24.092021 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:24.091998    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09052cb1-f1a8-4ccf-8583-d9b92bf9d897-kube-api-access-4bxlf" pod "09052cb1-f1a8-4ccf-8583-d9b92bf9d897" (UID: "09052cb1-f1a8-4ccf-8583-d9b92bf9d897"). InnerVolumeSpecName "kube-api-access-4bxlf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:03:24.093844 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:24.093818    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09052cb1-f1a8-4ccf-8583-d9b92bf9d897-maas-api-xks-ca" pod "09052cb1-f1a8-4ccf-8583-d9b92bf9d897" (UID: "09052cb1-f1a8-4ccf-8583-d9b92bf9d897"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:03:24.100786 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:24.100760    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:24.193146 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:24.193096    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bxlf\" (UniqueName: \"kubernetes.io/projected/09052cb1-f1a8-4ccf-8583-d9b92bf9d897-kube-api-access-4bxlf\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:24.193434 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:24.193412    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09052cb1-f1a8-4ccf-8583-d9b92bf9d897-maas-api-xks-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:24.193566 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:24.193532    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09052cb1-f1a8-4ccf-8583-d9b92bf9d897-maas-api-openshift-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:24.818072 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:24.818035    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c734f5-jf7w8" event={"ID":"09052cb1-f1a8-4ccf-8583-d9b92bf9d897","Type":"ContainerDied","Data":"402c6b15d33c4ac498ed2ea18acf365a1342ed295b9955f112021d9583b31965"}
Oct 31 20:03:24.818283 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:24.818128    4645 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="402c6b15d33c4ac498ed2ea18acf365a1342ed295b9955f112021d9583b31965"
Oct 31 20:03:24.818283 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:24.818046    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c734f5-jf7w8"
Oct 31 20:03:25.074410 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:25.068678    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:26.111486 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:26.111442    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:26.962920 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:26.962881    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:27.924159 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:27.924126    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:28.769213 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:28.769179    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:29.334455 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:29.334420    4645 patch_prober.go:29] interesting pod/e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.24:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 20:03:29.334929 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:29.334485    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb" podUID="ba05261e-eadf-44c3-abda-cb561b4ea6a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.24:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 20:03:29.761215 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:29.761180    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:30.549275 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:30.549235    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:31.316014 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:31.315874    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:31.564871 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:31.564793    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09c55086-fx4jv"]
Oct 31 20:03:31.565764 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:31.565728    4645 cpu_manager.go:414] "containerMap: removing container" podUID="09052cb1-f1a8-4ccf-8583-d9b92bf9d897" containerName="revoke-keys"
Oct 31 20:03:31.565764 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:31.565752    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09052cb1-f1a8-4ccf-8583-d9b92bf9d897" containerName="revoke-keys"
Oct 31 20:03:31.565952 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:31.565912    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="09052cb1-f1a8-4ccf-8583-d9b92bf9d897" containerName="revoke-keys"
Oct 31 20:03:31.568546 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:31.568524    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c55086-fx4jv"
Oct 31 20:03:31.574052 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:31.574021    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:03:31.574191 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:31.574062    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 20:03:31.579836 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:31.579814    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09c55086-fx4jv"]
Oct 31 20:03:31.579944 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:31.579925    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48a098c1-470e-4994-845a-3d34a8618ba9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09c55086-fx4jv\" (UID: \"48a098c1-470e-4994-845a-3d34a8618ba9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c55086-fx4jv"
Oct 31 20:03:31.580084 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:31.580056    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm42x\" (UniqueName: \"kubernetes.io/projected/48a098c1-470e-4994-845a-3d34a8618ba9-kube-api-access-zm42x\") pod \"maas-api-revoke-sub-09c55086-fx4jv\" (UID: \"48a098c1-470e-4994-845a-3d34a8618ba9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c55086-fx4jv"
Oct 31 20:03:31.580199 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:31.580175    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48a098c1-470e-4994-845a-3d34a8618ba9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09c55086-fx4jv\" (UID: \"48a098c1-470e-4994-845a-3d34a8618ba9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c55086-fx4jv"
Oct 31 20:03:31.681089 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:31.681063    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48a098c1-470e-4994-845a-3d34a8618ba9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09c55086-fx4jv\" (UID: \"48a098c1-470e-4994-845a-3d34a8618ba9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c55086-fx4jv"
Oct 31 20:03:31.681248 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:31.681126    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm42x\" (UniqueName: \"kubernetes.io/projected/48a098c1-470e-4994-845a-3d34a8618ba9-kube-api-access-zm42x\") pod \"maas-api-revoke-sub-09c55086-fx4jv\" (UID: \"48a098c1-470e-4994-845a-3d34a8618ba9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c55086-fx4jv"
Oct 31 20:03:31.681248 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:31.681228    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48a098c1-470e-4994-845a-3d34a8618ba9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09c55086-fx4jv\" (UID: \"48a098c1-470e-4994-845a-3d34a8618ba9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c55086-fx4jv"
Oct 31 20:03:31.681900 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:31.681875    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48a098c1-470e-4994-845a-3d34a8618ba9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09c55086-fx4jv\" (UID: \"48a098c1-470e-4994-845a-3d34a8618ba9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c55086-fx4jv"
Oct 31 20:03:31.683486 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:31.683467    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48a098c1-470e-4994-845a-3d34a8618ba9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09c55086-fx4jv\" (UID: \"48a098c1-470e-4994-845a-3d34a8618ba9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c55086-fx4jv"
Oct 31 20:03:31.693044 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:31.693023    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm42x\" (UniqueName: \"kubernetes.io/projected/48a098c1-470e-4994-845a-3d34a8618ba9-kube-api-access-zm42x\") pod \"maas-api-revoke-sub-09c55086-fx4jv\" (UID: \"48a098c1-470e-4994-845a-3d34a8618ba9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c55086-fx4jv"
Oct 31 20:03:31.789407 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:31.789386    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a84fcb-5465fc5b64-v2fc5"
Oct 31 20:03:31.790436 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:31.790416    4645 scope.go:122] "RemoveContainer" containerID="1edb64f1761ac82b92da42c3df5f7d5674bbac3840d46c187920d9ab1a8d6648"
Oct 31 20:03:31.801429 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:31.801405    4645 scope.go:122] "RemoveContainer" containerID="abd5bf92064b245b03b7801a3dd5a63c6a7f2b1eb9014bbe05b10b68f0f5ad60"
Oct 31 20:03:31.886883 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:31.886861    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c55086-fx4jv"
Oct 31 20:03:31.986554 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:31.985663    4645 generic.go:356] "Generic (PLEG): container finished" podID="ba05261e-eadf-44c3-abda-cb561b4ea6a5" containerID="28340e41fc47b32fbd85c43ddd7a92a8220d26df04ef0518263bdd162b889516" exitCode=0
Oct 31 20:03:31.986554 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:31.985816    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb" event={"ID":"ba05261e-eadf-44c3-abda-cb561b4ea6a5","Type":"ContainerDied","Data":"28340e41fc47b32fbd85c43ddd7a92a8220d26df04ef0518263bdd162b889516"}
Oct 31 20:03:31.986554 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:31.986149    4645 scope.go:122] "RemoveContainer" containerID="28340e41fc47b32fbd85c43ddd7a92a8220d26df04ef0518263bdd162b889516"
Oct 31 20:03:31.988534 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:31.988031    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:03:31.988534 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:31.988102    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:03:31.988534 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:31.988134    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:03:32.050373 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:32.049769    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:32.190725 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:32.190516    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09c55086-fx4jv"]
Oct 31 20:03:32.900949 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:32.900856    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:33.025032 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:33.024987    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c55086-fx4jv" event={"ID":"48a098c1-470e-4994-845a-3d34a8618ba9","Type":"ContainerStarted","Data":"3e10873a1057d46760e2c37f44d072523f0fcb8915b081c8d097f1d6a39376c2"}
Oct 31 20:03:33.025239 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:33.025058    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c55086-fx4jv" event={"ID":"48a098c1-470e-4994-845a-3d34a8618ba9","Type":"ContainerStarted","Data":"fb349de9792b255ac9d5d4afd835da5ec31e474ea5b389737a10717ab72c50f0"}
Oct 31 20:03:33.027260 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:33.027231    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb" event={"ID":"ba05261e-eadf-44c3-abda-cb561b4ea6a5","Type":"ContainerStarted","Data":"2d1e8615addbb51e1c72df9927e033ae0c2c9be3c100b66ea0db6468e04ecab7"}
Oct 31 20:03:33.044379 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:33.044292    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c55086-fx4jv" podStartSLOduration=2.044275387 podStartE2EDuration="2.044275387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:03:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:03:33.041865097 +0000 UTC m=+2643.615099227" watchObservedRunningTime="2026-10-31 20:03:33.044275387 +0000 UTC m=+2643.617509520"
Oct 31 20:03:33.840411 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:33.840372    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb"
Oct 31 20:03:33.840859 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:33.840836    4645 patch_prober.go:29] interesting pod/e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused" start-of-body=
Oct 31 20:03:33.840971 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:33.840886    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb" podUID="ba05261e-eadf-44c3-abda-cb561b4ea6a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused"
Oct 31 20:03:34.028862 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:34.028823    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:34.036833 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:34.036807    4645 generic.go:356] "Generic (PLEG): container finished" podID="48a098c1-470e-4994-845a-3d34a8618ba9" containerID="3e10873a1057d46760e2c37f44d072523f0fcb8915b081c8d097f1d6a39376c2" exitCode=0
Oct 31 20:03:34.036993 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:34.036901    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c55086-fx4jv" event={"ID":"48a098c1-470e-4994-845a-3d34a8618ba9","Type":"ContainerDied","Data":"3e10873a1057d46760e2c37f44d072523f0fcb8915b081c8d097f1d6a39376c2"}
Oct 31 20:03:34.839920 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:34.839889    4645 patch_prober.go:29] interesting pod/e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused" start-of-body=
Oct 31 20:03:34.840086 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:34.839938    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb" podUID="ba05261e-eadf-44c3-abda-cb561b4ea6a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused"
Oct 31 20:03:34.981974 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:34.981930    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:35.295206 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:35.295178    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c55086-fx4jv"
Oct 31 20:03:35.345821 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:35.345793    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48a098c1-470e-4994-845a-3d34a8618ba9-kube-api-access-zm42x\" (UniqueName: \"kubernetes.io/projected/48a098c1-470e-4994-845a-3d34a8618ba9-kube-api-access-zm42x\") pod \"48a098c1-470e-4994-845a-3d34a8618ba9\" (UID: \"48a098c1-470e-4994-845a-3d34a8618ba9\") "
Oct 31 20:03:35.346108 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:35.345926    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48a098c1-470e-4994-845a-3d34a8618ba9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48a098c1-470e-4994-845a-3d34a8618ba9-maas-api-openshift-ca\") pod \"48a098c1-470e-4994-845a-3d34a8618ba9\" (UID: \"48a098c1-470e-4994-845a-3d34a8618ba9\") "
Oct 31 20:03:35.346255 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:35.346162    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48a098c1-470e-4994-845a-3d34a8618ba9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48a098c1-470e-4994-845a-3d34a8618ba9-maas-api-xks-ca\") pod \"48a098c1-470e-4994-845a-3d34a8618ba9\" (UID: \"48a098c1-470e-4994-845a-3d34a8618ba9\") "
Oct 31 20:03:35.346255 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:35.346241    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48a098c1-470e-4994-845a-3d34a8618ba9-maas-api-openshift-ca" pod "48a098c1-470e-4994-845a-3d34a8618ba9" (UID: "48a098c1-470e-4994-845a-3d34a8618ba9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:03:35.346756 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:35.346731    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48a098c1-470e-4994-845a-3d34a8618ba9-maas-api-openshift-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:35.348384 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:35.348356    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48a098c1-470e-4994-845a-3d34a8618ba9-maas-api-xks-ca" pod "48a098c1-470e-4994-845a-3d34a8618ba9" (UID: "48a098c1-470e-4994-845a-3d34a8618ba9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:03:35.348789 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:35.348766    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48a098c1-470e-4994-845a-3d34a8618ba9-kube-api-access-zm42x" pod "48a098c1-470e-4994-845a-3d34a8618ba9" (UID: "48a098c1-470e-4994-845a-3d34a8618ba9"). InnerVolumeSpecName "kube-api-access-zm42x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:03:35.448041 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:35.448010    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48a098c1-470e-4994-845a-3d34a8618ba9-maas-api-xks-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:35.448210 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:35.448044    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zm42x\" (UniqueName: \"kubernetes.io/projected/48a098c1-470e-4994-845a-3d34a8618ba9-kube-api-access-zm42x\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:35.689624 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:35.689576    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9d3273d-b8v6k"]
Oct 31 20:03:35.690460 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:35.690440    4645 cpu_manager.go:414] "containerMap: removing container" podUID="48a098c1-470e-4994-845a-3d34a8618ba9" containerName="revoke-keys"
Oct 31 20:03:35.690534 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:35.690463    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48a098c1-470e-4994-845a-3d34a8618ba9" containerName="revoke-keys"
Oct 31 20:03:35.690619 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:35.690608    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="48a098c1-470e-4994-845a-3d34a8618ba9" containerName="revoke-keys"
Oct 31 20:03:35.694176 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:35.694155    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9d3273d-b8v6k"
Oct 31 20:03:35.709791 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:35.706823    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9d3273d-b8v6k"]
Oct 31 20:03:35.740567 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:35.739565    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4c5b9b6-wzn7j"]
Oct 31 20:03:35.751234 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:35.751200    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9slq2\" (UniqueName: \"kubernetes.io/projected/fcceb7d6-67ba-4407-9609-ffdc6758004e-kube-api-access-9slq2\") pod \"maas-api-revoke-sub-c9d3273d-b8v6k\" (UID: \"fcceb7d6-67ba-4407-9609-ffdc6758004e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9d3273d-b8v6k"
Oct 31 20:03:35.751429 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:35.751287    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcceb7d6-67ba-4407-9609-ffdc6758004e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9d3273d-b8v6k\" (UID: \"fcceb7d6-67ba-4407-9609-ffdc6758004e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9d3273d-b8v6k"
Oct 31 20:03:35.751728 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:35.751706    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcceb7d6-67ba-4407-9609-ffdc6758004e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9d3273d-b8v6k\" (UID: \"fcceb7d6-67ba-4407-9609-ffdc6758004e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9d3273d-b8v6k"
Oct 31 20:03:35.752349 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:35.752302    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c5b9b6-wzn7j"
Oct 31 20:03:35.783351 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:35.754793    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4c5b9b6-wzn7j"]
Oct 31 20:03:35.840141 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:35.839738    4645 patch_prober.go:29] interesting pod/e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused" start-of-body=
Oct 31 20:03:35.840141 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:35.839794    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb" podUID="ba05261e-eadf-44c3-abda-cb561b4ea6a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused"
Oct 31 20:03:35.853491 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:35.853468    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9841bd1b-461e-43fa-b433-793a0fd9fd97-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4c5b9b6-wzn7j\" (UID: \"9841bd1b-461e-43fa-b433-793a0fd9fd97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c5b9b6-wzn7j"
Oct 31 20:03:35.853586 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:35.853520    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxbw2\" (UniqueName: \"kubernetes.io/projected/9841bd1b-461e-43fa-b433-793a0fd9fd97-kube-api-access-zxbw2\") pod \"maas-api-revoke-sub-f4c5b9b6-wzn7j\" (UID: \"9841bd1b-461e-43fa-b433-793a0fd9fd97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c5b9b6-wzn7j"
Oct 31 20:03:35.853643 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:35.853620    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9841bd1b-461e-43fa-b433-793a0fd9fd97-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4c5b9b6-wzn7j\" (UID: \"9841bd1b-461e-43fa-b433-793a0fd9fd97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c5b9b6-wzn7j"
Oct 31 20:03:35.853692 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:35.853681    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcceb7d6-67ba-4407-9609-ffdc6758004e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9d3273d-b8v6k\" (UID: \"fcceb7d6-67ba-4407-9609-ffdc6758004e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9d3273d-b8v6k"
Oct 31 20:03:35.853780 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:35.853767    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9slq2\" (UniqueName: \"kubernetes.io/projected/fcceb7d6-67ba-4407-9609-ffdc6758004e-kube-api-access-9slq2\") pod \"maas-api-revoke-sub-c9d3273d-b8v6k\" (UID: \"fcceb7d6-67ba-4407-9609-ffdc6758004e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9d3273d-b8v6k"
Oct 31 20:03:35.853829 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:35.853812    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcceb7d6-67ba-4407-9609-ffdc6758004e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9d3273d-b8v6k\" (UID: \"fcceb7d6-67ba-4407-9609-ffdc6758004e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9d3273d-b8v6k"
Oct 31 20:03:35.854464 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:35.854442    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcceb7d6-67ba-4407-9609-ffdc6758004e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9d3273d-b8v6k\" (UID: \"fcceb7d6-67ba-4407-9609-ffdc6758004e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9d3273d-b8v6k"
Oct 31 20:03:35.883360 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:35.881762    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcceb7d6-67ba-4407-9609-ffdc6758004e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9d3273d-b8v6k\" (UID: \"fcceb7d6-67ba-4407-9609-ffdc6758004e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9d3273d-b8v6k"
Oct 31 20:03:35.888506 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:35.888478    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9slq2\" (UniqueName: \"kubernetes.io/projected/fcceb7d6-67ba-4407-9609-ffdc6758004e-kube-api-access-9slq2\") pod \"maas-api-revoke-sub-c9d3273d-b8v6k\" (UID: \"fcceb7d6-67ba-4407-9609-ffdc6758004e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9d3273d-b8v6k"
Oct 31 20:03:35.954797 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:35.954764    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9841bd1b-461e-43fa-b433-793a0fd9fd97-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4c5b9b6-wzn7j\" (UID: \"9841bd1b-461e-43fa-b433-793a0fd9fd97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c5b9b6-wzn7j"
Oct 31 20:03:35.954940 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:35.954840    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxbw2\" (UniqueName: \"kubernetes.io/projected/9841bd1b-461e-43fa-b433-793a0fd9fd97-kube-api-access-zxbw2\") pod \"maas-api-revoke-sub-f4c5b9b6-wzn7j\" (UID: \"9841bd1b-461e-43fa-b433-793a0fd9fd97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c5b9b6-wzn7j"
Oct 31 20:03:35.954940 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:35.954902    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9841bd1b-461e-43fa-b433-793a0fd9fd97-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4c5b9b6-wzn7j\" (UID: \"9841bd1b-461e-43fa-b433-793a0fd9fd97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c5b9b6-wzn7j"
Oct 31 20:03:35.980394 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:35.955441    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9841bd1b-461e-43fa-b433-793a0fd9fd97-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4c5b9b6-wzn7j\" (UID: \"9841bd1b-461e-43fa-b433-793a0fd9fd97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c5b9b6-wzn7j"
Oct 31 20:03:35.983595 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:35.983565    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9841bd1b-461e-43fa-b433-793a0fd9fd97-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4c5b9b6-wzn7j\" (UID: \"9841bd1b-461e-43fa-b433-793a0fd9fd97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c5b9b6-wzn7j"
Oct 31 20:03:35.984392 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:35.984369    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxbw2\" (UniqueName: \"kubernetes.io/projected/9841bd1b-461e-43fa-b433-793a0fd9fd97-kube-api-access-zxbw2\") pod \"maas-api-revoke-sub-f4c5b9b6-wzn7j\" (UID: \"9841bd1b-461e-43fa-b433-793a0fd9fd97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c5b9b6-wzn7j"
Oct 31 20:03:36.006450 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:36.006411    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:36.030348 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:36.030291    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9d3273d-b8v6k"
Oct 31 20:03:36.103286 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:36.101349    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c5b9b6-wzn7j"
Oct 31 20:03:36.148014 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:36.147977    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c55086-fx4jv" event={"ID":"48a098c1-470e-4994-845a-3d34a8618ba9","Type":"ContainerDied","Data":"fb349de9792b255ac9d5d4afd835da5ec31e474ea5b389737a10717ab72c50f0"}
Oct 31 20:03:36.148123 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:36.148082    4645 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb349de9792b255ac9d5d4afd835da5ec31e474ea5b389737a10717ab72c50f0"
Oct 31 20:03:36.148186 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:36.148166    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c55086-fx4jv"
Oct 31 20:03:36.291433 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:03:36.288318    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 20:03:36.439218 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:36.439183    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9d3273d-b8v6k"]
Oct 31 20:03:36.440016 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:03:36.439986    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcceb7d6_67ba_4407_9609_ffdc6758004e.slice/crio-075c8af9d0b4f1be1deb1df4b01d3796551bcca3a581a544f9c4879238065c44 WatchSource:0}: Error finding container 075c8af9d0b4f1be1deb1df4b01d3796551bcca3a581a544f9c4879238065c44: Status 404 returned error can't find the container with id 075c8af9d0b4f1be1deb1df4b01d3796551bcca3a581a544f9c4879238065c44
Oct 31 20:03:36.445340 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:36.445306    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4c5b9b6-wzn7j"]
Oct 31 20:03:36.447427 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:03:36.447405    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9841bd1b_461e_43fa_b433_793a0fd9fd97.slice/crio-9b136cbb9f92363dc752556e842a2e30ef5e97d44e29f65c3e6866fd49455555 WatchSource:0}: Error finding container 9b136cbb9f92363dc752556e842a2e30ef5e97d44e29f65c3e6866fd49455555: Status 404 returned error can't find the container with id 9b136cbb9f92363dc752556e842a2e30ef5e97d44e29f65c3e6866fd49455555
Oct 31 20:03:36.800157 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:36.800128    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:36.847376 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:36.847233    4645 patch_prober.go:29] interesting pod/e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused" start-of-body=
Oct 31 20:03:36.847376 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:36.847297    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb" podUID="ba05261e-eadf-44c3-abda-cb561b4ea6a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused"
Oct 31 20:03:37.187575 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:37.187537    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c5b9b6-wzn7j" event={"ID":"9841bd1b-461e-43fa-b433-793a0fd9fd97","Type":"ContainerStarted","Data":"0d14e2988c30c9a2204c0ae9b8625d5021f8c135f10fe67cd23beae095e5c238"}
Oct 31 20:03:37.187777 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:37.187607    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c5b9b6-wzn7j" event={"ID":"9841bd1b-461e-43fa-b433-793a0fd9fd97","Type":"ContainerStarted","Data":"9b136cbb9f92363dc752556e842a2e30ef5e97d44e29f65c3e6866fd49455555"}
Oct 31 20:03:37.189715 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:37.189684    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9d3273d-b8v6k" event={"ID":"fcceb7d6-67ba-4407-9609-ffdc6758004e","Type":"ContainerStarted","Data":"3d9c6e4480379edf4e051b7140bc29078012459e683a0c6dcac0af259d5a489c"}
Oct 31 20:03:37.189840 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:37.189733    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9d3273d-b8v6k" event={"ID":"fcceb7d6-67ba-4407-9609-ffdc6758004e","Type":"ContainerStarted","Data":"075c8af9d0b4f1be1deb1df4b01d3796551bcca3a581a544f9c4879238065c44"}
Oct 31 20:03:37.210630 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:37.210583    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c5b9b6-wzn7j" podStartSLOduration=2.210570916 podStartE2EDuration="2.210570916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:03:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:03:37.206011966 +0000 UTC m=+2647.779246097" watchObservedRunningTime="2026-10-31 20:03:37.210570916 +0000 UTC m=+2647.783805045"
Oct 31 20:03:37.224253 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:37.224213    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9d3273d-b8v6k" podStartSLOduration=2.224200563 podStartE2EDuration="2.224200563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:03:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:03:37.222876206 +0000 UTC m=+2647.796110335" watchObservedRunningTime="2026-10-31 20:03:37.224200563 +0000 UTC m=+2647.797434688"
Oct 31 20:03:37.607793 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:37.607766    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:37.840659 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:37.840582    4645 patch_prober.go:29] interesting pod/e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused" start-of-body=
Oct 31 20:03:37.840659 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:37.840632    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb" podUID="ba05261e-eadf-44c3-abda-cb561b4ea6a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.24:15021/healthz/ready\": dial tcp 10.133.1.24:15021: connect: connection refused"
Oct 31 20:03:38.202781 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:38.202740    4645 generic.go:356] "Generic (PLEG): container finished" podID="9841bd1b-461e-43fa-b433-793a0fd9fd97" containerID="0d14e2988c30c9a2204c0ae9b8625d5021f8c135f10fe67cd23beae095e5c238" exitCode=0
Oct 31 20:03:38.202993 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:38.202817    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c5b9b6-wzn7j" event={"ID":"9841bd1b-461e-43fa-b433-793a0fd9fd97","Type":"ContainerDied","Data":"0d14e2988c30c9a2204c0ae9b8625d5021f8c135f10fe67cd23beae095e5c238"}
Oct 31 20:03:38.205053 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:38.205027    4645 generic.go:356] "Generic (PLEG): container finished" podID="fcceb7d6-67ba-4407-9609-ffdc6758004e" containerID="3d9c6e4480379edf4e051b7140bc29078012459e683a0c6dcac0af259d5a489c" exitCode=0
Oct 31 20:03:38.205214 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:38.205165    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9d3273d-b8v6k" event={"ID":"fcceb7d6-67ba-4407-9609-ffdc6758004e","Type":"ContainerDied","Data":"3d9c6e4480379edf4e051b7140bc29078012459e683a0c6dcac0af259d5a489c"}
Oct 31 20:03:38.724357 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:38.721531    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:38.845623 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:38.845590    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb"
Oct 31 20:03:38.846105 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:38.846076    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb"
Oct 31 20:03:38.847282 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:38.847259    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb"
Oct 31 20:03:39.402626 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:39.400286    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c5b9b6-wzn7j"
Oct 31 20:03:39.446099 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:39.446075    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9d3273d-b8v6k"
Oct 31 20:03:39.454656 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:39.454625    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9841bd1b-461e-43fa-b433-793a0fd9fd97-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9841bd1b-461e-43fa-b433-793a0fd9fd97-maas-api-xks-ca\") pod \"9841bd1b-461e-43fa-b433-793a0fd9fd97\" (UID: \"9841bd1b-461e-43fa-b433-793a0fd9fd97\") "
Oct 31 20:03:39.454810 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:39.454743    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9841bd1b-461e-43fa-b433-793a0fd9fd97-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9841bd1b-461e-43fa-b433-793a0fd9fd97-maas-api-openshift-ca\") pod \"9841bd1b-461e-43fa-b433-793a0fd9fd97\" (UID: \"9841bd1b-461e-43fa-b433-793a0fd9fd97\") "
Oct 31 20:03:39.454888 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:39.454829    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9841bd1b-461e-43fa-b433-793a0fd9fd97-kube-api-access-zxbw2\" (UniqueName: \"kubernetes.io/projected/9841bd1b-461e-43fa-b433-793a0fd9fd97-kube-api-access-zxbw2\") pod \"9841bd1b-461e-43fa-b433-793a0fd9fd97\" (UID: \"9841bd1b-461e-43fa-b433-793a0fd9fd97\") "
Oct 31 20:03:39.455161 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:39.455133    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9841bd1b-461e-43fa-b433-793a0fd9fd97-maas-api-openshift-ca" pod "9841bd1b-461e-43fa-b433-793a0fd9fd97" (UID: "9841bd1b-461e-43fa-b433-793a0fd9fd97"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:03:39.455829 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:39.455804    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9841bd1b-461e-43fa-b433-793a0fd9fd97-maas-api-openshift-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:39.457606 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:39.457574    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9841bd1b-461e-43fa-b433-793a0fd9fd97-maas-api-xks-ca" pod "9841bd1b-461e-43fa-b433-793a0fd9fd97" (UID: "9841bd1b-461e-43fa-b433-793a0fd9fd97"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:03:39.457811 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:39.457791    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9841bd1b-461e-43fa-b433-793a0fd9fd97-kube-api-access-zxbw2" pod "9841bd1b-461e-43fa-b433-793a0fd9fd97" (UID: "9841bd1b-461e-43fa-b433-793a0fd9fd97"). InnerVolumeSpecName "kube-api-access-zxbw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:03:39.556304 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:39.556273    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcceb7d6-67ba-4407-9609-ffdc6758004e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcceb7d6-67ba-4407-9609-ffdc6758004e-maas-api-xks-ca\") pod \"fcceb7d6-67ba-4407-9609-ffdc6758004e\" (UID: \"fcceb7d6-67ba-4407-9609-ffdc6758004e\") "
Oct 31 20:03:39.556617 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:39.556601    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcceb7d6-67ba-4407-9609-ffdc6758004e-kube-api-access-9slq2\" (UniqueName: \"kubernetes.io/projected/fcceb7d6-67ba-4407-9609-ffdc6758004e-kube-api-access-9slq2\") pod \"fcceb7d6-67ba-4407-9609-ffdc6758004e\" (UID: \"fcceb7d6-67ba-4407-9609-ffdc6758004e\") "
Oct 31 20:03:39.556672 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:39.556648    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcceb7d6-67ba-4407-9609-ffdc6758004e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcceb7d6-67ba-4407-9609-ffdc6758004e-maas-api-openshift-ca\") pod \"fcceb7d6-67ba-4407-9609-ffdc6758004e\" (UID: \"fcceb7d6-67ba-4407-9609-ffdc6758004e\") "
Oct 31 20:03:39.557079 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:39.557050    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcceb7d6-67ba-4407-9609-ffdc6758004e-maas-api-openshift-ca" pod "fcceb7d6-67ba-4407-9609-ffdc6758004e" (UID: "fcceb7d6-67ba-4407-9609-ffdc6758004e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:03:39.557264 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:39.557249    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9841bd1b-461e-43fa-b433-793a0fd9fd97-maas-api-xks-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:39.557321 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:39.557267    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcceb7d6-67ba-4407-9609-ffdc6758004e-maas-api-openshift-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:39.557321 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:39.557277    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxbw2\" (UniqueName: \"kubernetes.io/projected/9841bd1b-461e-43fa-b433-793a0fd9fd97-kube-api-access-zxbw2\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:39.558487 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:39.558466    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcceb7d6-67ba-4407-9609-ffdc6758004e-maas-api-xks-ca" pod "fcceb7d6-67ba-4407-9609-ffdc6758004e" (UID: "fcceb7d6-67ba-4407-9609-ffdc6758004e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:03:39.558896 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:39.558877    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcceb7d6-67ba-4407-9609-ffdc6758004e-kube-api-access-9slq2" pod "fcceb7d6-67ba-4407-9609-ffdc6758004e" (UID: "fcceb7d6-67ba-4407-9609-ffdc6758004e"). InnerVolumeSpecName "kube-api-access-9slq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:03:39.659053 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:39.659022    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9slq2\" (UniqueName: \"kubernetes.io/projected/fcceb7d6-67ba-4407-9609-ffdc6758004e-kube-api-access-9slq2\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:39.659053 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:39.659051    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcceb7d6-67ba-4407-9609-ffdc6758004e-maas-api-xks-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:39.895656 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:39.895409    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:40.222161 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:40.222132    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9d3273d-b8v6k"
Oct 31 20:03:40.222365 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:40.222170    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9d3273d-b8v6k" event={"ID":"fcceb7d6-67ba-4407-9609-ffdc6758004e","Type":"ContainerDied","Data":"075c8af9d0b4f1be1deb1df4b01d3796551bcca3a581a544f9c4879238065c44"}
Oct 31 20:03:40.222365 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:40.222233    4645 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="075c8af9d0b4f1be1deb1df4b01d3796551bcca3a581a544f9c4879238065c44"
Oct 31 20:03:40.223725 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:40.223707    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c5b9b6-wzn7j"
Oct 31 20:03:40.223847 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:40.223735    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4c5b9b6-wzn7j" event={"ID":"9841bd1b-461e-43fa-b433-793a0fd9fd97","Type":"ContainerDied","Data":"9b136cbb9f92363dc752556e842a2e30ef5e97d44e29f65c3e6866fd49455555"}
Oct 31 20:03:40.223847 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:40.223772    4645 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b136cbb9f92363dc752556e842a2e30ef5e97d44e29f65c3e6866fd49455555"
Oct 31 20:03:40.836865 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:40.836828    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:41.789581 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:41.789539    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:42.211951 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:42.211918    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25607571-wl7gs"]
Oct 31 20:03:42.212746 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:42.212723    4645 cpu_manager.go:414] "containerMap: removing container" podUID="9841bd1b-461e-43fa-b433-793a0fd9fd97" containerName="revoke-keys"
Oct 31 20:03:42.212746 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:42.212745    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9841bd1b-461e-43fa-b433-793a0fd9fd97" containerName="revoke-keys"
Oct 31 20:03:42.212937 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:42.212793    4645 cpu_manager.go:414] "containerMap: removing container" podUID="fcceb7d6-67ba-4407-9609-ffdc6758004e" containerName="revoke-keys"
Oct 31 20:03:42.212937 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:42.212802    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcceb7d6-67ba-4407-9609-ffdc6758004e" containerName="revoke-keys"
Oct 31 20:03:42.213050 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:42.212982    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcceb7d6-67ba-4407-9609-ffdc6758004e" containerName="revoke-keys"
Oct 31 20:03:42.213050 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:42.213008    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="9841bd1b-461e-43fa-b433-793a0fd9fd97" containerName="revoke-keys"
Oct 31 20:03:42.216904 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:42.216883    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25607571-wl7gs"
Oct 31 20:03:42.220421 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:42.220398    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 20:03:42.220878 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:42.220760    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:03:42.226447 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:42.226425    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25607571-wl7gs"]
Oct 31 20:03:42.321740 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:42.321690    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hr58\" (UniqueName: \"kubernetes.io/projected/65db169f-9733-4f67-b773-a012c8c308aa-kube-api-access-9hr58\") pod \"maas-api-revoke-sub-25607571-wl7gs\" (UID: \"65db169f-9733-4f67-b773-a012c8c308aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25607571-wl7gs"
Oct 31 20:03:42.321935 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:42.321868    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65db169f-9733-4f67-b773-a012c8c308aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25607571-wl7gs\" (UID: \"65db169f-9733-4f67-b773-a012c8c308aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25607571-wl7gs"
Oct 31 20:03:42.322054 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:42.322037    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65db169f-9733-4f67-b773-a012c8c308aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25607571-wl7gs\" (UID: \"65db169f-9733-4f67-b773-a012c8c308aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25607571-wl7gs"
Oct 31 20:03:42.423115 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:42.423081    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65db169f-9733-4f67-b773-a012c8c308aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25607571-wl7gs\" (UID: \"65db169f-9733-4f67-b773-a012c8c308aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25607571-wl7gs"
Oct 31 20:03:42.423297 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:42.423193    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65db169f-9733-4f67-b773-a012c8c308aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25607571-wl7gs\" (UID: \"65db169f-9733-4f67-b773-a012c8c308aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25607571-wl7gs"
Oct 31 20:03:42.423383 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:42.423323    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hr58\" (UniqueName: \"kubernetes.io/projected/65db169f-9733-4f67-b773-a012c8c308aa-kube-api-access-9hr58\") pod \"maas-api-revoke-sub-25607571-wl7gs\" (UID: \"65db169f-9733-4f67-b773-a012c8c308aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25607571-wl7gs"
Oct 31 20:03:42.423901 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:42.423877    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65db169f-9733-4f67-b773-a012c8c308aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25607571-wl7gs\" (UID: \"65db169f-9733-4f67-b773-a012c8c308aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25607571-wl7gs"
Oct 31 20:03:42.425544 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:42.425514    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65db169f-9733-4f67-b773-a012c8c308aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25607571-wl7gs\" (UID: \"65db169f-9733-4f67-b773-a012c8c308aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25607571-wl7gs"
Oct 31 20:03:42.434631 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:42.434607    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hr58\" (UniqueName: \"kubernetes.io/projected/65db169f-9733-4f67-b773-a012c8c308aa-kube-api-access-9hr58\") pod \"maas-api-revoke-sub-25607571-wl7gs\" (UID: \"65db169f-9733-4f67-b773-a012c8c308aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25607571-wl7gs"
Oct 31 20:03:42.540430 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:42.540316    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25607571-wl7gs"
Oct 31 20:03:42.748910 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:42.748875    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:42.853640 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:42.853604    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25607571-wl7gs"]
Oct 31 20:03:43.254980 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:43.254942    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25607571-wl7gs" event={"ID":"65db169f-9733-4f67-b773-a012c8c308aa","Type":"ContainerStarted","Data":"491df582afe74d08057005503e8b6ab0dda154ee0246dc6e759498c70e6c0fe4"}
Oct 31 20:03:43.255188 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:43.255007    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25607571-wl7gs" event={"ID":"65db169f-9733-4f67-b773-a012c8c308aa","Type":"ContainerStarted","Data":"ae91f86248f79407e16e3aeaa559b9c135e084a7faaba803346ebc732e1ebc6f"}
Oct 31 20:03:43.275954 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:43.275898    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25607571-wl7gs" podStartSLOduration=1.275884687 podStartE2EDuration="1.275884687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:03:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:03:43.272545674 +0000 UTC m=+2653.845779802" watchObservedRunningTime="2026-10-31 20:03:43.275884687 +0000 UTC m=+2653.849118834"
Oct 31 20:03:43.475541 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:43.475426    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:44.165649 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:44.165610    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:44.166081 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:44.165721    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c734f5-684c4f6c9d-xdjpr"]
Oct 31 20:03:44.166081 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:44.165968    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-c734f5-684c4f6c9d-xdjpr" podUID="b8e321a4-db7b-4db9-ab08-d70604893cef" containerName="payload-processing" containerID="cri-o://f61a0e09d9fffc1217432055e1d7383a9fb3ea5c9354f1f41328c59f1136d3ff" gracePeriod=30
Oct 31 20:03:44.190756 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:03:44.183676    4645 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8e321a4_db7b_4db9_ab08_d70604893cef.slice/crio-f61a0e09d9fffc1217432055e1d7383a9fb3ea5c9354f1f41328c59f1136d3ff.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 20:03:44.190756 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:44.186437    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c734f5-786f585bf5-bs8jm"]
Oct 31 20:03:44.190756 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:44.186652    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c734f5-786f585bf5-bs8jm" podUID="c58165b3-72d7-4195-b703-17b3104d7207" containerName="payload-pre-processing" containerID="cri-o://4217234607ca6c91f1e0a1834bad73da3bd005576bfefcb1c6809b3cdea67264" gracePeriod=30
Oct 31 20:03:44.269562 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:44.269523    4645 generic.go:356] "Generic (PLEG): container finished" podID="65db169f-9733-4f67-b773-a012c8c308aa" containerID="491df582afe74d08057005503e8b6ab0dda154ee0246dc6e759498c70e6c0fe4" exitCode=0
Oct 31 20:03:44.269741 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:44.269591    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25607571-wl7gs" event={"ID":"65db169f-9733-4f67-b773-a012c8c308aa","Type":"ContainerDied","Data":"491df582afe74d08057005503e8b6ab0dda154ee0246dc6e759498c70e6c0fe4"}
Oct 31 20:03:44.342663 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:44.342593    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:44.653360 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:44.640096    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:44.730691 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:44.705946    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c734f5-jf7w8"]
Oct 31 20:03:44.750867 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:44.750831    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c734f5-jf7w8"]
Oct 31 20:03:44.753322 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:44.753281    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:44.798236 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:44.798205    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c734f5-684c4f6c9d-xdjpr"
Oct 31 20:03:44.898350 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:44.895000    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:44.948227 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:44.948193    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8e321a4-db7b-4db9-ab08-d70604893cef-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8e321a4-db7b-4db9-ab08-d70604893cef-plugins-config-volume\") pod \"b8e321a4-db7b-4db9-ab08-d70604893cef\" (UID: \"b8e321a4-db7b-4db9-ab08-d70604893cef\") "
Oct 31 20:03:44.948479 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:44.948459    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8e321a4-db7b-4db9-ab08-d70604893cef-kube-api-access-x7lzl\" (UniqueName: \"kubernetes.io/projected/b8e321a4-db7b-4db9-ab08-d70604893cef-kube-api-access-x7lzl\") pod \"b8e321a4-db7b-4db9-ab08-d70604893cef\" (UID: \"b8e321a4-db7b-4db9-ab08-d70604893cef\") "
Oct 31 20:03:44.949390 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:44.949312    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8e321a4-db7b-4db9-ab08-d70604893cef-plugins-config-volume" pod "b8e321a4-db7b-4db9-ab08-d70604893cef" (UID: "b8e321a4-db7b-4db9-ab08-d70604893cef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:03:44.955455 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:44.955419    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8e321a4-db7b-4db9-ab08-d70604893cef-kube-api-access-x7lzl" pod "b8e321a4-db7b-4db9-ab08-d70604893cef" (UID: "b8e321a4-db7b-4db9-ab08-d70604893cef"). InnerVolumeSpecName "kube-api-access-x7lzl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:03:44.988244 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:44.986962    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c734f5-786f585bf5-bs8jm"
Oct 31 20:03:45.050098 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:45.050069    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c58165b3-72d7-4195-b703-17b3104d7207-kube-api-access-wl64n\" (UniqueName: \"kubernetes.io/projected/c58165b3-72d7-4195-b703-17b3104d7207-kube-api-access-wl64n\") pod \"c58165b3-72d7-4195-b703-17b3104d7207\" (UID: \"c58165b3-72d7-4195-b703-17b3104d7207\") "
Oct 31 20:03:45.050256 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:45.050144    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c58165b3-72d7-4195-b703-17b3104d7207-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c58165b3-72d7-4195-b703-17b3104d7207-plugins-config-volume\") pod \"c58165b3-72d7-4195-b703-17b3104d7207\" (UID: \"c58165b3-72d7-4195-b703-17b3104d7207\") "
Oct 31 20:03:45.050547 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:45.050524    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c58165b3-72d7-4195-b703-17b3104d7207-plugins-config-volume" pod "c58165b3-72d7-4195-b703-17b3104d7207" (UID: "c58165b3-72d7-4195-b703-17b3104d7207"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:03:45.050675 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:45.050660    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7lzl\" (UniqueName: \"kubernetes.io/projected/b8e321a4-db7b-4db9-ab08-d70604893cef-kube-api-access-x7lzl\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:45.050719 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:45.050678    4645 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c58165b3-72d7-4195-b703-17b3104d7207-plugins-config-volume\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:45.050719 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:45.050688    4645 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8e321a4-db7b-4db9-ab08-d70604893cef-plugins-config-volume\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:45.052231 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:45.052209    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c58165b3-72d7-4195-b703-17b3104d7207-kube-api-access-wl64n" pod "c58165b3-72d7-4195-b703-17b3104d7207" (UID: "c58165b3-72d7-4195-b703-17b3104d7207"). InnerVolumeSpecName "kube-api-access-wl64n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:03:45.151878 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:45.151849    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wl64n\" (UniqueName: \"kubernetes.io/projected/c58165b3-72d7-4195-b703-17b3104d7207-kube-api-access-wl64n\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:45.240590 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:45.240531    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:45.279025 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:45.279002    4645 generic.go:356] "Generic (PLEG): container finished" podID="c58165b3-72d7-4195-b703-17b3104d7207" containerID="4217234607ca6c91f1e0a1834bad73da3bd005576bfefcb1c6809b3cdea67264" exitCode=0
Oct 31 20:03:45.279133 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:45.279049    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c734f5-786f585bf5-bs8jm"
Oct 31 20:03:45.279133 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:45.279089    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c734f5-786f585bf5-bs8jm" event={"ID":"c58165b3-72d7-4195-b703-17b3104d7207","Type":"ContainerDied","Data":"4217234607ca6c91f1e0a1834bad73da3bd005576bfefcb1c6809b3cdea67264"}
Oct 31 20:03:45.279228 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:45.279138    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c734f5-786f585bf5-bs8jm" event={"ID":"c58165b3-72d7-4195-b703-17b3104d7207","Type":"ContainerDied","Data":"f6936057dec68641affce77bba6c7799332cdb212eace4e20ca364d5ba493500"}
Oct 31 20:03:45.279228 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:45.279167    4645 scope.go:122] "RemoveContainer" containerID="4217234607ca6c91f1e0a1834bad73da3bd005576bfefcb1c6809b3cdea67264"
Oct 31 20:03:45.280667 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:45.280644    4645 generic.go:356] "Generic (PLEG): container finished" podID="b8e321a4-db7b-4db9-ab08-d70604893cef" containerID="f61a0e09d9fffc1217432055e1d7383a9fb3ea5c9354f1f41328c59f1136d3ff" exitCode=0
Oct 31 20:03:45.280773 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:45.280686    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c734f5-684c4f6c9d-xdjpr"
Oct 31 20:03:45.280773 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:45.280718    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c734f5-684c4f6c9d-xdjpr" event={"ID":"b8e321a4-db7b-4db9-ab08-d70604893cef","Type":"ContainerDied","Data":"f61a0e09d9fffc1217432055e1d7383a9fb3ea5c9354f1f41328c59f1136d3ff"}
Oct 31 20:03:45.280939 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:45.280910    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c734f5-684c4f6c9d-xdjpr" event={"ID":"b8e321a4-db7b-4db9-ab08-d70604893cef","Type":"ContainerDied","Data":"1df7e2f662ebe37d05b2955a077ebdde3c352f23fae8d9108c0c7ea7d971d3eb"}
Oct 31 20:03:45.293052 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:45.293032    4645 scope.go:122] "RemoveContainer" containerID="4217234607ca6c91f1e0a1834bad73da3bd005576bfefcb1c6809b3cdea67264"
Oct 31 20:03:45.293463 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:03:45.293436    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4217234607ca6c91f1e0a1834bad73da3bd005576bfefcb1c6809b3cdea67264\": container with ID starting with 4217234607ca6c91f1e0a1834bad73da3bd005576bfefcb1c6809b3cdea67264 not found: ID does not exist" containerID="4217234607ca6c91f1e0a1834bad73da3bd005576bfefcb1c6809b3cdea67264"
Oct 31 20:03:45.293570 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:45.293478    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4217234607ca6c91f1e0a1834bad73da3bd005576bfefcb1c6809b3cdea67264"} err="failed to get container status \"4217234607ca6c91f1e0a1834bad73da3bd005576bfefcb1c6809b3cdea67264\": rpc error: code = NotFound desc = could not find container \"4217234607ca6c91f1e0a1834bad73da3bd005576bfefcb1c6809b3cdea67264\": container with ID starting with 4217234607ca6c91f1e0a1834bad73da3bd005576bfefcb1c6809b3cdea67264 not found: ID does not exist"
Oct 31 20:03:45.293570 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:45.293496    4645 scope.go:122] "RemoveContainer" containerID="f61a0e09d9fffc1217432055e1d7383a9fb3ea5c9354f1f41328c59f1136d3ff"
Oct 31 20:03:45.309602 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:45.309500    4645 scope.go:122] "RemoveContainer" containerID="f61a0e09d9fffc1217432055e1d7383a9fb3ea5c9354f1f41328c59f1136d3ff"
Oct 31 20:03:45.310248 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:03:45.310222    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f61a0e09d9fffc1217432055e1d7383a9fb3ea5c9354f1f41328c59f1136d3ff\": container with ID starting with f61a0e09d9fffc1217432055e1d7383a9fb3ea5c9354f1f41328c59f1136d3ff not found: ID does not exist" containerID="f61a0e09d9fffc1217432055e1d7383a9fb3ea5c9354f1f41328c59f1136d3ff"
Oct 31 20:03:45.310357 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:45.310259    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f61a0e09d9fffc1217432055e1d7383a9fb3ea5c9354f1f41328c59f1136d3ff"} err="failed to get container status \"f61a0e09d9fffc1217432055e1d7383a9fb3ea5c9354f1f41328c59f1136d3ff\": rpc error: code = NotFound desc = could not find container \"f61a0e09d9fffc1217432055e1d7383a9fb3ea5c9354f1f41328c59f1136d3ff\": container with ID starting with f61a0e09d9fffc1217432055e1d7383a9fb3ea5c9354f1f41328c59f1136d3ff not found: ID does not exist"
Oct 31 20:03:45.312599 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:45.312577    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c734f5-786f585bf5-bs8jm"]
Oct 31 20:03:45.317971 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:45.317947    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c734f5-786f585bf5-bs8jm"]
Oct 31 20:03:45.329639 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:45.329616    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c734f5-684c4f6c9d-xdjpr"]
Oct 31 20:03:45.334683 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:45.334660    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c734f5-684c4f6c9d-xdjpr"]
Oct 31 20:03:45.445165 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:45.445140    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25607571-wl7gs"
Oct 31 20:03:45.456654 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:45.456632    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65db169f-9733-4f67-b773-a012c8c308aa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65db169f-9733-4f67-b773-a012c8c308aa-maas-api-openshift-ca\") pod \"65db169f-9733-4f67-b773-a012c8c308aa\" (UID: \"65db169f-9733-4f67-b773-a012c8c308aa\") "
Oct 31 20:03:45.456887 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:45.456872    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65db169f-9733-4f67-b773-a012c8c308aa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65db169f-9733-4f67-b773-a012c8c308aa-maas-api-xks-ca\") pod \"65db169f-9733-4f67-b773-a012c8c308aa\" (UID: \"65db169f-9733-4f67-b773-a012c8c308aa\") "
Oct 31 20:03:45.456937 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:45.456921    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65db169f-9733-4f67-b773-a012c8c308aa-kube-api-access-9hr58\" (UniqueName: \"kubernetes.io/projected/65db169f-9733-4f67-b773-a012c8c308aa-kube-api-access-9hr58\") pod \"65db169f-9733-4f67-b773-a012c8c308aa\" (UID: \"65db169f-9733-4f67-b773-a012c8c308aa\") "
Oct 31 20:03:45.456972 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:45.456933    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65db169f-9733-4f67-b773-a012c8c308aa-maas-api-openshift-ca" pod "65db169f-9733-4f67-b773-a012c8c308aa" (UID: "65db169f-9733-4f67-b773-a012c8c308aa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:03:45.457487 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:45.457465    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65db169f-9733-4f67-b773-a012c8c308aa-maas-api-openshift-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:45.458927 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:45.458902    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65db169f-9733-4f67-b773-a012c8c308aa-maas-api-xks-ca" pod "65db169f-9733-4f67-b773-a012c8c308aa" (UID: "65db169f-9733-4f67-b773-a012c8c308aa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:03:45.459014 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:45.458966    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65db169f-9733-4f67-b773-a012c8c308aa-kube-api-access-9hr58" pod "65db169f-9733-4f67-b773-a012c8c308aa" (UID: "65db169f-9733-4f67-b773-a012c8c308aa"). InnerVolumeSpecName "kube-api-access-9hr58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:03:45.558662 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:45.558604    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65db169f-9733-4f67-b773-a012c8c308aa-maas-api-xks-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:45.558662 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:45.558629    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hr58\" (UniqueName: \"kubernetes.io/projected/65db169f-9733-4f67-b773-a012c8c308aa-kube-api-access-9hr58\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:45.619798 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:45.619771    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09052cb1-f1a8-4ccf-8583-d9b92bf9d897" path="/var/lib/kubelet/pods/09052cb1-f1a8-4ccf-8583-d9b92bf9d897/volumes"
Oct 31 20:03:45.620234 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:45.620219    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8e321a4-db7b-4db9-ab08-d70604893cef" path="/var/lib/kubelet/pods/b8e321a4-db7b-4db9-ab08-d70604893cef/volumes"
Oct 31 20:03:45.620665 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:45.620651    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c58165b3-72d7-4195-b703-17b3104d7207" path="/var/lib/kubelet/pods/c58165b3-72d7-4195-b703-17b3104d7207/volumes"
Oct 31 20:03:45.995159 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:45.995116    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:46.290602 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:46.290521    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25607571-wl7gs"
Oct 31 20:03:46.290602 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:46.290559    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25607571-wl7gs" event={"ID":"65db169f-9733-4f67-b773-a012c8c308aa","Type":"ContainerDied","Data":"ae91f86248f79407e16e3aeaa559b9c135e084a7faaba803346ebc732e1ebc6f"}
Oct 31 20:03:46.291055 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:46.290610    4645 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae91f86248f79407e16e3aeaa559b9c135e084a7faaba803346ebc732e1ebc6f"
Oct 31 20:03:46.862137 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:46.862100    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:47.857892 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:47.857852    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:48.797190 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:48.797126    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:48.950977 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:48.950944    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:49.079284 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:49.076748    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:49.872800 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:49.872769    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:50.145430 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:50.144951    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:50.840582 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:50.840552    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:51.710361 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:51.710310    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:52.505581 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:52.505540    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:52.687342 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:52.687288    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8487c18-s65fh"]
Oct 31 20:03:52.688039 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:52.688022    4645 cpu_manager.go:414] "containerMap: removing container" podUID="c58165b3-72d7-4195-b703-17b3104d7207" containerName="payload-pre-processing"
Oct 31 20:03:52.688125 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:52.688040    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c58165b3-72d7-4195-b703-17b3104d7207" containerName="payload-pre-processing"
Oct 31 20:03:52.688125 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:52.688086    4645 cpu_manager.go:414] "containerMap: removing container" podUID="65db169f-9733-4f67-b773-a012c8c308aa" containerName="revoke-keys"
Oct 31 20:03:52.688125 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:52.688092    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65db169f-9733-4f67-b773-a012c8c308aa" containerName="revoke-keys"
Oct 31 20:03:52.688125 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:52.688108    4645 cpu_manager.go:414] "containerMap: removing container" podUID="b8e321a4-db7b-4db9-ab08-d70604893cef" containerName="payload-processing"
Oct 31 20:03:52.688125 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:52.688113    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8e321a4-db7b-4db9-ab08-d70604893cef" containerName="payload-processing"
Oct 31 20:03:52.688308 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:52.688233    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="c58165b3-72d7-4195-b703-17b3104d7207" containerName="payload-pre-processing"
Oct 31 20:03:52.688308 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:52.688246    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="65db169f-9733-4f67-b773-a012c8c308aa" containerName="revoke-keys"
Oct 31 20:03:52.688308 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:52.688253    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8e321a4-db7b-4db9-ab08-d70604893cef" containerName="payload-processing"
Oct 31 20:03:52.693179 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:52.693158    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8487c18-s65fh"
Oct 31 20:03:52.695278 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:52.695228    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:52.698070 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:52.698048    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:03:52.698563 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:52.698542    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 20:03:52.720253 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:52.720232    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8487c18-s65fh"]
Oct 31 20:03:52.825436 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:52.825350    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cpnn\" (UniqueName: \"kubernetes.io/projected/9e6f5b6b-6a09-4b98-82bd-cd6ad2899c0a-kube-api-access-7cpnn\") pod \"maas-api-revoke-sub-e8487c18-s65fh\" (UID: \"9e6f5b6b-6a09-4b98-82bd-cd6ad2899c0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8487c18-s65fh"
Oct 31 20:03:52.825594 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:52.825516    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e6f5b6b-6a09-4b98-82bd-cd6ad2899c0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8487c18-s65fh\" (UID: \"9e6f5b6b-6a09-4b98-82bd-cd6ad2899c0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8487c18-s65fh"
Oct 31 20:03:52.825811 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:52.825793    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e6f5b6b-6a09-4b98-82bd-cd6ad2899c0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8487c18-s65fh\" (UID: \"9e6f5b6b-6a09-4b98-82bd-cd6ad2899c0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8487c18-s65fh"
Oct 31 20:03:52.861888 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:52.861808    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:52.908634 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:52.908601    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w"]
Oct 31 20:03:52.909152 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:52.909119    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w" podUID="f655e4a8-4347-4492-b5c5-aa5e74e690ac" containerName="istio-proxy" containerID="cri-o://261c4acad2435340f349218a91ab931436d74fb224f2404d2a281cc939cfd258" gracePeriod=30
Oct 31 20:03:52.921482 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:52.921457    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w"]
Oct 31 20:03:52.921642 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:52.921625    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w" podUID="72bb253a-434b-4921-97f2-9f23df7249fe" containerName="istio-proxy" containerID="cri-o://c5963ca2e4f24daee898d56ffc52efd0c9db5a767b66dccd97a6914c2df41339" gracePeriod=30
Oct 31 20:03:52.927307 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:52.927278    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e6f5b6b-6a09-4b98-82bd-cd6ad2899c0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8487c18-s65fh\" (UID: \"9e6f5b6b-6a09-4b98-82bd-cd6ad2899c0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8487c18-s65fh"
Oct 31 20:03:52.927473 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:52.927455    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cpnn\" (UniqueName: \"kubernetes.io/projected/9e6f5b6b-6a09-4b98-82bd-cd6ad2899c0a-kube-api-access-7cpnn\") pod \"maas-api-revoke-sub-e8487c18-s65fh\" (UID: \"9e6f5b6b-6a09-4b98-82bd-cd6ad2899c0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8487c18-s65fh"
Oct 31 20:03:52.927566 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:52.927551    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e6f5b6b-6a09-4b98-82bd-cd6ad2899c0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8487c18-s65fh\" (UID: \"9e6f5b6b-6a09-4b98-82bd-cd6ad2899c0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8487c18-s65fh"
Oct 31 20:03:52.928171 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:52.928148    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e6f5b6b-6a09-4b98-82bd-cd6ad2899c0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8487c18-s65fh\" (UID: \"9e6f5b6b-6a09-4b98-82bd-cd6ad2899c0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8487c18-s65fh"
Oct 31 20:03:52.929655 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:52.929632    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e6f5b6b-6a09-4b98-82bd-cd6ad2899c0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8487c18-s65fh\" (UID: \"9e6f5b6b-6a09-4b98-82bd-cd6ad2899c0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8487c18-s65fh"
Oct 31 20:03:52.954496 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:52.954472    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cpnn\" (UniqueName: \"kubernetes.io/projected/9e6f5b6b-6a09-4b98-82bd-cd6ad2899c0a-kube-api-access-7cpnn\") pod \"maas-api-revoke-sub-e8487c18-s65fh\" (UID: \"9e6f5b6b-6a09-4b98-82bd-cd6ad2899c0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8487c18-s65fh"
Oct 31 20:03:53.013653 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:53.012681    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8487c18-s65fh"
Oct 31 20:03:53.090348 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:53.087255    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:53.492786 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:03:53.492757    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e6f5b6b_6a09_4b98_82bd_cd6ad2899c0a.slice/crio-015616971c886156df66ef92ed87c6f98e3c1bad02ce69d10bd1e301e378dee1 WatchSource:0}: Error finding container 015616971c886156df66ef92ed87c6f98e3c1bad02ce69d10bd1e301e378dee1: Status 404 returned error can't find the container with id 015616971c886156df66ef92ed87c6f98e3c1bad02ce69d10bd1e301e378dee1
Oct 31 20:03:53.492911 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:53.492879    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8487c18-s65fh"]
Oct 31 20:03:53.595948 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:53.595923    4645 patch_prober.go:29] interesting pod/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 20:03:53.596068 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:53.595978    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w" podUID="72bb253a-434b-4921-97f2-9f23df7249fe" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 20:03:54.005140 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:54.005108    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:54.362364 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:54.362293    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8487c18-s65fh" event={"ID":"9e6f5b6b-6a09-4b98-82bd-cd6ad2899c0a","Type":"ContainerStarted","Data":"b74947ef0513babd3d0cf7ff1ebfa0aeb5600e4869db5ed09e00075b04e5caf8"}
Oct 31 20:03:54.362568 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:54.362374    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8487c18-s65fh" event={"ID":"9e6f5b6b-6a09-4b98-82bd-cd6ad2899c0a","Type":"ContainerStarted","Data":"015616971c886156df66ef92ed87c6f98e3c1bad02ce69d10bd1e301e378dee1"}
Oct 31 20:03:54.383230 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:54.383180    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8487c18-s65fh" podStartSLOduration=2.38316558 podStartE2EDuration="2.38316558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:03:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:03:54.380748677 +0000 UTC m=+2664.953982807" watchObservedRunningTime="2026-10-31 20:03:54.38316558 +0000 UTC m=+2664.956399757"
Oct 31 20:03:54.950207 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:54.950168    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:55.371724 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:55.371640    4645 generic.go:356] "Generic (PLEG): container finished" podID="9e6f5b6b-6a09-4b98-82bd-cd6ad2899c0a" containerID="b74947ef0513babd3d0cf7ff1ebfa0aeb5600e4869db5ed09e00075b04e5caf8" exitCode=0
Oct 31 20:03:55.371724 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:55.371707    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8487c18-s65fh" event={"ID":"9e6f5b6b-6a09-4b98-82bd-cd6ad2899c0a","Type":"ContainerDied","Data":"b74947ef0513babd3d0cf7ff1ebfa0aeb5600e4869db5ed09e00075b04e5caf8"}
Oct 31 20:03:55.943040 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:55.943004    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:56.554990 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:56.554968    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8487c18-s65fh"
Oct 31 20:03:56.693777 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:56.693748    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e6f5b6b-6a09-4b98-82bd-cd6ad2899c0a-kube-api-access-7cpnn\" (UniqueName: \"kubernetes.io/projected/9e6f5b6b-6a09-4b98-82bd-cd6ad2899c0a-kube-api-access-7cpnn\") pod \"9e6f5b6b-6a09-4b98-82bd-cd6ad2899c0a\" (UID: \"9e6f5b6b-6a09-4b98-82bd-cd6ad2899c0a\") "
Oct 31 20:03:56.693968 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:56.693867    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e6f5b6b-6a09-4b98-82bd-cd6ad2899c0a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e6f5b6b-6a09-4b98-82bd-cd6ad2899c0a-maas-api-xks-ca\") pod \"9e6f5b6b-6a09-4b98-82bd-cd6ad2899c0a\" (UID: \"9e6f5b6b-6a09-4b98-82bd-cd6ad2899c0a\") "
Oct 31 20:03:56.694027 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:56.694002    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e6f5b6b-6a09-4b98-82bd-cd6ad2899c0a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e6f5b6b-6a09-4b98-82bd-cd6ad2899c0a-maas-api-openshift-ca\") pod \"9e6f5b6b-6a09-4b98-82bd-cd6ad2899c0a\" (UID: \"9e6f5b6b-6a09-4b98-82bd-cd6ad2899c0a\") "
Oct 31 20:03:56.694432 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:56.694400    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e6f5b6b-6a09-4b98-82bd-cd6ad2899c0a-maas-api-openshift-ca" pod "9e6f5b6b-6a09-4b98-82bd-cd6ad2899c0a" (UID: "9e6f5b6b-6a09-4b98-82bd-cd6ad2899c0a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:03:56.694719 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:56.694697    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e6f5b6b-6a09-4b98-82bd-cd6ad2899c0a-maas-api-openshift-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:56.696038 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:56.696007    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e6f5b6b-6a09-4b98-82bd-cd6ad2899c0a-maas-api-xks-ca" pod "9e6f5b6b-6a09-4b98-82bd-cd6ad2899c0a" (UID: "9e6f5b6b-6a09-4b98-82bd-cd6ad2899c0a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:03:56.696140 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:56.696124    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e6f5b6b-6a09-4b98-82bd-cd6ad2899c0a-kube-api-access-7cpnn" pod "9e6f5b6b-6a09-4b98-82bd-cd6ad2899c0a" (UID: "9e6f5b6b-6a09-4b98-82bd-cd6ad2899c0a"). InnerVolumeSpecName "kube-api-access-7cpnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:03:56.749788 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:56.749757    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:56.798350 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:56.797431    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cpnn\" (UniqueName: \"kubernetes.io/projected/9e6f5b6b-6a09-4b98-82bd-cd6ad2899c0a-kube-api-access-7cpnn\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:56.798350 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:56.797467    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e6f5b6b-6a09-4b98-82bd-cd6ad2899c0a-maas-api-xks-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:57.218103 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:57.218061    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:57.346762 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:57.346723    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:57.393997 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:57.393945    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8487c18-s65fh" event={"ID":"9e6f5b6b-6a09-4b98-82bd-cd6ad2899c0a","Type":"ContainerDied","Data":"015616971c886156df66ef92ed87c6f98e3c1bad02ce69d10bd1e301e378dee1"}
Oct 31 20:03:57.394206 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:57.394067    4645 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="015616971c886156df66ef92ed87c6f98e3c1bad02ce69d10bd1e301e378dee1"
Oct 31 20:03:57.394206 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:57.394157    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8487c18-s65fh"
Oct 31 20:03:57.588811 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:57.588723    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:58.231690 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:58.231648    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w"
Oct 31 20:03:58.251755 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:58.251727    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w"
Oct 31 20:03:58.348317 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:58.348281    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f655e4a8-4347-4492-b5c5-aa5e74e690ac-kube-api-access-vvdq6\" (UniqueName: \"kubernetes.io/projected/f655e4a8-4347-4492-b5c5-aa5e74e690ac-kube-api-access-vvdq6\") pod \"f655e4a8-4347-4492-b5c5-aa5e74e690ac\" (UID: \"f655e4a8-4347-4492-b5c5-aa5e74e690ac\") "
Oct 31 20:03:58.348503 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:58.348353    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f655e4a8-4347-4492-b5c5-aa5e74e690ac-istio-token\" (UniqueName: \"kubernetes.io/projected/f655e4a8-4347-4492-b5c5-aa5e74e690ac-istio-token\") pod \"f655e4a8-4347-4492-b5c5-aa5e74e690ac\" (UID: \"f655e4a8-4347-4492-b5c5-aa5e74e690ac\") "
Oct 31 20:03:58.348503 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:58.348392    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f655e4a8-4347-4492-b5c5-aa5e74e690ac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f655e4a8-4347-4492-b5c5-aa5e74e690ac-workload-socket\") pod \"f655e4a8-4347-4492-b5c5-aa5e74e690ac\" (UID: \"f655e4a8-4347-4492-b5c5-aa5e74e690ac\") "
Oct 31 20:03:58.348503 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:58.348444    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72bb253a-434b-4921-97f2-9f23df7249fe-kube-api-access-kt4hf\" (UniqueName: \"kubernetes.io/projected/72bb253a-434b-4921-97f2-9f23df7249fe-kube-api-access-kt4hf\") pod \"72bb253a-434b-4921-97f2-9f23df7249fe\" (UID: \"72bb253a-434b-4921-97f2-9f23df7249fe\") "
Oct 31 20:03:58.348503 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:58.348494    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72bb253a-434b-4921-97f2-9f23df7249fe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72bb253a-434b-4921-97f2-9f23df7249fe-istio-data\") pod \"72bb253a-434b-4921-97f2-9f23df7249fe\" (UID: \"72bb253a-434b-4921-97f2-9f23df7249fe\") "
Oct 31 20:03:58.348775 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:58.348527    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/72bb253a-434b-4921-97f2-9f23df7249fe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72bb253a-434b-4921-97f2-9f23df7249fe-istio-podinfo\") pod \"72bb253a-434b-4921-97f2-9f23df7249fe\" (UID: \"72bb253a-434b-4921-97f2-9f23df7249fe\") "
Oct 31 20:03:58.348775 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:58.348576    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72bb253a-434b-4921-97f2-9f23df7249fe-istio-token\" (UniqueName: \"kubernetes.io/projected/72bb253a-434b-4921-97f2-9f23df7249fe-istio-token\") pod \"72bb253a-434b-4921-97f2-9f23df7249fe\" (UID: \"72bb253a-434b-4921-97f2-9f23df7249fe\") "
Oct 31 20:03:58.348775 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:58.348620    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f655e4a8-4347-4492-b5c5-aa5e74e690ac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f655e4a8-4347-4492-b5c5-aa5e74e690ac-istio-envoy\") pod \"f655e4a8-4347-4492-b5c5-aa5e74e690ac\" (UID: \"f655e4a8-4347-4492-b5c5-aa5e74e690ac\") "
Oct 31 20:03:58.348775 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:58.348661    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72bb253a-434b-4921-97f2-9f23df7249fe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72bb253a-434b-4921-97f2-9f23df7249fe-istio-envoy\") pod \"72bb253a-434b-4921-97f2-9f23df7249fe\" (UID: \"72bb253a-434b-4921-97f2-9f23df7249fe\") "
Oct 31 20:03:58.348775 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:58.348750    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f655e4a8-4347-4492-b5c5-aa5e74e690ac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f655e4a8-4347-4492-b5c5-aa5e74e690ac-credential-socket\") pod \"f655e4a8-4347-4492-b5c5-aa5e74e690ac\" (UID: \"f655e4a8-4347-4492-b5c5-aa5e74e690ac\") "
Oct 31 20:03:58.349052 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:58.348813    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f655e4a8-4347-4492-b5c5-aa5e74e690ac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f655e4a8-4347-4492-b5c5-aa5e74e690ac-workload-certs\") pod \"f655e4a8-4347-4492-b5c5-aa5e74e690ac\" (UID: \"f655e4a8-4347-4492-b5c5-aa5e74e690ac\") "
Oct 31 20:03:58.349052 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:58.348883    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72bb253a-434b-4921-97f2-9f23df7249fe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72bb253a-434b-4921-97f2-9f23df7249fe-workload-certs\") pod \"72bb253a-434b-4921-97f2-9f23df7249fe\" (UID: \"72bb253a-434b-4921-97f2-9f23df7249fe\") "
Oct 31 20:03:58.349052 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:58.348900    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72bb253a-434b-4921-97f2-9f23df7249fe-istio-data" pod "72bb253a-434b-4921-97f2-9f23df7249fe" (UID: "72bb253a-434b-4921-97f2-9f23df7249fe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:03:58.349052 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:58.348982    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72bb253a-434b-4921-97f2-9f23df7249fe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72bb253a-434b-4921-97f2-9f23df7249fe-istiod-ca-cert\") pod \"72bb253a-434b-4921-97f2-9f23df7249fe\" (UID: \"72bb253a-434b-4921-97f2-9f23df7249fe\") "
Oct 31 20:03:58.349263 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:58.349092    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f655e4a8-4347-4492-b5c5-aa5e74e690ac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f655e4a8-4347-4492-b5c5-aa5e74e690ac-istio-data\") pod \"f655e4a8-4347-4492-b5c5-aa5e74e690ac\" (UID: \"f655e4a8-4347-4492-b5c5-aa5e74e690ac\") "
Oct 31 20:03:58.349263 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:58.349123    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f655e4a8-4347-4492-b5c5-aa5e74e690ac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f655e4a8-4347-4492-b5c5-aa5e74e690ac-istio-podinfo\") pod \"f655e4a8-4347-4492-b5c5-aa5e74e690ac\" (UID: \"f655e4a8-4347-4492-b5c5-aa5e74e690ac\") "
Oct 31 20:03:58.349263 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:58.349152    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f655e4a8-4347-4492-b5c5-aa5e74e690ac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f655e4a8-4347-4492-b5c5-aa5e74e690ac-istiod-ca-cert\") pod \"f655e4a8-4347-4492-b5c5-aa5e74e690ac\" (UID: \"f655e4a8-4347-4492-b5c5-aa5e74e690ac\") "
Oct 31 20:03:58.349263 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:58.349222    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72bb253a-434b-4921-97f2-9f23df7249fe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72bb253a-434b-4921-97f2-9f23df7249fe-workload-socket\") pod \"72bb253a-434b-4921-97f2-9f23df7249fe\" (UID: \"72bb253a-434b-4921-97f2-9f23df7249fe\") "
Oct 31 20:03:58.349263 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:58.349260    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72bb253a-434b-4921-97f2-9f23df7249fe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72bb253a-434b-4921-97f2-9f23df7249fe-credential-socket\") pod \"72bb253a-434b-4921-97f2-9f23df7249fe\" (UID: \"72bb253a-434b-4921-97f2-9f23df7249fe\") "
Oct 31 20:03:58.350114 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:58.350084    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72bb253a-434b-4921-97f2-9f23df7249fe-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:58.350622 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:58.350595    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f655e4a8-4347-4492-b5c5-aa5e74e690ac-credential-socket" pod "f655e4a8-4347-4492-b5c5-aa5e74e690ac" (UID: "f655e4a8-4347-4492-b5c5-aa5e74e690ac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:03:58.350837 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:58.350813    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f655e4a8-4347-4492-b5c5-aa5e74e690ac-istio-data" pod "f655e4a8-4347-4492-b5c5-aa5e74e690ac" (UID: "f655e4a8-4347-4492-b5c5-aa5e74e690ac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:03:58.353565 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:58.351233    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f655e4a8-4347-4492-b5c5-aa5e74e690ac-workload-certs" pod "f655e4a8-4347-4492-b5c5-aa5e74e690ac" (UID: "f655e4a8-4347-4492-b5c5-aa5e74e690ac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:03:58.353565 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:58.352029    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72bb253a-434b-4921-97f2-9f23df7249fe-istio-token" pod "72bb253a-434b-4921-97f2-9f23df7249fe" (UID: "72bb253a-434b-4921-97f2-9f23df7249fe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:03:58.353565 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:58.352141    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f655e4a8-4347-4492-b5c5-aa5e74e690ac-istio-token" pod "f655e4a8-4347-4492-b5c5-aa5e74e690ac" (UID: "f655e4a8-4347-4492-b5c5-aa5e74e690ac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:03:58.353565 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:58.352639    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72bb253a-434b-4921-97f2-9f23df7249fe-istiod-ca-cert" pod "72bb253a-434b-4921-97f2-9f23df7249fe" (UID: "72bb253a-434b-4921-97f2-9f23df7249fe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:03:58.353565 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:58.352703    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f655e4a8-4347-4492-b5c5-aa5e74e690ac-workload-socket" pod "f655e4a8-4347-4492-b5c5-aa5e74e690ac" (UID: "f655e4a8-4347-4492-b5c5-aa5e74e690ac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:03:58.353565 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:58.352755    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72bb253a-434b-4921-97f2-9f23df7249fe-workload-socket" pod "72bb253a-434b-4921-97f2-9f23df7249fe" (UID: "72bb253a-434b-4921-97f2-9f23df7249fe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:03:58.353565 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:58.352812    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72bb253a-434b-4921-97f2-9f23df7249fe-credential-socket" pod "72bb253a-434b-4921-97f2-9f23df7249fe" (UID: "72bb253a-434b-4921-97f2-9f23df7249fe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:03:58.353565 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:58.352860    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72bb253a-434b-4921-97f2-9f23df7249fe-workload-certs" pod "72bb253a-434b-4921-97f2-9f23df7249fe" (UID: "72bb253a-434b-4921-97f2-9f23df7249fe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:03:58.353565 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:58.353164    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f655e4a8-4347-4492-b5c5-aa5e74e690ac-istio-envoy" pod "f655e4a8-4347-4492-b5c5-aa5e74e690ac" (UID: "f655e4a8-4347-4492-b5c5-aa5e74e690ac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:03:58.353565 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:58.353404    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f655e4a8-4347-4492-b5c5-aa5e74e690ac-istiod-ca-cert" pod "f655e4a8-4347-4492-b5c5-aa5e74e690ac" (UID: "f655e4a8-4347-4492-b5c5-aa5e74e690ac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:03:58.354272 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:58.354247    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72bb253a-434b-4921-97f2-9f23df7249fe-kube-api-access-kt4hf" pod "72bb253a-434b-4921-97f2-9f23df7249fe" (UID: "72bb253a-434b-4921-97f2-9f23df7249fe"). InnerVolumeSpecName "kube-api-access-kt4hf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:03:58.354450 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:58.354307    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/72bb253a-434b-4921-97f2-9f23df7249fe-istio-podinfo" pod "72bb253a-434b-4921-97f2-9f23df7249fe" (UID: "72bb253a-434b-4921-97f2-9f23df7249fe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 20:03:58.354450 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:58.354373    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72bb253a-434b-4921-97f2-9f23df7249fe-istio-envoy" pod "72bb253a-434b-4921-97f2-9f23df7249fe" (UID: "72bb253a-434b-4921-97f2-9f23df7249fe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:03:58.354953 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:58.354932    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f655e4a8-4347-4492-b5c5-aa5e74e690ac-istio-podinfo" pod "f655e4a8-4347-4492-b5c5-aa5e74e690ac" (UID: "f655e4a8-4347-4492-b5c5-aa5e74e690ac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 20:03:58.355695 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:58.355671    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f655e4a8-4347-4492-b5c5-aa5e74e690ac-kube-api-access-vvdq6" pod "f655e4a8-4347-4492-b5c5-aa5e74e690ac" (UID: "f655e4a8-4347-4492-b5c5-aa5e74e690ac"). InnerVolumeSpecName "kube-api-access-vvdq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:03:58.405572 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:58.405541    4645 generic.go:356] "Generic (PLEG): container finished" podID="f655e4a8-4347-4492-b5c5-aa5e74e690ac" containerID="261c4acad2435340f349218a91ab931436d74fb224f2404d2a281cc939cfd258" exitCode=0
Oct 31 20:03:58.405740 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:58.405590    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w"
Oct 31 20:03:58.405740 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:58.405629    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w" event={"ID":"f655e4a8-4347-4492-b5c5-aa5e74e690ac","Type":"ContainerDied","Data":"261c4acad2435340f349218a91ab931436d74fb224f2404d2a281cc939cfd258"}
Oct 31 20:03:58.405740 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:58.405691    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w" event={"ID":"f655e4a8-4347-4492-b5c5-aa5e74e690ac","Type":"ContainerDied","Data":"4125fb93d7c6bae961f06d0efb606531ddc1f18d605c56c865f3943a691a03b7"}
Oct 31 20:03:58.405740 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:58.405721    4645 scope.go:122] "RemoveContainer" containerID="261c4acad2435340f349218a91ab931436d74fb224f2404d2a281cc939cfd258"
Oct 31 20:03:58.408095 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:58.408068    4645 generic.go:356] "Generic (PLEG): container finished" podID="72bb253a-434b-4921-97f2-9f23df7249fe" containerID="c5963ca2e4f24daee898d56ffc52efd0c9db5a767b66dccd97a6914c2df41339" exitCode=0
Oct 31 20:03:58.408219 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:58.408184    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w" event={"ID":"72bb253a-434b-4921-97f2-9f23df7249fe","Type":"ContainerDied","Data":"c5963ca2e4f24daee898d56ffc52efd0c9db5a767b66dccd97a6914c2df41339"}
Oct 31 20:03:58.408400 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:58.408376    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w" event={"ID":"72bb253a-434b-4921-97f2-9f23df7249fe","Type":"ContainerDied","Data":"c3fd0928bb115b5dcaac8194c6f386f53ade179f10b100a88cc8ff671d491751"}
Oct 31 20:03:58.408544 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:58.408526    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w"
Oct 31 20:03:58.422838 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:58.422821    4645 scope.go:122] "RemoveContainer" containerID="261c4acad2435340f349218a91ab931436d74fb224f2404d2a281cc939cfd258"
Oct 31 20:03:58.423231 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:03:58.423200    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"261c4acad2435340f349218a91ab931436d74fb224f2404d2a281cc939cfd258\": container with ID starting with 261c4acad2435340f349218a91ab931436d74fb224f2404d2a281cc939cfd258 not found: ID does not exist" containerID="261c4acad2435340f349218a91ab931436d74fb224f2404d2a281cc939cfd258"
Oct 31 20:03:58.423343 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:58.423242    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"261c4acad2435340f349218a91ab931436d74fb224f2404d2a281cc939cfd258"} err="failed to get container status \"261c4acad2435340f349218a91ab931436d74fb224f2404d2a281cc939cfd258\": rpc error: code = NotFound desc = could not find container \"261c4acad2435340f349218a91ab931436d74fb224f2404d2a281cc939cfd258\": container with ID starting with 261c4acad2435340f349218a91ab931436d74fb224f2404d2a281cc939cfd258 not found: ID does not exist"
Oct 31 20:03:58.423343 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:58.423266    4645 scope.go:122] "RemoveContainer" containerID="c5963ca2e4f24daee898d56ffc52efd0c9db5a767b66dccd97a6914c2df41339"
Oct 31 20:03:58.436913 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:58.436890    4645 scope.go:122] "RemoveContainer" containerID="c5963ca2e4f24daee898d56ffc52efd0c9db5a767b66dccd97a6914c2df41339"
Oct 31 20:03:58.437258 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:03:58.437223    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5963ca2e4f24daee898d56ffc52efd0c9db5a767b66dccd97a6914c2df41339\": container with ID starting with c5963ca2e4f24daee898d56ffc52efd0c9db5a767b66dccd97a6914c2df41339 not found: ID does not exist" containerID="c5963ca2e4f24daee898d56ffc52efd0c9db5a767b66dccd97a6914c2df41339"
Oct 31 20:03:58.437393 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:58.437257    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5963ca2e4f24daee898d56ffc52efd0c9db5a767b66dccd97a6914c2df41339"} err="failed to get container status \"c5963ca2e4f24daee898d56ffc52efd0c9db5a767b66dccd97a6914c2df41339\": rpc error: code = NotFound desc = could not find container \"c5963ca2e4f24daee898d56ffc52efd0c9db5a767b66dccd97a6914c2df41339\": container with ID starting with c5963ca2e4f24daee898d56ffc52efd0c9db5a767b66dccd97a6914c2df41339 not found: ID does not exist"
Oct 31 20:03:58.441439 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:58.441413    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:58.442403 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:58.442312    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w"]
Oct 31 20:03:58.447535 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:58.447486    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fhr8w"]
Oct 31 20:03:58.451582 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:58.451558    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72bb253a-434b-4921-97f2-9f23df7249fe-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:58.451582 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:58.451581    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f655e4a8-4347-4492-b5c5-aa5e74e690ac-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:58.451713 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:58.451592    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72bb253a-434b-4921-97f2-9f23df7249fe-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:58.451713 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:58.451601    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f655e4a8-4347-4492-b5c5-aa5e74e690ac-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:58.451713 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:58.451610    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f655e4a8-4347-4492-b5c5-aa5e74e690ac-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:58.451713 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:58.451620    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72bb253a-434b-4921-97f2-9f23df7249fe-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:58.451713 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:58.451628    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72bb253a-434b-4921-97f2-9f23df7249fe-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:58.451713 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:58.451636    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f655e4a8-4347-4492-b5c5-aa5e74e690ac-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:58.451713 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:58.451644    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f655e4a8-4347-4492-b5c5-aa5e74e690ac-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:58.451713 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:58.451653    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f655e4a8-4347-4492-b5c5-aa5e74e690ac-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:58.451713 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:58.451661    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72bb253a-434b-4921-97f2-9f23df7249fe-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:58.451713 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:58.451671    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72bb253a-434b-4921-97f2-9f23df7249fe-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:58.451713 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:58.451680    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvdq6\" (UniqueName: \"kubernetes.io/projected/f655e4a8-4347-4492-b5c5-aa5e74e690ac-kube-api-access-vvdq6\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:58.451713 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:58.451689    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f655e4a8-4347-4492-b5c5-aa5e74e690ac-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:58.451713 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:58.451698    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f655e4a8-4347-4492-b5c5-aa5e74e690ac-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:58.451713 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:58.451707    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kt4hf\" (UniqueName: \"kubernetes.io/projected/72bb253a-434b-4921-97f2-9f23df7249fe-kube-api-access-kt4hf\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:58.451713 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:58.451715    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72bb253a-434b-4921-97f2-9f23df7249fe-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:03:58.460122 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:58.460099    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w"]
Oct 31 20:03:58.468230 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:58.468209    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-c734f5-openshift-default-7f99c45bc-fdk2w"]
Oct 31 20:03:59.184391 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:59.184355    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:03:59.618991 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:59.618961    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72bb253a-434b-4921-97f2-9f23df7249fe" path="/var/lib/kubelet/pods/72bb253a-434b-4921-97f2-9f23df7249fe/volumes"
Oct 31 20:03:59.619491 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:59.619476    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f655e4a8-4347-4492-b5c5-aa5e74e690ac" path="/var/lib/kubelet/pods/f655e4a8-4347-4492-b5c5-aa5e74e690ac/volumes"
Oct 31 20:03:59.969610 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:03:59.969576    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:00.728500 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:00.728463    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-23843f-6fc9d79f67-clgcx"]
Oct 31 20:04:00.728972 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:00.728703    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-23843f-6fc9d79f67-clgcx" podUID="56586a65-8824-40b4-9e94-b517f0307311" containerName="maas-api" containerID="cri-o://c75614b1ba6ef41d671864f39c92f108893048539cf56b32ac65d42b96425232" gracePeriod=30
Oct 31 20:04:00.784351 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:00.784299    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:00.958509 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:00.958462    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:00.995428 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:00.995314    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bd1c9f-65b6b696bd-2bqtm"]
Oct 31 20:04:00.995599 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:00.995570    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-bd1c9f-65b6b696bd-2bqtm" podUID="9c653fdd-5545-415b-b7f9-cf5dcadff326" containerName="payload-processing" containerID="cri-o://eb74d2f583a1da1d6ad93983665726fcfda433c433fef3aaa9ec5119606415ae" gracePeriod=30
Oct 31 20:04:01.006229 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:01.006204    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bd1c9f-745586fb75-mzfnm"]
Oct 31 20:04:01.006463 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:01.006432    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd1c9f-745586fb75-mzfnm" podUID="140d8a5f-7b8a-409b-9fcc-5a050fdeff9c" containerName="payload-pre-processing" containerID="cri-o://d21b66ba02111d09f39482fad8118cba23c585fda475f6e7878adc46fb0f7775" gracePeriod=30
Oct 31 20:04:01.051456 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:01.051396    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-23843f-976pz"]
Oct 31 20:04:01.056186 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:01.056149    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-23843f-976pz"]
Oct 31 20:04:01.082589 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:01.082565    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:01.150400 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:01.150374    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-23843f-6fc9d79f67-clgcx"
Oct 31 20:04:01.210447 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:01.210177    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56586a65-8824-40b4-9e94-b517f0307311-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56586a65-8824-40b4-9e94-b517f0307311-maas-api-tls\") pod \"56586a65-8824-40b4-9e94-b517f0307311\" (UID: \"56586a65-8824-40b4-9e94-b517f0307311\") "
Oct 31 20:04:01.210447 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:01.210321    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56586a65-8824-40b4-9e94-b517f0307311-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56586a65-8824-40b4-9e94-b517f0307311-openshift-service-ca\") pod \"56586a65-8824-40b4-9e94-b517f0307311\" (UID: \"56586a65-8824-40b4-9e94-b517f0307311\") "
Oct 31 20:04:01.210447 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:01.210366    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56586a65-8824-40b4-9e94-b517f0307311-kube-api-access-dcv8c\" (UniqueName: \"kubernetes.io/projected/56586a65-8824-40b4-9e94-b517f0307311-kube-api-access-dcv8c\") pod \"56586a65-8824-40b4-9e94-b517f0307311\" (UID: \"56586a65-8824-40b4-9e94-b517f0307311\") "
Oct 31 20:04:01.210447 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:01.210416    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56586a65-8824-40b4-9e94-b517f0307311-metrics-certs\" (UniqueName: \"kubernetes.io/secret/56586a65-8824-40b4-9e94-b517f0307311-metrics-certs\") pod \"56586a65-8824-40b4-9e94-b517f0307311\" (UID: \"56586a65-8824-40b4-9e94-b517f0307311\") "
Oct 31 20:04:01.215924 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:01.215871    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56586a65-8824-40b4-9e94-b517f0307311-openshift-service-ca" pod "56586a65-8824-40b4-9e94-b517f0307311" (UID: "56586a65-8824-40b4-9e94-b517f0307311"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:04:01.218709 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:01.218683    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56586a65-8824-40b4-9e94-b517f0307311-kube-api-access-dcv8c" pod "56586a65-8824-40b4-9e94-b517f0307311" (UID: "56586a65-8824-40b4-9e94-b517f0307311"). InnerVolumeSpecName "kube-api-access-dcv8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:04:01.219818 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:01.219793    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56586a65-8824-40b4-9e94-b517f0307311-metrics-certs" pod "56586a65-8824-40b4-9e94-b517f0307311" (UID: "56586a65-8824-40b4-9e94-b517f0307311"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:04:01.219904 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:01.219869    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56586a65-8824-40b4-9e94-b517f0307311-maas-api-tls" pod "56586a65-8824-40b4-9e94-b517f0307311" (UID: "56586a65-8824-40b4-9e94-b517f0307311"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:04:01.231528 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:01.231503    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:01.279483 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:01.279460    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bd1c9f-65b6b696bd-2bqtm"
Oct 31 20:04:01.363084 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:01.362594    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56586a65-8824-40b4-9e94-b517f0307311-maas-api-tls\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:01.363084 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:01.362633    4645 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56586a65-8824-40b4-9e94-b517f0307311-openshift-service-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:01.363084 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:01.362648    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcv8c\" (UniqueName: \"kubernetes.io/projected/56586a65-8824-40b4-9e94-b517f0307311-kube-api-access-dcv8c\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:01.363084 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:01.362665    4645 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56586a65-8824-40b4-9e94-b517f0307311-metrics-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:01.382351 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:01.375707    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:01.463898 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:01.463858    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c653fdd-5545-415b-b7f9-cf5dcadff326-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c653fdd-5545-415b-b7f9-cf5dcadff326-plugins-config-volume\") pod \"9c653fdd-5545-415b-b7f9-cf5dcadff326\" (UID: \"9c653fdd-5545-415b-b7f9-cf5dcadff326\") "
Oct 31 20:04:01.465998 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:01.465970    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c653fdd-5545-415b-b7f9-cf5dcadff326-kube-api-access-rgrpf\" (UniqueName: \"kubernetes.io/projected/9c653fdd-5545-415b-b7f9-cf5dcadff326-kube-api-access-rgrpf\") pod \"9c653fdd-5545-415b-b7f9-cf5dcadff326\" (UID: \"9c653fdd-5545-415b-b7f9-cf5dcadff326\") "
Oct 31 20:04:01.466294 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:01.466262    4645 generic.go:356] "Generic (PLEG): container finished" podID="9c653fdd-5545-415b-b7f9-cf5dcadff326" containerID="eb74d2f583a1da1d6ad93983665726fcfda433c433fef3aaa9ec5119606415ae" exitCode=0
Oct 31 20:04:01.467341 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:01.464573    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c653fdd-5545-415b-b7f9-cf5dcadff326-plugins-config-volume" pod "9c653fdd-5545-415b-b7f9-cf5dcadff326" (UID: "9c653fdd-5545-415b-b7f9-cf5dcadff326"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:04:01.468366 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:01.467665    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bd1c9f-65b6b696bd-2bqtm"
Oct 31 20:04:01.468921 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:01.467686    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bd1c9f-65b6b696bd-2bqtm" event={"ID":"9c653fdd-5545-415b-b7f9-cf5dcadff326","Type":"ContainerDied","Data":"eb74d2f583a1da1d6ad93983665726fcfda433c433fef3aaa9ec5119606415ae"}
Oct 31 20:04:01.468921 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:01.468596    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bd1c9f-65b6b696bd-2bqtm" event={"ID":"9c653fdd-5545-415b-b7f9-cf5dcadff326","Type":"ContainerDied","Data":"f743761bff9418ee54f2602a121bf0d6b79b00e03241a4ece114a3cfa97970b3"}
Oct 31 20:04:01.468921 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:01.468632    4645 scope.go:122] "RemoveContainer" containerID="eb74d2f583a1da1d6ad93983665726fcfda433c433fef3aaa9ec5119606415ae"
Oct 31 20:04:01.476165 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:01.476139    4645 generic.go:356] "Generic (PLEG): container finished" podID="140d8a5f-7b8a-409b-9fcc-5a050fdeff9c" containerID="d21b66ba02111d09f39482fad8118cba23c585fda475f6e7878adc46fb0f7775" exitCode=0
Oct 31 20:04:01.476451 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:01.476426    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd1c9f-745586fb75-mzfnm" event={"ID":"140d8a5f-7b8a-409b-9fcc-5a050fdeff9c","Type":"ContainerDied","Data":"d21b66ba02111d09f39482fad8118cba23c585fda475f6e7878adc46fb0f7775"}
Oct 31 20:04:01.483643 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:01.483588    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c653fdd-5545-415b-b7f9-cf5dcadff326-kube-api-access-rgrpf" pod "9c653fdd-5545-415b-b7f9-cf5dcadff326" (UID: "9c653fdd-5545-415b-b7f9-cf5dcadff326"). InnerVolumeSpecName "kube-api-access-rgrpf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:04:01.484496 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:01.484456    4645 generic.go:356] "Generic (PLEG): container finished" podID="56586a65-8824-40b4-9e94-b517f0307311" containerID="c75614b1ba6ef41d671864f39c92f108893048539cf56b32ac65d42b96425232" exitCode=0
Oct 31 20:04:01.484847 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:01.484822    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-23843f-6fc9d79f67-clgcx" event={"ID":"56586a65-8824-40b4-9e94-b517f0307311","Type":"ContainerDied","Data":"c75614b1ba6ef41d671864f39c92f108893048539cf56b32ac65d42b96425232"}
Oct 31 20:04:01.485016 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:01.484995    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-23843f-6fc9d79f67-clgcx" event={"ID":"56586a65-8824-40b4-9e94-b517f0307311","Type":"ContainerDied","Data":"8f9ad2115d76936579f787d15893a81c850ae5b937256c1a3e0b67d59565ec0c"}
Oct 31 20:04:01.485218 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:01.485200    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-23843f-6fc9d79f67-clgcx"
Oct 31 20:04:01.503371 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:01.503231    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd1c9f-745586fb75-mzfnm"
Oct 31 20:04:01.514292 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:01.514174    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:01.543552 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:01.543520    4645 scope.go:122] "RemoveContainer" containerID="eb74d2f583a1da1d6ad93983665726fcfda433c433fef3aaa9ec5119606415ae"
Oct 31 20:04:01.547245 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:04:01.547211    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb74d2f583a1da1d6ad93983665726fcfda433c433fef3aaa9ec5119606415ae\": container with ID starting with eb74d2f583a1da1d6ad93983665726fcfda433c433fef3aaa9ec5119606415ae not found: ID does not exist" containerID="eb74d2f583a1da1d6ad93983665726fcfda433c433fef3aaa9ec5119606415ae"
Oct 31 20:04:01.547443 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:01.547411    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb74d2f583a1da1d6ad93983665726fcfda433c433fef3aaa9ec5119606415ae"} err="failed to get container status \"eb74d2f583a1da1d6ad93983665726fcfda433c433fef3aaa9ec5119606415ae\": rpc error: code = NotFound desc = could not find container \"eb74d2f583a1da1d6ad93983665726fcfda433c433fef3aaa9ec5119606415ae\": container with ID starting with eb74d2f583a1da1d6ad93983665726fcfda433c433fef3aaa9ec5119606415ae not found: ID does not exist"
Oct 31 20:04:01.547585 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:01.547569    4645 scope.go:122] "RemoveContainer" containerID="c75614b1ba6ef41d671864f39c92f108893048539cf56b32ac65d42b96425232"
Oct 31 20:04:01.570384 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:01.570344    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-23843f-6fc9d79f67-clgcx"]
Oct 31 20:04:01.572540 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:01.572516    4645 scope.go:122] "RemoveContainer" containerID="c75614b1ba6ef41d671864f39c92f108893048539cf56b32ac65d42b96425232"
Oct 31 20:04:01.572902 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:04:01.572876    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c75614b1ba6ef41d671864f39c92f108893048539cf56b32ac65d42b96425232\": container with ID starting with c75614b1ba6ef41d671864f39c92f108893048539cf56b32ac65d42b96425232 not found: ID does not exist" containerID="c75614b1ba6ef41d671864f39c92f108893048539cf56b32ac65d42b96425232"
Oct 31 20:04:01.573006 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:01.572914    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c75614b1ba6ef41d671864f39c92f108893048539cf56b32ac65d42b96425232"} err="failed to get container status \"c75614b1ba6ef41d671864f39c92f108893048539cf56b32ac65d42b96425232\": rpc error: code = NotFound desc = could not find container \"c75614b1ba6ef41d671864f39c92f108893048539cf56b32ac65d42b96425232\": container with ID starting with c75614b1ba6ef41d671864f39c92f108893048539cf56b32ac65d42b96425232 not found: ID does not exist"
Oct 31 20:04:01.575042 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:01.575013    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/140d8a5f-7b8a-409b-9fcc-5a050fdeff9c-kube-api-access-bflwg\" (UniqueName: \"kubernetes.io/projected/140d8a5f-7b8a-409b-9fcc-5a050fdeff9c-kube-api-access-bflwg\") pod \"140d8a5f-7b8a-409b-9fcc-5a050fdeff9c\" (UID: \"140d8a5f-7b8a-409b-9fcc-5a050fdeff9c\") "
Oct 31 20:04:01.575165 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:01.575145    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/140d8a5f-7b8a-409b-9fcc-5a050fdeff9c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/140d8a5f-7b8a-409b-9fcc-5a050fdeff9c-plugins-config-volume\") pod \"140d8a5f-7b8a-409b-9fcc-5a050fdeff9c\" (UID: \"140d8a5f-7b8a-409b-9fcc-5a050fdeff9c\") "
Oct 31 20:04:01.576305 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:01.576280    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgrpf\" (UniqueName: \"kubernetes.io/projected/9c653fdd-5545-415b-b7f9-cf5dcadff326-kube-api-access-rgrpf\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:01.576429 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:01.576309    4645 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c653fdd-5545-415b-b7f9-cf5dcadff326-plugins-config-volume\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:01.578598 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:01.578568    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/140d8a5f-7b8a-409b-9fcc-5a050fdeff9c-plugins-config-volume" pod "140d8a5f-7b8a-409b-9fcc-5a050fdeff9c" (UID: "140d8a5f-7b8a-409b-9fcc-5a050fdeff9c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:04:01.578725 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:01.578710    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-23843f-6fc9d79f67-clgcx"]
Oct 31 20:04:01.582100 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:01.582073    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/140d8a5f-7b8a-409b-9fcc-5a050fdeff9c-kube-api-access-bflwg" pod "140d8a5f-7b8a-409b-9fcc-5a050fdeff9c" (UID: "140d8a5f-7b8a-409b-9fcc-5a050fdeff9c"). InnerVolumeSpecName "kube-api-access-bflwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:04:01.663119 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:01.663078    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56586a65-8824-40b4-9e94-b517f0307311" path="/var/lib/kubelet/pods/56586a65-8824-40b4-9e94-b517f0307311/volumes"
Oct 31 20:04:01.665712 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:01.665680    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3c4c540-6b69-4c81-bea8-6c4176cf621c" path="/var/lib/kubelet/pods/a3c4c540-6b69-4c81-bea8-6c4176cf621c/volumes"
Oct 31 20:04:01.679925 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:01.679893    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bflwg\" (UniqueName: \"kubernetes.io/projected/140d8a5f-7b8a-409b-9fcc-5a050fdeff9c-kube-api-access-bflwg\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:01.680150 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:01.680135    4645 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/140d8a5f-7b8a-409b-9fcc-5a050fdeff9c-plugins-config-volume\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:01.680983 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:01.680936    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:01.815171 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:01.810242    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bd1c9f-65b6b696bd-2bqtm"]
Oct 31 20:04:01.816535 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:01.816504    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bd1c9f-65b6b696bd-2bqtm"]
Oct 31 20:04:02.009447 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:02.009413    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:02.495561 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:02.495529    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd1c9f-745586fb75-mzfnm"
Oct 31 20:04:02.495561 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:02.495540    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bd1c9f-745586fb75-mzfnm" event={"ID":"140d8a5f-7b8a-409b-9fcc-5a050fdeff9c","Type":"ContainerDied","Data":"a5c33810c2aa344e7056b56c2ff0a340838c46ab8b385b42950f24e6e40cc5f6"}
Oct 31 20:04:02.495834 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:02.495606    4645 scope.go:122] "RemoveContainer" containerID="d21b66ba02111d09f39482fad8118cba23c585fda475f6e7878adc46fb0f7775"
Oct 31 20:04:02.515480 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:02.515451    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bd1c9f-745586fb75-mzfnm"]
Oct 31 20:04:02.525538 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:02.525513    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bd1c9f-745586fb75-mzfnm"]
Oct 31 20:04:02.832225 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:02.832097    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:03.499854 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:03.499808    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:03.833398 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:03.819359    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="140d8a5f-7b8a-409b-9fcc-5a050fdeff9c" path="/var/lib/kubelet/pods/140d8a5f-7b8a-409b-9fcc-5a050fdeff9c/volumes"
Oct 31 20:04:03.833398 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:03.820793    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c653fdd-5545-415b-b7f9-cf5dcadff326" path="/var/lib/kubelet/pods/9c653fdd-5545-415b-b7f9-cf5dcadff326/volumes"
Oct 31 20:04:04.145943 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:04.145853    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:05.039355 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:05.034617    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:05.227720 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:05.209420    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d7647c-66bbb74874-4dsdn"]
Oct 31 20:04:05.227720 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:05.209676    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d7647c-85d9ccfd7f-qvj7p"]
Oct 31 20:04:05.227720 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:05.209870    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7647c-85d9ccfd7f-qvj7p" podUID="04e10b50-5ded-4487-831a-09edcb6e9f9e" containerName="payload-pre-processing" containerID="cri-o://84065f0c7da8fbbcdaa813339c3435854555b01727a616f01208b00f9e13c2cd" gracePeriod=30
Oct 31 20:04:05.227720 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:05.224503    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:05.227720 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:05.209681    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-d7647c-66bbb74874-4dsdn" podUID="daf67342-fb86-4a20-96b5-c753798fa0cd" containerName="payload-processing" containerID="cri-o://54440c3d95effcd198df4c326fdd97df16cd243c6dacd267effc83feb2d8e344" gracePeriod=30
Oct 31 20:04:05.492842 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:05.492375    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:05.749605 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:05.749525    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:05.878460 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:05.878063    4645 generic.go:356] "Generic (PLEG): container finished" podID="04e10b50-5ded-4487-831a-09edcb6e9f9e" containerID="84065f0c7da8fbbcdaa813339c3435854555b01727a616f01208b00f9e13c2cd" exitCode=0
Oct 31 20:04:05.878460 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:05.878290    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7647c-85d9ccfd7f-qvj7p" event={"ID":"04e10b50-5ded-4487-831a-09edcb6e9f9e","Type":"ContainerDied","Data":"84065f0c7da8fbbcdaa813339c3435854555b01727a616f01208b00f9e13c2cd"}
Oct 31 20:04:05.881593 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:05.881266    4645 generic.go:356] "Generic (PLEG): container finished" podID="daf67342-fb86-4a20-96b5-c753798fa0cd" containerID="54440c3d95effcd198df4c326fdd97df16cd243c6dacd267effc83feb2d8e344" exitCode=0
Oct 31 20:04:05.881593 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:05.881510    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d7647c-66bbb74874-4dsdn" event={"ID":"daf67342-fb86-4a20-96b5-c753798fa0cd","Type":"ContainerDied","Data":"54440c3d95effcd198df4c326fdd97df16cd243c6dacd267effc83feb2d8e344"}
Oct 31 20:04:06.045190 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:06.045163    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:06.127766 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:06.127737    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7647c-85d9ccfd7f-qvj7p"
Oct 31 20:04:06.185046 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:06.183522    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d7647c-66bbb74874-4dsdn"
Oct 31 20:04:06.254367 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:06.244760    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/daf67342-fb86-4a20-96b5-c753798fa0cd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/daf67342-fb86-4a20-96b5-c753798fa0cd-plugins-config-volume\") pod \"daf67342-fb86-4a20-96b5-c753798fa0cd\" (UID: \"daf67342-fb86-4a20-96b5-c753798fa0cd\") "
Oct 31 20:04:06.254367 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:06.244818    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04e10b50-5ded-4487-831a-09edcb6e9f9e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04e10b50-5ded-4487-831a-09edcb6e9f9e-plugins-config-volume\") pod \"04e10b50-5ded-4487-831a-09edcb6e9f9e\" (UID: \"04e10b50-5ded-4487-831a-09edcb6e9f9e\") "
Oct 31 20:04:06.254367 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:06.244964    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/daf67342-fb86-4a20-96b5-c753798fa0cd-kube-api-access-x6zp6\" (UniqueName: \"kubernetes.io/projected/daf67342-fb86-4a20-96b5-c753798fa0cd-kube-api-access-x6zp6\") pod \"daf67342-fb86-4a20-96b5-c753798fa0cd\" (UID: \"daf67342-fb86-4a20-96b5-c753798fa0cd\") "
Oct 31 20:04:06.254367 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:06.245110    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04e10b50-5ded-4487-831a-09edcb6e9f9e-kube-api-access-gmwft\" (UniqueName: \"kubernetes.io/projected/04e10b50-5ded-4487-831a-09edcb6e9f9e-kube-api-access-gmwft\") pod \"04e10b50-5ded-4487-831a-09edcb6e9f9e\" (UID: \"04e10b50-5ded-4487-831a-09edcb6e9f9e\") "
Oct 31 20:04:06.280415 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:06.280354    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daf67342-fb86-4a20-96b5-c753798fa0cd-plugins-config-volume" pod "daf67342-fb86-4a20-96b5-c753798fa0cd" (UID: "daf67342-fb86-4a20-96b5-c753798fa0cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:04:06.283009 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:06.282881    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04e10b50-5ded-4487-831a-09edcb6e9f9e-plugins-config-volume" pod "04e10b50-5ded-4487-831a-09edcb6e9f9e" (UID: "04e10b50-5ded-4487-831a-09edcb6e9f9e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:04:06.289510 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:06.289475    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daf67342-fb86-4a20-96b5-c753798fa0cd-kube-api-access-x6zp6" pod "daf67342-fb86-4a20-96b5-c753798fa0cd" (UID: "daf67342-fb86-4a20-96b5-c753798fa0cd"). InnerVolumeSpecName "kube-api-access-x6zp6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:04:06.289671 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:06.289600    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04e10b50-5ded-4487-831a-09edcb6e9f9e-kube-api-access-gmwft" pod "04e10b50-5ded-4487-831a-09edcb6e9f9e" (UID: "04e10b50-5ded-4487-831a-09edcb6e9f9e"). InnerVolumeSpecName "kube-api-access-gmwft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:04:06.330056 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:04:06.330017    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 20:04:06.348008 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:06.347976    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6zp6\" (UniqueName: \"kubernetes.io/projected/daf67342-fb86-4a20-96b5-c753798fa0cd-kube-api-access-x6zp6\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:06.348167 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:06.348012    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmwft\" (UniqueName: \"kubernetes.io/projected/04e10b50-5ded-4487-831a-09edcb6e9f9e-kube-api-access-gmwft\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:06.348167 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:06.348029    4645 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/daf67342-fb86-4a20-96b5-c753798fa0cd-plugins-config-volume\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:06.348167 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:06.348043    4645 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04e10b50-5ded-4487-831a-09edcb6e9f9e-plugins-config-volume\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:06.896694 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:06.896665    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7647c-85d9ccfd7f-qvj7p"
Oct 31 20:04:06.897526 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:06.897474    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d7647c-85d9ccfd7f-qvj7p" event={"ID":"04e10b50-5ded-4487-831a-09edcb6e9f9e","Type":"ContainerDied","Data":"d9a03a9817d3f0c56504fab43129e4f349e577013065682062b3a8fffadec275"}
Oct 31 20:04:06.897792 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:06.897753    4645 scope.go:122] "RemoveContainer" containerID="84065f0c7da8fbbcdaa813339c3435854555b01727a616f01208b00f9e13c2cd"
Oct 31 20:04:06.899036 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:06.899007    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d7647c-66bbb74874-4dsdn" event={"ID":"daf67342-fb86-4a20-96b5-c753798fa0cd","Type":"ContainerDied","Data":"f392b2bcb1f3a193fa2261224fd62ab0d10fab180137dced165fce7395c31615"}
Oct 31 20:04:06.899166 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:06.899103    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d7647c-66bbb74874-4dsdn"
Oct 31 20:04:06.918272 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:06.918246    4645 scope.go:122] "RemoveContainer" containerID="54440c3d95effcd198df4c326fdd97df16cd243c6dacd267effc83feb2d8e344"
Oct 31 20:04:06.941991 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:06.941969    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d7647c-85d9ccfd7f-qvj7p"]
Oct 31 20:04:06.948401 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:06.948379    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d7647c-85d9ccfd7f-qvj7p"]
Oct 31 20:04:06.971781 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:06.971755    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d7647c-66bbb74874-4dsdn"]
Oct 31 20:04:06.976198 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:06.976161    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d7647c-66bbb74874-4dsdn"]
Oct 31 20:04:06.983603 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:06.983471    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:07.623081 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:07.623039    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04e10b50-5ded-4487-831a-09edcb6e9f9e" path="/var/lib/kubelet/pods/04e10b50-5ded-4487-831a-09edcb6e9f9e/volumes"
Oct 31 20:04:07.623788 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:07.623766    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="daf67342-fb86-4a20-96b5-c753798fa0cd" path="/var/lib/kubelet/pods/daf67342-fb86-4a20-96b5-c753798fa0cd/volumes"
Oct 31 20:04:07.966140 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:07.966084    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:08.877358 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:08.873853    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:09.839353 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:09.839302    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:09.952106 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:09.952072    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:10.197307 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:10.197274    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:10.295352 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:10.293112    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6"]
Oct 31 20:04:10.295352 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:10.293722    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6" podUID="679339df-b722-4990-b395-b14fea7f6458" containerName="istio-proxy" containerID="cri-o://f95e143bc1d14b498d313fe8ff37414d1995008395a2383c52a2073a864a1859" gracePeriod=30
Oct 31 20:04:10.328980 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:10.328936    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:10.466414 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:10.459883    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:10.841560 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:10.841484    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd"]
Oct 31 20:04:10.841728 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:10.841701    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd" podUID="ba844f48-656b-492b-8a60-f1e85d33d977" containerName="istio-proxy" containerID="cri-o://8aa46e2c2b335792a0224ab9f712b47604539d50f509b5ba4c83b5cf44720ee0" gracePeriod=30
Oct 31 20:04:10.893608 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:10.893576    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a84fcb-6hhlc"]
Oct 31 20:04:10.894198 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:10.894184    4645 cpu_manager.go:414] "containerMap: removing container" podUID="72bb253a-434b-4921-97f2-9f23df7249fe" containerName="istio-proxy"
Oct 31 20:04:10.894244 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:10.894200    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72bb253a-434b-4921-97f2-9f23df7249fe" containerName="istio-proxy"
Oct 31 20:04:10.894244 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:10.894219    4645 cpu_manager.go:414] "containerMap: removing container" podUID="9e6f5b6b-6a09-4b98-82bd-cd6ad2899c0a" containerName="revoke-keys"
Oct 31 20:04:10.894244 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:10.894226    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e6f5b6b-6a09-4b98-82bd-cd6ad2899c0a" containerName="revoke-keys"
Oct 31 20:04:10.894244 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:10.894234    4645 cpu_manager.go:414] "containerMap: removing container" podUID="04e10b50-5ded-4487-831a-09edcb6e9f9e" containerName="payload-pre-processing"
Oct 31 20:04:10.894244 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:10.894240    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04e10b50-5ded-4487-831a-09edcb6e9f9e" containerName="payload-pre-processing"
Oct 31 20:04:10.894458 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:10.894252    4645 cpu_manager.go:414] "containerMap: removing container" podUID="140d8a5f-7b8a-409b-9fcc-5a050fdeff9c" containerName="payload-pre-processing"
Oct 31 20:04:10.894458 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:10.894257    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="140d8a5f-7b8a-409b-9fcc-5a050fdeff9c" containerName="payload-pre-processing"
Oct 31 20:04:10.894458 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:10.894267    4645 cpu_manager.go:414] "containerMap: removing container" podUID="daf67342-fb86-4a20-96b5-c753798fa0cd" containerName="payload-processing"
Oct 31 20:04:10.894458 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:10.894273    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daf67342-fb86-4a20-96b5-c753798fa0cd" containerName="payload-processing"
Oct 31 20:04:10.894458 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:10.894280    4645 cpu_manager.go:414] "containerMap: removing container" podUID="56586a65-8824-40b4-9e94-b517f0307311" containerName="maas-api"
Oct 31 20:04:10.894458 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:10.894284    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56586a65-8824-40b4-9e94-b517f0307311" containerName="maas-api"
Oct 31 20:04:10.894458 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:10.894295    4645 cpu_manager.go:414] "containerMap: removing container" podUID="f655e4a8-4347-4492-b5c5-aa5e74e690ac" containerName="istio-proxy"
Oct 31 20:04:10.894458 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:10.894300    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f655e4a8-4347-4492-b5c5-aa5e74e690ac" containerName="istio-proxy"
Oct 31 20:04:10.894458 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:10.894341    4645 cpu_manager.go:414] "containerMap: removing container" podUID="9c653fdd-5545-415b-b7f9-cf5dcadff326" containerName="payload-processing"
Oct 31 20:04:10.894458 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:10.894346    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c653fdd-5545-415b-b7f9-cf5dcadff326" containerName="payload-processing"
Oct 31 20:04:10.894458 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:10.894442    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c653fdd-5545-415b-b7f9-cf5dcadff326" containerName="payload-processing"
Oct 31 20:04:10.894458 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:10.894453    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="72bb253a-434b-4921-97f2-9f23df7249fe" containerName="istio-proxy"
Oct 31 20:04:10.894458 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:10.894461    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e6f5b6b-6a09-4b98-82bd-cd6ad2899c0a" containerName="revoke-keys"
Oct 31 20:04:10.894879 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:10.894467    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="04e10b50-5ded-4487-831a-09edcb6e9f9e" containerName="payload-pre-processing"
Oct 31 20:04:10.894879 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:10.894474    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="140d8a5f-7b8a-409b-9fcc-5a050fdeff9c" containerName="payload-pre-processing"
Oct 31 20:04:10.894879 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:10.894480    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="daf67342-fb86-4a20-96b5-c753798fa0cd" containerName="payload-processing"
Oct 31 20:04:10.894879 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:10.894486    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="56586a65-8824-40b4-9e94-b517f0307311" containerName="maas-api"
Oct 31 20:04:10.894879 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:10.894492    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="f655e4a8-4347-4492-b5c5-aa5e74e690ac" containerName="istio-proxy"
Oct 31 20:04:10.899887 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:10.899871    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a84fcb-6hhlc"
Oct 31 20:04:10.916682 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:10.916665    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:04:10.916811 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:10.916797    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 20:04:10.943967 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:10.943943    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a84fcb-6hhlc"]
Oct 31 20:04:10.947814 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:10.947787    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:11.062350 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:11.061520    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a438ba9d-8191-4a01-bf6b-3633ee87a6de-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a84fcb-6hhlc\" (UID: \"a438ba9d-8191-4a01-bf6b-3633ee87a6de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a84fcb-6hhlc"
Oct 31 20:04:11.062350 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:11.061702    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a438ba9d-8191-4a01-bf6b-3633ee87a6de-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a84fcb-6hhlc\" (UID: \"a438ba9d-8191-4a01-bf6b-3633ee87a6de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a84fcb-6hhlc"
Oct 31 20:04:11.062350 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:11.061814    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrfhd\" (UniqueName: \"kubernetes.io/projected/a438ba9d-8191-4a01-bf6b-3633ee87a6de-kube-api-access-vrfhd\") pod \"maas-api-revoke-keys-e2e-ipp-b-a84fcb-6hhlc\" (UID: \"a438ba9d-8191-4a01-bf6b-3633ee87a6de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a84fcb-6hhlc"
Oct 31 20:04:11.117485 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:11.112806    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:11.168876 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:11.163417    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a438ba9d-8191-4a01-bf6b-3633ee87a6de-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a84fcb-6hhlc\" (UID: \"a438ba9d-8191-4a01-bf6b-3633ee87a6de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a84fcb-6hhlc"
Oct 31 20:04:11.168876 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:11.163553    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrfhd\" (UniqueName: \"kubernetes.io/projected/a438ba9d-8191-4a01-bf6b-3633ee87a6de-kube-api-access-vrfhd\") pod \"maas-api-revoke-keys-e2e-ipp-b-a84fcb-6hhlc\" (UID: \"a438ba9d-8191-4a01-bf6b-3633ee87a6de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a84fcb-6hhlc"
Oct 31 20:04:11.168876 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:11.163722    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a438ba9d-8191-4a01-bf6b-3633ee87a6de-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a84fcb-6hhlc\" (UID: \"a438ba9d-8191-4a01-bf6b-3633ee87a6de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a84fcb-6hhlc"
Oct 31 20:04:11.168876 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:11.168067    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a438ba9d-8191-4a01-bf6b-3633ee87a6de-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a84fcb-6hhlc\" (UID: \"a438ba9d-8191-4a01-bf6b-3633ee87a6de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a84fcb-6hhlc"
Oct 31 20:04:11.192351 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:11.189689    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a438ba9d-8191-4a01-bf6b-3633ee87a6de-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-a84fcb-6hhlc\" (UID: \"a438ba9d-8191-4a01-bf6b-3633ee87a6de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a84fcb-6hhlc"
Oct 31 20:04:11.192351 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:11.190693    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrfhd\" (UniqueName: \"kubernetes.io/projected/a438ba9d-8191-4a01-bf6b-3633ee87a6de-kube-api-access-vrfhd\") pod \"maas-api-revoke-keys-e2e-ipp-b-a84fcb-6hhlc\" (UID: \"a438ba9d-8191-4a01-bf6b-3633ee87a6de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a84fcb-6hhlc"
Oct 31 20:04:11.216232 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:11.216195    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a84fcb-6hhlc"
Oct 31 20:04:11.529545 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:11.529515    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a84fcb-6hhlc"]
Oct 31 20:04:11.530166 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:04:11.530140    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda438ba9d_8191_4a01_bf6b_3633ee87a6de.slice/crio-445fd41707d83bf1cac6e0a9f2f790eb2d7259ab5146bf0bd01376cd0d32ea53 WatchSource:0}: Error finding container 445fd41707d83bf1cac6e0a9f2f790eb2d7259ab5146bf0bd01376cd0d32ea53: Status 404 returned error can't find the container with id 445fd41707d83bf1cac6e0a9f2f790eb2d7259ab5146bf0bd01376cd0d32ea53
Oct 31 20:04:11.969808 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:11.969774    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a84fcb-6hhlc" event={"ID":"a438ba9d-8191-4a01-bf6b-3633ee87a6de","Type":"ContainerStarted","Data":"7f78b2a9b929d5bceca795372443e33b46a80a12870f7bf5ea75967eb8e965f5"}
Oct 31 20:04:11.969958 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:11.969826    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a84fcb-6hhlc" event={"ID":"a438ba9d-8191-4a01-bf6b-3633ee87a6de","Type":"ContainerStarted","Data":"445fd41707d83bf1cac6e0a9f2f790eb2d7259ab5146bf0bd01376cd0d32ea53"}
Oct 31 20:04:11.988764 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:11.988723    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a84fcb-6hhlc" podStartSLOduration=1.988706936 podStartE2EDuration="1.988706936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:04:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:04:11.987017291 +0000 UTC m=+2682.560251431" watchObservedRunningTime="2026-10-31 20:04:11.988706936 +0000 UTC m=+2682.561941065"
Oct 31 20:04:11.994038 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:11.994010    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:12.457760 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:12.457727    4645 patch_prober.go:29] interesting pod/e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 20:04:12.458070 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:12.457781    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6" podUID="679339df-b722-4990-b395-b14fea7f6458" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 20:04:12.982199 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:12.982166    4645 generic.go:356] "Generic (PLEG): container finished" podID="a438ba9d-8191-4a01-bf6b-3633ee87a6de" containerID="7f78b2a9b929d5bceca795372443e33b46a80a12870f7bf5ea75967eb8e965f5" exitCode=0
Oct 31 20:04:12.982426 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:12.982400    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a84fcb-6hhlc" event={"ID":"a438ba9d-8191-4a01-bf6b-3633ee87a6de","Type":"ContainerDied","Data":"7f78b2a9b929d5bceca795372443e33b46a80a12870f7bf5ea75967eb8e965f5"}
Oct 31 20:04:13.489569 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:13.489539    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:13.693943 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:13.693908    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:14.166308 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:14.166276    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a84fcb-6hhlc"
Oct 31 20:04:14.300480 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:14.300399    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a438ba9d-8191-4a01-bf6b-3633ee87a6de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a438ba9d-8191-4a01-bf6b-3633ee87a6de-maas-api-xks-ca\") pod \"a438ba9d-8191-4a01-bf6b-3633ee87a6de\" (UID: \"a438ba9d-8191-4a01-bf6b-3633ee87a6de\") "
Oct 31 20:04:14.300650 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:14.300497    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a438ba9d-8191-4a01-bf6b-3633ee87a6de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a438ba9d-8191-4a01-bf6b-3633ee87a6de-maas-api-openshift-ca\") pod \"a438ba9d-8191-4a01-bf6b-3633ee87a6de\" (UID: \"a438ba9d-8191-4a01-bf6b-3633ee87a6de\") "
Oct 31 20:04:14.300650 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:14.300624    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a438ba9d-8191-4a01-bf6b-3633ee87a6de-kube-api-access-vrfhd\" (UniqueName: \"kubernetes.io/projected/a438ba9d-8191-4a01-bf6b-3633ee87a6de-kube-api-access-vrfhd\") pod \"a438ba9d-8191-4a01-bf6b-3633ee87a6de\" (UID: \"a438ba9d-8191-4a01-bf6b-3633ee87a6de\") "
Oct 31 20:04:14.300882 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:14.300859    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a438ba9d-8191-4a01-bf6b-3633ee87a6de-maas-api-openshift-ca" pod "a438ba9d-8191-4a01-bf6b-3633ee87a6de" (UID: "a438ba9d-8191-4a01-bf6b-3633ee87a6de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:04:14.301129 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:14.301109    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a438ba9d-8191-4a01-bf6b-3633ee87a6de-maas-api-openshift-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:14.302625 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:14.302601    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a438ba9d-8191-4a01-bf6b-3633ee87a6de-maas-api-xks-ca" pod "a438ba9d-8191-4a01-bf6b-3633ee87a6de" (UID: "a438ba9d-8191-4a01-bf6b-3633ee87a6de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:04:14.302776 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:14.302754    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a438ba9d-8191-4a01-bf6b-3633ee87a6de-kube-api-access-vrfhd" pod "a438ba9d-8191-4a01-bf6b-3633ee87a6de" (UID: "a438ba9d-8191-4a01-bf6b-3633ee87a6de"). InnerVolumeSpecName "kube-api-access-vrfhd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:04:14.362108 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:14.362064    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:14.420513 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:14.415075    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a438ba9d-8191-4a01-bf6b-3633ee87a6de-maas-api-xks-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:14.420513 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:14.415112    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrfhd\" (UniqueName: \"kubernetes.io/projected/a438ba9d-8191-4a01-bf6b-3633ee87a6de-kube-api-access-vrfhd\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:14.472099 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:14.472066    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:14.671243 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:14.671212    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:14.843106 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:14.843069    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:15.080403 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.075170    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a84fcb-6hhlc" event={"ID":"a438ba9d-8191-4a01-bf6b-3633ee87a6de","Type":"ContainerDied","Data":"445fd41707d83bf1cac6e0a9f2f790eb2d7259ab5146bf0bd01376cd0d32ea53"}
Oct 31 20:04:15.080403 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.075286    4645 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="445fd41707d83bf1cac6e0a9f2f790eb2d7259ab5146bf0bd01376cd0d32ea53"
Oct 31 20:04:15.080403 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.075379    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a84fcb-6hhlc"
Oct 31 20:04:15.258515 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.258478    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e618e1-7bjdh"]
Oct 31 20:04:15.259148 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.259131    4645 cpu_manager.go:414] "containerMap: removing container" podUID="a438ba9d-8191-4a01-bf6b-3633ee87a6de" containerName="revoke-keys"
Oct 31 20:04:15.259148 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.259148    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a438ba9d-8191-4a01-bf6b-3633ee87a6de" containerName="revoke-keys"
Oct 31 20:04:15.259291 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.259255    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="a438ba9d-8191-4a01-bf6b-3633ee87a6de" containerName="revoke-keys"
Oct 31 20:04:15.262957 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.262935    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e618e1-7bjdh"
Oct 31 20:04:15.276118 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.276073    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:04:15.276560 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.276516    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 20:04:15.282666 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.282426    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e618e1-7bjdh"]
Oct 31 20:04:15.284745 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.284715    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l789c\" (UniqueName: \"kubernetes.io/projected/83ad495b-4d10-447c-ba58-0657d7c66403-kube-api-access-l789c\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-e618e1-7bjdh\" (UID: \"83ad495b-4d10-447c-ba58-0657d7c66403\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e618e1-7bjdh"
Oct 31 20:04:15.284910 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.284890    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83ad495b-4d10-447c-ba58-0657d7c66403-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-e618e1-7bjdh\" (UID: \"83ad495b-4d10-447c-ba58-0657d7c66403\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e618e1-7bjdh"
Oct 31 20:04:15.285025 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.285008    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83ad495b-4d10-447c-ba58-0657d7c66403-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-e618e1-7bjdh\" (UID: \"83ad495b-4d10-447c-ba58-0657d7c66403\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e618e1-7bjdh"
Oct 31 20:04:15.311381 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.311309    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:15.385837 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.385809    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l789c\" (UniqueName: \"kubernetes.io/projected/83ad495b-4d10-447c-ba58-0657d7c66403-kube-api-access-l789c\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-e618e1-7bjdh\" (UID: \"83ad495b-4d10-447c-ba58-0657d7c66403\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e618e1-7bjdh"
Oct 31 20:04:15.385953 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.385921    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83ad495b-4d10-447c-ba58-0657d7c66403-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-e618e1-7bjdh\" (UID: \"83ad495b-4d10-447c-ba58-0657d7c66403\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e618e1-7bjdh"
Oct 31 20:04:15.386037 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.385998    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83ad495b-4d10-447c-ba58-0657d7c66403-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-e618e1-7bjdh\" (UID: \"83ad495b-4d10-447c-ba58-0657d7c66403\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e618e1-7bjdh"
Oct 31 20:04:15.386721 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.386697    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83ad495b-4d10-447c-ba58-0657d7c66403-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-e618e1-7bjdh\" (UID: \"83ad495b-4d10-447c-ba58-0657d7c66403\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e618e1-7bjdh"
Oct 31 20:04:15.388289 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.388262    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83ad495b-4d10-447c-ba58-0657d7c66403-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-e618e1-7bjdh\" (UID: \"83ad495b-4d10-447c-ba58-0657d7c66403\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e618e1-7bjdh"
Oct 31 20:04:15.403062 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.403035    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l789c\" (UniqueName: \"kubernetes.io/projected/83ad495b-4d10-447c-ba58-0657d7c66403-kube-api-access-l789c\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-e618e1-7bjdh\" (UID: \"83ad495b-4d10-447c-ba58-0657d7c66403\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e618e1-7bjdh"
Oct 31 20:04:15.578916 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.578892    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6"
Oct 31 20:04:15.582911 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.582885    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e618e1-7bjdh"
Oct 31 20:04:15.588109 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.588079    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/679339df-b722-4990-b395-b14fea7f6458-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/679339df-b722-4990-b395-b14fea7f6458-istiod-ca-cert\") pod \"679339df-b722-4990-b395-b14fea7f6458\" (UID: \"679339df-b722-4990-b395-b14fea7f6458\") "
Oct 31 20:04:15.588234 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.588117    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/679339df-b722-4990-b395-b14fea7f6458-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/679339df-b722-4990-b395-b14fea7f6458-workload-socket\") pod \"679339df-b722-4990-b395-b14fea7f6458\" (UID: \"679339df-b722-4990-b395-b14fea7f6458\") "
Oct 31 20:04:15.588234 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.588171    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/679339df-b722-4990-b395-b14fea7f6458-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/679339df-b722-4990-b395-b14fea7f6458-istio-podinfo\") pod \"679339df-b722-4990-b395-b14fea7f6458\" (UID: \"679339df-b722-4990-b395-b14fea7f6458\") "
Oct 31 20:04:15.588234 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.588221    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/679339df-b722-4990-b395-b14fea7f6458-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/679339df-b722-4990-b395-b14fea7f6458-credential-socket\") pod \"679339df-b722-4990-b395-b14fea7f6458\" (UID: \"679339df-b722-4990-b395-b14fea7f6458\") "
Oct 31 20:04:15.588475 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.588248    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/679339df-b722-4990-b395-b14fea7f6458-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/679339df-b722-4990-b395-b14fea7f6458-istio-data\") pod \"679339df-b722-4990-b395-b14fea7f6458\" (UID: \"679339df-b722-4990-b395-b14fea7f6458\") "
Oct 31 20:04:15.588475 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.588423    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/679339df-b722-4990-b395-b14fea7f6458-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/679339df-b722-4990-b395-b14fea7f6458-istio-envoy\") pod \"679339df-b722-4990-b395-b14fea7f6458\" (UID: \"679339df-b722-4990-b395-b14fea7f6458\") "
Oct 31 20:04:15.588596 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.588482    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/679339df-b722-4990-b395-b14fea7f6458-kube-api-access-ltmpk\" (UniqueName: \"kubernetes.io/projected/679339df-b722-4990-b395-b14fea7f6458-kube-api-access-ltmpk\") pod \"679339df-b722-4990-b395-b14fea7f6458\" (UID: \"679339df-b722-4990-b395-b14fea7f6458\") "
Oct 31 20:04:15.588596 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.588525    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/679339df-b722-4990-b395-b14fea7f6458-istio-token\" (UniqueName: \"kubernetes.io/projected/679339df-b722-4990-b395-b14fea7f6458-istio-token\") pod \"679339df-b722-4990-b395-b14fea7f6458\" (UID: \"679339df-b722-4990-b395-b14fea7f6458\") "
Oct 31 20:04:15.588596 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.588568    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/679339df-b722-4990-b395-b14fea7f6458-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/679339df-b722-4990-b395-b14fea7f6458-workload-certs\") pod \"679339df-b722-4990-b395-b14fea7f6458\" (UID: \"679339df-b722-4990-b395-b14fea7f6458\") "
Oct 31 20:04:15.588596 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.588481    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/679339df-b722-4990-b395-b14fea7f6458-workload-socket" pod "679339df-b722-4990-b395-b14fea7f6458" (UID: "679339df-b722-4990-b395-b14fea7f6458"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:04:15.588833 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.588503    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/679339df-b722-4990-b395-b14fea7f6458-istiod-ca-cert" pod "679339df-b722-4990-b395-b14fea7f6458" (UID: "679339df-b722-4990-b395-b14fea7f6458"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:04:15.588833 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.588582    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/679339df-b722-4990-b395-b14fea7f6458-credential-socket" pod "679339df-b722-4990-b395-b14fea7f6458" (UID: "679339df-b722-4990-b395-b14fea7f6458"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:04:15.588962 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.588890    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/679339df-b722-4990-b395-b14fea7f6458-workload-certs" pod "679339df-b722-4990-b395-b14fea7f6458" (UID: "679339df-b722-4990-b395-b14fea7f6458"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:04:15.589104 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.589078    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/679339df-b722-4990-b395-b14fea7f6458-istio-data" pod "679339df-b722-4990-b395-b14fea7f6458" (UID: "679339df-b722-4990-b395-b14fea7f6458"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:04:15.589385 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.589364    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/679339df-b722-4990-b395-b14fea7f6458-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:15.589456 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.589391    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/679339df-b722-4990-b395-b14fea7f6458-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:15.589456 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.589406    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/679339df-b722-4990-b395-b14fea7f6458-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:15.589456 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.589422    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/679339df-b722-4990-b395-b14fea7f6458-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:15.589456 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.589437    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/679339df-b722-4990-b395-b14fea7f6458-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:15.591164 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.591140    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/679339df-b722-4990-b395-b14fea7f6458-istio-token" pod "679339df-b722-4990-b395-b14fea7f6458" (UID: "679339df-b722-4990-b395-b14fea7f6458"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:04:15.591164 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.591151    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/679339df-b722-4990-b395-b14fea7f6458-istio-envoy" pod "679339df-b722-4990-b395-b14fea7f6458" (UID: "679339df-b722-4990-b395-b14fea7f6458"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:04:15.591408 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.591282    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/679339df-b722-4990-b395-b14fea7f6458-istio-podinfo" pod "679339df-b722-4990-b395-b14fea7f6458" (UID: "679339df-b722-4990-b395-b14fea7f6458"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 20:04:15.591652 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.591612    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/679339df-b722-4990-b395-b14fea7f6458-kube-api-access-ltmpk" pod "679339df-b722-4990-b395-b14fea7f6458" (UID: "679339df-b722-4990-b395-b14fea7f6458"). InnerVolumeSpecName "kube-api-access-ltmpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:04:15.693862 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.690722    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc"]
Oct 31 20:04:15.693862 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.691625    4645 cpu_manager.go:414] "containerMap: removing container" podUID="679339df-b722-4990-b395-b14fea7f6458" containerName="istio-proxy"
Oct 31 20:04:15.693862 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.691642    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="679339df-b722-4990-b395-b14fea7f6458" containerName="istio-proxy"
Oct 31 20:04:15.693862 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.691688    4645 cpu_manager.go:414] "containerMap: removing container" podUID="679339df-b722-4990-b395-b14fea7f6458" containerName="istio-proxy"
Oct 31 20:04:15.693862 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.691696    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="679339df-b722-4990-b395-b14fea7f6458" containerName="istio-proxy"
Oct 31 20:04:15.693862 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.691840    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="679339df-b722-4990-b395-b14fea7f6458" containerName="istio-proxy"
Oct 31 20:04:15.696999 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.696537    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/679339df-b722-4990-b395-b14fea7f6458-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:15.696999 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.696570    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltmpk\" (UniqueName: \"kubernetes.io/projected/679339df-b722-4990-b395-b14fea7f6458-kube-api-access-ltmpk\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:15.696999 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.696587    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/679339df-b722-4990-b395-b14fea7f6458-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:15.696999 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.696603    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/679339df-b722-4990-b395-b14fea7f6458-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:15.714138 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.702603    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc"
Oct 31 20:04:15.714138 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.712312    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc"]
Oct 31 20:04:15.723633 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.723603    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-f28d48-openshift-default-dockercfg-mg2rh\""
Oct 31 20:04:15.779428 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.779319    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:15.797481 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.797454    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7309f62d-7bc0-4677-a81d-3b306f5eca82-workload-certs\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc\" (UID: \"7309f62d-7bc0-4677-a81d-3b306f5eca82\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc"
Oct 31 20:04:15.797621 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.797490    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7309f62d-7bc0-4677-a81d-3b306f5eca82-istiod-ca-cert\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc\" (UID: \"7309f62d-7bc0-4677-a81d-3b306f5eca82\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc"
Oct 31 20:04:15.797621 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.797591    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7309f62d-7bc0-4677-a81d-3b306f5eca82-istio-token\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc\" (UID: \"7309f62d-7bc0-4677-a81d-3b306f5eca82\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc"
Oct 31 20:04:15.797711 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.797652    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7309f62d-7bc0-4677-a81d-3b306f5eca82-workload-socket\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc\" (UID: \"7309f62d-7bc0-4677-a81d-3b306f5eca82\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc"
Oct 31 20:04:15.797763 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.797742    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7309f62d-7bc0-4677-a81d-3b306f5eca82-istio-podinfo\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc\" (UID: \"7309f62d-7bc0-4677-a81d-3b306f5eca82\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc"
Oct 31 20:04:15.797806 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.797774    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7309f62d-7bc0-4677-a81d-3b306f5eca82-istio-envoy\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc\" (UID: \"7309f62d-7bc0-4677-a81d-3b306f5eca82\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc"
Oct 31 20:04:15.797806 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.797793    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7309f62d-7bc0-4677-a81d-3b306f5eca82-credential-socket\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc\" (UID: \"7309f62d-7bc0-4677-a81d-3b306f5eca82\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc"
Oct 31 20:04:15.797937 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.797917    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhsdt\" (UniqueName: \"kubernetes.io/projected/7309f62d-7bc0-4677-a81d-3b306f5eca82-kube-api-access-bhsdt\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc\" (UID: \"7309f62d-7bc0-4677-a81d-3b306f5eca82\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc"
Oct 31 20:04:15.797979 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.797948    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7309f62d-7bc0-4677-a81d-3b306f5eca82-istio-data\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc\" (UID: \"7309f62d-7bc0-4677-a81d-3b306f5eca82\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc"
Oct 31 20:04:15.920919 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.920883    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7309f62d-7bc0-4677-a81d-3b306f5eca82-istio-token\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc\" (UID: \"7309f62d-7bc0-4677-a81d-3b306f5eca82\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc"
Oct 31 20:04:15.921208 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.921189    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7309f62d-7bc0-4677-a81d-3b306f5eca82-workload-socket\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc\" (UID: \"7309f62d-7bc0-4677-a81d-3b306f5eca82\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc"
Oct 31 20:04:15.921424 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.921404    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7309f62d-7bc0-4677-a81d-3b306f5eca82-istio-podinfo\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc\" (UID: \"7309f62d-7bc0-4677-a81d-3b306f5eca82\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc"
Oct 31 20:04:15.921576 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.921558    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7309f62d-7bc0-4677-a81d-3b306f5eca82-istio-envoy\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc\" (UID: \"7309f62d-7bc0-4677-a81d-3b306f5eca82\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc"
Oct 31 20:04:15.921699 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.921684    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7309f62d-7bc0-4677-a81d-3b306f5eca82-credential-socket\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc\" (UID: \"7309f62d-7bc0-4677-a81d-3b306f5eca82\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc"
Oct 31 20:04:15.921899 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.921883    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhsdt\" (UniqueName: \"kubernetes.io/projected/7309f62d-7bc0-4677-a81d-3b306f5eca82-kube-api-access-bhsdt\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc\" (UID: \"7309f62d-7bc0-4677-a81d-3b306f5eca82\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc"
Oct 31 20:04:15.922019 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.922005    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7309f62d-7bc0-4677-a81d-3b306f5eca82-istio-data\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc\" (UID: \"7309f62d-7bc0-4677-a81d-3b306f5eca82\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc"
Oct 31 20:04:15.922413 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.922383    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7309f62d-7bc0-4677-a81d-3b306f5eca82-workload-certs\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc\" (UID: \"7309f62d-7bc0-4677-a81d-3b306f5eca82\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc"
Oct 31 20:04:15.922581 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.922562    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7309f62d-7bc0-4677-a81d-3b306f5eca82-istiod-ca-cert\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc\" (UID: \"7309f62d-7bc0-4677-a81d-3b306f5eca82\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc"
Oct 31 20:04:15.923641 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.923620    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:15.924782 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.924762    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7309f62d-7bc0-4677-a81d-3b306f5eca82-workload-certs\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc\" (UID: \"7309f62d-7bc0-4677-a81d-3b306f5eca82\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc"
Oct 31 20:04:15.947628 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.926900    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7309f62d-7bc0-4677-a81d-3b306f5eca82-istio-data\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc\" (UID: \"7309f62d-7bc0-4677-a81d-3b306f5eca82\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc"
Oct 31 20:04:15.947628 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.928805    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7309f62d-7bc0-4677-a81d-3b306f5eca82-workload-socket\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc\" (UID: \"7309f62d-7bc0-4677-a81d-3b306f5eca82\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc"
Oct 31 20:04:15.947628 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.937492    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7309f62d-7bc0-4677-a81d-3b306f5eca82-credential-socket\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc\" (UID: \"7309f62d-7bc0-4677-a81d-3b306f5eca82\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc"
Oct 31 20:04:15.947628 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.939260    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7309f62d-7bc0-4677-a81d-3b306f5eca82-istiod-ca-cert\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc\" (UID: \"7309f62d-7bc0-4677-a81d-3b306f5eca82\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc"
Oct 31 20:04:15.959106 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.959051    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7309f62d-7bc0-4677-a81d-3b306f5eca82-istio-envoy\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc\" (UID: \"7309f62d-7bc0-4677-a81d-3b306f5eca82\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc"
Oct 31 20:04:15.999891 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:15.999801    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e618e1-7bjdh"]
Oct 31 20:04:16.013536 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:16.002866    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7309f62d-7bc0-4677-a81d-3b306f5eca82-istio-podinfo\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc\" (UID: \"7309f62d-7bc0-4677-a81d-3b306f5eca82\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc"
Oct 31 20:04:16.028886 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:16.027049    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhsdt\" (UniqueName: \"kubernetes.io/projected/7309f62d-7bc0-4677-a81d-3b306f5eca82-kube-api-access-bhsdt\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc\" (UID: \"7309f62d-7bc0-4677-a81d-3b306f5eca82\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc"
Oct 31 20:04:16.064818 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:16.064742    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7309f62d-7bc0-4677-a81d-3b306f5eca82-istio-token\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc\" (UID: \"7309f62d-7bc0-4677-a81d-3b306f5eca82\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc"
Oct 31 20:04:16.137250 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:16.131715    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e618e1-7bjdh" event={"ID":"83ad495b-4d10-447c-ba58-0657d7c66403","Type":"ContainerStarted","Data":"1828c4f531dcbcdb8f9bed7f5c85f2e91517757480128b2c92d336f09af1284a"}
Oct 31 20:04:16.144345 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:16.143949    4645 generic.go:356] "Generic (PLEG): container finished" podID="679339df-b722-4990-b395-b14fea7f6458" containerID="f95e143bc1d14b498d313fe8ff37414d1995008395a2383c52a2073a864a1859" exitCode=0
Oct 31 20:04:16.144345 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:16.144060    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6" event={"ID":"679339df-b722-4990-b395-b14fea7f6458","Type":"ContainerDied","Data":"f95e143bc1d14b498d313fe8ff37414d1995008395a2383c52a2073a864a1859"}
Oct 31 20:04:16.144345 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:16.144121    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6" event={"ID":"679339df-b722-4990-b395-b14fea7f6458","Type":"ContainerDied","Data":"b88c7fb6aefcc24f7ce4b491dd43d4f4e81d9dd76867d164159c77045b303fc2"}
Oct 31 20:04:16.144345 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:16.144154    4645 scope.go:122] "RemoveContainer" containerID="f95e143bc1d14b498d313fe8ff37414d1995008395a2383c52a2073a864a1859"
Oct 31 20:04:16.144345 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:16.144221    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6"
Oct 31 20:04:16.150371 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:16.147670    4645 generic.go:356] "Generic (PLEG): container finished" podID="ba844f48-656b-492b-8a60-f1e85d33d977" containerID="8aa46e2c2b335792a0224ab9f712b47604539d50f509b5ba4c83b5cf44720ee0" exitCode=0
Oct 31 20:04:16.150371 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:16.147877    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd" event={"ID":"ba844f48-656b-492b-8a60-f1e85d33d977","Type":"ContainerDied","Data":"8aa46e2c2b335792a0224ab9f712b47604539d50f509b5ba4c83b5cf44720ee0"}
Oct 31 20:04:16.161690 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:16.161594    4645 scope.go:122] "RemoveContainer" containerID="56fe3396ecf1c21d316da758dac8576fc1887b1d90379d62b9cb2b8754cb241b"
Oct 31 20:04:16.177480 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:16.177451    4645 scope.go:122] "RemoveContainer" containerID="f95e143bc1d14b498d313fe8ff37414d1995008395a2383c52a2073a864a1859"
Oct 31 20:04:16.178702 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:04:16.178675    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f95e143bc1d14b498d313fe8ff37414d1995008395a2383c52a2073a864a1859\": container with ID starting with f95e143bc1d14b498d313fe8ff37414d1995008395a2383c52a2073a864a1859 not found: ID does not exist" containerID="f95e143bc1d14b498d313fe8ff37414d1995008395a2383c52a2073a864a1859"
Oct 31 20:04:16.178817 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:16.178714    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f95e143bc1d14b498d313fe8ff37414d1995008395a2383c52a2073a864a1859"} err="failed to get container status \"f95e143bc1d14b498d313fe8ff37414d1995008395a2383c52a2073a864a1859\": rpc error: code = NotFound desc = could not find container \"f95e143bc1d14b498d313fe8ff37414d1995008395a2383c52a2073a864a1859\": container with ID starting with f95e143bc1d14b498d313fe8ff37414d1995008395a2383c52a2073a864a1859 not found: ID does not exist"
Oct 31 20:04:16.178817 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:16.178738    4645 scope.go:122] "RemoveContainer" containerID="56fe3396ecf1c21d316da758dac8576fc1887b1d90379d62b9cb2b8754cb241b"
Oct 31 20:04:16.179291 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:04:16.179198    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56fe3396ecf1c21d316da758dac8576fc1887b1d90379d62b9cb2b8754cb241b\": container with ID starting with 56fe3396ecf1c21d316da758dac8576fc1887b1d90379d62b9cb2b8754cb241b not found: ID does not exist" containerID="56fe3396ecf1c21d316da758dac8576fc1887b1d90379d62b9cb2b8754cb241b"
Oct 31 20:04:16.179291 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:16.179228    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56fe3396ecf1c21d316da758dac8576fc1887b1d90379d62b9cb2b8754cb241b"} err="failed to get container status \"56fe3396ecf1c21d316da758dac8576fc1887b1d90379d62b9cb2b8754cb241b\": rpc error: code = NotFound desc = could not find container \"56fe3396ecf1c21d316da758dac8576fc1887b1d90379d62b9cb2b8754cb241b\": container with ID starting with 56fe3396ecf1c21d316da758dac8576fc1887b1d90379d62b9cb2b8754cb241b not found: ID does not exist"
Oct 31 20:04:16.188277 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:16.188244    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6"]
Oct 31 20:04:16.193943 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:16.193921    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-bd1c9f-openshift-default-74d884b45c-7dhw6"]
Oct 31 20:04:16.297130 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:16.297101    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd"
Oct 31 20:04:16.329171 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:16.329148    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc"
Oct 31 20:04:16.338967 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:16.338795    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba844f48-656b-492b-8a60-f1e85d33d977-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba844f48-656b-492b-8a60-f1e85d33d977-workload-socket\") pod \"ba844f48-656b-492b-8a60-f1e85d33d977\" (UID: \"ba844f48-656b-492b-8a60-f1e85d33d977\") "
Oct 31 20:04:16.338967 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:16.338923    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba844f48-656b-492b-8a60-f1e85d33d977-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba844f48-656b-492b-8a60-f1e85d33d977-credential-socket\") pod \"ba844f48-656b-492b-8a60-f1e85d33d977\" (UID: \"ba844f48-656b-492b-8a60-f1e85d33d977\") "
Oct 31 20:04:16.339191 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:16.338973    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba844f48-656b-492b-8a60-f1e85d33d977-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba844f48-656b-492b-8a60-f1e85d33d977-istio-envoy\") pod \"ba844f48-656b-492b-8a60-f1e85d33d977\" (UID: \"ba844f48-656b-492b-8a60-f1e85d33d977\") "
Oct 31 20:04:16.339191 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:16.339005    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba844f48-656b-492b-8a60-f1e85d33d977-kube-api-access-5psls\" (UniqueName: \"kubernetes.io/projected/ba844f48-656b-492b-8a60-f1e85d33d977-kube-api-access-5psls\") pod \"ba844f48-656b-492b-8a60-f1e85d33d977\" (UID: \"ba844f48-656b-492b-8a60-f1e85d33d977\") "
Oct 31 20:04:16.339191 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:16.339031    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba844f48-656b-492b-8a60-f1e85d33d977-workload-socket" pod "ba844f48-656b-492b-8a60-f1e85d33d977" (UID: "ba844f48-656b-492b-8a60-f1e85d33d977"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:04:16.339191 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:16.339103    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba844f48-656b-492b-8a60-f1e85d33d977-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba844f48-656b-492b-8a60-f1e85d33d977-istio-data\") pod \"ba844f48-656b-492b-8a60-f1e85d33d977\" (UID: \"ba844f48-656b-492b-8a60-f1e85d33d977\") "
Oct 31 20:04:16.339191 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:16.339151    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ba844f48-656b-492b-8a60-f1e85d33d977-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba844f48-656b-492b-8a60-f1e85d33d977-istio-podinfo\") pod \"ba844f48-656b-492b-8a60-f1e85d33d977\" (UID: \"ba844f48-656b-492b-8a60-f1e85d33d977\") "
Oct 31 20:04:16.339588 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:16.339188    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba844f48-656b-492b-8a60-f1e85d33d977-credential-socket" pod "ba844f48-656b-492b-8a60-f1e85d33d977" (UID: "ba844f48-656b-492b-8a60-f1e85d33d977"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:04:16.339588 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:16.339496    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba844f48-656b-492b-8a60-f1e85d33d977-istio-data" pod "ba844f48-656b-492b-8a60-f1e85d33d977" (UID: "ba844f48-656b-492b-8a60-f1e85d33d977"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:04:16.339777 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:16.339663    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba844f48-656b-492b-8a60-f1e85d33d977-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba844f48-656b-492b-8a60-f1e85d33d977-istiod-ca-cert\") pod \"ba844f48-656b-492b-8a60-f1e85d33d977\" (UID: \"ba844f48-656b-492b-8a60-f1e85d33d977\") "
Oct 31 20:04:16.339777 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:16.339712    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba844f48-656b-492b-8a60-f1e85d33d977-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba844f48-656b-492b-8a60-f1e85d33d977-workload-certs\") pod \"ba844f48-656b-492b-8a60-f1e85d33d977\" (UID: \"ba844f48-656b-492b-8a60-f1e85d33d977\") "
Oct 31 20:04:16.339777 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:16.339741    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba844f48-656b-492b-8a60-f1e85d33d977-istio-token\" (UniqueName: \"kubernetes.io/projected/ba844f48-656b-492b-8a60-f1e85d33d977-istio-token\") pod \"ba844f48-656b-492b-8a60-f1e85d33d977\" (UID: \"ba844f48-656b-492b-8a60-f1e85d33d977\") "
Oct 31 20:04:16.340195 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:16.340161    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba844f48-656b-492b-8a60-f1e85d33d977-workload-certs" pod "ba844f48-656b-492b-8a60-f1e85d33d977" (UID: "ba844f48-656b-492b-8a60-f1e85d33d977"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:04:16.340678 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:16.340655    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba844f48-656b-492b-8a60-f1e85d33d977-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:16.340792 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:16.340680    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba844f48-656b-492b-8a60-f1e85d33d977-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:16.340792 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:16.340695    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba844f48-656b-492b-8a60-f1e85d33d977-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:16.340792 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:16.340703    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba844f48-656b-492b-8a60-f1e85d33d977-istiod-ca-cert" pod "ba844f48-656b-492b-8a60-f1e85d33d977" (UID: "ba844f48-656b-492b-8a60-f1e85d33d977"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:04:16.340792 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:16.340709    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba844f48-656b-492b-8a60-f1e85d33d977-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:16.341933 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:16.341901    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba844f48-656b-492b-8a60-f1e85d33d977-istio-podinfo" pod "ba844f48-656b-492b-8a60-f1e85d33d977" (UID: "ba844f48-656b-492b-8a60-f1e85d33d977"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 20:04:16.342830 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:16.342804    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba844f48-656b-492b-8a60-f1e85d33d977-istio-token" pod "ba844f48-656b-492b-8a60-f1e85d33d977" (UID: "ba844f48-656b-492b-8a60-f1e85d33d977"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:04:16.343272 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:16.343236    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba844f48-656b-492b-8a60-f1e85d33d977-kube-api-access-5psls" pod "ba844f48-656b-492b-8a60-f1e85d33d977" (UID: "ba844f48-656b-492b-8a60-f1e85d33d977"). InnerVolumeSpecName "kube-api-access-5psls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:04:16.343502 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:16.343402    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba844f48-656b-492b-8a60-f1e85d33d977-istio-envoy" pod "ba844f48-656b-492b-8a60-f1e85d33d977" (UID: "ba844f48-656b-492b-8a60-f1e85d33d977"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:04:16.442370 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:16.442323    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba844f48-656b-492b-8a60-f1e85d33d977-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:16.442475 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:16.442377    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5psls\" (UniqueName: \"kubernetes.io/projected/ba844f48-656b-492b-8a60-f1e85d33d977-kube-api-access-5psls\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:16.442475 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:16.442395    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba844f48-656b-492b-8a60-f1e85d33d977-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:16.442475 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:16.442410    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba844f48-656b-492b-8a60-f1e85d33d977-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:16.442475 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:16.442425    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba844f48-656b-492b-8a60-f1e85d33d977-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:16.661210 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:04:16.661182    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7309f62d_7bc0_4677_a81d_3b306f5eca82.slice/crio-be87b0bfca961aa4010714e627f42dfd4973491edfc3e5716f2a57bb6707fd1a WatchSource:0}: Error finding container be87b0bfca961aa4010714e627f42dfd4973491edfc3e5716f2a57bb6707fd1a: Status 404 returned error can't find the container with id be87b0bfca961aa4010714e627f42dfd4973491edfc3e5716f2a57bb6707fd1a
Oct 31 20:04:16.662362 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:16.662319    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc"]
Oct 31 20:04:16.664026 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:16.663990    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:04:16.664127 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:16.664059    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:04:16.664127 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:16.664100    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:04:17.063458 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:17.063342    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:17.158347 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:17.158302    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e618e1-7bjdh" event={"ID":"83ad495b-4d10-447c-ba58-0657d7c66403","Type":"ContainerStarted","Data":"22256a7bcfc34444d4ebc822f63eed8c817aa87fd15bcb69ba53a472f960e054"}
Oct 31 20:04:17.161216 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:17.161192    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd"
Oct 31 20:04:17.161430 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:17.161246    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd" event={"ID":"ba844f48-656b-492b-8a60-f1e85d33d977","Type":"ContainerDied","Data":"850a79bf90284e00599330603d201ac141e9fb3c8179e32cefa818d8d29d9c14"}
Oct 31 20:04:17.161430 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:17.161304    4645 scope.go:122] "RemoveContainer" containerID="8aa46e2c2b335792a0224ab9f712b47604539d50f509b5ba4c83b5cf44720ee0"
Oct 31 20:04:17.163323 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:17.163294    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc" event={"ID":"7309f62d-7bc0-4677-a81d-3b306f5eca82","Type":"ContainerStarted","Data":"766a6cd8f5a7309349b6d7d5323cb51a83328f800aa67b77d9a7f37492975905"}
Oct 31 20:04:17.163562 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:17.163462    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc" event={"ID":"7309f62d-7bc0-4677-a81d-3b306f5eca82","Type":"ContainerStarted","Data":"be87b0bfca961aa4010714e627f42dfd4973491edfc3e5716f2a57bb6707fd1a"}
Oct 31 20:04:17.184567 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:17.184514    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e618e1-7bjdh" podStartSLOduration=2.184496173 podStartE2EDuration="2.184496173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:04:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:04:17.182388469 +0000 UTC m=+2687.755622596" watchObservedRunningTime="2026-10-31 20:04:17.184496173 +0000 UTC m=+2687.757730303"
Oct 31 20:04:17.215623 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:17.215584    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc" podStartSLOduration=2.215570416 podStartE2EDuration="2.215570416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:04:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:04:17.210349203 +0000 UTC m=+2687.783583325" watchObservedRunningTime="2026-10-31 20:04:17.215570416 +0000 UTC m=+2687.788804545"
Oct 31 20:04:17.232368 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:17.232339    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd"]
Oct 31 20:04:17.260303 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:17.260276    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-23843f-openshift-default-54cf54885-bzzjd"]
Oct 31 20:04:17.333572 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:17.333491    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc"
Oct 31 20:04:17.382807 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:17.382763    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc"
Oct 31 20:04:17.620723 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:17.620644    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="679339df-b722-4990-b395-b14fea7f6458" path="/var/lib/kubelet/pods/679339df-b722-4990-b395-b14fea7f6458/volumes"
Oct 31 20:04:17.621416 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:17.621395    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba844f48-656b-492b-8a60-f1e85d33d977" path="/var/lib/kubelet/pods/ba844f48-656b-492b-8a60-f1e85d33d977/volumes"
Oct 31 20:04:18.172801 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:18.172770    4645 generic.go:356] "Generic (PLEG): container finished" podID="83ad495b-4d10-447c-ba58-0657d7c66403" containerID="22256a7bcfc34444d4ebc822f63eed8c817aa87fd15bcb69ba53a472f960e054" exitCode=0
Oct 31 20:04:18.173250 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:18.172857    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e618e1-7bjdh" event={"ID":"83ad495b-4d10-447c-ba58-0657d7c66403","Type":"ContainerDied","Data":"22256a7bcfc34444d4ebc822f63eed8c817aa87fd15bcb69ba53a472f960e054"}
Oct 31 20:04:18.174375 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:18.174355    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc"
Oct 31 20:04:18.175320 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:18.175300    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc"
Oct 31 20:04:18.182212 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:18.182191    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:18.331886 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:18.331842    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:19.035643 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:19.035609    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e02c2e34-rq6jc"]
Oct 31 20:04:19.040576 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:19.040545    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e02c2e34-rq6jc"]
Oct 31 20:04:19.187112 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:19.187081    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:19.303153 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:19.303075    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:19.359422 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:19.359401    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e618e1-7bjdh"
Oct 31 20:04:19.397482 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:19.397446    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83ad495b-4d10-447c-ba58-0657d7c66403-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83ad495b-4d10-447c-ba58-0657d7c66403-maas-api-xks-ca\") pod \"83ad495b-4d10-447c-ba58-0657d7c66403\" (UID: \"83ad495b-4d10-447c-ba58-0657d7c66403\") "
Oct 31 20:04:19.397625 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:19.397581    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83ad495b-4d10-447c-ba58-0657d7c66403-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83ad495b-4d10-447c-ba58-0657d7c66403-maas-api-openshift-ca\") pod \"83ad495b-4d10-447c-ba58-0657d7c66403\" (UID: \"83ad495b-4d10-447c-ba58-0657d7c66403\") "
Oct 31 20:04:19.397759 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:19.397742    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83ad495b-4d10-447c-ba58-0657d7c66403-kube-api-access-l789c\" (UniqueName: \"kubernetes.io/projected/83ad495b-4d10-447c-ba58-0657d7c66403-kube-api-access-l789c\") pod \"83ad495b-4d10-447c-ba58-0657d7c66403\" (UID: \"83ad495b-4d10-447c-ba58-0657d7c66403\") "
Oct 31 20:04:19.399348 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:19.397999    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83ad495b-4d10-447c-ba58-0657d7c66403-maas-api-openshift-ca" pod "83ad495b-4d10-447c-ba58-0657d7c66403" (UID: "83ad495b-4d10-447c-ba58-0657d7c66403"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:04:19.399348 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:19.398504    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83ad495b-4d10-447c-ba58-0657d7c66403-maas-api-openshift-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:19.403788 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:19.403762    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83ad495b-4d10-447c-ba58-0657d7c66403-maas-api-xks-ca" pod "83ad495b-4d10-447c-ba58-0657d7c66403" (UID: "83ad495b-4d10-447c-ba58-0657d7c66403"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:04:19.403902 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:19.403854    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83ad495b-4d10-447c-ba58-0657d7c66403-kube-api-access-l789c" pod "83ad495b-4d10-447c-ba58-0657d7c66403" (UID: "83ad495b-4d10-447c-ba58-0657d7c66403"). InnerVolumeSpecName "kube-api-access-l789c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:04:19.500205 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:19.500171    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83ad495b-4d10-447c-ba58-0657d7c66403-maas-api-xks-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:19.500350 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:19.500209    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l789c\" (UniqueName: \"kubernetes.io/projected/83ad495b-4d10-447c-ba58-0657d7c66403-kube-api-access-l789c\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:19.578813 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:19.578685    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:19.621788 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:19.621750    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d14cdb3-49a7-4bed-ac45-b18e6452ebbe" path="/var/lib/kubelet/pods/1d14cdb3-49a7-4bed-ac45-b18e6452ebbe/volumes"
Oct 31 20:04:19.725482 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:19.725452    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:20.136223 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:20.136188    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:20.219554 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:20.219512    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e618e1-7bjdh"
Oct 31 20:04:20.219960 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:20.219552    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e618e1-7bjdh" event={"ID":"83ad495b-4d10-447c-ba58-0657d7c66403","Type":"ContainerDied","Data":"1828c4f531dcbcdb8f9bed7f5c85f2e91517757480128b2c92d336f09af1284a"}
Oct 31 20:04:20.219960 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:20.219613    4645 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1828c4f531dcbcdb8f9bed7f5c85f2e91517757480128b2c92d336f09af1284a"
Oct 31 20:04:20.347054 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:20.347021    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:20.790247 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:20.790192    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:20.964526 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:20.964493    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:21.303795 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:21.303756    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:22.487393 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:22.487352    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:23.541603 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:23.541562    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:24.496052 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:24.496014    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:25.030667 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:25.030633    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89846e5e-82ghj"]
Oct 31 20:04:25.034549 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:25.034527    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89846e5e-82ghj"]
Oct 31 20:04:25.446914 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:25.446879    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:25.554578 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:25.554544    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:25.620062 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:25.620033    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d5386a8-12b7-4668-a255-7d0861f01989" path="/var/lib/kubelet/pods/6d5386a8-12b7-4668-a255-7d0861f01989/volumes"
Oct 31 20:04:26.274819 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:26.274786    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:27.106579 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:27.106540    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:27.958052 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:27.958013    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:29.153563 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:29.153531    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:29.942275 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:29.942242    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:30.055459 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:30.055422    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b66e50dd-pbgn9"]
Oct 31 20:04:30.059253 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:30.059218    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b66e50dd-pbgn9"]
Oct 31 20:04:30.659634 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:30.659603    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:31.537268 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:31.537229    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:31.618881 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:31.618849    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e725d80-b1c1-45e1-8036-65313087d735" path="/var/lib/kubelet/pods/9e725d80-b1c1-45e1-8036-65313087d735/volumes"
Oct 31 20:04:31.859039 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:31.858963    4645 scope.go:122] "RemoveContainer" containerID="ecc3a2c7dddafc62fe88a0dea019adff03ee5b1f8a7a89917487c455745321e7"
Oct 31 20:04:32.250132 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:32.250100    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:32.967427 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:32.967387    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:33.313567 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:33.313314    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:33.428091 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:33.428057    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:33.472797 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:33.472757    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e618e1-7bjdh"]
Oct 31 20:04:33.480948 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:33.480910    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e618e1-7bjdh"]
Oct 31 20:04:33.543731 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:33.543696    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w6-a891ce/rate-test-model-a891ce-kserve-55f654969c-tf5cj"]
Oct 31 20:04:33.553295 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:33.553262    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:33.618926 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:33.618846    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83ad495b-4d10-447c-ba58-0657d7c66403" path="/var/lib/kubelet/pods/83ad495b-4d10-447c-ba58-0657d7c66403/volumes"
Oct 31 20:04:33.675100 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:33.675066    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:34.054153 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:34.054118    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:34.472153 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:34.472121    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:35.368640 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:35.368587    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:36.035148 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:36.035111    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f70b2521-7sgrn"]
Oct 31 20:04:36.039166 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:36.039138    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f70b2521-7sgrn"]
Oct 31 20:04:36.169191 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:36.169156    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:36.246125 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:04:36.246093    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 20:04:36.308402 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:36.308257    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:37.163264 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:37.163218    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:37.618175 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:37.618102    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18543794-fe65-4afa-8335-54df6a28f6db" path="/var/lib/kubelet/pods/18543794-fe65-4afa-8335-54df6a28f6db/volumes"
Oct 31 20:04:37.997483 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:37.997455    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:38.716463 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:38.716421    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:38.746902 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:38.746867    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a84fcb-5465fc5b64-v2fc5"]
Oct 31 20:04:38.747101 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:38.747054    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-a84fcb-5465fc5b64-v2fc5" podUID="ed6a7560-203e-4f9a-92ba-aa98b0564e09" containerName="payload-processing" containerID="cri-o://b83f31a519e22805250dea81c103115de8a452c333e9ef32af9a054a9328a056" gracePeriod=30
Oct 31 20:04:38.870210 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:38.870162    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:39.251910 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:39.251879    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a84fcb-6hhlc"]
Oct 31 20:04:39.252856 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:39.252829    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a84fcb-5465fc5b64-v2fc5"
Oct 31 20:04:39.256768 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:39.256740    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-a84fcb-6hhlc"]
Oct 31 20:04:39.356356 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:39.354035    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed6a7560-203e-4f9a-92ba-aa98b0564e09-plugins-config-volume" pod "ed6a7560-203e-4f9a-92ba-aa98b0564e09" (UID: "ed6a7560-203e-4f9a-92ba-aa98b0564e09"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:04:39.361452 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:39.361421    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed6a7560-203e-4f9a-92ba-aa98b0564e09-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed6a7560-203e-4f9a-92ba-aa98b0564e09-plugins-config-volume\") pod \"ed6a7560-203e-4f9a-92ba-aa98b0564e09\" (UID: \"ed6a7560-203e-4f9a-92ba-aa98b0564e09\") "
Oct 31 20:04:39.361625 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:39.361544    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed6a7560-203e-4f9a-92ba-aa98b0564e09-kube-api-access-kkrlt\" (UniqueName: \"kubernetes.io/projected/ed6a7560-203e-4f9a-92ba-aa98b0564e09-kube-api-access-kkrlt\") pod \"ed6a7560-203e-4f9a-92ba-aa98b0564e09\" (UID: \"ed6a7560-203e-4f9a-92ba-aa98b0564e09\") "
Oct 31 20:04:39.363303 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:39.362755    4645 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed6a7560-203e-4f9a-92ba-aa98b0564e09-plugins-config-volume\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:39.368713 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:39.368684    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed6a7560-203e-4f9a-92ba-aa98b0564e09-kube-api-access-kkrlt" pod "ed6a7560-203e-4f9a-92ba-aa98b0564e09" (UID: "ed6a7560-203e-4f9a-92ba-aa98b0564e09"). InnerVolumeSpecName "kube-api-access-kkrlt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:04:39.413081 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:39.413044    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:39.420782 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:39.420746    4645 generic.go:356] "Generic (PLEG): container finished" podID="ed6a7560-203e-4f9a-92ba-aa98b0564e09" containerID="b83f31a519e22805250dea81c103115de8a452c333e9ef32af9a054a9328a056" exitCode=0
Oct 31 20:04:39.420959 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:39.420792    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a84fcb-5465fc5b64-v2fc5"
Oct 31 20:04:39.420959 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:39.420825    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a84fcb-5465fc5b64-v2fc5" event={"ID":"ed6a7560-203e-4f9a-92ba-aa98b0564e09","Type":"ContainerDied","Data":"b83f31a519e22805250dea81c103115de8a452c333e9ef32af9a054a9328a056"}
Oct 31 20:04:39.420959 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:39.420879    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a84fcb-5465fc5b64-v2fc5" event={"ID":"ed6a7560-203e-4f9a-92ba-aa98b0564e09","Type":"ContainerDied","Data":"c3654a652310eff178d41e1b35393641c400ab6639db2ec6fb1bad5eb8d77c15"}
Oct 31 20:04:39.420959 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:39.420906    4645 scope.go:122] "RemoveContainer" containerID="b83f31a519e22805250dea81c103115de8a452c333e9ef32af9a054a9328a056"
Oct 31 20:04:39.435272 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:39.435254    4645 scope.go:122] "RemoveContainer" containerID="b83f31a519e22805250dea81c103115de8a452c333e9ef32af9a054a9328a056"
Oct 31 20:04:39.435628 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:04:39.435584    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b83f31a519e22805250dea81c103115de8a452c333e9ef32af9a054a9328a056\": container with ID starting with b83f31a519e22805250dea81c103115de8a452c333e9ef32af9a054a9328a056 not found: ID does not exist" containerID="b83f31a519e22805250dea81c103115de8a452c333e9ef32af9a054a9328a056"
Oct 31 20:04:39.435628 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:39.435610    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b83f31a519e22805250dea81c103115de8a452c333e9ef32af9a054a9328a056"} err="failed to get container status \"b83f31a519e22805250dea81c103115de8a452c333e9ef32af9a054a9328a056\": rpc error: code = NotFound desc = could not find container \"b83f31a519e22805250dea81c103115de8a452c333e9ef32af9a054a9328a056\": container with ID starting with b83f31a519e22805250dea81c103115de8a452c333e9ef32af9a054a9328a056 not found: ID does not exist"
Oct 31 20:04:39.460134 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:39.460114    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a84fcb-5465fc5b64-v2fc5"]
Oct 31 20:04:39.465096 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:39.465070    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkrlt\" (UniqueName: \"kubernetes.io/projected/ed6a7560-203e-4f9a-92ba-aa98b0564e09-kube-api-access-kkrlt\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:39.467630 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:39.467610    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a84fcb-5465fc5b64-v2fc5"]
Oct 31 20:04:39.616750 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:39.616718    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:39.619598 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:39.619573    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a438ba9d-8191-4a01-bf6b-3633ee87a6de" path="/var/lib/kubelet/pods/a438ba9d-8191-4a01-bf6b-3633ee87a6de/volumes"
Oct 31 20:04:39.620049 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:39.620034    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed6a7560-203e-4f9a-92ba-aa98b0564e09" path="/var/lib/kubelet/pods/ed6a7560-203e-4f9a-92ba-aa98b0564e09/volumes"
Oct 31 20:04:39.804212 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:39.804096    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:40.737975 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:40.737934    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:41.392299 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:41.392263    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:41.514381 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:41.514349    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:41.624914 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:41.624869    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:42.381730 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:42.381697    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:42.665470 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:42.665436    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w6-e1d060/rate-test-model-e1d060-kserve-5969df469-4mfjn"]
Oct 31 20:04:42.696762 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:42.696731    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:42.900109 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:42.900073    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a891ce-fpmrl"]
Oct 31 20:04:42.900735 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:42.900716    4645 cpu_manager.go:414] "containerMap: removing container" podUID="83ad495b-4d10-447c-ba58-0657d7c66403" containerName="revoke-keys"
Oct 31 20:04:42.900735 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:42.900732    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83ad495b-4d10-447c-ba58-0657d7c66403" containerName="revoke-keys"
Oct 31 20:04:42.900933 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:42.900782    4645 cpu_manager.go:414] "containerMap: removing container" podUID="ed6a7560-203e-4f9a-92ba-aa98b0564e09" containerName="payload-processing"
Oct 31 20:04:42.900933 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:42.900791    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed6a7560-203e-4f9a-92ba-aa98b0564e09" containerName="payload-processing"
Oct 31 20:04:42.900933 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:42.900805    4645 cpu_manager.go:414] "containerMap: removing container" podUID="ba844f48-656b-492b-8a60-f1e85d33d977" containerName="istio-proxy"
Oct 31 20:04:42.900933 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:42.900813    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba844f48-656b-492b-8a60-f1e85d33d977" containerName="istio-proxy"
Oct 31 20:04:42.900933 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:42.900935    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="83ad495b-4d10-447c-ba58-0657d7c66403" containerName="revoke-keys"
Oct 31 20:04:42.901098 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:42.900949    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="679339df-b722-4990-b395-b14fea7f6458" containerName="istio-proxy"
Oct 31 20:04:42.901098 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:42.900957    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed6a7560-203e-4f9a-92ba-aa98b0564e09" containerName="payload-processing"
Oct 31 20:04:42.901098 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:42.900962    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba844f48-656b-492b-8a60-f1e85d33d977" containerName="istio-proxy"
Oct 31 20:04:42.905947 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:42.905926    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a891ce-fpmrl"
Oct 31 20:04:42.911569 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:42.911527    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:04:42.911732 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:42.911630    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 20:04:42.917371 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:42.917278    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a891ce-fpmrl"]
Oct 31 20:04:43.034826 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:43.034795    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82b3f50e-f79d-4f62-a212-1138dd4fd528-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-a891ce-fpmrl\" (UID: \"82b3f50e-f79d-4f62-a212-1138dd4fd528\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a891ce-fpmrl"
Oct 31 20:04:43.034980 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:43.034831    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dvj8\" (UniqueName: \"kubernetes.io/projected/82b3f50e-f79d-4f62-a212-1138dd4fd528-kube-api-access-7dvj8\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-a891ce-fpmrl\" (UID: \"82b3f50e-f79d-4f62-a212-1138dd4fd528\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a891ce-fpmrl"
Oct 31 20:04:43.035074 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:43.035052    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82b3f50e-f79d-4f62-a212-1138dd4fd528-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-a891ce-fpmrl\" (UID: \"82b3f50e-f79d-4f62-a212-1138dd4fd528\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a891ce-fpmrl"
Oct 31 20:04:43.136630 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:43.136603    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82b3f50e-f79d-4f62-a212-1138dd4fd528-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-a891ce-fpmrl\" (UID: \"82b3f50e-f79d-4f62-a212-1138dd4fd528\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a891ce-fpmrl"
Oct 31 20:04:43.136782 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:43.136700    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82b3f50e-f79d-4f62-a212-1138dd4fd528-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-a891ce-fpmrl\" (UID: \"82b3f50e-f79d-4f62-a212-1138dd4fd528\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a891ce-fpmrl"
Oct 31 20:04:43.136782 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:43.136719    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dvj8\" (UniqueName: \"kubernetes.io/projected/82b3f50e-f79d-4f62-a212-1138dd4fd528-kube-api-access-7dvj8\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-a891ce-fpmrl\" (UID: \"82b3f50e-f79d-4f62-a212-1138dd4fd528\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a891ce-fpmrl"
Oct 31 20:04:43.137498 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:43.137473    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82b3f50e-f79d-4f62-a212-1138dd4fd528-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-a891ce-fpmrl\" (UID: \"82b3f50e-f79d-4f62-a212-1138dd4fd528\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a891ce-fpmrl"
Oct 31 20:04:43.139309 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:43.139287    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82b3f50e-f79d-4f62-a212-1138dd4fd528-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-a891ce-fpmrl\" (UID: \"82b3f50e-f79d-4f62-a212-1138dd4fd528\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a891ce-fpmrl"
Oct 31 20:04:43.148529 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:43.148507    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dvj8\" (UniqueName: \"kubernetes.io/projected/82b3f50e-f79d-4f62-a212-1138dd4fd528-kube-api-access-7dvj8\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-a891ce-fpmrl\" (UID: \"82b3f50e-f79d-4f62-a212-1138dd4fd528\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a891ce-fpmrl"
Oct 31 20:04:43.221501 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:43.221433    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a891ce-fpmrl"
Oct 31 20:04:43.251473 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:43.251441    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:43.386302 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:43.386266    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:43.534321 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:43.534290    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a891ce-fpmrl"]
Oct 31 20:04:43.535424 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:04:43.535389    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82b3f50e_f79d_4f62_a212_1138dd4fd528.slice/crio-83d3b85012bfcc4d7cb6bfba49e94032516cd80756d03cb53b5e9d99b0f19883 WatchSource:0}: Error finding container 83d3b85012bfcc4d7cb6bfba49e94032516cd80756d03cb53b5e9d99b0f19883: Status 404 returned error can't find the container with id 83d3b85012bfcc4d7cb6bfba49e94032516cd80756d03cb53b5e9d99b0f19883
Oct 31 20:04:43.565079 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:43.565053    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr"]
Oct 31 20:04:43.565316 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:43.565282    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr" podUID="52d535c0-6d59-4788-9258-bbee438df233" containerName="istio-proxy" containerID="cri-o://c17be81049cf163d701039e21a9a7baf8ab010531b946507691e08439cf34bd5" gracePeriod=30
Oct 31 20:04:43.571985 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:43.571963    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph"]
Oct 31 20:04:43.572134 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:43.572115    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph" podUID="b24f33df-8939-405b-83a6-8d7af872e7da" containerName="istio-proxy" containerID="cri-o://b3422fc5e1f05bc8edd9545342a5050b0eb6813883c69fb12fe63c3bbc81ae57" gracePeriod=30
Oct 31 20:04:43.589495 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:43.589458    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8"]
Oct 31 20:04:43.589772 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:43.589744    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8" podUID="ede30242-d4ae-4caa-9c4b-8918314a1ead" containerName="istio-proxy" containerID="cri-o://85880c2b644d01275aa9674a5d67f076e83cac0aa9206136cca068e69f29023f" gracePeriod=30
Oct 31 20:04:43.598633 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:43.598594    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:43.715352 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:43.701908    4645 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 20:04:43.715352 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:43.701970    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8" podUID="ede30242-d4ae-4caa-9c4b-8918314a1ead" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 20:04:43.738373 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:43.730440    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:44.465704 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:44.465668    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a891ce-fpmrl" event={"ID":"82b3f50e-f79d-4f62-a212-1138dd4fd528","Type":"ContainerStarted","Data":"83592c7b557b846803e645ebeb8f0000ef2ee39cc5c79d07175b3cf4f00d8bda"}
Oct 31 20:04:44.465889 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:44.465725    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a891ce-fpmrl" event={"ID":"82b3f50e-f79d-4f62-a212-1138dd4fd528","Type":"ContainerStarted","Data":"83d3b85012bfcc4d7cb6bfba49e94032516cd80756d03cb53b5e9d99b0f19883"}
Oct 31 20:04:44.493969 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:44.493925    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a891ce-fpmrl" podStartSLOduration=2.493913841 podStartE2EDuration="2.493913841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:04:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:04:44.489939162 +0000 UTC m=+2715.063173291" watchObservedRunningTime="2026-10-31 20:04:44.493913841 +0000 UTC m=+2715.067147969"
Oct 31 20:04:44.674766 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:44.674736    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:45.476021 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:45.475982    4645 generic.go:356] "Generic (PLEG): container finished" podID="82b3f50e-f79d-4f62-a212-1138dd4fd528" containerID="83592c7b557b846803e645ebeb8f0000ef2ee39cc5c79d07175b3cf4f00d8bda" exitCode=0
Oct 31 20:04:45.476021 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:45.476016    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a891ce-fpmrl" event={"ID":"82b3f50e-f79d-4f62-a212-1138dd4fd528","Type":"ContainerDied","Data":"83592c7b557b846803e645ebeb8f0000ef2ee39cc5c79d07175b3cf4f00d8bda"}
Oct 31 20:04:45.682592 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:45.682557    4645 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 20:04:45.682857 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:45.682826    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr" podUID="52d535c0-6d59-4788-9258-bbee438df233" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 20:04:45.764277 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:45.764197    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:46.039869 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:46.039777    4645 patch_prober.go:29] interesting pod/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 20:04:46.039869 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:46.039846    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph" podUID="b24f33df-8939-405b-83a6-8d7af872e7da" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 20:04:46.634743 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:46.634716    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a891ce-fpmrl"
Oct 31 20:04:46.685663 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:46.685628    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82b3f50e-f79d-4f62-a212-1138dd4fd528-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82b3f50e-f79d-4f62-a212-1138dd4fd528-maas-api-xks-ca\") pod \"82b3f50e-f79d-4f62-a212-1138dd4fd528\" (UID: \"82b3f50e-f79d-4f62-a212-1138dd4fd528\") "
Oct 31 20:04:46.685840 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:46.685764    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82b3f50e-f79d-4f62-a212-1138dd4fd528-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82b3f50e-f79d-4f62-a212-1138dd4fd528-maas-api-openshift-ca\") pod \"82b3f50e-f79d-4f62-a212-1138dd4fd528\" (UID: \"82b3f50e-f79d-4f62-a212-1138dd4fd528\") "
Oct 31 20:04:46.685840 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:46.685836    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82b3f50e-f79d-4f62-a212-1138dd4fd528-kube-api-access-7dvj8\" (UniqueName: \"kubernetes.io/projected/82b3f50e-f79d-4f62-a212-1138dd4fd528-kube-api-access-7dvj8\") pod \"82b3f50e-f79d-4f62-a212-1138dd4fd528\" (UID: \"82b3f50e-f79d-4f62-a212-1138dd4fd528\") "
Oct 31 20:04:46.686746 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:46.686712    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82b3f50e-f79d-4f62-a212-1138dd4fd528-maas-api-openshift-ca" pod "82b3f50e-f79d-4f62-a212-1138dd4fd528" (UID: "82b3f50e-f79d-4f62-a212-1138dd4fd528"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:04:46.689299 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:46.689270    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82b3f50e-f79d-4f62-a212-1138dd4fd528-kube-api-access-7dvj8" pod "82b3f50e-f79d-4f62-a212-1138dd4fd528" (UID: "82b3f50e-f79d-4f62-a212-1138dd4fd528"). InnerVolumeSpecName "kube-api-access-7dvj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:04:46.689795 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:46.689770    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82b3f50e-f79d-4f62-a212-1138dd4fd528-maas-api-xks-ca" pod "82b3f50e-f79d-4f62-a212-1138dd4fd528" (UID: "82b3f50e-f79d-4f62-a212-1138dd4fd528"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:04:46.787543 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:46.787482    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dvj8\" (UniqueName: \"kubernetes.io/projected/82b3f50e-f79d-4f62-a212-1138dd4fd528-kube-api-access-7dvj8\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:46.787543 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:46.787508    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82b3f50e-f79d-4f62-a212-1138dd4fd528-maas-api-xks-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:46.787543 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:46.787517    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82b3f50e-f79d-4f62-a212-1138dd4fd528-maas-api-openshift-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:46.845861 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:46.845816    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:47.037403 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:47.037365    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b1ba135-ndbx5"]
Oct 31 20:04:47.041041 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:47.040988    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b1ba135-ndbx5"]
Oct 31 20:04:47.482510 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:47.482477    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:47.491060 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:47.491038    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a891ce-fpmrl"
Oct 31 20:04:47.491256 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:47.491041    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a891ce-fpmrl" event={"ID":"82b3f50e-f79d-4f62-a212-1138dd4fd528","Type":"ContainerDied","Data":"83d3b85012bfcc4d7cb6bfba49e94032516cd80756d03cb53b5e9d99b0f19883"}
Oct 31 20:04:47.491256 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:47.491200    4645 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83d3b85012bfcc4d7cb6bfba49e94032516cd80756d03cb53b5e9d99b0f19883"
Oct 31 20:04:47.605568 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:47.605533    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:47.618815 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:47.618788    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1ea88a5-c67d-4dc3-818b-e93c61a03a82" path="/var/lib/kubelet/pods/f1ea88a5-c67d-4dc3-818b-e93c61a03a82/volumes"
Oct 31 20:04:47.743009 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:47.742928    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:47.794224 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:47.794191    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb"]
Oct 31 20:04:47.794485 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:47.794458    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb" podUID="ba05261e-eadf-44c3-abda-cb561b4ea6a5" containerName="istio-proxy" containerID="cri-o://2d1e8615addbb51e1c72df9927e033ae0c2c9be3c100b66ea0db6468e04ecab7" gracePeriod=30
Oct 31 20:04:47.857948 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:47.857918    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:48.447246 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:48.447214    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4bac5a-slrwq"]
Oct 31 20:04:48.447882 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:48.447864    4645 cpu_manager.go:414] "containerMap: removing container" podUID="82b3f50e-f79d-4f62-a212-1138dd4fd528" containerName="revoke-keys"
Oct 31 20:04:48.447949 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:48.447883    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82b3f50e-f79d-4f62-a212-1138dd4fd528" containerName="revoke-keys"
Oct 31 20:04:48.448001 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:48.447991    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="82b3f50e-f79d-4f62-a212-1138dd4fd528" containerName="revoke-keys"
Oct 31 20:04:48.452655 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:48.452637    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4bac5a-slrwq"
Oct 31 20:04:48.457246 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:48.457222    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 20:04:48.457400 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:48.457258    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:04:48.470812 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:48.470787    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4bac5a-slrwq"]
Oct 31 20:04:48.539708 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:48.539677    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43f468d0-65fb-4e70-be20-2c4b66d67034-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-4bac5a-slrwq\" (UID: \"43f468d0-65fb-4e70-be20-2c4b66d67034\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4bac5a-slrwq"
Oct 31 20:04:48.539860 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:48.539719    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvlsd\" (UniqueName: \"kubernetes.io/projected/43f468d0-65fb-4e70-be20-2c4b66d67034-kube-api-access-pvlsd\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-4bac5a-slrwq\" (UID: \"43f468d0-65fb-4e70-be20-2c4b66d67034\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4bac5a-slrwq"
Oct 31 20:04:48.539860 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:48.539811    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43f468d0-65fb-4e70-be20-2c4b66d67034-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-4bac5a-slrwq\" (UID: \"43f468d0-65fb-4e70-be20-2c4b66d67034\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4bac5a-slrwq"
Oct 31 20:04:48.573772 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:48.573741    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w6-a891ce/rate-test-model-a891ce-kserve-55f654969c-tf5cj" podUID="f1b1d76a-8047-46ad-b08d-fdfb89ea668c" containerName="main" containerID="cri-o://799fdc8bd35fa54f1a4abc49ccd072bf42032a9145b8a73c718f3472c8a3c25a" gracePeriod=45
Oct 31 20:04:48.641624 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:48.641597    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43f468d0-65fb-4e70-be20-2c4b66d67034-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-4bac5a-slrwq\" (UID: \"43f468d0-65fb-4e70-be20-2c4b66d67034\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4bac5a-slrwq"
Oct 31 20:04:48.641734 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:48.641639    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvlsd\" (UniqueName: \"kubernetes.io/projected/43f468d0-65fb-4e70-be20-2c4b66d67034-kube-api-access-pvlsd\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-4bac5a-slrwq\" (UID: \"43f468d0-65fb-4e70-be20-2c4b66d67034\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4bac5a-slrwq"
Oct 31 20:04:48.641734 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:48.641687    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43f468d0-65fb-4e70-be20-2c4b66d67034-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-4bac5a-slrwq\" (UID: \"43f468d0-65fb-4e70-be20-2c4b66d67034\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4bac5a-slrwq"
Oct 31 20:04:48.642372 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:48.642282    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43f468d0-65fb-4e70-be20-2c4b66d67034-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-4bac5a-slrwq\" (UID: \"43f468d0-65fb-4e70-be20-2c4b66d67034\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4bac5a-slrwq"
Oct 31 20:04:48.644033 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:48.644016    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43f468d0-65fb-4e70-be20-2c4b66d67034-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-4bac5a-slrwq\" (UID: \"43f468d0-65fb-4e70-be20-2c4b66d67034\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4bac5a-slrwq"
Oct 31 20:04:48.653209 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:48.653189    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvlsd\" (UniqueName: \"kubernetes.io/projected/43f468d0-65fb-4e70-be20-2c4b66d67034-kube-api-access-pvlsd\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-4bac5a-slrwq\" (UID: \"43f468d0-65fb-4e70-be20-2c4b66d67034\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4bac5a-slrwq"
Oct 31 20:04:48.767370 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:48.766519    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4bac5a-slrwq"
Oct 31 20:04:48.887148 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:48.886802    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr"
Oct 31 20:04:48.945215 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:48.945156    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8"
Oct 31 20:04:48.946523 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:48.946493    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52d535c0-6d59-4788-9258-bbee438df233-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52d535c0-6d59-4788-9258-bbee438df233-istio-data\") pod \"52d535c0-6d59-4788-9258-bbee438df233\" (UID: \"52d535c0-6d59-4788-9258-bbee438df233\") "
Oct 31 20:04:48.946635 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:48.946549    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52d535c0-6d59-4788-9258-bbee438df233-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52d535c0-6d59-4788-9258-bbee438df233-credential-socket\") pod \"52d535c0-6d59-4788-9258-bbee438df233\" (UID: \"52d535c0-6d59-4788-9258-bbee438df233\") "
Oct 31 20:04:48.946635 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:48.946615    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52d535c0-6d59-4788-9258-bbee438df233-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52d535c0-6d59-4788-9258-bbee438df233-istio-envoy\") pod \"52d535c0-6d59-4788-9258-bbee438df233\" (UID: \"52d535c0-6d59-4788-9258-bbee438df233\") "
Oct 31 20:04:48.946730 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:48.946687    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52d535c0-6d59-4788-9258-bbee438df233-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52d535c0-6d59-4788-9258-bbee438df233-workload-socket\") pod \"52d535c0-6d59-4788-9258-bbee438df233\" (UID: \"52d535c0-6d59-4788-9258-bbee438df233\") "
Oct 31 20:04:48.946783 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:48.946765    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52d535c0-6d59-4788-9258-bbee438df233-kube-api-access-qf2lc\" (UniqueName: \"kubernetes.io/projected/52d535c0-6d59-4788-9258-bbee438df233-kube-api-access-qf2lc\") pod \"52d535c0-6d59-4788-9258-bbee438df233\" (UID: \"52d535c0-6d59-4788-9258-bbee438df233\") "
Oct 31 20:04:48.946900 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:48.946885    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52d535c0-6d59-4788-9258-bbee438df233-istio-token\" (UniqueName: \"kubernetes.io/projected/52d535c0-6d59-4788-9258-bbee438df233-istio-token\") pod \"52d535c0-6d59-4788-9258-bbee438df233\" (UID: \"52d535c0-6d59-4788-9258-bbee438df233\") "
Oct 31 20:04:48.946949 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:48.946926    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/52d535c0-6d59-4788-9258-bbee438df233-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52d535c0-6d59-4788-9258-bbee438df233-istio-podinfo\") pod \"52d535c0-6d59-4788-9258-bbee438df233\" (UID: \"52d535c0-6d59-4788-9258-bbee438df233\") "
Oct 31 20:04:48.947004 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:48.946990    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52d535c0-6d59-4788-9258-bbee438df233-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52d535c0-6d59-4788-9258-bbee438df233-workload-certs\") pod \"52d535c0-6d59-4788-9258-bbee438df233\" (UID: \"52d535c0-6d59-4788-9258-bbee438df233\") "
Oct 31 20:04:48.947157 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:48.947138    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52d535c0-6d59-4788-9258-bbee438df233-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52d535c0-6d59-4788-9258-bbee438df233-istiod-ca-cert\") pod \"52d535c0-6d59-4788-9258-bbee438df233\" (UID: \"52d535c0-6d59-4788-9258-bbee438df233\") "
Oct 31 20:04:48.948122 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:48.947734    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52d535c0-6d59-4788-9258-bbee438df233-istio-data" pod "52d535c0-6d59-4788-9258-bbee438df233" (UID: "52d535c0-6d59-4788-9258-bbee438df233"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:04:48.948122 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:48.947704    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52d535c0-6d59-4788-9258-bbee438df233-credential-socket" pod "52d535c0-6d59-4788-9258-bbee438df233" (UID: "52d535c0-6d59-4788-9258-bbee438df233"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:04:48.948122 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:48.948049    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52d535c0-6d59-4788-9258-bbee438df233-workload-certs" pod "52d535c0-6d59-4788-9258-bbee438df233" (UID: "52d535c0-6d59-4788-9258-bbee438df233"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:04:48.948122 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:48.948114    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52d535c0-6d59-4788-9258-bbee438df233-istiod-ca-cert" pod "52d535c0-6d59-4788-9258-bbee438df233" (UID: "52d535c0-6d59-4788-9258-bbee438df233"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:04:48.948350 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:48.948144    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52d535c0-6d59-4788-9258-bbee438df233-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:48.948350 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:48.948163    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52d535c0-6d59-4788-9258-bbee438df233-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:48.948468 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:48.948409    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52d535c0-6d59-4788-9258-bbee438df233-workload-socket" pod "52d535c0-6d59-4788-9258-bbee438df233" (UID: "52d535c0-6d59-4788-9258-bbee438df233"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:04:48.950459 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:48.950426    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph"
Oct 31 20:04:48.951203 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:48.951157    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52d535c0-6d59-4788-9258-bbee438df233-kube-api-access-qf2lc" pod "52d535c0-6d59-4788-9258-bbee438df233" (UID: "52d535c0-6d59-4788-9258-bbee438df233"). InnerVolumeSpecName "kube-api-access-qf2lc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:04:48.952907 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:48.952887    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52d535c0-6d59-4788-9258-bbee438df233-istio-token" pod "52d535c0-6d59-4788-9258-bbee438df233" (UID: "52d535c0-6d59-4788-9258-bbee438df233"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:04:48.953059 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:48.953031    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52d535c0-6d59-4788-9258-bbee438df233-istio-envoy" pod "52d535c0-6d59-4788-9258-bbee438df233" (UID: "52d535c0-6d59-4788-9258-bbee438df233"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:04:48.953154 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:48.953091    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/52d535c0-6d59-4788-9258-bbee438df233-istio-podinfo" pod "52d535c0-6d59-4788-9258-bbee438df233" (UID: "52d535c0-6d59-4788-9258-bbee438df233"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 20:04:48.953641 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:48.953628    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-a891ce/rate-test-model-a891ce-kserve-55f654969c-tf5cj"
Oct 31 20:04:49.035991 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.027966    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:49.055976 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.050983    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b24f33df-8939-405b-83a6-8d7af872e7da-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24f33df-8939-405b-83a6-8d7af872e7da-credential-socket\") pod \"b24f33df-8939-405b-83a6-8d7af872e7da\" (UID: \"b24f33df-8939-405b-83a6-8d7af872e7da\") "
Oct 31 20:04:49.055976 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.051046    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b24f33df-8939-405b-83a6-8d7af872e7da-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b24f33df-8939-405b-83a6-8d7af872e7da-istio-envoy\") pod \"b24f33df-8939-405b-83a6-8d7af872e7da\" (UID: \"b24f33df-8939-405b-83a6-8d7af872e7da\") "
Oct 31 20:04:49.055976 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.051132    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b24f33df-8939-405b-83a6-8d7af872e7da-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b24f33df-8939-405b-83a6-8d7af872e7da-istio-podinfo\") pod \"b24f33df-8939-405b-83a6-8d7af872e7da\" (UID: \"b24f33df-8939-405b-83a6-8d7af872e7da\") "
Oct 31 20:04:49.055976 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.051182    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1b1d76a-8047-46ad-b08d-fdfb89ea668c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1b1d76a-8047-46ad-b08d-fdfb89ea668c-dshm\") pod \"f1b1d76a-8047-46ad-b08d-fdfb89ea668c\" (UID: \"f1b1d76a-8047-46ad-b08d-fdfb89ea668c\") "
Oct 31 20:04:49.055976 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.051241    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1b1d76a-8047-46ad-b08d-fdfb89ea668c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1b1d76a-8047-46ad-b08d-fdfb89ea668c-model-cache\") pod \"f1b1d76a-8047-46ad-b08d-fdfb89ea668c\" (UID: \"f1b1d76a-8047-46ad-b08d-fdfb89ea668c\") "
Oct 31 20:04:49.055976 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.051278    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ede30242-d4ae-4caa-9c4b-8918314a1ead-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ede30242-d4ae-4caa-9c4b-8918314a1ead-istio-data\") pod \"ede30242-d4ae-4caa-9c4b-8918314a1ead\" (UID: \"ede30242-d4ae-4caa-9c4b-8918314a1ead\") "
Oct 31 20:04:49.055976 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.051312    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b24f33df-8939-405b-83a6-8d7af872e7da-istio-token\" (UniqueName: \"kubernetes.io/projected/b24f33df-8939-405b-83a6-8d7af872e7da-istio-token\") pod \"b24f33df-8939-405b-83a6-8d7af872e7da\" (UID: \"b24f33df-8939-405b-83a6-8d7af872e7da\") "
Oct 31 20:04:49.055976 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.051367    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1b1d76a-8047-46ad-b08d-fdfb89ea668c-home\" (UniqueName: \"kubernetes.io/empty-dir/f1b1d76a-8047-46ad-b08d-fdfb89ea668c-home\") pod \"f1b1d76a-8047-46ad-b08d-fdfb89ea668c\" (UID: \"f1b1d76a-8047-46ad-b08d-fdfb89ea668c\") "
Oct 31 20:04:49.055976 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.051406    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ede30242-d4ae-4caa-9c4b-8918314a1ead-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ede30242-d4ae-4caa-9c4b-8918314a1ead-workload-certs\") pod \"ede30242-d4ae-4caa-9c4b-8918314a1ead\" (UID: \"ede30242-d4ae-4caa-9c4b-8918314a1ead\") "
Oct 31 20:04:49.055976 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.051450    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ede30242-d4ae-4caa-9c4b-8918314a1ead-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ede30242-d4ae-4caa-9c4b-8918314a1ead-istio-envoy\") pod \"ede30242-d4ae-4caa-9c4b-8918314a1ead\" (UID: \"ede30242-d4ae-4caa-9c4b-8918314a1ead\") "
Oct 31 20:04:49.055976 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.051478    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ede30242-d4ae-4caa-9c4b-8918314a1ead-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ede30242-d4ae-4caa-9c4b-8918314a1ead-istio-podinfo\") pod \"ede30242-d4ae-4caa-9c4b-8918314a1ead\" (UID: \"ede30242-d4ae-4caa-9c4b-8918314a1ead\") "
Oct 31 20:04:49.055976 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.051519    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b24f33df-8939-405b-83a6-8d7af872e7da-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24f33df-8939-405b-83a6-8d7af872e7da-workload-socket\") pod \"b24f33df-8939-405b-83a6-8d7af872e7da\" (UID: \"b24f33df-8939-405b-83a6-8d7af872e7da\") "
Oct 31 20:04:49.055976 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.051628    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b24f33df-8939-405b-83a6-8d7af872e7da-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b24f33df-8939-405b-83a6-8d7af872e7da-workload-certs\") pod \"b24f33df-8939-405b-83a6-8d7af872e7da\" (UID: \"b24f33df-8939-405b-83a6-8d7af872e7da\") "
Oct 31 20:04:49.055976 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.051675    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ede30242-d4ae-4caa-9c4b-8918314a1ead-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede30242-d4ae-4caa-9c4b-8918314a1ead-workload-socket\") pod \"ede30242-d4ae-4caa-9c4b-8918314a1ead\" (UID: \"ede30242-d4ae-4caa-9c4b-8918314a1ead\") "
Oct 31 20:04:49.055976 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.051747    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ede30242-d4ae-4caa-9c4b-8918314a1ead-istio-token\" (UniqueName: \"kubernetes.io/projected/ede30242-d4ae-4caa-9c4b-8918314a1ead-istio-token\") pod \"ede30242-d4ae-4caa-9c4b-8918314a1ead\" (UID: \"ede30242-d4ae-4caa-9c4b-8918314a1ead\") "
Oct 31 20:04:49.055976 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.051797    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b24f33df-8939-405b-83a6-8d7af872e7da-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b24f33df-8939-405b-83a6-8d7af872e7da-istiod-ca-cert\") pod \"b24f33df-8939-405b-83a6-8d7af872e7da\" (UID: \"b24f33df-8939-405b-83a6-8d7af872e7da\") "
Oct 31 20:04:49.055976 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.051840    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ede30242-d4ae-4caa-9c4b-8918314a1ead-kube-api-access-x8gnc\" (UniqueName: \"kubernetes.io/projected/ede30242-d4ae-4caa-9c4b-8918314a1ead-kube-api-access-x8gnc\") pod \"ede30242-d4ae-4caa-9c4b-8918314a1ead\" (UID: \"ede30242-d4ae-4caa-9c4b-8918314a1ead\") "
Oct 31 20:04:49.055976 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.051913    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ede30242-d4ae-4caa-9c4b-8918314a1ead-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ede30242-d4ae-4caa-9c4b-8918314a1ead-istiod-ca-cert\") pod \"ede30242-d4ae-4caa-9c4b-8918314a1ead\" (UID: \"ede30242-d4ae-4caa-9c4b-8918314a1ead\") "
Oct 31 20:04:49.055976 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.051952    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1b1d76a-8047-46ad-b08d-fdfb89ea668c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1b1d76a-8047-46ad-b08d-fdfb89ea668c-tmp-dir\") pod \"f1b1d76a-8047-46ad-b08d-fdfb89ea668c\" (UID: \"f1b1d76a-8047-46ad-b08d-fdfb89ea668c\") "
Oct 31 20:04:49.055976 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.051984    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b24f33df-8939-405b-83a6-8d7af872e7da-kube-api-access-w87jq\" (UniqueName: \"kubernetes.io/projected/b24f33df-8939-405b-83a6-8d7af872e7da-kube-api-access-w87jq\") pod \"b24f33df-8939-405b-83a6-8d7af872e7da\" (UID: \"b24f33df-8939-405b-83a6-8d7af872e7da\") "
Oct 31 20:04:49.055976 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.052269    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b24f33df-8939-405b-83a6-8d7af872e7da-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b24f33df-8939-405b-83a6-8d7af872e7da-istio-data\") pod \"b24f33df-8939-405b-83a6-8d7af872e7da\" (UID: \"b24f33df-8939-405b-83a6-8d7af872e7da\") "
Oct 31 20:04:49.055976 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.052353    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1b1d76a-8047-46ad-b08d-fdfb89ea668c-tls-certs\" (UniqueName: \"kubernetes.io/secret/f1b1d76a-8047-46ad-b08d-fdfb89ea668c-tls-certs\") pod \"f1b1d76a-8047-46ad-b08d-fdfb89ea668c\" (UID: \"f1b1d76a-8047-46ad-b08d-fdfb89ea668c\") "
Oct 31 20:04:49.055976 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.052410    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ede30242-d4ae-4caa-9c4b-8918314a1ead-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede30242-d4ae-4caa-9c4b-8918314a1ead-credential-socket\") pod \"ede30242-d4ae-4caa-9c4b-8918314a1ead\" (UID: \"ede30242-d4ae-4caa-9c4b-8918314a1ead\") "
Oct 31 20:04:49.055976 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.052446    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1b1d76a-8047-46ad-b08d-fdfb89ea668c-kube-api-access-cn5zv\" (UniqueName: \"kubernetes.io/projected/f1b1d76a-8047-46ad-b08d-fdfb89ea668c-kube-api-access-cn5zv\") pod \"f1b1d76a-8047-46ad-b08d-fdfb89ea668c\" (UID: \"f1b1d76a-8047-46ad-b08d-fdfb89ea668c\") "
Oct 31 20:04:49.055976 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.053468    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52d535c0-6d59-4788-9258-bbee438df233-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:49.055976 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.053493    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52d535c0-6d59-4788-9258-bbee438df233-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:49.055976 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.053507    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52d535c0-6d59-4788-9258-bbee438df233-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:49.055976 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.053521    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52d535c0-6d59-4788-9258-bbee438df233-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:49.055976 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.053534    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52d535c0-6d59-4788-9258-bbee438df233-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:49.055976 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.053548    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52d535c0-6d59-4788-9258-bbee438df233-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:49.055976 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.053562    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qf2lc\" (UniqueName: \"kubernetes.io/projected/52d535c0-6d59-4788-9258-bbee438df233-kube-api-access-qf2lc\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:49.064502 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.056491    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1b1d76a-8047-46ad-b08d-fdfb89ea668c-kube-api-access-cn5zv" pod "f1b1d76a-8047-46ad-b08d-fdfb89ea668c" (UID: "f1b1d76a-8047-46ad-b08d-fdfb89ea668c"). InnerVolumeSpecName "kube-api-access-cn5zv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:04:49.064502 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.056751    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b24f33df-8939-405b-83a6-8d7af872e7da-credential-socket" pod "b24f33df-8939-405b-83a6-8d7af872e7da" (UID: "b24f33df-8939-405b-83a6-8d7af872e7da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:04:49.064502 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.059367    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b24f33df-8939-405b-83a6-8d7af872e7da-istio-envoy" pod "b24f33df-8939-405b-83a6-8d7af872e7da" (UID: "b24f33df-8939-405b-83a6-8d7af872e7da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:04:49.069705 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.064685    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b24f33df-8939-405b-83a6-8d7af872e7da-istio-podinfo" pod "b24f33df-8939-405b-83a6-8d7af872e7da" (UID: "b24f33df-8939-405b-83a6-8d7af872e7da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 20:04:49.072486 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.072446    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1b1d76a-8047-46ad-b08d-fdfb89ea668c-dshm" pod "f1b1d76a-8047-46ad-b08d-fdfb89ea668c" (UID: "f1b1d76a-8047-46ad-b08d-fdfb89ea668c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:04:49.072700 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.072672    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ede30242-d4ae-4caa-9c4b-8918314a1ead-workload-socket" pod "ede30242-d4ae-4caa-9c4b-8918314a1ead" (UID: "ede30242-d4ae-4caa-9c4b-8918314a1ead"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:04:49.072887 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.072871    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1b1d76a-8047-46ad-b08d-fdfb89ea668c-model-cache" pod "f1b1d76a-8047-46ad-b08d-fdfb89ea668c" (UID: "f1b1d76a-8047-46ad-b08d-fdfb89ea668c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:04:49.073061 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.073036    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ede30242-d4ae-4caa-9c4b-8918314a1ead-istio-data" pod "ede30242-d4ae-4caa-9c4b-8918314a1ead" (UID: "ede30242-d4ae-4caa-9c4b-8918314a1ead"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:04:49.073960 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.073928    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1b1d76a-8047-46ad-b08d-fdfb89ea668c-home" pod "f1b1d76a-8047-46ad-b08d-fdfb89ea668c" (UID: "f1b1d76a-8047-46ad-b08d-fdfb89ea668c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:04:49.074190 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.074168    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ede30242-d4ae-4caa-9c4b-8918314a1ead-workload-certs" pod "ede30242-d4ae-4caa-9c4b-8918314a1ead" (UID: "ede30242-d4ae-4caa-9c4b-8918314a1ead"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:04:49.077156 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.077125    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b24f33df-8939-405b-83a6-8d7af872e7da-istio-token" pod "b24f33df-8939-405b-83a6-8d7af872e7da" (UID: "b24f33df-8939-405b-83a6-8d7af872e7da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:04:49.077430 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.077406    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1b1d76a-8047-46ad-b08d-fdfb89ea668c-tmp-dir" pod "f1b1d76a-8047-46ad-b08d-fdfb89ea668c" (UID: "f1b1d76a-8047-46ad-b08d-fdfb89ea668c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:04:49.078211 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.077771    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b24f33df-8939-405b-83a6-8d7af872e7da-istio-data" pod "b24f33df-8939-405b-83a6-8d7af872e7da" (UID: "b24f33df-8939-405b-83a6-8d7af872e7da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:04:49.078211 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.077904    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b24f33df-8939-405b-83a6-8d7af872e7da-istiod-ca-cert" pod "b24f33df-8939-405b-83a6-8d7af872e7da" (UID: "b24f33df-8939-405b-83a6-8d7af872e7da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:04:49.078697 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.078649    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b24f33df-8939-405b-83a6-8d7af872e7da-workload-certs" pod "b24f33df-8939-405b-83a6-8d7af872e7da" (UID: "b24f33df-8939-405b-83a6-8d7af872e7da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:04:49.079094 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.079075    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b24f33df-8939-405b-83a6-8d7af872e7da-workload-socket" pod "b24f33df-8939-405b-83a6-8d7af872e7da" (UID: "b24f33df-8939-405b-83a6-8d7af872e7da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:04:49.079855 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.079828    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ede30242-d4ae-4caa-9c4b-8918314a1ead-credential-socket" pod "ede30242-d4ae-4caa-9c4b-8918314a1ead" (UID: "ede30242-d4ae-4caa-9c4b-8918314a1ead"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:04:49.079973 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.079890    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ede30242-d4ae-4caa-9c4b-8918314a1ead-istiod-ca-cert" pod "ede30242-d4ae-4caa-9c4b-8918314a1ead" (UID: "ede30242-d4ae-4caa-9c4b-8918314a1ead"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:04:49.082683 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.082655    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ede30242-d4ae-4caa-9c4b-8918314a1ead-istio-token" pod "ede30242-d4ae-4caa-9c4b-8918314a1ead" (UID: "ede30242-d4ae-4caa-9c4b-8918314a1ead"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:04:49.086518 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.086478    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ede30242-d4ae-4caa-9c4b-8918314a1ead-istio-podinfo" pod "ede30242-d4ae-4caa-9c4b-8918314a1ead" (UID: "ede30242-d4ae-4caa-9c4b-8918314a1ead"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 20:04:49.086610 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.086531    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ede30242-d4ae-4caa-9c4b-8918314a1ead-istio-envoy" pod "ede30242-d4ae-4caa-9c4b-8918314a1ead" (UID: "ede30242-d4ae-4caa-9c4b-8918314a1ead"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:04:49.087713 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.087690    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1b1d76a-8047-46ad-b08d-fdfb89ea668c-tls-certs" pod "f1b1d76a-8047-46ad-b08d-fdfb89ea668c" (UID: "f1b1d76a-8047-46ad-b08d-fdfb89ea668c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:04:49.088074 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.088051    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ede30242-d4ae-4caa-9c4b-8918314a1ead-kube-api-access-x8gnc" pod "ede30242-d4ae-4caa-9c4b-8918314a1ead" (UID: "ede30242-d4ae-4caa-9c4b-8918314a1ead"). InnerVolumeSpecName "kube-api-access-x8gnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:04:49.088568 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.088546    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b24f33df-8939-405b-83a6-8d7af872e7da-kube-api-access-w87jq" pod "b24f33df-8939-405b-83a6-8d7af872e7da" (UID: "b24f33df-8939-405b-83a6-8d7af872e7da"). InnerVolumeSpecName "kube-api-access-w87jq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:04:49.112569 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.112543    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4bac5a-slrwq"]
Oct 31 20:04:49.114401 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:04:49.114379    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43f468d0_65fb_4e70_be20_2c4b66d67034.slice/crio-2a69cf7f081db9a1fdaa80e9c5577a9f05c22efca1998c86c4b7e8a2f448bd95 WatchSource:0}: Error finding container 2a69cf7f081db9a1fdaa80e9c5577a9f05c22efca1998c86c4b7e8a2f448bd95: Status 404 returned error can't find the container with id 2a69cf7f081db9a1fdaa80e9c5577a9f05c22efca1998c86c4b7e8a2f448bd95
Oct 31 20:04:49.154882 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.154858    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b24f33df-8939-405b-83a6-8d7af872e7da-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:49.154976 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.154885    4645 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1b1d76a-8047-46ad-b08d-fdfb89ea668c-tls-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:49.154976 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.154901    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede30242-d4ae-4caa-9c4b-8918314a1ead-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:49.154976 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.154915    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cn5zv\" (UniqueName: \"kubernetes.io/projected/f1b1d76a-8047-46ad-b08d-fdfb89ea668c-kube-api-access-cn5zv\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:49.154976 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.154929    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24f33df-8939-405b-83a6-8d7af872e7da-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:49.154976 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.154945    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b24f33df-8939-405b-83a6-8d7af872e7da-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:49.154976 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.154959    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b24f33df-8939-405b-83a6-8d7af872e7da-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:49.154976 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.154972    4645 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1b1d76a-8047-46ad-b08d-fdfb89ea668c-dshm\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:49.155302 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.154986    4645 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1b1d76a-8047-46ad-b08d-fdfb89ea668c-model-cache\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:49.155302 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.155000    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ede30242-d4ae-4caa-9c4b-8918314a1ead-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:49.155302 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.155013    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b24f33df-8939-405b-83a6-8d7af872e7da-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:49.155302 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.155027    4645 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1b1d76a-8047-46ad-b08d-fdfb89ea668c-home\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:49.155302 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.155041    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ede30242-d4ae-4caa-9c4b-8918314a1ead-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:49.155302 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.155054    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ede30242-d4ae-4caa-9c4b-8918314a1ead-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:49.155302 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.155068    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ede30242-d4ae-4caa-9c4b-8918314a1ead-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:49.155302 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.155085    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b24f33df-8939-405b-83a6-8d7af872e7da-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:49.155302 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.155099    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b24f33df-8939-405b-83a6-8d7af872e7da-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:49.155302 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.155112    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ede30242-d4ae-4caa-9c4b-8918314a1ead-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:49.155302 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.155127    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ede30242-d4ae-4caa-9c4b-8918314a1ead-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:49.155302 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.155141    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b24f33df-8939-405b-83a6-8d7af872e7da-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:49.155302 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.155154    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8gnc\" (UniqueName: \"kubernetes.io/projected/ede30242-d4ae-4caa-9c4b-8918314a1ead-kube-api-access-x8gnc\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:49.155302 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.155168    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ede30242-d4ae-4caa-9c4b-8918314a1ead-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:49.155302 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.155184    4645 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1b1d76a-8047-46ad-b08d-fdfb89ea668c-tmp-dir\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:49.155302 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.155197    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w87jq\" (UniqueName: \"kubernetes.io/projected/b24f33df-8939-405b-83a6-8d7af872e7da-kube-api-access-w87jq\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:49.509981 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.509952    4645 generic.go:356] "Generic (PLEG): container finished" podID="b24f33df-8939-405b-83a6-8d7af872e7da" containerID="b3422fc5e1f05bc8edd9545342a5050b0eb6813883c69fb12fe63c3bbc81ae57" exitCode=0
Oct 31 20:04:49.510133 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.509995    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph"
Oct 31 20:04:49.510133 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.510043    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph" event={"ID":"b24f33df-8939-405b-83a6-8d7af872e7da","Type":"ContainerDied","Data":"b3422fc5e1f05bc8edd9545342a5050b0eb6813883c69fb12fe63c3bbc81ae57"}
Oct 31 20:04:49.510133 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.510097    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph" event={"ID":"b24f33df-8939-405b-83a6-8d7af872e7da","Type":"ContainerDied","Data":"634c3a98781b99971bc36b5fa61c77cb20e62c33e29ec55ce6f80ecd4750559f"}
Oct 31 20:04:49.510313 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.510127    4645 scope.go:122] "RemoveContainer" containerID="b3422fc5e1f05bc8edd9545342a5050b0eb6813883c69fb12fe63c3bbc81ae57"
Oct 31 20:04:49.511857 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.511834    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4bac5a-slrwq" event={"ID":"43f468d0-65fb-4e70-be20-2c4b66d67034","Type":"ContainerStarted","Data":"7c73a5ff8bfc6ad062ba730349acd817c0efd4be88c24702906b5e71c7b32da7"}
Oct 31 20:04:49.511990 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.511886    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4bac5a-slrwq" event={"ID":"43f468d0-65fb-4e70-be20-2c4b66d67034","Type":"ContainerStarted","Data":"2a69cf7f081db9a1fdaa80e9c5577a9f05c22efca1998c86c4b7e8a2f448bd95"}
Oct 31 20:04:49.513725 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.513697    4645 generic.go:356] "Generic (PLEG): container finished" podID="ede30242-d4ae-4caa-9c4b-8918314a1ead" containerID="85880c2b644d01275aa9674a5d67f076e83cac0aa9206136cca068e69f29023f" exitCode=0
Oct 31 20:04:49.513831 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.513726    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8" event={"ID":"ede30242-d4ae-4caa-9c4b-8918314a1ead","Type":"ContainerDied","Data":"85880c2b644d01275aa9674a5d67f076e83cac0aa9206136cca068e69f29023f"}
Oct 31 20:04:49.513831 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.513775    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8" event={"ID":"ede30242-d4ae-4caa-9c4b-8918314a1ead","Type":"ContainerDied","Data":"af3f9275b982225bbbb8387f3810b4073305442900f79b67a498b1965515c87a"}
Oct 31 20:04:49.513831 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.513737    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8"
Oct 31 20:04:49.515512 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.515460    4645 generic.go:356] "Generic (PLEG): container finished" podID="f1b1d76a-8047-46ad-b08d-fdfb89ea668c" containerID="799fdc8bd35fa54f1a4abc49ccd072bf42032a9145b8a73c718f3472c8a3c25a" exitCode=0
Oct 31 20:04:49.515512 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.515499    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-a891ce/rate-test-model-a891ce-kserve-55f654969c-tf5cj"
Oct 31 20:04:49.515702 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.515642    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-a891ce/rate-test-model-a891ce-kserve-55f654969c-tf5cj" event={"ID":"f1b1d76a-8047-46ad-b08d-fdfb89ea668c","Type":"ContainerDied","Data":"799fdc8bd35fa54f1a4abc49ccd072bf42032a9145b8a73c718f3472c8a3c25a"}
Oct 31 20:04:49.515702 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.515673    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-a891ce/rate-test-model-a891ce-kserve-55f654969c-tf5cj" event={"ID":"f1b1d76a-8047-46ad-b08d-fdfb89ea668c","Type":"ContainerDied","Data":"eb361ad5c7ffe721b80c4c39ae1334fe6474b3ded8dc2587fa831d8a62c91775"}
Oct 31 20:04:49.517550 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.517524    4645 generic.go:356] "Generic (PLEG): container finished" podID="52d535c0-6d59-4788-9258-bbee438df233" containerID="c17be81049cf163d701039e21a9a7baf8ab010531b946507691e08439cf34bd5" exitCode=0
Oct 31 20:04:49.517666 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.517574    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr" event={"ID":"52d535c0-6d59-4788-9258-bbee438df233","Type":"ContainerDied","Data":"c17be81049cf163d701039e21a9a7baf8ab010531b946507691e08439cf34bd5"}
Oct 31 20:04:49.517666 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.517614    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr" event={"ID":"52d535c0-6d59-4788-9258-bbee438df233","Type":"ContainerDied","Data":"3d21ae2a7db75457a68c444d8c1c9429e68b403f06211ee15fb34c9853d8560f"}
Oct 31 20:04:49.517784 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.517682    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr"
Oct 31 20:04:49.524856 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.524831    4645 scope.go:122] "RemoveContainer" containerID="b3422fc5e1f05bc8edd9545342a5050b0eb6813883c69fb12fe63c3bbc81ae57"
Oct 31 20:04:49.530509 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:04:49.530483    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3422fc5e1f05bc8edd9545342a5050b0eb6813883c69fb12fe63c3bbc81ae57\": container with ID starting with b3422fc5e1f05bc8edd9545342a5050b0eb6813883c69fb12fe63c3bbc81ae57 not found: ID does not exist" containerID="b3422fc5e1f05bc8edd9545342a5050b0eb6813883c69fb12fe63c3bbc81ae57"
Oct 31 20:04:49.530623 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.530527    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3422fc5e1f05bc8edd9545342a5050b0eb6813883c69fb12fe63c3bbc81ae57"} err="failed to get container status \"b3422fc5e1f05bc8edd9545342a5050b0eb6813883c69fb12fe63c3bbc81ae57\": rpc error: code = NotFound desc = could not find container \"b3422fc5e1f05bc8edd9545342a5050b0eb6813883c69fb12fe63c3bbc81ae57\": container with ID starting with b3422fc5e1f05bc8edd9545342a5050b0eb6813883c69fb12fe63c3bbc81ae57 not found: ID does not exist"
Oct 31 20:04:49.530623 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.530553    4645 scope.go:122] "RemoveContainer" containerID="85880c2b644d01275aa9674a5d67f076e83cac0aa9206136cca068e69f29023f"
Oct 31 20:04:49.546462 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.546431    4645 scope.go:122] "RemoveContainer" containerID="22276da2e572fbe9afeabcf73cd849af3e094556271c14915e031ea48e2f852a"
Oct 31 20:04:49.548808 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.548763    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4bac5a-slrwq" podStartSLOduration=1.5487516430000001 podStartE2EDuration="1.548751643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:04:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:04:49.545522995 +0000 UTC m=+2720.118757208" watchObservedRunningTime="2026-10-31 20:04:49.548751643 +0000 UTC m=+2720.121985772"
Oct 31 20:04:49.559004 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.558985    4645 scope.go:122] "RemoveContainer" containerID="85880c2b644d01275aa9674a5d67f076e83cac0aa9206136cca068e69f29023f"
Oct 31 20:04:49.559319 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:04:49.559293    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85880c2b644d01275aa9674a5d67f076e83cac0aa9206136cca068e69f29023f\": container with ID starting with 85880c2b644d01275aa9674a5d67f076e83cac0aa9206136cca068e69f29023f not found: ID does not exist" containerID="85880c2b644d01275aa9674a5d67f076e83cac0aa9206136cca068e69f29023f"
Oct 31 20:04:49.559441 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.559323    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85880c2b644d01275aa9674a5d67f076e83cac0aa9206136cca068e69f29023f"} err="failed to get container status \"85880c2b644d01275aa9674a5d67f076e83cac0aa9206136cca068e69f29023f\": rpc error: code = NotFound desc = could not find container \"85880c2b644d01275aa9674a5d67f076e83cac0aa9206136cca068e69f29023f\": container with ID starting with 85880c2b644d01275aa9674a5d67f076e83cac0aa9206136cca068e69f29023f not found: ID does not exist"
Oct 31 20:04:49.559441 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.559379    4645 scope.go:122] "RemoveContainer" containerID="22276da2e572fbe9afeabcf73cd849af3e094556271c14915e031ea48e2f852a"
Oct 31 20:04:49.559661 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:04:49.559638    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22276da2e572fbe9afeabcf73cd849af3e094556271c14915e031ea48e2f852a\": container with ID starting with 22276da2e572fbe9afeabcf73cd849af3e094556271c14915e031ea48e2f852a not found: ID does not exist" containerID="22276da2e572fbe9afeabcf73cd849af3e094556271c14915e031ea48e2f852a"
Oct 31 20:04:49.559755 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.559667    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22276da2e572fbe9afeabcf73cd849af3e094556271c14915e031ea48e2f852a"} err="failed to get container status \"22276da2e572fbe9afeabcf73cd849af3e094556271c14915e031ea48e2f852a\": rpc error: code = NotFound desc = could not find container \"22276da2e572fbe9afeabcf73cd849af3e094556271c14915e031ea48e2f852a\": container with ID starting with 22276da2e572fbe9afeabcf73cd849af3e094556271c14915e031ea48e2f852a not found: ID does not exist"
Oct 31 20:04:49.559755 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.559683    4645 scope.go:122] "RemoveContainer" containerID="799fdc8bd35fa54f1a4abc49ccd072bf42032a9145b8a73c718f3472c8a3c25a"
Oct 31 20:04:49.574369 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.574341    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr"]
Oct 31 20:04:49.575479 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.574688    4645 scope.go:122] "RemoveContainer" containerID="799fdc8bd35fa54f1a4abc49ccd072bf42032a9145b8a73c718f3472c8a3c25a"
Oct 31 20:04:49.575479 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:04:49.575042    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"799fdc8bd35fa54f1a4abc49ccd072bf42032a9145b8a73c718f3472c8a3c25a\": container with ID starting with 799fdc8bd35fa54f1a4abc49ccd072bf42032a9145b8a73c718f3472c8a3c25a not found: ID does not exist" containerID="799fdc8bd35fa54f1a4abc49ccd072bf42032a9145b8a73c718f3472c8a3c25a"
Oct 31 20:04:49.575479 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.575072    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"799fdc8bd35fa54f1a4abc49ccd072bf42032a9145b8a73c718f3472c8a3c25a"} err="failed to get container status \"799fdc8bd35fa54f1a4abc49ccd072bf42032a9145b8a73c718f3472c8a3c25a\": rpc error: code = NotFound desc = could not find container \"799fdc8bd35fa54f1a4abc49ccd072bf42032a9145b8a73c718f3472c8a3c25a\": container with ID starting with 799fdc8bd35fa54f1a4abc49ccd072bf42032a9145b8a73c718f3472c8a3c25a not found: ID does not exist"
Oct 31 20:04:49.575479 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.575093    4645 scope.go:122] "RemoveContainer" containerID="c17be81049cf163d701039e21a9a7baf8ab010531b946507691e08439cf34bd5"
Oct 31 20:04:49.581679 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.581653    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-hltgr"]
Oct 31 20:04:49.599910 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.599884    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w6-a891ce/rate-test-model-a891ce-kserve-55f654969c-tf5cj"]
Oct 31 20:04:49.621278 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.621221    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52d535c0-6d59-4788-9258-bbee438df233" path="/var/lib/kubelet/pods/52d535c0-6d59-4788-9258-bbee438df233/volumes"
Oct 31 20:04:49.621640 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.621618    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w6-a891ce/rate-test-model-a891ce-kserve-55f654969c-tf5cj"]
Oct 31 20:04:49.631758 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.631731    4645 scope.go:122] "RemoveContainer" containerID="c17be81049cf163d701039e21a9a7baf8ab010531b946507691e08439cf34bd5"
Oct 31 20:04:49.632247 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:04:49.632216    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c17be81049cf163d701039e21a9a7baf8ab010531b946507691e08439cf34bd5\": container with ID starting with c17be81049cf163d701039e21a9a7baf8ab010531b946507691e08439cf34bd5 not found: ID does not exist" containerID="c17be81049cf163d701039e21a9a7baf8ab010531b946507691e08439cf34bd5"
Oct 31 20:04:49.632490 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.632455    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c17be81049cf163d701039e21a9a7baf8ab010531b946507691e08439cf34bd5"} err="failed to get container status \"c17be81049cf163d701039e21a9a7baf8ab010531b946507691e08439cf34bd5\": rpc error: code = NotFound desc = could not find container \"c17be81049cf163d701039e21a9a7baf8ab010531b946507691e08439cf34bd5\": container with ID starting with c17be81049cf163d701039e21a9a7baf8ab010531b946507691e08439cf34bd5 not found: ID does not exist"
Oct 31 20:04:49.635015 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.634986    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph"]
Oct 31 20:04:49.638712 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.638690    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-7flph"]
Oct 31 20:04:49.650242 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.650222    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8"]
Oct 31 20:04:49.664364 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:49.664319    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-e618e1-openshift-default-6448cf6987-v9ds8"]
Oct 31 20:04:50.116920 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:50.116881    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:50.586105 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:50.586068    4645 generic.go:356] "Generic (PLEG): container finished" podID="43f468d0-65fb-4e70-be20-2c4b66d67034" containerID="7c73a5ff8bfc6ad062ba730349acd817c0efd4be88c24702906b5e71c7b32da7" exitCode=0
Oct 31 20:04:50.586444 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:50.586401    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4bac5a-slrwq" event={"ID":"43f468d0-65fb-4e70-be20-2c4b66d67034","Type":"ContainerDied","Data":"7c73a5ff8bfc6ad062ba730349acd817c0efd4be88c24702906b5e71c7b32da7"}
Oct 31 20:04:51.515785 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:51.515750    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:51.620777 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:51.620735    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b24f33df-8939-405b-83a6-8d7af872e7da" path="/var/lib/kubelet/pods/b24f33df-8939-405b-83a6-8d7af872e7da/volumes"
Oct 31 20:04:51.621455 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:51.621432    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ede30242-d4ae-4caa-9c4b-8918314a1ead" path="/var/lib/kubelet/pods/ede30242-d4ae-4caa-9c4b-8918314a1ead/volumes"
Oct 31 20:04:51.621980 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:51.621959    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1b1d76a-8047-46ad-b08d-fdfb89ea668c" path="/var/lib/kubelet/pods/f1b1d76a-8047-46ad-b08d-fdfb89ea668c/volumes"
Oct 31 20:04:51.761255 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:51.761230    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4bac5a-slrwq"
Oct 31 20:04:51.829719 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:51.829647    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43f468d0-65fb-4e70-be20-2c4b66d67034-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43f468d0-65fb-4e70-be20-2c4b66d67034-maas-api-xks-ca\") pod \"43f468d0-65fb-4e70-be20-2c4b66d67034\" (UID: \"43f468d0-65fb-4e70-be20-2c4b66d67034\") "
Oct 31 20:04:51.829887 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:51.829867    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43f468d0-65fb-4e70-be20-2c4b66d67034-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43f468d0-65fb-4e70-be20-2c4b66d67034-maas-api-openshift-ca\") pod \"43f468d0-65fb-4e70-be20-2c4b66d67034\" (UID: \"43f468d0-65fb-4e70-be20-2c4b66d67034\") "
Oct 31 20:04:51.830005 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:51.829988    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43f468d0-65fb-4e70-be20-2c4b66d67034-kube-api-access-pvlsd\" (UniqueName: \"kubernetes.io/projected/43f468d0-65fb-4e70-be20-2c4b66d67034-kube-api-access-pvlsd\") pod \"43f468d0-65fb-4e70-be20-2c4b66d67034\" (UID: \"43f468d0-65fb-4e70-be20-2c4b66d67034\") "
Oct 31 20:04:51.830320 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:51.830284    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43f468d0-65fb-4e70-be20-2c4b66d67034-maas-api-openshift-ca" pod "43f468d0-65fb-4e70-be20-2c4b66d67034" (UID: "43f468d0-65fb-4e70-be20-2c4b66d67034"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:04:51.830796 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:51.830769    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43f468d0-65fb-4e70-be20-2c4b66d67034-maas-api-openshift-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:51.832322 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:51.832293    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43f468d0-65fb-4e70-be20-2c4b66d67034-kube-api-access-pvlsd" pod "43f468d0-65fb-4e70-be20-2c4b66d67034" (UID: "43f468d0-65fb-4e70-be20-2c4b66d67034"). InnerVolumeSpecName "kube-api-access-pvlsd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:04:51.832526 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:51.832471    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43f468d0-65fb-4e70-be20-2c4b66d67034-maas-api-xks-ca" pod "43f468d0-65fb-4e70-be20-2c4b66d67034" (UID: "43f468d0-65fb-4e70-be20-2c4b66d67034"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:04:51.932648 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:51.932616    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvlsd\" (UniqueName: \"kubernetes.io/projected/43f468d0-65fb-4e70-be20-2c4b66d67034-kube-api-access-pvlsd\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:51.932648 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:51.932647    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43f468d0-65fb-4e70-be20-2c4b66d67034-maas-api-xks-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:52.382500 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:52.382467    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:52.605802 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:52.605764    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4bac5a-slrwq" event={"ID":"43f468d0-65fb-4e70-be20-2c4b66d67034","Type":"ContainerDied","Data":"2a69cf7f081db9a1fdaa80e9c5577a9f05c22efca1998c86c4b7e8a2f448bd95"}
Oct 31 20:04:52.605802 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:52.605794    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4bac5a-slrwq"
Oct 31 20:04:52.606218 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:52.605818    4645 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a69cf7f081db9a1fdaa80e9c5577a9f05c22efca1998c86c4b7e8a2f448bd95"
Oct 31 20:04:53.078878 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:53.078787    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb"
Oct 31 20:04:53.142636 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:53.142603    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:53.151711 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:53.151654    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba05261e-eadf-44c3-abda-cb561b4ea6a5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba05261e-eadf-44c3-abda-cb561b4ea6a5-istio-data\") pod \"ba05261e-eadf-44c3-abda-cb561b4ea6a5\" (UID: \"ba05261e-eadf-44c3-abda-cb561b4ea6a5\") "
Oct 31 20:04:53.151894 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:53.151753    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba05261e-eadf-44c3-abda-cb561b4ea6a5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba05261e-eadf-44c3-abda-cb561b4ea6a5-credential-socket\") pod \"ba05261e-eadf-44c3-abda-cb561b4ea6a5\" (UID: \"ba05261e-eadf-44c3-abda-cb561b4ea6a5\") "
Oct 31 20:04:53.151894 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:53.151830    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba05261e-eadf-44c3-abda-cb561b4ea6a5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba05261e-eadf-44c3-abda-cb561b4ea6a5-istio-envoy\") pod \"ba05261e-eadf-44c3-abda-cb561b4ea6a5\" (UID: \"ba05261e-eadf-44c3-abda-cb561b4ea6a5\") "
Oct 31 20:04:53.152048 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:53.151915    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba05261e-eadf-44c3-abda-cb561b4ea6a5-istio-token\" (UniqueName: \"kubernetes.io/projected/ba05261e-eadf-44c3-abda-cb561b4ea6a5-istio-token\") pod \"ba05261e-eadf-44c3-abda-cb561b4ea6a5\" (UID: \"ba05261e-eadf-44c3-abda-cb561b4ea6a5\") "
Oct 31 20:04:53.152111 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:53.152049    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba05261e-eadf-44c3-abda-cb561b4ea6a5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba05261e-eadf-44c3-abda-cb561b4ea6a5-workload-certs\") pod \"ba05261e-eadf-44c3-abda-cb561b4ea6a5\" (UID: \"ba05261e-eadf-44c3-abda-cb561b4ea6a5\") "
Oct 31 20:04:53.152111 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:53.152051    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba05261e-eadf-44c3-abda-cb561b4ea6a5-istio-data" pod "ba05261e-eadf-44c3-abda-cb561b4ea6a5" (UID: "ba05261e-eadf-44c3-abda-cb561b4ea6a5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:04:53.152111 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:53.152089    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba05261e-eadf-44c3-abda-cb561b4ea6a5-kube-api-access-tm7pw\" (UniqueName: \"kubernetes.io/projected/ba05261e-eadf-44c3-abda-cb561b4ea6a5-kube-api-access-tm7pw\") pod \"ba05261e-eadf-44c3-abda-cb561b4ea6a5\" (UID: \"ba05261e-eadf-44c3-abda-cb561b4ea6a5\") "
Oct 31 20:04:53.152111 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:53.152100    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba05261e-eadf-44c3-abda-cb561b4ea6a5-credential-socket" pod "ba05261e-eadf-44c3-abda-cb561b4ea6a5" (UID: "ba05261e-eadf-44c3-abda-cb561b4ea6a5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:04:53.152385 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:53.152133    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba05261e-eadf-44c3-abda-cb561b4ea6a5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba05261e-eadf-44c3-abda-cb561b4ea6a5-workload-socket\") pod \"ba05261e-eadf-44c3-abda-cb561b4ea6a5\" (UID: \"ba05261e-eadf-44c3-abda-cb561b4ea6a5\") "
Oct 31 20:04:53.152385 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:53.152215    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ba05261e-eadf-44c3-abda-cb561b4ea6a5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba05261e-eadf-44c3-abda-cb561b4ea6a5-istio-podinfo\") pod \"ba05261e-eadf-44c3-abda-cb561b4ea6a5\" (UID: \"ba05261e-eadf-44c3-abda-cb561b4ea6a5\") "
Oct 31 20:04:53.152385 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:53.152268    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba05261e-eadf-44c3-abda-cb561b4ea6a5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba05261e-eadf-44c3-abda-cb561b4ea6a5-istiod-ca-cert\") pod \"ba05261e-eadf-44c3-abda-cb561b4ea6a5\" (UID: \"ba05261e-eadf-44c3-abda-cb561b4ea6a5\") "
Oct 31 20:04:53.152385 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:53.152320    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba05261e-eadf-44c3-abda-cb561b4ea6a5-workload-certs" pod "ba05261e-eadf-44c3-abda-cb561b4ea6a5" (UID: "ba05261e-eadf-44c3-abda-cb561b4ea6a5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:04:53.152603 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:53.152527    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba05261e-eadf-44c3-abda-cb561b4ea6a5-workload-socket" pod "ba05261e-eadf-44c3-abda-cb561b4ea6a5" (UID: "ba05261e-eadf-44c3-abda-cb561b4ea6a5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:04:53.152805 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:53.152782    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba05261e-eadf-44c3-abda-cb561b4ea6a5-istiod-ca-cert" pod "ba05261e-eadf-44c3-abda-cb561b4ea6a5" (UID: "ba05261e-eadf-44c3-abda-cb561b4ea6a5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:04:53.153062 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:53.153039    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba05261e-eadf-44c3-abda-cb561b4ea6a5-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:53.153147 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:53.153067    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba05261e-eadf-44c3-abda-cb561b4ea6a5-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:53.153147 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:53.153084    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba05261e-eadf-44c3-abda-cb561b4ea6a5-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:53.153147 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:53.153098    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba05261e-eadf-44c3-abda-cb561b4ea6a5-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:53.153147 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:53.153111    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba05261e-eadf-44c3-abda-cb561b4ea6a5-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:53.154864 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:53.154832    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba05261e-eadf-44c3-abda-cb561b4ea6a5-istio-envoy" pod "ba05261e-eadf-44c3-abda-cb561b4ea6a5" (UID: "ba05261e-eadf-44c3-abda-cb561b4ea6a5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:04:53.154974 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:53.154930    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba05261e-eadf-44c3-abda-cb561b4ea6a5-istio-podinfo" pod "ba05261e-eadf-44c3-abda-cb561b4ea6a5" (UID: "ba05261e-eadf-44c3-abda-cb561b4ea6a5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 20:04:53.155080 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:53.155063    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba05261e-eadf-44c3-abda-cb561b4ea6a5-istio-token" pod "ba05261e-eadf-44c3-abda-cb561b4ea6a5" (UID: "ba05261e-eadf-44c3-abda-cb561b4ea6a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:04:53.156750 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:53.156724    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba05261e-eadf-44c3-abda-cb561b4ea6a5-kube-api-access-tm7pw" pod "ba05261e-eadf-44c3-abda-cb561b4ea6a5" (UID: "ba05261e-eadf-44c3-abda-cb561b4ea6a5"). InnerVolumeSpecName "kube-api-access-tm7pw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:04:53.250473 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:53.250431    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:53.255008 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:53.254967    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tm7pw\" (UniqueName: \"kubernetes.io/projected/ba05261e-eadf-44c3-abda-cb561b4ea6a5-kube-api-access-tm7pw\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:53.255008 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:53.255007    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba05261e-eadf-44c3-abda-cb561b4ea6a5-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:53.255218 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:53.255021    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba05261e-eadf-44c3-abda-cb561b4ea6a5-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:53.255218 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:53.255033    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba05261e-eadf-44c3-abda-cb561b4ea6a5-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:53.615534 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:53.615489    4645 generic.go:356] "Generic (PLEG): container finished" podID="ba05261e-eadf-44c3-abda-cb561b4ea6a5" containerID="2d1e8615addbb51e1c72df9927e033ae0c2c9be3c100b66ea0db6468e04ecab7" exitCode=0
Oct 31 20:04:53.616046 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:53.615622    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb"
Oct 31 20:04:53.620369 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:53.620317    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb" event={"ID":"ba05261e-eadf-44c3-abda-cb561b4ea6a5","Type":"ContainerDied","Data":"2d1e8615addbb51e1c72df9927e033ae0c2c9be3c100b66ea0db6468e04ecab7"}
Oct 31 20:04:53.620530 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:53.620401    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb" event={"ID":"ba05261e-eadf-44c3-abda-cb561b4ea6a5","Type":"ContainerDied","Data":"10a9f2f047601abbb0068e9ee3ddbac9f74f9b69bc8c0997b269cf901ad8dca9"}
Oct 31 20:04:53.620530 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:53.620437    4645 scope.go:122] "RemoveContainer" containerID="2d1e8615addbb51e1c72df9927e033ae0c2c9be3c100b66ea0db6468e04ecab7"
Oct 31 20:04:53.632574 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:53.632548    4645 scope.go:122] "RemoveContainer" containerID="28340e41fc47b32fbd85c43ddd7a92a8220d26df04ef0518263bdd162b889516"
Oct 31 20:04:53.643728 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:53.643700    4645 scope.go:122] "RemoveContainer" containerID="2d1e8615addbb51e1c72df9927e033ae0c2c9be3c100b66ea0db6468e04ecab7"
Oct 31 20:04:53.643863 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:53.643843    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb"]
Oct 31 20:04:53.644063 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:04:53.644047    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d1e8615addbb51e1c72df9927e033ae0c2c9be3c100b66ea0db6468e04ecab7\": container with ID starting with 2d1e8615addbb51e1c72df9927e033ae0c2c9be3c100b66ea0db6468e04ecab7 not found: ID does not exist" containerID="2d1e8615addbb51e1c72df9927e033ae0c2c9be3c100b66ea0db6468e04ecab7"
Oct 31 20:04:53.644108 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:53.644074    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d1e8615addbb51e1c72df9927e033ae0c2c9be3c100b66ea0db6468e04ecab7"} err="failed to get container status \"2d1e8615addbb51e1c72df9927e033ae0c2c9be3c100b66ea0db6468e04ecab7\": rpc error: code = NotFound desc = could not find container \"2d1e8615addbb51e1c72df9927e033ae0c2c9be3c100b66ea0db6468e04ecab7\": container with ID starting with 2d1e8615addbb51e1c72df9927e033ae0c2c9be3c100b66ea0db6468e04ecab7 not found: ID does not exist"
Oct 31 20:04:53.644108 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:53.644099    4645 scope.go:122] "RemoveContainer" containerID="28340e41fc47b32fbd85c43ddd7a92a8220d26df04ef0518263bdd162b889516"
Oct 31 20:04:53.644450 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:04:53.644426    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28340e41fc47b32fbd85c43ddd7a92a8220d26df04ef0518263bdd162b889516\": container with ID starting with 28340e41fc47b32fbd85c43ddd7a92a8220d26df04ef0518263bdd162b889516 not found: ID does not exist" containerID="28340e41fc47b32fbd85c43ddd7a92a8220d26df04ef0518263bdd162b889516"
Oct 31 20:04:53.644538 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:53.644460    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28340e41fc47b32fbd85c43ddd7a92a8220d26df04ef0518263bdd162b889516"} err="failed to get container status \"28340e41fc47b32fbd85c43ddd7a92a8220d26df04ef0518263bdd162b889516\": rpc error: code = NotFound desc = could not find container \"28340e41fc47b32fbd85c43ddd7a92a8220d26df04ef0518263bdd162b889516\": container with ID starting with 28340e41fc47b32fbd85c43ddd7a92a8220d26df04ef0518263bdd162b889516 not found: ID does not exist"
Oct 31 20:04:53.650514 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:53.650478    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-a84fcb-openshift-default-5778f8f47c-l4tjb"]
Oct 31 20:04:53.865476 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:53.865428    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:54.198715 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:54.198668    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-d9c048-79ff7865f5-qq5qp"]
Oct 31 20:04:54.203323 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:54.203288    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d9c048-5459f7f8dc-rhrkn"]
Oct 31 20:04:54.269437 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:54.269403    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:54.378260 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:54.378215    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:54.701894 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:54.701828    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:55.436475 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:55.436437    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:55.668351 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:55.658233    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba05261e-eadf-44c3-abda-cb561b4ea6a5" path="/var/lib/kubelet/pods/ba05261e-eadf-44c3-abda-cb561b4ea6a5/volumes"
Oct 31 20:04:55.782438 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:55.781803    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:55.932113 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:55.932052    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:56.047983 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:56.047894    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:56.391723 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:56.391617    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:56.909388 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:56.909341    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:57.686842 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:57.686805    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w6-e1d060/rate-test-model-e1d060-kserve-5969df469-4mfjn" podUID="8b0c4b8a-bf5b-4af7-b272-aeefb1149b03" containerName="main" containerID="cri-o://8d8ef7c5b454d39dfbe4433c26006191b791f35bf1fb59e88d73007f15de6d05" gracePeriod=45
Oct 31 20:04:57.806270 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:57.806240    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:58.007162 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:58.007131    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-e1d060/rate-test-model-e1d060-kserve-5969df469-4mfjn"
Oct 31 20:04:58.059027 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:58.058951    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf94617d-rhhp2"]
Oct 31 20:04:58.063372 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:58.063339    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf94617d-rhhp2"]
Oct 31 20:04:58.173729 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:58.173691    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b0c4b8a-bf5b-4af7-b272-aeefb1149b03-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b0c4b8a-bf5b-4af7-b272-aeefb1149b03-dshm\") pod \"8b0c4b8a-bf5b-4af7-b272-aeefb1149b03\" (UID: \"8b0c4b8a-bf5b-4af7-b272-aeefb1149b03\") "
Oct 31 20:04:58.173892 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:58.173833    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b0c4b8a-bf5b-4af7-b272-aeefb1149b03-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b0c4b8a-bf5b-4af7-b272-aeefb1149b03-tmp-dir\") pod \"8b0c4b8a-bf5b-4af7-b272-aeefb1149b03\" (UID: \"8b0c4b8a-bf5b-4af7-b272-aeefb1149b03\") "
Oct 31 20:04:58.173969 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:58.173946    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b0c4b8a-bf5b-4af7-b272-aeefb1149b03-tls-certs\" (UniqueName: \"kubernetes.io/secret/8b0c4b8a-bf5b-4af7-b272-aeefb1149b03-tls-certs\") pod \"8b0c4b8a-bf5b-4af7-b272-aeefb1149b03\" (UID: \"8b0c4b8a-bf5b-4af7-b272-aeefb1149b03\") "
Oct 31 20:04:58.174023 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:58.174006    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b0c4b8a-bf5b-4af7-b272-aeefb1149b03-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b0c4b8a-bf5b-4af7-b272-aeefb1149b03-model-cache\") pod \"8b0c4b8a-bf5b-4af7-b272-aeefb1149b03\" (UID: \"8b0c4b8a-bf5b-4af7-b272-aeefb1149b03\") "
Oct 31 20:04:58.174140 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:58.174108    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b0c4b8a-bf5b-4af7-b272-aeefb1149b03-kube-api-access-9qw8v\" (UniqueName: \"kubernetes.io/projected/8b0c4b8a-bf5b-4af7-b272-aeefb1149b03-kube-api-access-9qw8v\") pod \"8b0c4b8a-bf5b-4af7-b272-aeefb1149b03\" (UID: \"8b0c4b8a-bf5b-4af7-b272-aeefb1149b03\") "
Oct 31 20:04:58.174140 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:58.174128    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b0c4b8a-bf5b-4af7-b272-aeefb1149b03-tmp-dir" pod "8b0c4b8a-bf5b-4af7-b272-aeefb1149b03" (UID: "8b0c4b8a-bf5b-4af7-b272-aeefb1149b03"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:04:58.174377 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:58.174211    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b0c4b8a-bf5b-4af7-b272-aeefb1149b03-home\" (UniqueName: \"kubernetes.io/empty-dir/8b0c4b8a-bf5b-4af7-b272-aeefb1149b03-home\") pod \"8b0c4b8a-bf5b-4af7-b272-aeefb1149b03\" (UID: \"8b0c4b8a-bf5b-4af7-b272-aeefb1149b03\") "
Oct 31 20:04:58.174377 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:58.174312    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b0c4b8a-bf5b-4af7-b272-aeefb1149b03-model-cache" pod "8b0c4b8a-bf5b-4af7-b272-aeefb1149b03" (UID: "8b0c4b8a-bf5b-4af7-b272-aeefb1149b03"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:04:58.174513 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:58.174469    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b0c4b8a-bf5b-4af7-b272-aeefb1149b03-home" pod "8b0c4b8a-bf5b-4af7-b272-aeefb1149b03" (UID: "8b0c4b8a-bf5b-4af7-b272-aeefb1149b03"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:04:58.174784 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:58.174766    4645 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b0c4b8a-bf5b-4af7-b272-aeefb1149b03-tmp-dir\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:58.174851 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:58.174786    4645 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b0c4b8a-bf5b-4af7-b272-aeefb1149b03-model-cache\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:58.174851 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:58.174795    4645 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b0c4b8a-bf5b-4af7-b272-aeefb1149b03-home\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:58.176098 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:58.176062    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b0c4b8a-bf5b-4af7-b272-aeefb1149b03-tls-certs" pod "8b0c4b8a-bf5b-4af7-b272-aeefb1149b03" (UID: "8b0c4b8a-bf5b-4af7-b272-aeefb1149b03"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:04:58.176392 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:58.176375    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b0c4b8a-bf5b-4af7-b272-aeefb1149b03-dshm" pod "8b0c4b8a-bf5b-4af7-b272-aeefb1149b03" (UID: "8b0c4b8a-bf5b-4af7-b272-aeefb1149b03"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:04:58.176464 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:58.176445    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b0c4b8a-bf5b-4af7-b272-aeefb1149b03-kube-api-access-9qw8v" pod "8b0c4b8a-bf5b-4af7-b272-aeefb1149b03" (UID: "8b0c4b8a-bf5b-4af7-b272-aeefb1149b03"). InnerVolumeSpecName "kube-api-access-9qw8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:04:58.275978 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:58.275901    4645 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b0c4b8a-bf5b-4af7-b272-aeefb1149b03-tls-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:58.275978 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:58.275932    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qw8v\" (UniqueName: \"kubernetes.io/projected/8b0c4b8a-bf5b-4af7-b272-aeefb1149b03-kube-api-access-9qw8v\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:58.275978 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:58.275942    4645 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b0c4b8a-bf5b-4af7-b272-aeefb1149b03-dshm\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:04:58.546087 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:58.545998    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:58.571413 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:58.571385    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-4bac5a-78664677cc-6shsd"]
Oct 31 20:04:58.672601 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:58.672563    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:58.749719 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:58.747513    4645 generic.go:356] "Generic (PLEG): container finished" podID="8b0c4b8a-bf5b-4af7-b272-aeefb1149b03" containerID="8d8ef7c5b454d39dfbe4433c26006191b791f35bf1fb59e88d73007f15de6d05" exitCode=0
Oct 31 20:04:58.749719 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:58.747852    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-e1d060/rate-test-model-e1d060-kserve-5969df469-4mfjn"
Oct 31 20:04:58.749719 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:58.747867    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-e1d060/rate-test-model-e1d060-kserve-5969df469-4mfjn" event={"ID":"8b0c4b8a-bf5b-4af7-b272-aeefb1149b03","Type":"ContainerDied","Data":"8d8ef7c5b454d39dfbe4433c26006191b791f35bf1fb59e88d73007f15de6d05"}
Oct 31 20:04:58.749719 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:58.747915    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-e1d060/rate-test-model-e1d060-kserve-5969df469-4mfjn" event={"ID":"8b0c4b8a-bf5b-4af7-b272-aeefb1149b03","Type":"ContainerDied","Data":"393715bd38586726f501e893d1276f50ce6a3c79e30216f8f1634051eed56a43"}
Oct 31 20:04:58.749719 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:58.747937    4645 scope.go:122] "RemoveContainer" containerID="8d8ef7c5b454d39dfbe4433c26006191b791f35bf1fb59e88d73007f15de6d05"
Oct 31 20:04:58.758941 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:58.758914    4645 scope.go:122] "RemoveContainer" containerID="8d8ef7c5b454d39dfbe4433c26006191b791f35bf1fb59e88d73007f15de6d05"
Oct 31 20:04:58.759211 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:04:58.759194    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d8ef7c5b454d39dfbe4433c26006191b791f35bf1fb59e88d73007f15de6d05\": container with ID starting with 8d8ef7c5b454d39dfbe4433c26006191b791f35bf1fb59e88d73007f15de6d05 not found: ID does not exist" containerID="8d8ef7c5b454d39dfbe4433c26006191b791f35bf1fb59e88d73007f15de6d05"
Oct 31 20:04:58.759286 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:58.759220    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d8ef7c5b454d39dfbe4433c26006191b791f35bf1fb59e88d73007f15de6d05"} err="failed to get container status \"8d8ef7c5b454d39dfbe4433c26006191b791f35bf1fb59e88d73007f15de6d05\": rpc error: code = NotFound desc = could not find container \"8d8ef7c5b454d39dfbe4433c26006191b791f35bf1fb59e88d73007f15de6d05\": container with ID starting with 8d8ef7c5b454d39dfbe4433c26006191b791f35bf1fb59e88d73007f15de6d05 not found: ID does not exist"
Oct 31 20:04:58.780044 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:58.780010    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w6-e1d060/rate-test-model-e1d060-kserve-5969df469-4mfjn"]
Oct 31 20:04:58.784748 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:58.784726    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w6-e1d060/rate-test-model-e1d060-kserve-5969df469-4mfjn"]
Oct 31 20:04:58.820066 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:58.820016    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:59.199895 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:59.199841    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d9c048-79ff7865f5-qq5qp" podUID="b4677f66-6fc4-4c4a-8539-ede1dfe8f503" containerName="payload-processing" containerID="cri-o://4c0a2d2968e63a1cbf5e9c590794fbb50ff640b4998c60b08ceb9dc5aed8bff6" gracePeriod=25
Oct 31 20:04:59.203621 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:59.203564    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d9c048-5459f7f8dc-rhrkn" podUID="7d33bec5-7276-4b99-9a02-6ac2eaaf770f" containerName="payload-pre-processing" containerID="cri-o://93e922dc887ee5e527ed9b2837c331ca8a034671b2ddbd9fa2eb01e78364c7d3" gracePeriod=25
Oct 31 20:04:59.481378 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:59.481274    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:04:59.618571 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:59.618528    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43db5dd0-3e61-485e-9d44-b141096e887a" path="/var/lib/kubelet/pods/43db5dd0-3e61-485e-9d44-b141096e887a/volumes"
Oct 31 20:04:59.619162 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:04:59.619140    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b0c4b8a-bf5b-4af7-b272-aeefb1149b03" path="/var/lib/kubelet/pods/8b0c4b8a-bf5b-4af7-b272-aeefb1149b03/volumes"
Oct 31 20:05:00.255774 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:00.255736    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:00.524416 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:00.522842    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:00.632115 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:00.632080    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:01.277644 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:01.277607    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:02.159118 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:02.159083    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:03.070368 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:03.070323    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:03.571924 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:03.571891    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4bac5a-78664677cc-6shsd" podUID="aa76773f-998f-408d-a804-31f6c4921774" containerName="payload-processing" containerID="cri-o://e88fd7cd9d0e1fc554eb036fb0aa37effec4b66817bff1b01a519b8a943229d2" gracePeriod=25
Oct 31 20:05:04.035724 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:04.035690    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:04.642925 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:04.642886    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d9c048-5459f7f8dc-rhrkn" podUID="7d33bec5-7276-4b99-9a02-6ac2eaaf770f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:05:04.768082 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:04.768057    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:04.905223 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:04.905141    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:05.025353 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:05.021715    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:05.253607 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:05.253565    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk"]
Oct 31 20:05:05.253841 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:05.253808    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk" podUID="bd191bcb-3c5e-49ff-accd-8a706eef5bd9" containerName="istio-proxy" containerID="cri-o://b7800bf245a0dd9aeec96bdbcdbcf45b1b443b61f10a69633b95126ed0f10543" gracePeriod=30
Oct 31 20:05:05.263437 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:05.263409    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl"]
Oct 31 20:05:05.263850 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:05.263823    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl" podUID="9deed825-0c5f-4950-96e4-098a01b928f4" containerName="istio-proxy" containerID="cri-o://d61f9ae3975309d54fda8264fda0f42f298e67f4b14db44a3365d1ad9257cf4b" gracePeriod=30
Oct 31 20:05:05.300037 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:05.299994    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:05.651464 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:05.651376    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d9c048-79ff7865f5-qq5qp" podUID="b4677f66-6fc4-4c4a-8539-ede1dfe8f503" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:05:05.721976 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:05.721945    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:05.846213 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:05.846182    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:05.989142 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:05.989108    4645 patch_prober.go:29] interesting pod/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 20:05:05.989308 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:05.989170    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl" podUID="9deed825-0c5f-4950-96e4-098a01b928f4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 20:05:06.245988 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:05:06.245912    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 20:05:06.682196 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:06.682161    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:06.886363 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:06.885103    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:07.105405 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:07.105294    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a891ce-fpmrl"]
Oct 31 20:05:07.108798 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:07.108766    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-a891ce-fpmrl"]
Oct 31 20:05:07.135651 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:07.135611    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:07.297273 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:07.297234    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:07.618892 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:07.618833    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82b3f50e-f79d-4f62-a212-1138dd4fd528" path="/var/lib/kubelet/pods/82b3f50e-f79d-4f62-a212-1138dd4fd528/volumes"
Oct 31 20:05:07.628697 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:07.628666    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:08.053168 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:08.053134    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:08.875316 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:08.875279    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:09.066905 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:09.066868    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:09.201354 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:09.192704    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:09.250731 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:09.249395    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4bac5a-slrwq"]
Oct 31 20:05:09.308057 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:09.308018    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4bac5a-slrwq"]
Oct 31 20:05:09.311913 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:09.311862    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:09.618452 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:09.618360    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43f468d0-65fb-4e70-be20-2c4b66d67034" path="/var/lib/kubelet/pods/43f468d0-65fb-4e70-be20-2c4b66d67034/volumes"
Oct 31 20:05:09.651720 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:09.651684    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:09.833178 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:09.833143    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4bac5a-78664677cc-6shsd" podUID="aa76773f-998f-408d-a804-31f6c4921774" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:05:09.948734 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:09.948704    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:10.049458 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.049420    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:10.146979 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.146942    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f28d48-5bbcbb6f5-8jbqq"]
Oct 31 20:05:10.147728 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.147706    4645 cpu_manager.go:414] "containerMap: removing container" podUID="ba05261e-eadf-44c3-abda-cb561b4ea6a5" containerName="istio-proxy"
Oct 31 20:05:10.147842 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.147729    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba05261e-eadf-44c3-abda-cb561b4ea6a5" containerName="istio-proxy"
Oct 31 20:05:10.147842 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.147755    4645 cpu_manager.go:414] "containerMap: removing container" podUID="f1b1d76a-8047-46ad-b08d-fdfb89ea668c" containerName="main"
Oct 31 20:05:10.147842 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.147762    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1b1d76a-8047-46ad-b08d-fdfb89ea668c" containerName="main"
Oct 31 20:05:10.147842 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.147777    4645 cpu_manager.go:414] "containerMap: removing container" podUID="8b0c4b8a-bf5b-4af7-b272-aeefb1149b03" containerName="main"
Oct 31 20:05:10.147842 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.147784    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b0c4b8a-bf5b-4af7-b272-aeefb1149b03" containerName="main"
Oct 31 20:05:10.147842 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.147796    4645 cpu_manager.go:414] "containerMap: removing container" podUID="ede30242-d4ae-4caa-9c4b-8918314a1ead" containerName="istio-proxy"
Oct 31 20:05:10.147842 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.147804    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ede30242-d4ae-4caa-9c4b-8918314a1ead" containerName="istio-proxy"
Oct 31 20:05:10.147842 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.147815    4645 cpu_manager.go:414] "containerMap: removing container" podUID="ba05261e-eadf-44c3-abda-cb561b4ea6a5" containerName="istio-proxy"
Oct 31 20:05:10.147842 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.147822    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba05261e-eadf-44c3-abda-cb561b4ea6a5" containerName="istio-proxy"
Oct 31 20:05:10.148324 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.147853    4645 cpu_manager.go:414] "containerMap: removing container" podUID="52d535c0-6d59-4788-9258-bbee438df233" containerName="istio-proxy"
Oct 31 20:05:10.148324 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.147860    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52d535c0-6d59-4788-9258-bbee438df233" containerName="istio-proxy"
Oct 31 20:05:10.148324 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.147899    4645 cpu_manager.go:414] "containerMap: removing container" podUID="b24f33df-8939-405b-83a6-8d7af872e7da" containerName="istio-proxy"
Oct 31 20:05:10.148324 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.147906    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b24f33df-8939-405b-83a6-8d7af872e7da" containerName="istio-proxy"
Oct 31 20:05:10.148324 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.147925    4645 cpu_manager.go:414] "containerMap: removing container" podUID="43f468d0-65fb-4e70-be20-2c4b66d67034" containerName="revoke-keys"
Oct 31 20:05:10.148324 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.147933    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43f468d0-65fb-4e70-be20-2c4b66d67034" containerName="revoke-keys"
Oct 31 20:05:10.148324 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.148068    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="ede30242-d4ae-4caa-9c4b-8918314a1ead" containerName="istio-proxy"
Oct 31 20:05:10.148324 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.148080    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="b24f33df-8939-405b-83a6-8d7af872e7da" containerName="istio-proxy"
Oct 31 20:05:10.148324 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.148094    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="43f468d0-65fb-4e70-be20-2c4b66d67034" containerName="revoke-keys"
Oct 31 20:05:10.148324 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.148103    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba05261e-eadf-44c3-abda-cb561b4ea6a5" containerName="istio-proxy"
Oct 31 20:05:10.148324 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.148112    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1b1d76a-8047-46ad-b08d-fdfb89ea668c" containerName="main"
Oct 31 20:05:10.148324 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.148123    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b0c4b8a-bf5b-4af7-b272-aeefb1149b03" containerName="main"
Oct 31 20:05:10.148324 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.148132    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="ede30242-d4ae-4caa-9c4b-8918314a1ead" containerName="istio-proxy"
Oct 31 20:05:10.148324 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.148146    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="52d535c0-6d59-4788-9258-bbee438df233" containerName="istio-proxy"
Oct 31 20:05:10.153214 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.153192    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f28d48-5bbcbb6f5-8jbqq"
Oct 31 20:05:10.162046 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.162025    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f28d48\""
Oct 31 20:05:10.162163 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.162144    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f28d48-dockercfg-8p8j5\""
Oct 31 20:05:10.163211 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.163188    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f28d48-5bbcbb6f5-8jbqq"]
Oct 31 20:05:10.326361 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.299119    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7599e792-03f8-41bf-9cda-aca650a9022b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f28d48-5bbcbb6f5-8jbqq\" (UID: \"7599e792-03f8-41bf-9cda-aca650a9022b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f28d48-5bbcbb6f5-8jbqq"
Oct 31 20:05:10.345826 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.330164    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f28d48-6fc75df778-4x22w"]
Oct 31 20:05:10.355394 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.349696    4645 cpu_manager.go:414] "containerMap: removing container" podUID="ede30242-d4ae-4caa-9c4b-8918314a1ead" containerName="istio-proxy"
Oct 31 20:05:10.355601 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.355576    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ede30242-d4ae-4caa-9c4b-8918314a1ead" containerName="istio-proxy"
Oct 31 20:05:10.355960 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.355941    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba05261e-eadf-44c3-abda-cb561b4ea6a5" containerName="istio-proxy"
Oct 31 20:05:10.358796 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.358764    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnpd8\" (UniqueName: \"kubernetes.io/projected/7599e792-03f8-41bf-9cda-aca650a9022b-kube-api-access-tnpd8\") pod \"payload-pre-processing-e2e-ipp-a-f28d48-5bbcbb6f5-8jbqq\" (UID: \"7599e792-03f8-41bf-9cda-aca650a9022b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f28d48-5bbcbb6f5-8jbqq"
Oct 31 20:05:10.362887 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.362859    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f28d48-6fc75df778-4x22w"]
Oct 31 20:05:10.363146 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.363122    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f28d48-6fc75df778-4x22w"
Oct 31 20:05:10.462894 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.462858    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnpd8\" (UniqueName: \"kubernetes.io/projected/7599e792-03f8-41bf-9cda-aca650a9022b-kube-api-access-tnpd8\") pod \"payload-pre-processing-e2e-ipp-a-f28d48-5bbcbb6f5-8jbqq\" (UID: \"7599e792-03f8-41bf-9cda-aca650a9022b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f28d48-5bbcbb6f5-8jbqq"
Oct 31 20:05:10.463015 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.462988    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4swf\" (UniqueName: \"kubernetes.io/projected/2dd3d751-d8d8-4fa7-b1bf-5c2da531f07d-kube-api-access-v4swf\") pod \"payload-processing-e2e-ipp-a-f28d48-6fc75df778-4x22w\" (UID: \"2dd3d751-d8d8-4fa7-b1bf-5c2da531f07d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f28d48-6fc75df778-4x22w"
Oct 31 20:05:10.463133 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.463117    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dd3d751-d8d8-4fa7-b1bf-5c2da531f07d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f28d48-6fc75df778-4x22w\" (UID: \"2dd3d751-d8d8-4fa7-b1bf-5c2da531f07d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f28d48-6fc75df778-4x22w"
Oct 31 20:05:10.463254 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.463238    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7599e792-03f8-41bf-9cda-aca650a9022b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f28d48-5bbcbb6f5-8jbqq\" (UID: \"7599e792-03f8-41bf-9cda-aca650a9022b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f28d48-5bbcbb6f5-8jbqq"
Oct 31 20:05:10.464235 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.464175    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7599e792-03f8-41bf-9cda-aca650a9022b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f28d48-5bbcbb6f5-8jbqq\" (UID: \"7599e792-03f8-41bf-9cda-aca650a9022b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f28d48-5bbcbb6f5-8jbqq"
Oct 31 20:05:10.476376 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.476306    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnpd8\" (UniqueName: \"kubernetes.io/projected/7599e792-03f8-41bf-9cda-aca650a9022b-kube-api-access-tnpd8\") pod \"payload-pre-processing-e2e-ipp-a-f28d48-5bbcbb6f5-8jbqq\" (UID: \"7599e792-03f8-41bf-9cda-aca650a9022b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f28d48-5bbcbb6f5-8jbqq"
Oct 31 20:05:10.564037 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.564012    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4swf\" (UniqueName: \"kubernetes.io/projected/2dd3d751-d8d8-4fa7-b1bf-5c2da531f07d-kube-api-access-v4swf\") pod \"payload-processing-e2e-ipp-a-f28d48-6fc75df778-4x22w\" (UID: \"2dd3d751-d8d8-4fa7-b1bf-5c2da531f07d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f28d48-6fc75df778-4x22w"
Oct 31 20:05:10.564176 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.564114    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dd3d751-d8d8-4fa7-b1bf-5c2da531f07d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f28d48-6fc75df778-4x22w\" (UID: \"2dd3d751-d8d8-4fa7-b1bf-5c2da531f07d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f28d48-6fc75df778-4x22w"
Oct 31 20:05:10.564860 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.564828    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dd3d751-d8d8-4fa7-b1bf-5c2da531f07d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f28d48-6fc75df778-4x22w\" (UID: \"2dd3d751-d8d8-4fa7-b1bf-5c2da531f07d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f28d48-6fc75df778-4x22w"
Oct 31 20:05:10.565639 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.565611    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:10.565930 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.565912    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl"
Oct 31 20:05:10.575759 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.575737    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4swf\" (UniqueName: \"kubernetes.io/projected/2dd3d751-d8d8-4fa7-b1bf-5c2da531f07d-kube-api-access-v4swf\") pod \"payload-processing-e2e-ipp-a-f28d48-6fc75df778-4x22w\" (UID: \"2dd3d751-d8d8-4fa7-b1bf-5c2da531f07d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f28d48-6fc75df778-4x22w"
Oct 31 20:05:10.747254 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.747219    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n"]
Oct 31 20:05:10.766722 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.766693    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f28d48-6fc75df778-4x22w"
Oct 31 20:05:10.768751 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.768718    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f28d48-5bbcbb6f5-8jbqq"
Oct 31 20:05:10.771363 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.771307    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9deed825-0c5f-4950-96e4-098a01b928f4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9deed825-0c5f-4950-96e4-098a01b928f4-istio-envoy\") pod \"9deed825-0c5f-4950-96e4-098a01b928f4\" (UID: \"9deed825-0c5f-4950-96e4-098a01b928f4\") "
Oct 31 20:05:10.771478 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.771453    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9deed825-0c5f-4950-96e4-098a01b928f4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9deed825-0c5f-4950-96e4-098a01b928f4-istio-podinfo\") pod \"9deed825-0c5f-4950-96e4-098a01b928f4\" (UID: \"9deed825-0c5f-4950-96e4-098a01b928f4\") "
Oct 31 20:05:10.771553 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.771493    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9deed825-0c5f-4950-96e4-098a01b928f4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9deed825-0c5f-4950-96e4-098a01b928f4-istio-data\") pod \"9deed825-0c5f-4950-96e4-098a01b928f4\" (UID: \"9deed825-0c5f-4950-96e4-098a01b928f4\") "
Oct 31 20:05:10.771624 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.771566    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9deed825-0c5f-4950-96e4-098a01b928f4-istio-token\" (UniqueName: \"kubernetes.io/projected/9deed825-0c5f-4950-96e4-098a01b928f4-istio-token\") pod \"9deed825-0c5f-4950-96e4-098a01b928f4\" (UID: \"9deed825-0c5f-4950-96e4-098a01b928f4\") "
Oct 31 20:05:10.771624 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.771604    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9deed825-0c5f-4950-96e4-098a01b928f4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9deed825-0c5f-4950-96e4-098a01b928f4-workload-socket\") pod \"9deed825-0c5f-4950-96e4-098a01b928f4\" (UID: \"9deed825-0c5f-4950-96e4-098a01b928f4\") "
Oct 31 20:05:10.771748 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.771663    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9deed825-0c5f-4950-96e4-098a01b928f4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9deed825-0c5f-4950-96e4-098a01b928f4-istiod-ca-cert\") pod \"9deed825-0c5f-4950-96e4-098a01b928f4\" (UID: \"9deed825-0c5f-4950-96e4-098a01b928f4\") "
Oct 31 20:05:10.771748 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.771727    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9deed825-0c5f-4950-96e4-098a01b928f4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9deed825-0c5f-4950-96e4-098a01b928f4-workload-certs\") pod \"9deed825-0c5f-4950-96e4-098a01b928f4\" (UID: \"9deed825-0c5f-4950-96e4-098a01b928f4\") "
Oct 31 20:05:10.771866 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.771773    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9deed825-0c5f-4950-96e4-098a01b928f4-kube-api-access-hrnk2\" (UniqueName: \"kubernetes.io/projected/9deed825-0c5f-4950-96e4-098a01b928f4-kube-api-access-hrnk2\") pod \"9deed825-0c5f-4950-96e4-098a01b928f4\" (UID: \"9deed825-0c5f-4950-96e4-098a01b928f4\") "
Oct 31 20:05:10.771866 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.771833    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9deed825-0c5f-4950-96e4-098a01b928f4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9deed825-0c5f-4950-96e4-098a01b928f4-credential-socket\") pod \"9deed825-0c5f-4950-96e4-098a01b928f4\" (UID: \"9deed825-0c5f-4950-96e4-098a01b928f4\") "
Oct 31 20:05:10.772707 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.772679    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9deed825-0c5f-4950-96e4-098a01b928f4-credential-socket" pod "9deed825-0c5f-4950-96e4-098a01b928f4" (UID: "9deed825-0c5f-4950-96e4-098a01b928f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:10.773764 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.773558    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9deed825-0c5f-4950-96e4-098a01b928f4-workload-certs" pod "9deed825-0c5f-4950-96e4-098a01b928f4" (UID: "9deed825-0c5f-4950-96e4-098a01b928f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:10.774477 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.774229    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9deed825-0c5f-4950-96e4-098a01b928f4-istiod-ca-cert" pod "9deed825-0c5f-4950-96e4-098a01b928f4" (UID: "9deed825-0c5f-4950-96e4-098a01b928f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:05:10.774591 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.774509    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9deed825-0c5f-4950-96e4-098a01b928f4-istio-data" pod "9deed825-0c5f-4950-96e4-098a01b928f4" (UID: "9deed825-0c5f-4950-96e4-098a01b928f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:10.776402 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.774732    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9deed825-0c5f-4950-96e4-098a01b928f4-workload-socket" pod "9deed825-0c5f-4950-96e4-098a01b928f4" (UID: "9deed825-0c5f-4950-96e4-098a01b928f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:10.776402 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.776268    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9deed825-0c5f-4950-96e4-098a01b928f4-istio-token" pod "9deed825-0c5f-4950-96e4-098a01b928f4" (UID: "9deed825-0c5f-4950-96e4-098a01b928f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:05:10.778365 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.778307    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9deed825-0c5f-4950-96e4-098a01b928f4-kube-api-access-hrnk2" pod "9deed825-0c5f-4950-96e4-098a01b928f4" (UID: "9deed825-0c5f-4950-96e4-098a01b928f4"). InnerVolumeSpecName "kube-api-access-hrnk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:05:10.778631 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.778606    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9deed825-0c5f-4950-96e4-098a01b928f4-istio-podinfo" pod "9deed825-0c5f-4950-96e4-098a01b928f4" (UID: "9deed825-0c5f-4950-96e4-098a01b928f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 20:05:10.791896 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.791839    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9deed825-0c5f-4950-96e4-098a01b928f4-istio-envoy" pod "9deed825-0c5f-4950-96e4-098a01b928f4" (UID: "9deed825-0c5f-4950-96e4-098a01b928f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:10.842351 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.841855    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:10.879793 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.878772    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9deed825-0c5f-4950-96e4-098a01b928f4-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:10.879793 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.878812    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9deed825-0c5f-4950-96e4-098a01b928f4-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:10.879793 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.878840    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9deed825-0c5f-4950-96e4-098a01b928f4-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:10.879793 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.878856    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9deed825-0c5f-4950-96e4-098a01b928f4-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:10.879793 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.879623    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrnk2\" (UniqueName: \"kubernetes.io/projected/9deed825-0c5f-4950-96e4-098a01b928f4-kube-api-access-hrnk2\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:10.879793 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.879658    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9deed825-0c5f-4950-96e4-098a01b928f4-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:10.879793 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.879684    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9deed825-0c5f-4950-96e4-098a01b928f4-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:10.879793 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.879697    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9deed825-0c5f-4950-96e4-098a01b928f4-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:10.879793 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.879710    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9deed825-0c5f-4950-96e4-098a01b928f4-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:10.977815 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.977780    4645 generic.go:356] "Generic (PLEG): container finished" podID="bd191bcb-3c5e-49ff-accd-8a706eef5bd9" containerID="b7800bf245a0dd9aeec96bdbcdbcf45b1b443b61f10a69633b95126ed0f10543" exitCode=0
Oct 31 20:05:10.978370 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.977919    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk" event={"ID":"bd191bcb-3c5e-49ff-accd-8a706eef5bd9","Type":"ContainerDied","Data":"b7800bf245a0dd9aeec96bdbcdbcf45b1b443b61f10a69633b95126ed0f10543"}
Oct 31 20:05:10.979945 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.979914    4645 generic.go:356] "Generic (PLEG): container finished" podID="9deed825-0c5f-4950-96e4-098a01b928f4" containerID="d61f9ae3975309d54fda8264fda0f42f298e67f4b14db44a3365d1ad9257cf4b" exitCode=0
Oct 31 20:05:10.980091 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.979968    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl" event={"ID":"9deed825-0c5f-4950-96e4-098a01b928f4","Type":"ContainerDied","Data":"d61f9ae3975309d54fda8264fda0f42f298e67f4b14db44a3365d1ad9257cf4b"}
Oct 31 20:05:10.980091 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.979990    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl"
Oct 31 20:05:10.980091 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.980013    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl" event={"ID":"9deed825-0c5f-4950-96e4-098a01b928f4","Type":"ContainerDied","Data":"e1d1970e38322b81a328aa1bfa7c8115d30bd19b5303dfa7fe63400d6faa690d"}
Oct 31 20:05:10.980091 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.980042    4645 scope.go:122] "RemoveContainer" containerID="d61f9ae3975309d54fda8264fda0f42f298e67f4b14db44a3365d1ad9257cf4b"
Oct 31 20:05:10.995019 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.994996    4645 scope.go:122] "RemoveContainer" containerID="d61f9ae3975309d54fda8264fda0f42f298e67f4b14db44a3365d1ad9257cf4b"
Oct 31 20:05:10.995344 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:05:10.995304    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d61f9ae3975309d54fda8264fda0f42f298e67f4b14db44a3365d1ad9257cf4b\": container with ID starting with d61f9ae3975309d54fda8264fda0f42f298e67f4b14db44a3365d1ad9257cf4b not found: ID does not exist" containerID="d61f9ae3975309d54fda8264fda0f42f298e67f4b14db44a3365d1ad9257cf4b"
Oct 31 20:05:10.995442 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:10.995365    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d61f9ae3975309d54fda8264fda0f42f298e67f4b14db44a3365d1ad9257cf4b"} err="failed to get container status \"d61f9ae3975309d54fda8264fda0f42f298e67f4b14db44a3365d1ad9257cf4b\": rpc error: code = NotFound desc = could not find container \"d61f9ae3975309d54fda8264fda0f42f298e67f4b14db44a3365d1ad9257cf4b\": container with ID starting with d61f9ae3975309d54fda8264fda0f42f298e67f4b14db44a3365d1ad9257cf4b not found: ID does not exist"
Oct 31 20:05:11.006404 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:11.006379    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl"]
Oct 31 20:05:11.018229 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:11.018210    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk"
Oct 31 20:05:11.023263 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:11.023240    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-4krfl"]
Oct 31 20:05:11.082235 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:11.082168    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bd191bcb-3c5e-49ff-accd-8a706eef5bd9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd191bcb-3c5e-49ff-accd-8a706eef5bd9-istio-podinfo\") pod \"bd191bcb-3c5e-49ff-accd-8a706eef5bd9\" (UID: \"bd191bcb-3c5e-49ff-accd-8a706eef5bd9\") "
Oct 31 20:05:11.082235 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:11.082216    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd191bcb-3c5e-49ff-accd-8a706eef5bd9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd191bcb-3c5e-49ff-accd-8a706eef5bd9-istio-envoy\") pod \"bd191bcb-3c5e-49ff-accd-8a706eef5bd9\" (UID: \"bd191bcb-3c5e-49ff-accd-8a706eef5bd9\") "
Oct 31 20:05:11.082481 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:11.082255    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd191bcb-3c5e-49ff-accd-8a706eef5bd9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd191bcb-3c5e-49ff-accd-8a706eef5bd9-istio-data\") pod \"bd191bcb-3c5e-49ff-accd-8a706eef5bd9\" (UID: \"bd191bcb-3c5e-49ff-accd-8a706eef5bd9\") "
Oct 31 20:05:11.082481 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:11.082410    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd191bcb-3c5e-49ff-accd-8a706eef5bd9-istio-token\" (UniqueName: \"kubernetes.io/projected/bd191bcb-3c5e-49ff-accd-8a706eef5bd9-istio-token\") pod \"bd191bcb-3c5e-49ff-accd-8a706eef5bd9\" (UID: \"bd191bcb-3c5e-49ff-accd-8a706eef5bd9\") "
Oct 31 20:05:11.082481 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:11.082465    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd191bcb-3c5e-49ff-accd-8a706eef5bd9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd191bcb-3c5e-49ff-accd-8a706eef5bd9-workload-certs\") pod \"bd191bcb-3c5e-49ff-accd-8a706eef5bd9\" (UID: \"bd191bcb-3c5e-49ff-accd-8a706eef5bd9\") "
Oct 31 20:05:11.082658 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:11.082550    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd191bcb-3c5e-49ff-accd-8a706eef5bd9-kube-api-access-hgzn7\" (UniqueName: \"kubernetes.io/projected/bd191bcb-3c5e-49ff-accd-8a706eef5bd9-kube-api-access-hgzn7\") pod \"bd191bcb-3c5e-49ff-accd-8a706eef5bd9\" (UID: \"bd191bcb-3c5e-49ff-accd-8a706eef5bd9\") "
Oct 31 20:05:11.082658 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:11.082596    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd191bcb-3c5e-49ff-accd-8a706eef5bd9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd191bcb-3c5e-49ff-accd-8a706eef5bd9-istiod-ca-cert\") pod \"bd191bcb-3c5e-49ff-accd-8a706eef5bd9\" (UID: \"bd191bcb-3c5e-49ff-accd-8a706eef5bd9\") "
Oct 31 20:05:11.082774 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:11.082665    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd191bcb-3c5e-49ff-accd-8a706eef5bd9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd191bcb-3c5e-49ff-accd-8a706eef5bd9-workload-socket\") pod \"bd191bcb-3c5e-49ff-accd-8a706eef5bd9\" (UID: \"bd191bcb-3c5e-49ff-accd-8a706eef5bd9\") "
Oct 31 20:05:11.082839 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:11.082787    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd191bcb-3c5e-49ff-accd-8a706eef5bd9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd191bcb-3c5e-49ff-accd-8a706eef5bd9-credential-socket\") pod \"bd191bcb-3c5e-49ff-accd-8a706eef5bd9\" (UID: \"bd191bcb-3c5e-49ff-accd-8a706eef5bd9\") "
Oct 31 20:05:11.083468 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:11.083440    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd191bcb-3c5e-49ff-accd-8a706eef5bd9-istio-data" pod "bd191bcb-3c5e-49ff-accd-8a706eef5bd9" (UID: "bd191bcb-3c5e-49ff-accd-8a706eef5bd9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:11.083468 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:11.083452    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd191bcb-3c5e-49ff-accd-8a706eef5bd9-workload-socket" pod "bd191bcb-3c5e-49ff-accd-8a706eef5bd9" (UID: "bd191bcb-3c5e-49ff-accd-8a706eef5bd9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:11.083650 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:11.083532    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd191bcb-3c5e-49ff-accd-8a706eef5bd9-istiod-ca-cert" pod "bd191bcb-3c5e-49ff-accd-8a706eef5bd9" (UID: "bd191bcb-3c5e-49ff-accd-8a706eef5bd9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:05:11.083650 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:11.083634    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd191bcb-3c5e-49ff-accd-8a706eef5bd9-credential-socket" pod "bd191bcb-3c5e-49ff-accd-8a706eef5bd9" (UID: "bd191bcb-3c5e-49ff-accd-8a706eef5bd9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:11.083770 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:11.083651    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd191bcb-3c5e-49ff-accd-8a706eef5bd9-workload-certs" pod "bd191bcb-3c5e-49ff-accd-8a706eef5bd9" (UID: "bd191bcb-3c5e-49ff-accd-8a706eef5bd9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:11.083919 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:11.083894    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd191bcb-3c5e-49ff-accd-8a706eef5bd9-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:11.083982 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:11.083926    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd191bcb-3c5e-49ff-accd-8a706eef5bd9-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:11.083982 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:11.083943    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd191bcb-3c5e-49ff-accd-8a706eef5bd9-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:11.083982 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:11.083958    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd191bcb-3c5e-49ff-accd-8a706eef5bd9-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:11.083982 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:11.083972    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd191bcb-3c5e-49ff-accd-8a706eef5bd9-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:11.085475 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:11.085429    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bd191bcb-3c5e-49ff-accd-8a706eef5bd9-istio-podinfo" pod "bd191bcb-3c5e-49ff-accd-8a706eef5bd9" (UID: "bd191bcb-3c5e-49ff-accd-8a706eef5bd9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 20:05:11.085654 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:11.085627    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd191bcb-3c5e-49ff-accd-8a706eef5bd9-istio-token" pod "bd191bcb-3c5e-49ff-accd-8a706eef5bd9" (UID: "bd191bcb-3c5e-49ff-accd-8a706eef5bd9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:05:11.086036 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:11.086007    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd191bcb-3c5e-49ff-accd-8a706eef5bd9-istio-envoy" pod "bd191bcb-3c5e-49ff-accd-8a706eef5bd9" (UID: "bd191bcb-3c5e-49ff-accd-8a706eef5bd9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:11.086196 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:11.086176    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd191bcb-3c5e-49ff-accd-8a706eef5bd9-kube-api-access-hgzn7" pod "bd191bcb-3c5e-49ff-accd-8a706eef5bd9" (UID: "bd191bcb-3c5e-49ff-accd-8a706eef5bd9"). InnerVolumeSpecName "kube-api-access-hgzn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:05:11.087871 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:11.087852    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f28d48-6fc75df778-4x22w"]
Oct 31 20:05:11.091246 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:05:11.091221    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dd3d751_d8d8_4fa7_b1bf_5c2da531f07d.slice/crio-fb972368091a8a5c896c348e417cf9aa6bcfe7f61ae687132d1c924818609ec5 WatchSource:0}: Error finding container fb972368091a8a5c896c348e417cf9aa6bcfe7f61ae687132d1c924818609ec5: Status 404 returned error can't find the container with id fb972368091a8a5c896c348e417cf9aa6bcfe7f61ae687132d1c924818609ec5
Oct 31 20:05:11.095261 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:05:11.094959    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7599e792_03f8_41bf_9cda_aca650a9022b.slice/crio-eed3ccf5717df7b26c9deeeed230fa5a7c4f2ed8ea2243b585443a9bb475cbd5 WatchSource:0}: Error finding container eed3ccf5717df7b26c9deeeed230fa5a7c4f2ed8ea2243b585443a9bb475cbd5: Status 404 returned error can't find the container with id eed3ccf5717df7b26c9deeeed230fa5a7c4f2ed8ea2243b585443a9bb475cbd5
Oct 31 20:05:11.095706 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:11.095671    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f28d48-5bbcbb6f5-8jbqq"]
Oct 31 20:05:11.185219 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:11.185190    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd191bcb-3c5e-49ff-accd-8a706eef5bd9-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:11.185573 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:11.185228    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgzn7\" (UniqueName: \"kubernetes.io/projected/bd191bcb-3c5e-49ff-accd-8a706eef5bd9-kube-api-access-hgzn7\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:11.185573 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:11.185244    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd191bcb-3c5e-49ff-accd-8a706eef5bd9-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:11.185573 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:11.185259    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd191bcb-3c5e-49ff-accd-8a706eef5bd9-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:11.445378 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:11.445342    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:11.618790 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:11.618746    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9deed825-0c5f-4950-96e4-098a01b928f4" path="/var/lib/kubelet/pods/9deed825-0c5f-4950-96e4-098a01b928f4/volumes"
Oct 31 20:05:11.991397 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:11.991351    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f28d48-5bbcbb6f5-8jbqq" event={"ID":"7599e792-03f8-41bf-9cda-aca650a9022b","Type":"ContainerStarted","Data":"76d7cc3b1c1bd8a8e68b5626855048a60998412d230e7c0b8a4a74ed307019f1"}
Oct 31 20:05:11.991616 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:11.991415    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f28d48-5bbcbb6f5-8jbqq" event={"ID":"7599e792-03f8-41bf-9cda-aca650a9022b","Type":"ContainerStarted","Data":"eed3ccf5717df7b26c9deeeed230fa5a7c4f2ed8ea2243b585443a9bb475cbd5"}
Oct 31 20:05:11.991616 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:11.991526    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f28d48-5bbcbb6f5-8jbqq"
Oct 31 20:05:11.993049 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:11.993024    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk"
Oct 31 20:05:11.993172 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:11.993062    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk" event={"ID":"bd191bcb-3c5e-49ff-accd-8a706eef5bd9","Type":"ContainerDied","Data":"98e41f1339d8b077eb29842725638af1c4cb3ddfc261af970d8df04e52afa18a"}
Oct 31 20:05:11.993172 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:11.993123    4645 scope.go:122] "RemoveContainer" containerID="b7800bf245a0dd9aeec96bdbcdbcf45b1b443b61f10a69633b95126ed0f10543"
Oct 31 20:05:11.997122 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:11.997094    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f28d48-6fc75df778-4x22w" event={"ID":"2dd3d751-d8d8-4fa7-b1bf-5c2da531f07d","Type":"ContainerStarted","Data":"44ff49ee1dabc8669a574a379e718960648e053ef65a77745bdb00971fbb3adf"}
Oct 31 20:05:11.997242 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:11.997140    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f28d48-6fc75df778-4x22w" event={"ID":"2dd3d751-d8d8-4fa7-b1bf-5c2da531f07d","Type":"ContainerStarted","Data":"fb972368091a8a5c896c348e417cf9aa6bcfe7f61ae687132d1c924818609ec5"}
Oct 31 20:05:11.997242 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:11.997169    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f28d48-6fc75df778-4x22w"
Oct 31 20:05:12.009494 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:12.009445    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f28d48-5bbcbb6f5-8jbqq" podStartSLOduration=2.009431395 podStartE2EDuration="2.009431395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:05:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:05:12.008491055 +0000 UTC m=+2742.581725183" watchObservedRunningTime="2026-10-31 20:05:12.009431395 +0000 UTC m=+2742.582665529"
Oct 31 20:05:12.025624 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:12.025600    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk"]
Oct 31 20:05:12.033721 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:12.033700    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-d9c048-openshift-default-5dbbb74c57-t8wfk"]
Oct 31 20:05:12.048225 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:12.048177    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-f28d48-6fc75df778-4x22w" podStartSLOduration=2.048165107 podStartE2EDuration="2.048165107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:05:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:05:12.046020988 +0000 UTC m=+2742.619255118" watchObservedRunningTime="2026-10-31 20:05:12.048165107 +0000 UTC m=+2742.621399275"
Oct 31 20:05:12.345054 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:12.344970    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:13.086220 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:13.086185    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:13.618521 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:13.618482    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd191bcb-3c5e-49ff-accd-8a706eef5bd9" path="/var/lib/kubelet/pods/bd191bcb-3c5e-49ff-accd-8a706eef5bd9/volumes"
Oct 31 20:05:13.850610 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:13.850580    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:14.642461 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:14.642425    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d9c048-5459f7f8dc-rhrkn" podUID="7d33bec5-7276-4b99-9a02-6ac2eaaf770f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:05:14.731060 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:14.731028    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:15.445212 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:15.445173    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:15.651228 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:15.651183    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d9c048-79ff7865f5-qq5qp" podUID="b4677f66-6fc4-4c4a-8539-ede1dfe8f503" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:05:15.780270 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:15.780195    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:15.902828 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:15.902797    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:16.152704 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:16.152605    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6"]
Oct 31 20:05:16.153012 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:16.152835    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6" podUID="45ebe81b-1559-4c1b-860e-619f8e9e608e" containerName="istio-proxy" containerID="cri-o://b775495befc3cafaca74868b49ea438b27da6e77be98147c8111bd69d47fcfc5" gracePeriod=30
Oct 31 20:05:16.153012 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:16.152907    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:16.265652 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:16.265622    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:16.448357 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:16.447952    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:16.906856 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:16.906781    4645 patch_prober.go:29] interesting pod/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 20:05:16.906856 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:16.906830    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6" podUID="45ebe81b-1559-4c1b-860e-619f8e9e608e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 20:05:17.765077 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:17.765044    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:18.242489 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:18.242457    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:18.382029 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:18.381983    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:18.629101 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:18.629003    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99"]
Oct 31 20:05:18.629323 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:18.629295    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99" podUID="726d2f42-5772-4a85-ae19-2fcf878567d5" containerName="istio-proxy" containerID="cri-o://b1c4b5407b6eac7792ec915125e7714b46ffeec197e61582ec1682b4120690b5" gracePeriod=30
Oct 31 20:05:18.672273 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:18.672241    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:18.805688 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:18.805652    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:19.139880 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:19.139844    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ae2e60-g9w27"]
Oct 31 20:05:19.140485 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:19.140469    4645 cpu_manager.go:414] "containerMap: removing container" podUID="9deed825-0c5f-4950-96e4-098a01b928f4" containerName="istio-proxy"
Oct 31 20:05:19.140540 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:19.140486    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9deed825-0c5f-4950-96e4-098a01b928f4" containerName="istio-proxy"
Oct 31 20:05:19.140540 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:19.140523    4645 cpu_manager.go:414] "containerMap: removing container" podUID="bd191bcb-3c5e-49ff-accd-8a706eef5bd9" containerName="istio-proxy"
Oct 31 20:05:19.140540 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:19.140528    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd191bcb-3c5e-49ff-accd-8a706eef5bd9" containerName="istio-proxy"
Oct 31 20:05:19.140645 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:19.140608    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="9deed825-0c5f-4950-96e4-098a01b928f4" containerName="istio-proxy"
Oct 31 20:05:19.140645 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:19.140624    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd191bcb-3c5e-49ff-accd-8a706eef5bd9" containerName="istio-proxy"
Oct 31 20:05:19.145391 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:19.145372    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ae2e60-g9w27"
Oct 31 20:05:19.149648 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:19.149615    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:05:19.149968 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:19.149952    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 20:05:19.158226 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:19.158203    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ae2e60-g9w27"]
Oct 31 20:05:19.255277 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:19.255246    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de7f86dd-e3a8-4cf2-9b14-83f2a2e53dc3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-ae2e60-g9w27\" (UID: \"de7f86dd-e3a8-4cf2-9b14-83f2a2e53dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ae2e60-g9w27"
Oct 31 20:05:19.255702 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:19.255439    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9jnf\" (UniqueName: \"kubernetes.io/projected/de7f86dd-e3a8-4cf2-9b14-83f2a2e53dc3-kube-api-access-g9jnf\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-ae2e60-g9w27\" (UID: \"de7f86dd-e3a8-4cf2-9b14-83f2a2e53dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ae2e60-g9w27"
Oct 31 20:05:19.255702 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:19.255469    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de7f86dd-e3a8-4cf2-9b14-83f2a2e53dc3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-ae2e60-g9w27\" (UID: \"de7f86dd-e3a8-4cf2-9b14-83f2a2e53dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ae2e60-g9w27"
Oct 31 20:05:19.339511 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:19.339471    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:19.357559 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:19.357518    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de7f86dd-e3a8-4cf2-9b14-83f2a2e53dc3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-ae2e60-g9w27\" (UID: \"de7f86dd-e3a8-4cf2-9b14-83f2a2e53dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ae2e60-g9w27"
Oct 31 20:05:19.357717 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:19.357684    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9jnf\" (UniqueName: \"kubernetes.io/projected/de7f86dd-e3a8-4cf2-9b14-83f2a2e53dc3-kube-api-access-g9jnf\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-ae2e60-g9w27\" (UID: \"de7f86dd-e3a8-4cf2-9b14-83f2a2e53dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ae2e60-g9w27"
Oct 31 20:05:19.357770 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:19.357716    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de7f86dd-e3a8-4cf2-9b14-83f2a2e53dc3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-ae2e60-g9w27\" (UID: \"de7f86dd-e3a8-4cf2-9b14-83f2a2e53dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ae2e60-g9w27"
Oct 31 20:05:19.358630 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:19.358605    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de7f86dd-e3a8-4cf2-9b14-83f2a2e53dc3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-ae2e60-g9w27\" (UID: \"de7f86dd-e3a8-4cf2-9b14-83f2a2e53dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ae2e60-g9w27"
Oct 31 20:05:19.361068 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:19.361029    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de7f86dd-e3a8-4cf2-9b14-83f2a2e53dc3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-ae2e60-g9w27\" (UID: \"de7f86dd-e3a8-4cf2-9b14-83f2a2e53dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ae2e60-g9w27"
Oct 31 20:05:19.370401 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:19.370373    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9jnf\" (UniqueName: \"kubernetes.io/projected/de7f86dd-e3a8-4cf2-9b14-83f2a2e53dc3-kube-api-access-g9jnf\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-ae2e60-g9w27\" (UID: \"de7f86dd-e3a8-4cf2-9b14-83f2a2e53dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ae2e60-g9w27"
Oct 31 20:05:19.461663 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:19.461635    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ae2e60-g9w27"
Oct 31 20:05:19.478051 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:19.478020    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d9c048-79ff7865f5-qq5qp"
Oct 31 20:05:19.506370 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:19.506341    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d9c048-5459f7f8dc-rhrkn"
Oct 31 20:05:19.559969 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:19.559930    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4677f66-6fc4-4c4a-8539-ede1dfe8f503-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4677f66-6fc4-4c4a-8539-ede1dfe8f503-plugins-config-volume\") pod \"b4677f66-6fc4-4c4a-8539-ede1dfe8f503\" (UID: \"b4677f66-6fc4-4c4a-8539-ede1dfe8f503\") "
Oct 31 20:05:19.560137 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:19.560115    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4677f66-6fc4-4c4a-8539-ede1dfe8f503-kube-api-access-2wdwr\" (UniqueName: \"kubernetes.io/projected/b4677f66-6fc4-4c4a-8539-ede1dfe8f503-kube-api-access-2wdwr\") pod \"b4677f66-6fc4-4c4a-8539-ede1dfe8f503\" (UID: \"b4677f66-6fc4-4c4a-8539-ede1dfe8f503\") "
Oct 31 20:05:19.561612 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:19.561582    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4677f66-6fc4-4c4a-8539-ede1dfe8f503-plugins-config-volume" pod "b4677f66-6fc4-4c4a-8539-ede1dfe8f503" (UID: "b4677f66-6fc4-4c4a-8539-ede1dfe8f503"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:05:19.563554 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:19.563530    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4677f66-6fc4-4c4a-8539-ede1dfe8f503-kube-api-access-2wdwr" pod "b4677f66-6fc4-4c4a-8539-ede1dfe8f503" (UID: "b4677f66-6fc4-4c4a-8539-ede1dfe8f503"). InnerVolumeSpecName "kube-api-access-2wdwr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:05:19.644027 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:19.643990    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:19.661993 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:19.661960    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d33bec5-7276-4b99-9a02-6ac2eaaf770f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d33bec5-7276-4b99-9a02-6ac2eaaf770f-plugins-config-volume\") pod \"7d33bec5-7276-4b99-9a02-6ac2eaaf770f\" (UID: \"7d33bec5-7276-4b99-9a02-6ac2eaaf770f\") "
Oct 31 20:05:19.662251 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:19.662208    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d33bec5-7276-4b99-9a02-6ac2eaaf770f-kube-api-access-b4pxv\" (UniqueName: \"kubernetes.io/projected/7d33bec5-7276-4b99-9a02-6ac2eaaf770f-kube-api-access-b4pxv\") pod \"7d33bec5-7276-4b99-9a02-6ac2eaaf770f\" (UID: \"7d33bec5-7276-4b99-9a02-6ac2eaaf770f\") "
Oct 31 20:05:19.662424 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:19.662404    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d33bec5-7276-4b99-9a02-6ac2eaaf770f-plugins-config-volume" pod "7d33bec5-7276-4b99-9a02-6ac2eaaf770f" (UID: "7d33bec5-7276-4b99-9a02-6ac2eaaf770f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:05:19.662705 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:19.662688    4645 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d33bec5-7276-4b99-9a02-6ac2eaaf770f-plugins-config-volume\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:19.662748 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:19.662709    4645 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4677f66-6fc4-4c4a-8539-ede1dfe8f503-plugins-config-volume\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:19.662748 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:19.662719    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wdwr\" (UniqueName: \"kubernetes.io/projected/b4677f66-6fc4-4c4a-8539-ede1dfe8f503-kube-api-access-2wdwr\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:19.664235 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:19.664215    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d33bec5-7276-4b99-9a02-6ac2eaaf770f-kube-api-access-b4pxv" pod "7d33bec5-7276-4b99-9a02-6ac2eaaf770f" (UID: "7d33bec5-7276-4b99-9a02-6ac2eaaf770f"). InnerVolumeSpecName "kube-api-access-b4pxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:05:19.764551 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:19.764268    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4pxv\" (UniqueName: \"kubernetes.io/projected/7d33bec5-7276-4b99-9a02-6ac2eaaf770f-kube-api-access-b4pxv\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:19.766481 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:19.766450    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ae2e60-g9w27"]
Oct 31 20:05:19.767682 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:05:19.767658    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde7f86dd_e3a8_4cf2_9b14_83f2a2e53dc3.slice/crio-5729ddcb38ac910e421b421e56d00661b5d33e8347b36b0004905f905aa0dedc WatchSource:0}: Error finding container 5729ddcb38ac910e421b421e56d00661b5d33e8347b36b0004905f905aa0dedc: Status 404 returned error can't find the container with id 5729ddcb38ac910e421b421e56d00661b5d33e8347b36b0004905f905aa0dedc
Oct 31 20:05:19.832452 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:19.832416    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4bac5a-78664677cc-6shsd" podUID="aa76773f-998f-408d-a804-31f6c4921774" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:05:20.059451 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:20.059288    4645 generic.go:356] "Generic (PLEG): container finished" podID="7d33bec5-7276-4b99-9a02-6ac2eaaf770f" containerID="93e922dc887ee5e527ed9b2837c331ca8a034671b2ddbd9fa2eb01e78364c7d3" exitCode=0
Oct 31 20:05:20.059660 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:20.059530    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d9c048-5459f7f8dc-rhrkn" event={"ID":"7d33bec5-7276-4b99-9a02-6ac2eaaf770f","Type":"ContainerDied","Data":"93e922dc887ee5e527ed9b2837c331ca8a034671b2ddbd9fa2eb01e78364c7d3"}
Oct 31 20:05:20.059660 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:20.059591    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d9c048-5459f7f8dc-rhrkn" event={"ID":"7d33bec5-7276-4b99-9a02-6ac2eaaf770f","Type":"ContainerDied","Data":"60ada9036ba294374cfc05df9265a7308622326c8a8031498a5b9eb91f932abe"}
Oct 31 20:05:20.059801 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:20.059717    4645 scope.go:122] "RemoveContainer" containerID="93e922dc887ee5e527ed9b2837c331ca8a034671b2ddbd9fa2eb01e78364c7d3"
Oct 31 20:05:20.059801 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:20.059723    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d9c048-5459f7f8dc-rhrkn"
Oct 31 20:05:20.061474 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:20.061155    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ae2e60-g9w27" event={"ID":"de7f86dd-e3a8-4cf2-9b14-83f2a2e53dc3","Type":"ContainerStarted","Data":"5729ddcb38ac910e421b421e56d00661b5d33e8347b36b0004905f905aa0dedc"}
Oct 31 20:05:20.062894 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:20.062865    4645 generic.go:356] "Generic (PLEG): container finished" podID="b4677f66-6fc4-4c4a-8539-ede1dfe8f503" containerID="4c0a2d2968e63a1cbf5e9c590794fbb50ff640b4998c60b08ceb9dc5aed8bff6" exitCode=0
Oct 31 20:05:20.063003 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:20.062908    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d9c048-79ff7865f5-qq5qp" event={"ID":"b4677f66-6fc4-4c4a-8539-ede1dfe8f503","Type":"ContainerDied","Data":"4c0a2d2968e63a1cbf5e9c590794fbb50ff640b4998c60b08ceb9dc5aed8bff6"}
Oct 31 20:05:20.063003 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:20.062944    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d9c048-79ff7865f5-qq5qp" event={"ID":"b4677f66-6fc4-4c4a-8539-ede1dfe8f503","Type":"ContainerDied","Data":"36c1a19e36d791dfc728bf70fcde9e4dbef4c44e056c56c402d16988d1e35d25"}
Oct 31 20:05:20.063003 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:20.062982    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-d9c048-79ff7865f5-qq5qp"
Oct 31 20:05:20.073798 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:20.073777    4645 scope.go:122] "RemoveContainer" containerID="93e922dc887ee5e527ed9b2837c331ca8a034671b2ddbd9fa2eb01e78364c7d3"
Oct 31 20:05:20.074059 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:05:20.074038    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93e922dc887ee5e527ed9b2837c331ca8a034671b2ddbd9fa2eb01e78364c7d3\": container with ID starting with 93e922dc887ee5e527ed9b2837c331ca8a034671b2ddbd9fa2eb01e78364c7d3 not found: ID does not exist" containerID="93e922dc887ee5e527ed9b2837c331ca8a034671b2ddbd9fa2eb01e78364c7d3"
Oct 31 20:05:20.074136 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:20.074065    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93e922dc887ee5e527ed9b2837c331ca8a034671b2ddbd9fa2eb01e78364c7d3"} err="failed to get container status \"93e922dc887ee5e527ed9b2837c331ca8a034671b2ddbd9fa2eb01e78364c7d3\": rpc error: code = NotFound desc = could not find container \"93e922dc887ee5e527ed9b2837c331ca8a034671b2ddbd9fa2eb01e78364c7d3\": container with ID starting with 93e922dc887ee5e527ed9b2837c331ca8a034671b2ddbd9fa2eb01e78364c7d3 not found: ID does not exist"
Oct 31 20:05:20.074136 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:20.074082    4645 scope.go:122] "RemoveContainer" containerID="4c0a2d2968e63a1cbf5e9c590794fbb50ff640b4998c60b08ceb9dc5aed8bff6"
Oct 31 20:05:20.083121 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:20.083099    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-d9c048-79ff7865f5-qq5qp"]
Oct 31 20:05:20.087075 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:20.087052    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-d9c048-79ff7865f5-qq5qp"]
Oct 31 20:05:20.088172 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:20.088144    4645 scope.go:122] "RemoveContainer" containerID="4c0a2d2968e63a1cbf5e9c590794fbb50ff640b4998c60b08ceb9dc5aed8bff6"
Oct 31 20:05:20.088476 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:05:20.088456    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c0a2d2968e63a1cbf5e9c590794fbb50ff640b4998c60b08ceb9dc5aed8bff6\": container with ID starting with 4c0a2d2968e63a1cbf5e9c590794fbb50ff640b4998c60b08ceb9dc5aed8bff6 not found: ID does not exist" containerID="4c0a2d2968e63a1cbf5e9c590794fbb50ff640b4998c60b08ceb9dc5aed8bff6"
Oct 31 20:05:20.088534 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:20.088485    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c0a2d2968e63a1cbf5e9c590794fbb50ff640b4998c60b08ceb9dc5aed8bff6"} err="failed to get container status \"4c0a2d2968e63a1cbf5e9c590794fbb50ff640b4998c60b08ceb9dc5aed8bff6\": rpc error: code = NotFound desc = could not find container \"4c0a2d2968e63a1cbf5e9c590794fbb50ff640b4998c60b08ceb9dc5aed8bff6\": container with ID starting with 4c0a2d2968e63a1cbf5e9c590794fbb50ff640b4998c60b08ceb9dc5aed8bff6 not found: ID does not exist"
Oct 31 20:05:20.099476 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:20.099446    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d9c048-5459f7f8dc-rhrkn"]
Oct 31 20:05:20.103515 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:20.103494    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-d9c048-5459f7f8dc-rhrkn"]
Oct 31 20:05:20.378682 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:20.378605    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:20.560869 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:20.560820    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:21.071560 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:21.071461    4645 generic.go:356] "Generic (PLEG): container finished" podID="de7f86dd-e3a8-4cf2-9b14-83f2a2e53dc3" containerID="f31a16feab5f034af33f270b2ad2eff05abb3b6b0f076d8d629ec3a8a84e63c9" exitCode=0
Oct 31 20:05:21.071717 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:21.071549    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ae2e60-g9w27" event={"ID":"de7f86dd-e3a8-4cf2-9b14-83f2a2e53dc3","Type":"ContainerDied","Data":"f31a16feab5f034af33f270b2ad2eff05abb3b6b0f076d8d629ec3a8a84e63c9"}
Oct 31 20:05:21.443113 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:21.443081    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6"
Oct 31 20:05:21.443792 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:21.443764    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:21.596691 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:21.596606    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45ebe81b-1559-4c1b-860e-619f8e9e608e-istio-token\" (UniqueName: \"kubernetes.io/projected/45ebe81b-1559-4c1b-860e-619f8e9e608e-istio-token\") pod \"45ebe81b-1559-4c1b-860e-619f8e9e608e\" (UID: \"45ebe81b-1559-4c1b-860e-619f8e9e608e\") "
Oct 31 20:05:21.596871 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:21.596717    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45ebe81b-1559-4c1b-860e-619f8e9e608e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45ebe81b-1559-4c1b-860e-619f8e9e608e-workload-socket\") pod \"45ebe81b-1559-4c1b-860e-619f8e9e608e\" (UID: \"45ebe81b-1559-4c1b-860e-619f8e9e608e\") "
Oct 31 20:05:21.596871 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:21.596764    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45ebe81b-1559-4c1b-860e-619f8e9e608e-kube-api-access-fw255\" (UniqueName: \"kubernetes.io/projected/45ebe81b-1559-4c1b-860e-619f8e9e608e-kube-api-access-fw255\") pod \"45ebe81b-1559-4c1b-860e-619f8e9e608e\" (UID: \"45ebe81b-1559-4c1b-860e-619f8e9e608e\") "
Oct 31 20:05:21.596871 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:21.596796    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45ebe81b-1559-4c1b-860e-619f8e9e608e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45ebe81b-1559-4c1b-860e-619f8e9e608e-istio-data\") pod \"45ebe81b-1559-4c1b-860e-619f8e9e608e\" (UID: \"45ebe81b-1559-4c1b-860e-619f8e9e608e\") "
Oct 31 20:05:21.596871 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:21.596854    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45ebe81b-1559-4c1b-860e-619f8e9e608e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45ebe81b-1559-4c1b-860e-619f8e9e608e-istiod-ca-cert\") pod \"45ebe81b-1559-4c1b-860e-619f8e9e608e\" (UID: \"45ebe81b-1559-4c1b-860e-619f8e9e608e\") "
Oct 31 20:05:21.597101 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:21.596973    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/45ebe81b-1559-4c1b-860e-619f8e9e608e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45ebe81b-1559-4c1b-860e-619f8e9e608e-istio-podinfo\") pod \"45ebe81b-1559-4c1b-860e-619f8e9e608e\" (UID: \"45ebe81b-1559-4c1b-860e-619f8e9e608e\") "
Oct 31 20:05:21.597101 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:21.597021    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45ebe81b-1559-4c1b-860e-619f8e9e608e-workload-socket" pod "45ebe81b-1559-4c1b-860e-619f8e9e608e" (UID: "45ebe81b-1559-4c1b-860e-619f8e9e608e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:21.597101 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:21.597053    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45ebe81b-1559-4c1b-860e-619f8e9e608e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45ebe81b-1559-4c1b-860e-619f8e9e608e-credential-socket\") pod \"45ebe81b-1559-4c1b-860e-619f8e9e608e\" (UID: \"45ebe81b-1559-4c1b-860e-619f8e9e608e\") "
Oct 31 20:05:21.597401 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:21.597109    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45ebe81b-1559-4c1b-860e-619f8e9e608e-istio-data" pod "45ebe81b-1559-4c1b-860e-619f8e9e608e" (UID: "45ebe81b-1559-4c1b-860e-619f8e9e608e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:21.597401 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:21.597133    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45ebe81b-1559-4c1b-860e-619f8e9e608e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45ebe81b-1559-4c1b-860e-619f8e9e608e-istio-envoy\") pod \"45ebe81b-1559-4c1b-860e-619f8e9e608e\" (UID: \"45ebe81b-1559-4c1b-860e-619f8e9e608e\") "
Oct 31 20:05:21.597401 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:21.597171    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45ebe81b-1559-4c1b-860e-619f8e9e608e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45ebe81b-1559-4c1b-860e-619f8e9e608e-workload-certs\") pod \"45ebe81b-1559-4c1b-860e-619f8e9e608e\" (UID: \"45ebe81b-1559-4c1b-860e-619f8e9e608e\") "
Oct 31 20:05:21.597401 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:21.597269    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45ebe81b-1559-4c1b-860e-619f8e9e608e-istiod-ca-cert" pod "45ebe81b-1559-4c1b-860e-619f8e9e608e" (UID: "45ebe81b-1559-4c1b-860e-619f8e9e608e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:05:21.597709 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:21.597653    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45ebe81b-1559-4c1b-860e-619f8e9e608e-credential-socket" pod "45ebe81b-1559-4c1b-860e-619f8e9e608e" (UID: "45ebe81b-1559-4c1b-860e-619f8e9e608e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:21.597887 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:21.597861    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45ebe81b-1559-4c1b-860e-619f8e9e608e-workload-certs" pod "45ebe81b-1559-4c1b-860e-619f8e9e608e" (UID: "45ebe81b-1559-4c1b-860e-619f8e9e608e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:21.598044 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:21.597951    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45ebe81b-1559-4c1b-860e-619f8e9e608e-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:21.598303 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:21.598056    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45ebe81b-1559-4c1b-860e-619f8e9e608e-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:21.598303 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:21.598074    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45ebe81b-1559-4c1b-860e-619f8e9e608e-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:21.598303 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:21.598090    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45ebe81b-1559-4c1b-860e-619f8e9e608e-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:21.599443 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:21.599410    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45ebe81b-1559-4c1b-860e-619f8e9e608e-kube-api-access-fw255" pod "45ebe81b-1559-4c1b-860e-619f8e9e608e" (UID: "45ebe81b-1559-4c1b-860e-619f8e9e608e"). InnerVolumeSpecName "kube-api-access-fw255". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:05:21.599560 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:21.599513    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45ebe81b-1559-4c1b-860e-619f8e9e608e-istio-token" pod "45ebe81b-1559-4c1b-860e-619f8e9e608e" (UID: "45ebe81b-1559-4c1b-860e-619f8e9e608e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:05:21.599822 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:21.599796    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/45ebe81b-1559-4c1b-860e-619f8e9e608e-istio-podinfo" pod "45ebe81b-1559-4c1b-860e-619f8e9e608e" (UID: "45ebe81b-1559-4c1b-860e-619f8e9e608e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 20:05:21.599935 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:21.599864    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45ebe81b-1559-4c1b-860e-619f8e9e608e-istio-envoy" pod "45ebe81b-1559-4c1b-860e-619f8e9e608e" (UID: "45ebe81b-1559-4c1b-860e-619f8e9e608e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:21.617938 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:21.617911    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d33bec5-7276-4b99-9a02-6ac2eaaf770f" path="/var/lib/kubelet/pods/7d33bec5-7276-4b99-9a02-6ac2eaaf770f/volumes"
Oct 31 20:05:21.618367 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:21.618353    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4677f66-6fc4-4c4a-8539-ede1dfe8f503" path="/var/lib/kubelet/pods/b4677f66-6fc4-4c4a-8539-ede1dfe8f503/volumes"
Oct 31 20:05:21.699624 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:21.699589    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fw255\" (UniqueName: \"kubernetes.io/projected/45ebe81b-1559-4c1b-860e-619f8e9e608e-kube-api-access-fw255\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:21.699624 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:21.699622    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45ebe81b-1559-4c1b-860e-619f8e9e608e-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:21.699831 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:21.699637    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45ebe81b-1559-4c1b-860e-619f8e9e608e-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:21.699831 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:21.699649    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45ebe81b-1559-4c1b-860e-619f8e9e608e-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:21.699831 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:21.699661    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45ebe81b-1559-4c1b-860e-619f8e9e608e-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:22.082189 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:22.082155    4645 generic.go:356] "Generic (PLEG): container finished" podID="45ebe81b-1559-4c1b-860e-619f8e9e608e" containerID="b775495befc3cafaca74868b49ea438b27da6e77be98147c8111bd69d47fcfc5" exitCode=0
Oct 31 20:05:22.082386 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:22.082209    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6"
Oct 31 20:05:22.082386 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:22.082235    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6" event={"ID":"45ebe81b-1559-4c1b-860e-619f8e9e608e","Type":"ContainerDied","Data":"b775495befc3cafaca74868b49ea438b27da6e77be98147c8111bd69d47fcfc5"}
Oct 31 20:05:22.082386 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:22.082296    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6" event={"ID":"45ebe81b-1559-4c1b-860e-619f8e9e608e","Type":"ContainerDied","Data":"7a12fb767b5c65ff7f512838e1af96b776d8f8cc1f205b6c9b4f080e7fc733d0"}
Oct 31 20:05:22.082386 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:22.082320    4645 scope.go:122] "RemoveContainer" containerID="b775495befc3cafaca74868b49ea438b27da6e77be98147c8111bd69d47fcfc5"
Oct 31 20:05:22.093022 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:22.092998    4645 scope.go:122] "RemoveContainer" containerID="b775495befc3cafaca74868b49ea438b27da6e77be98147c8111bd69d47fcfc5"
Oct 31 20:05:22.093304 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:05:22.093285    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b775495befc3cafaca74868b49ea438b27da6e77be98147c8111bd69d47fcfc5\": container with ID starting with b775495befc3cafaca74868b49ea438b27da6e77be98147c8111bd69d47fcfc5 not found: ID does not exist" containerID="b775495befc3cafaca74868b49ea438b27da6e77be98147c8111bd69d47fcfc5"
Oct 31 20:05:22.093362 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:22.093314    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b775495befc3cafaca74868b49ea438b27da6e77be98147c8111bd69d47fcfc5"} err="failed to get container status \"b775495befc3cafaca74868b49ea438b27da6e77be98147c8111bd69d47fcfc5\": rpc error: code = NotFound desc = could not find container \"b775495befc3cafaca74868b49ea438b27da6e77be98147c8111bd69d47fcfc5\": container with ID starting with b775495befc3cafaca74868b49ea438b27da6e77be98147c8111bd69d47fcfc5 not found: ID does not exist"
Oct 31 20:05:22.108090 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:22.108057    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6"]
Oct 31 20:05:22.115992 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:22.115960    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-a891ce-openshift-default-5645c95f65-cqnc6"]
Oct 31 20:05:22.243568 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:22.243543    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ae2e60-g9w27"
Oct 31 20:05:22.275678 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:22.275619    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:22.432498 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:22.432461    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de7f86dd-e3a8-4cf2-9b14-83f2a2e53dc3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de7f86dd-e3a8-4cf2-9b14-83f2a2e53dc3-maas-api-openshift-ca\") pod \"de7f86dd-e3a8-4cf2-9b14-83f2a2e53dc3\" (UID: \"de7f86dd-e3a8-4cf2-9b14-83f2a2e53dc3\") "
Oct 31 20:05:22.432712 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:22.432555    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de7f86dd-e3a8-4cf2-9b14-83f2a2e53dc3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de7f86dd-e3a8-4cf2-9b14-83f2a2e53dc3-maas-api-xks-ca\") pod \"de7f86dd-e3a8-4cf2-9b14-83f2a2e53dc3\" (UID: \"de7f86dd-e3a8-4cf2-9b14-83f2a2e53dc3\") "
Oct 31 20:05:22.432712 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:22.432693    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de7f86dd-e3a8-4cf2-9b14-83f2a2e53dc3-kube-api-access-g9jnf\" (UniqueName: \"kubernetes.io/projected/de7f86dd-e3a8-4cf2-9b14-83f2a2e53dc3-kube-api-access-g9jnf\") pod \"de7f86dd-e3a8-4cf2-9b14-83f2a2e53dc3\" (UID: \"de7f86dd-e3a8-4cf2-9b14-83f2a2e53dc3\") "
Oct 31 20:05:22.432926 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:22.432902    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de7f86dd-e3a8-4cf2-9b14-83f2a2e53dc3-maas-api-openshift-ca" pod "de7f86dd-e3a8-4cf2-9b14-83f2a2e53dc3" (UID: "de7f86dd-e3a8-4cf2-9b14-83f2a2e53dc3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:05:22.433102 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:22.433084    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de7f86dd-e3a8-4cf2-9b14-83f2a2e53dc3-maas-api-openshift-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:22.434656 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:22.434633    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de7f86dd-e3a8-4cf2-9b14-83f2a2e53dc3-maas-api-xks-ca" pod "de7f86dd-e3a8-4cf2-9b14-83f2a2e53dc3" (UID: "de7f86dd-e3a8-4cf2-9b14-83f2a2e53dc3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:05:22.434755 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:22.434738    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de7f86dd-e3a8-4cf2-9b14-83f2a2e53dc3-kube-api-access-g9jnf" pod "de7f86dd-e3a8-4cf2-9b14-83f2a2e53dc3" (UID: "de7f86dd-e3a8-4cf2-9b14-83f2a2e53dc3"). InnerVolumeSpecName "kube-api-access-g9jnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:05:22.534842 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:22.534809    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de7f86dd-e3a8-4cf2-9b14-83f2a2e53dc3-maas-api-xks-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:22.534842 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:22.534839    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9jnf\" (UniqueName: \"kubernetes.io/projected/de7f86dd-e3a8-4cf2-9b14-83f2a2e53dc3-kube-api-access-g9jnf\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:23.008339 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:23.008284    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f28d48-6fc75df778-4x22w"
Oct 31 20:05:23.008577 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:23.008561    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f28d48-5bbcbb6f5-8jbqq"
Oct 31 20:05:23.095420 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:23.094266    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ae2e60-g9w27"
Oct 31 20:05:23.095420 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:23.094294    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ae2e60-g9w27" event={"ID":"de7f86dd-e3a8-4cf2-9b14-83f2a2e53dc3","Type":"ContainerDied","Data":"5729ddcb38ac910e421b421e56d00661b5d33e8347b36b0004905f905aa0dedc"}
Oct 31 20:05:23.095420 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:23.094391    4645 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5729ddcb38ac910e421b421e56d00661b5d33e8347b36b0004905f905aa0dedc"
Oct 31 20:05:23.265777 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:23.265740    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:23.689764 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:23.689736    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45ebe81b-1559-4c1b-860e-619f8e9e608e" path="/var/lib/kubelet/pods/45ebe81b-1559-4c1b-860e-619f8e9e608e/volumes"
Oct 31 20:05:24.154722 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:24.154682    4645 generic.go:356] "Generic (PLEG): container finished" podID="aa76773f-998f-408d-a804-31f6c4921774" containerID="e88fd7cd9d0e1fc554eb036fb0aa37effec4b66817bff1b01a519b8a943229d2" exitCode=0
Oct 31 20:05:24.154978 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:24.154959    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4bac5a-78664677cc-6shsd" event={"ID":"aa76773f-998f-408d-a804-31f6c4921774","Type":"ContainerDied","Data":"e88fd7cd9d0e1fc554eb036fb0aa37effec4b66817bff1b01a519b8a943229d2"}
Oct 31 20:05:24.170807 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:24.170765    4645 generic.go:356] "Generic (PLEG): container finished" podID="726d2f42-5772-4a85-ae19-2fcf878567d5" containerID="b1c4b5407b6eac7792ec915125e7714b46ffeec197e61582ec1682b4120690b5" exitCode=0
Oct 31 20:05:24.171091 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:24.171026    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99" event={"ID":"726d2f42-5772-4a85-ae19-2fcf878567d5","Type":"ContainerDied","Data":"b1c4b5407b6eac7792ec915125e7714b46ffeec197e61582ec1682b4120690b5"}
Oct 31 20:05:24.188445 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:24.187101    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4bac5a-78664677cc-6shsd"
Oct 31 20:05:24.234466 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:24.234437    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99"
Oct 31 20:05:24.317752 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:24.317718    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/726d2f42-5772-4a85-ae19-2fcf878567d5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/726d2f42-5772-4a85-ae19-2fcf878567d5-istio-envoy\") pod \"726d2f42-5772-4a85-ae19-2fcf878567d5\" (UID: \"726d2f42-5772-4a85-ae19-2fcf878567d5\") "
Oct 31 20:05:24.318050 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:24.318034    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/726d2f42-5772-4a85-ae19-2fcf878567d5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/726d2f42-5772-4a85-ae19-2fcf878567d5-istio-podinfo\") pod \"726d2f42-5772-4a85-ae19-2fcf878567d5\" (UID: \"726d2f42-5772-4a85-ae19-2fcf878567d5\") "
Oct 31 20:05:24.318218 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:24.318204    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/726d2f42-5772-4a85-ae19-2fcf878567d5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/726d2f42-5772-4a85-ae19-2fcf878567d5-istio-data\") pod \"726d2f42-5772-4a85-ae19-2fcf878567d5\" (UID: \"726d2f42-5772-4a85-ae19-2fcf878567d5\") "
Oct 31 20:05:24.318449 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:24.318430    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/726d2f42-5772-4a85-ae19-2fcf878567d5-kube-api-access-bpq9n\" (UniqueName: \"kubernetes.io/projected/726d2f42-5772-4a85-ae19-2fcf878567d5-kube-api-access-bpq9n\") pod \"726d2f42-5772-4a85-ae19-2fcf878567d5\" (UID: \"726d2f42-5772-4a85-ae19-2fcf878567d5\") "
Oct 31 20:05:24.318577 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:24.318562    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/726d2f42-5772-4a85-ae19-2fcf878567d5-istio-token\" (UniqueName: \"kubernetes.io/projected/726d2f42-5772-4a85-ae19-2fcf878567d5-istio-token\") pod \"726d2f42-5772-4a85-ae19-2fcf878567d5\" (UID: \"726d2f42-5772-4a85-ae19-2fcf878567d5\") "
Oct 31 20:05:24.318688 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:24.318675    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/726d2f42-5772-4a85-ae19-2fcf878567d5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/726d2f42-5772-4a85-ae19-2fcf878567d5-credential-socket\") pod \"726d2f42-5772-4a85-ae19-2fcf878567d5\" (UID: \"726d2f42-5772-4a85-ae19-2fcf878567d5\") "
Oct 31 20:05:24.318824 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:24.318811    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa76773f-998f-408d-a804-31f6c4921774-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa76773f-998f-408d-a804-31f6c4921774-plugins-config-volume\") pod \"aa76773f-998f-408d-a804-31f6c4921774\" (UID: \"aa76773f-998f-408d-a804-31f6c4921774\") "
Oct 31 20:05:24.318918 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:24.318905    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/726d2f42-5772-4a85-ae19-2fcf878567d5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/726d2f42-5772-4a85-ae19-2fcf878567d5-workload-certs\") pod \"726d2f42-5772-4a85-ae19-2fcf878567d5\" (UID: \"726d2f42-5772-4a85-ae19-2fcf878567d5\") "
Oct 31 20:05:24.319021 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:24.319008    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/726d2f42-5772-4a85-ae19-2fcf878567d5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/726d2f42-5772-4a85-ae19-2fcf878567d5-istiod-ca-cert\") pod \"726d2f42-5772-4a85-ae19-2fcf878567d5\" (UID: \"726d2f42-5772-4a85-ae19-2fcf878567d5\") "
Oct 31 20:05:24.319167 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:24.319155    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa76773f-998f-408d-a804-31f6c4921774-kube-api-access-89xtb\" (UniqueName: \"kubernetes.io/projected/aa76773f-998f-408d-a804-31f6c4921774-kube-api-access-89xtb\") pod \"aa76773f-998f-408d-a804-31f6c4921774\" (UID: \"aa76773f-998f-408d-a804-31f6c4921774\") "
Oct 31 20:05:24.319274 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:24.319261    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/726d2f42-5772-4a85-ae19-2fcf878567d5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/726d2f42-5772-4a85-ae19-2fcf878567d5-workload-socket\") pod \"726d2f42-5772-4a85-ae19-2fcf878567d5\" (UID: \"726d2f42-5772-4a85-ae19-2fcf878567d5\") "
Oct 31 20:05:24.320177 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:24.320145    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/726d2f42-5772-4a85-ae19-2fcf878567d5-workload-socket" pod "726d2f42-5772-4a85-ae19-2fcf878567d5" (UID: "726d2f42-5772-4a85-ae19-2fcf878567d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:24.320827 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:24.320804    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/726d2f42-5772-4a85-ae19-2fcf878567d5-credential-socket" pod "726d2f42-5772-4a85-ae19-2fcf878567d5" (UID: "726d2f42-5772-4a85-ae19-2fcf878567d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:24.322801 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:24.322092    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/726d2f42-5772-4a85-ae19-2fcf878567d5-istiod-ca-cert" pod "726d2f42-5772-4a85-ae19-2fcf878567d5" (UID: "726d2f42-5772-4a85-ae19-2fcf878567d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:05:24.322801 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:24.322524    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/726d2f42-5772-4a85-ae19-2fcf878567d5-workload-certs" pod "726d2f42-5772-4a85-ae19-2fcf878567d5" (UID: "726d2f42-5772-4a85-ae19-2fcf878567d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:24.322801 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:24.322716    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa76773f-998f-408d-a804-31f6c4921774-plugins-config-volume" pod "aa76773f-998f-408d-a804-31f6c4921774" (UID: "aa76773f-998f-408d-a804-31f6c4921774"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:05:24.322801 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:24.322774    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/726d2f42-5772-4a85-ae19-2fcf878567d5-istio-data" pod "726d2f42-5772-4a85-ae19-2fcf878567d5" (UID: "726d2f42-5772-4a85-ae19-2fcf878567d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:24.324318 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:24.324291    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/726d2f42-5772-4a85-ae19-2fcf878567d5-kube-api-access-bpq9n" pod "726d2f42-5772-4a85-ae19-2fcf878567d5" (UID: "726d2f42-5772-4a85-ae19-2fcf878567d5"). InnerVolumeSpecName "kube-api-access-bpq9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:05:24.331418 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:24.326730    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/726d2f42-5772-4a85-ae19-2fcf878567d5-istio-token" pod "726d2f42-5772-4a85-ae19-2fcf878567d5" (UID: "726d2f42-5772-4a85-ae19-2fcf878567d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:05:24.331418 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:24.327926    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/726d2f42-5772-4a85-ae19-2fcf878567d5-istio-podinfo" pod "726d2f42-5772-4a85-ae19-2fcf878567d5" (UID: "726d2f42-5772-4a85-ae19-2fcf878567d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 20:05:24.331418 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:24.329731    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa76773f-998f-408d-a804-31f6c4921774-kube-api-access-89xtb" pod "aa76773f-998f-408d-a804-31f6c4921774" (UID: "aa76773f-998f-408d-a804-31f6c4921774"). InnerVolumeSpecName "kube-api-access-89xtb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:05:24.331642 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:24.331624    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/726d2f42-5772-4a85-ae19-2fcf878567d5-istio-envoy" pod "726d2f42-5772-4a85-ae19-2fcf878567d5" (UID: "726d2f42-5772-4a85-ae19-2fcf878567d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:24.397891 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:24.397849    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:24.450373 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:24.437528    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpq9n\" (UniqueName: \"kubernetes.io/projected/726d2f42-5772-4a85-ae19-2fcf878567d5-kube-api-access-bpq9n\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:24.450373 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:24.437570    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/726d2f42-5772-4a85-ae19-2fcf878567d5-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:24.450373 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:24.437586    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/726d2f42-5772-4a85-ae19-2fcf878567d5-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:24.450373 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:24.437600    4645 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa76773f-998f-408d-a804-31f6c4921774-plugins-config-volume\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:24.450373 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:24.437613    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/726d2f42-5772-4a85-ae19-2fcf878567d5-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:24.450373 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:24.437626    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/726d2f42-5772-4a85-ae19-2fcf878567d5-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:24.450373 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:24.437639    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89xtb\" (UniqueName: \"kubernetes.io/projected/aa76773f-998f-408d-a804-31f6c4921774-kube-api-access-89xtb\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:24.450373 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:24.437653    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/726d2f42-5772-4a85-ae19-2fcf878567d5-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:24.450373 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:24.437665    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/726d2f42-5772-4a85-ae19-2fcf878567d5-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:24.450373 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:24.437677    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/726d2f42-5772-4a85-ae19-2fcf878567d5-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:24.450373 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:24.437688    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/726d2f42-5772-4a85-ae19-2fcf878567d5-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:25.190879 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:25.190842    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4bac5a-78664677cc-6shsd"
Oct 31 20:05:25.190879 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:25.190862    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4bac5a-78664677cc-6shsd" event={"ID":"aa76773f-998f-408d-a804-31f6c4921774","Type":"ContainerDied","Data":"c657e70ab77bb8468a78a3e1946667bafea487d69899a3a1f473ed0119c424eb"}
Oct 31 20:05:25.191537 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:25.190940    4645 scope.go:122] "RemoveContainer" containerID="e88fd7cd9d0e1fc554eb036fb0aa37effec4b66817bff1b01a519b8a943229d2"
Oct 31 20:05:25.194015 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:25.193982    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99" event={"ID":"726d2f42-5772-4a85-ae19-2fcf878567d5","Type":"ContainerDied","Data":"8306699aa9e18cfa1c7c45c8a2cac5816574fe7c6cd0bdc0bc280609526fefb1"}
Oct 31 20:05:25.194143 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:25.194093    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99"
Oct 31 20:05:25.213488 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:25.213462    4645 scope.go:122] "RemoveContainer" containerID="b1c4b5407b6eac7792ec915125e7714b46ffeec197e61582ec1682b4120690b5"
Oct 31 20:05:25.223868 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:25.223846    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-4bac5a-78664677cc-6shsd"]
Oct 31 20:05:25.227696 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:25.227672    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-4bac5a-78664677cc-6shsd"]
Oct 31 20:05:25.239072 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:25.239048    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99"]
Oct 31 20:05:25.245437 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:25.245414    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-4bac5a-openshift-default-598ccc8647-r5j99"]
Oct 31 20:05:25.448870 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:25.448779    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:25.617941 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:25.617907    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="726d2f42-5772-4a85-ae19-2fcf878567d5" path="/var/lib/kubelet/pods/726d2f42-5772-4a85-ae19-2fcf878567d5/volumes"
Oct 31 20:05:25.618321 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:25.618308    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa76773f-998f-408d-a804-31f6c4921774" path="/var/lib/kubelet/pods/aa76773f-998f-408d-a804-31f6c4921774/volumes"
Oct 31 20:05:25.778734 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:25.778634    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n" podUID="dbb97168-db93-4757-9a1b-f9c5eb704fc5" containerName="main" containerID="cri-o://9672227baf80eaf23cdc05e00b5d9277b804295a45104e8114f256dda17f4a11" gracePeriod=45
Oct 31 20:05:26.100515 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:26.100483    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n"
Oct 31 20:05:26.210553 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:26.210511    4645 generic.go:356] "Generic (PLEG): container finished" podID="dbb97168-db93-4757-9a1b-f9c5eb704fc5" containerID="9672227baf80eaf23cdc05e00b5d9277b804295a45104e8114f256dda17f4a11" exitCode=0
Oct 31 20:05:26.211054 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:26.210562    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n"
Oct 31 20:05:26.211054 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:26.210598    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n" event={"ID":"dbb97168-db93-4757-9a1b-f9c5eb704fc5","Type":"ContainerDied","Data":"9672227baf80eaf23cdc05e00b5d9277b804295a45104e8114f256dda17f4a11"}
Oct 31 20:05:26.211054 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:26.210650    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n" event={"ID":"dbb97168-db93-4757-9a1b-f9c5eb704fc5","Type":"ContainerDied","Data":"4d400e34f50a429f9efd96d72791662367965d6259aa9e299ab395bacb12533f"}
Oct 31 20:05:26.211054 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:26.210676    4645 scope.go:122] "RemoveContainer" containerID="9672227baf80eaf23cdc05e00b5d9277b804295a45104e8114f256dda17f4a11"
Oct 31 20:05:26.223831 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:26.223808    4645 scope.go:122] "RemoveContainer" containerID="9672227baf80eaf23cdc05e00b5d9277b804295a45104e8114f256dda17f4a11"
Oct 31 20:05:26.224226 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:05:26.224197    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9672227baf80eaf23cdc05e00b5d9277b804295a45104e8114f256dda17f4a11\": container with ID starting with 9672227baf80eaf23cdc05e00b5d9277b804295a45104e8114f256dda17f4a11 not found: ID does not exist" containerID="9672227baf80eaf23cdc05e00b5d9277b804295a45104e8114f256dda17f4a11"
Oct 31 20:05:26.224347 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:26.224238    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9672227baf80eaf23cdc05e00b5d9277b804295a45104e8114f256dda17f4a11"} err="failed to get container status \"9672227baf80eaf23cdc05e00b5d9277b804295a45104e8114f256dda17f4a11\": rpc error: code = NotFound desc = could not find container \"9672227baf80eaf23cdc05e00b5d9277b804295a45104e8114f256dda17f4a11\": container with ID starting with 9672227baf80eaf23cdc05e00b5d9277b804295a45104e8114f256dda17f4a11 not found: ID does not exist"
Oct 31 20:05:26.288965 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:26.288935    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbb97168-db93-4757-9a1b-f9c5eb704fc5-tls-certs\" (UniqueName: \"kubernetes.io/secret/dbb97168-db93-4757-9a1b-f9c5eb704fc5-tls-certs\") pod \"dbb97168-db93-4757-9a1b-f9c5eb704fc5\" (UID: \"dbb97168-db93-4757-9a1b-f9c5eb704fc5\") "
Oct 31 20:05:26.289117 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:26.289052    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbb97168-db93-4757-9a1b-f9c5eb704fc5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbb97168-db93-4757-9a1b-f9c5eb704fc5-dshm\") pod \"dbb97168-db93-4757-9a1b-f9c5eb704fc5\" (UID: \"dbb97168-db93-4757-9a1b-f9c5eb704fc5\") "
Oct 31 20:05:26.289117 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:26.289085    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbb97168-db93-4757-9a1b-f9c5eb704fc5-kube-api-access-7rnc9\" (UniqueName: \"kubernetes.io/projected/dbb97168-db93-4757-9a1b-f9c5eb704fc5-kube-api-access-7rnc9\") pod \"dbb97168-db93-4757-9a1b-f9c5eb704fc5\" (UID: \"dbb97168-db93-4757-9a1b-f9c5eb704fc5\") "
Oct 31 20:05:26.289214 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:26.289163    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbb97168-db93-4757-9a1b-f9c5eb704fc5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbb97168-db93-4757-9a1b-f9c5eb704fc5-tmp-dir\") pod \"dbb97168-db93-4757-9a1b-f9c5eb704fc5\" (UID: \"dbb97168-db93-4757-9a1b-f9c5eb704fc5\") "
Oct 31 20:05:26.289268 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:26.289253    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbb97168-db93-4757-9a1b-f9c5eb704fc5-home\" (UniqueName: \"kubernetes.io/empty-dir/dbb97168-db93-4757-9a1b-f9c5eb704fc5-home\") pod \"dbb97168-db93-4757-9a1b-f9c5eb704fc5\" (UID: \"dbb97168-db93-4757-9a1b-f9c5eb704fc5\") "
Oct 31 20:05:26.289308 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:26.289291    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbb97168-db93-4757-9a1b-f9c5eb704fc5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbb97168-db93-4757-9a1b-f9c5eb704fc5-model-cache\") pod \"dbb97168-db93-4757-9a1b-f9c5eb704fc5\" (UID: \"dbb97168-db93-4757-9a1b-f9c5eb704fc5\") "
Oct 31 20:05:26.289563 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:26.289533    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbb97168-db93-4757-9a1b-f9c5eb704fc5-tmp-dir" pod "dbb97168-db93-4757-9a1b-f9c5eb704fc5" (UID: "dbb97168-db93-4757-9a1b-f9c5eb704fc5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:26.289673 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:26.289568    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbb97168-db93-4757-9a1b-f9c5eb704fc5-home" pod "dbb97168-db93-4757-9a1b-f9c5eb704fc5" (UID: "dbb97168-db93-4757-9a1b-f9c5eb704fc5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:26.289673 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:26.289622    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbb97168-db93-4757-9a1b-f9c5eb704fc5-model-cache" pod "dbb97168-db93-4757-9a1b-f9c5eb704fc5" (UID: "dbb97168-db93-4757-9a1b-f9c5eb704fc5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:26.289970 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:26.289950    4645 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbb97168-db93-4757-9a1b-f9c5eb704fc5-tmp-dir\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:26.290072 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:26.289976    4645 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbb97168-db93-4757-9a1b-f9c5eb704fc5-home\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:26.290072 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:26.289990    4645 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbb97168-db93-4757-9a1b-f9c5eb704fc5-model-cache\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:26.291369 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:26.291347    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbb97168-db93-4757-9a1b-f9c5eb704fc5-kube-api-access-7rnc9" pod "dbb97168-db93-4757-9a1b-f9c5eb704fc5" (UID: "dbb97168-db93-4757-9a1b-f9c5eb704fc5"). InnerVolumeSpecName "kube-api-access-7rnc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:05:26.291441 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:26.291400    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbb97168-db93-4757-9a1b-f9c5eb704fc5-tls-certs" pod "dbb97168-db93-4757-9a1b-f9c5eb704fc5" (UID: "dbb97168-db93-4757-9a1b-f9c5eb704fc5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:05:26.291613 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:26.291591    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbb97168-db93-4757-9a1b-f9c5eb704fc5-dshm" pod "dbb97168-db93-4757-9a1b-f9c5eb704fc5" (UID: "dbb97168-db93-4757-9a1b-f9c5eb704fc5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:26.341277 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:26.341159    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:26.391862 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:26.391830    4645 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbb97168-db93-4757-9a1b-f9c5eb704fc5-tls-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:26.391862 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:26.391858    4645 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbb97168-db93-4757-9a1b-f9c5eb704fc5-dshm\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:26.391862 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:26.391870    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rnc9\" (UniqueName: \"kubernetes.io/projected/dbb97168-db93-4757-9a1b-f9c5eb704fc5-kube-api-access-7rnc9\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:26.544349 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:26.544285    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n"]
Oct 31 20:05:26.547641 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:26.547613    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-996c32/premium-w4-996c32-kserve-864499dc8b-l9t9n"]
Oct 31 20:05:26.962363 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:26.955550    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:27.108115 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:27.108067    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:27.220616 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:27.220516    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:27.366852 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:27.366816    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:27.617482 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:27.617401    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbb97168-db93-4757-9a1b-f9c5eb704fc5" path="/var/lib/kubelet/pods/dbb97168-db93-4757-9a1b-f9c5eb704fc5/volumes"
Oct 31 20:05:27.794386 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:27.794132    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd"]
Oct 31 20:05:27.797855 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:27.795004    4645 cpu_manager.go:414] "containerMap: removing container" podUID="45ebe81b-1559-4c1b-860e-619f8e9e608e" containerName="istio-proxy"
Oct 31 20:05:27.797855 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:27.795029    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45ebe81b-1559-4c1b-860e-619f8e9e608e" containerName="istio-proxy"
Oct 31 20:05:27.797855 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:27.795052    4645 cpu_manager.go:414] "containerMap: removing container" podUID="b4677f66-6fc4-4c4a-8539-ede1dfe8f503" containerName="payload-processing"
Oct 31 20:05:27.797855 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:27.795060    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4677f66-6fc4-4c4a-8539-ede1dfe8f503" containerName="payload-processing"
Oct 31 20:05:27.797855 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:27.795087    4645 cpu_manager.go:414] "containerMap: removing container" podUID="de7f86dd-e3a8-4cf2-9b14-83f2a2e53dc3" containerName="revoke-keys"
Oct 31 20:05:27.797855 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:27.795096    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de7f86dd-e3a8-4cf2-9b14-83f2a2e53dc3" containerName="revoke-keys"
Oct 31 20:05:27.797855 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:27.795135    4645 cpu_manager.go:414] "containerMap: removing container" podUID="726d2f42-5772-4a85-ae19-2fcf878567d5" containerName="istio-proxy"
Oct 31 20:05:27.797855 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:27.795144    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="726d2f42-5772-4a85-ae19-2fcf878567d5" containerName="istio-proxy"
Oct 31 20:05:27.797855 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:27.795159    4645 cpu_manager.go:414] "containerMap: removing container" podUID="aa76773f-998f-408d-a804-31f6c4921774" containerName="payload-processing"
Oct 31 20:05:27.797855 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:27.795167    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa76773f-998f-408d-a804-31f6c4921774" containerName="payload-processing"
Oct 31 20:05:27.797855 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:27.795181    4645 cpu_manager.go:414] "containerMap: removing container" podUID="dbb97168-db93-4757-9a1b-f9c5eb704fc5" containerName="main"
Oct 31 20:05:27.797855 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:27.795188    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbb97168-db93-4757-9a1b-f9c5eb704fc5" containerName="main"
Oct 31 20:05:27.797855 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:27.795210    4645 cpu_manager.go:414] "containerMap: removing container" podUID="7d33bec5-7276-4b99-9a02-6ac2eaaf770f" containerName="payload-pre-processing"
Oct 31 20:05:27.797855 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:27.795217    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d33bec5-7276-4b99-9a02-6ac2eaaf770f" containerName="payload-pre-processing"
Oct 31 20:05:27.797855 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:27.795397    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d33bec5-7276-4b99-9a02-6ac2eaaf770f" containerName="payload-pre-processing"
Oct 31 20:05:27.797855 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:27.795410    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="45ebe81b-1559-4c1b-860e-619f8e9e608e" containerName="istio-proxy"
Oct 31 20:05:27.797855 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:27.795419    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4677f66-6fc4-4c4a-8539-ede1dfe8f503" containerName="payload-processing"
Oct 31 20:05:27.797855 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:27.795429    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="de7f86dd-e3a8-4cf2-9b14-83f2a2e53dc3" containerName="revoke-keys"
Oct 31 20:05:27.797855 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:27.795439    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="726d2f42-5772-4a85-ae19-2fcf878567d5" containerName="istio-proxy"
Oct 31 20:05:27.797855 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:27.795447    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa76773f-998f-408d-a804-31f6c4921774" containerName="payload-processing"
Oct 31 20:05:27.797855 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:27.795453    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbb97168-db93-4757-9a1b-f9c5eb704fc5" containerName="main"
Oct 31 20:05:27.801554 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:27.801525    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd"]
Oct 31 20:05:27.801698 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:27.801648    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd"
Oct 31 20:05:27.805854 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:27.805829    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-7fe845-openshift-default-dockercfg-tfcnw\""
Oct 31 20:05:27.825416 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:27.825374    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:27.957369 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:27.949308    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pftr\" (UniqueName: \"kubernetes.io/projected/05f487dd-a849-4cb8-b5f0-1368f60518a3-kube-api-access-2pftr\") pod \"e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd\" (UID: \"05f487dd-a849-4cb8-b5f0-1368f60518a3\") " pod="openshift-ingress/e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd"
Oct 31 20:05:27.957369 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:27.949438    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05f487dd-a849-4cb8-b5f0-1368f60518a3-istio-token\") pod \"e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd\" (UID: \"05f487dd-a849-4cb8-b5f0-1368f60518a3\") " pod="openshift-ingress/e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd"
Oct 31 20:05:27.957369 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:27.949482    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05f487dd-a849-4cb8-b5f0-1368f60518a3-istiod-ca-cert\") pod \"e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd\" (UID: \"05f487dd-a849-4cb8-b5f0-1368f60518a3\") " pod="openshift-ingress/e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd"
Oct 31 20:05:27.957369 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:27.949534    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f487dd-a849-4cb8-b5f0-1368f60518a3-workload-socket\") pod \"e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd\" (UID: \"05f487dd-a849-4cb8-b5f0-1368f60518a3\") " pod="openshift-ingress/e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd"
Oct 31 20:05:27.957369 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:27.949628    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f487dd-a849-4cb8-b5f0-1368f60518a3-credential-socket\") pod \"e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd\" (UID: \"05f487dd-a849-4cb8-b5f0-1368f60518a3\") " pod="openshift-ingress/e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd"
Oct 31 20:05:27.957369 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:27.949670    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05f487dd-a849-4cb8-b5f0-1368f60518a3-istio-envoy\") pod \"e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd\" (UID: \"05f487dd-a849-4cb8-b5f0-1368f60518a3\") " pod="openshift-ingress/e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd"
Oct 31 20:05:27.957369 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:27.949747    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05f487dd-a849-4cb8-b5f0-1368f60518a3-workload-certs\") pod \"e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd\" (UID: \"05f487dd-a849-4cb8-b5f0-1368f60518a3\") " pod="openshift-ingress/e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd"
Oct 31 20:05:27.957369 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:27.949784    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05f487dd-a849-4cb8-b5f0-1368f60518a3-istio-podinfo\") pod \"e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd\" (UID: \"05f487dd-a849-4cb8-b5f0-1368f60518a3\") " pod="openshift-ingress/e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd"
Oct 31 20:05:27.957369 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:27.949883    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05f487dd-a849-4cb8-b5f0-1368f60518a3-istio-data\") pod \"e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd\" (UID: \"05f487dd-a849-4cb8-b5f0-1368f60518a3\") " pod="openshift-ingress/e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd"
Oct 31 20:05:28.068352 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:28.061583    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f487dd-a849-4cb8-b5f0-1368f60518a3-credential-socket\") pod \"e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd\" (UID: \"05f487dd-a849-4cb8-b5f0-1368f60518a3\") " pod="openshift-ingress/e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd"
Oct 31 20:05:28.068352 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:28.061655    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05f487dd-a849-4cb8-b5f0-1368f60518a3-istio-envoy\") pod \"e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd\" (UID: \"05f487dd-a849-4cb8-b5f0-1368f60518a3\") " pod="openshift-ingress/e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd"
Oct 31 20:05:28.068352 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:28.061740    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05f487dd-a849-4cb8-b5f0-1368f60518a3-workload-certs\") pod \"e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd\" (UID: \"05f487dd-a849-4cb8-b5f0-1368f60518a3\") " pod="openshift-ingress/e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd"
Oct 31 20:05:28.068352 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:28.061784    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05f487dd-a849-4cb8-b5f0-1368f60518a3-istio-podinfo\") pod \"e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd\" (UID: \"05f487dd-a849-4cb8-b5f0-1368f60518a3\") " pod="openshift-ingress/e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd"
Oct 31 20:05:28.068352 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:28.061871    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05f487dd-a849-4cb8-b5f0-1368f60518a3-istio-data\") pod \"e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd\" (UID: \"05f487dd-a849-4cb8-b5f0-1368f60518a3\") " pod="openshift-ingress/e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd"
Oct 31 20:05:28.068352 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:28.061935    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pftr\" (UniqueName: \"kubernetes.io/projected/05f487dd-a849-4cb8-b5f0-1368f60518a3-kube-api-access-2pftr\") pod \"e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd\" (UID: \"05f487dd-a849-4cb8-b5f0-1368f60518a3\") " pod="openshift-ingress/e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd"
Oct 31 20:05:28.068352 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:28.061967    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05f487dd-a849-4cb8-b5f0-1368f60518a3-istio-token\") pod \"e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd\" (UID: \"05f487dd-a849-4cb8-b5f0-1368f60518a3\") " pod="openshift-ingress/e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd"
Oct 31 20:05:28.068352 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:28.062001    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05f487dd-a849-4cb8-b5f0-1368f60518a3-istiod-ca-cert\") pod \"e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd\" (UID: \"05f487dd-a849-4cb8-b5f0-1368f60518a3\") " pod="openshift-ingress/e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd"
Oct 31 20:05:28.068352 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:28.062046    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f487dd-a849-4cb8-b5f0-1368f60518a3-workload-socket\") pod \"e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd\" (UID: \"05f487dd-a849-4cb8-b5f0-1368f60518a3\") " pod="openshift-ingress/e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd"
Oct 31 20:05:28.068352 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:28.062576    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f487dd-a849-4cb8-b5f0-1368f60518a3-workload-socket\") pod \"e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd\" (UID: \"05f487dd-a849-4cb8-b5f0-1368f60518a3\") " pod="openshift-ingress/e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd"
Oct 31 20:05:28.068352 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:28.062834    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f487dd-a849-4cb8-b5f0-1368f60518a3-credential-socket\") pod \"e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd\" (UID: \"05f487dd-a849-4cb8-b5f0-1368f60518a3\") " pod="openshift-ingress/e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd"
Oct 31 20:05:28.068352 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:28.063746    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05f487dd-a849-4cb8-b5f0-1368f60518a3-istio-data\") pod \"e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd\" (UID: \"05f487dd-a849-4cb8-b5f0-1368f60518a3\") " pod="openshift-ingress/e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd"
Oct 31 20:05:28.068352 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:28.063977    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05f487dd-a849-4cb8-b5f0-1368f60518a3-workload-certs\") pod \"e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd\" (UID: \"05f487dd-a849-4cb8-b5f0-1368f60518a3\") " pod="openshift-ingress/e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd"
Oct 31 20:05:28.068352 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:28.065649    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05f487dd-a849-4cb8-b5f0-1368f60518a3-istiod-ca-cert\") pod \"e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd\" (UID: \"05f487dd-a849-4cb8-b5f0-1368f60518a3\") " pod="openshift-ingress/e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd"
Oct 31 20:05:28.098967 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:28.098882    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pftr\" (UniqueName: \"kubernetes.io/projected/05f487dd-a849-4cb8-b5f0-1368f60518a3-kube-api-access-2pftr\") pod \"e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd\" (UID: \"05f487dd-a849-4cb8-b5f0-1368f60518a3\") " pod="openshift-ingress/e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd"
Oct 31 20:05:28.101304 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:28.101252    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05f487dd-a849-4cb8-b5f0-1368f60518a3-istio-token\") pod \"e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd\" (UID: \"05f487dd-a849-4cb8-b5f0-1368f60518a3\") " pod="openshift-ingress/e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd"
Oct 31 20:05:28.103074 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:28.102276    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05f487dd-a849-4cb8-b5f0-1368f60518a3-istio-podinfo\") pod \"e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd\" (UID: \"05f487dd-a849-4cb8-b5f0-1368f60518a3\") " pod="openshift-ingress/e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd"
Oct 31 20:05:28.103074 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:28.103010    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05f487dd-a849-4cb8-b5f0-1368f60518a3-istio-envoy\") pod \"e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd\" (UID: \"05f487dd-a849-4cb8-b5f0-1368f60518a3\") " pod="openshift-ingress/e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd"
Oct 31 20:05:28.118821 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:28.117787    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd"
Oct 31 20:05:28.143629 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:28.143500    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:28.342828 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:28.342788    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:28.490733 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:28.490704    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd"]
Oct 31 20:05:28.492238 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:05:28.492209    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05f487dd_a849_4cb8_b5f0_1368f60518a3.slice/crio-f91951887d7d5796e9fbb5f92ef6d869aa876367cb29ba1a7a645b42ca21d6a1 WatchSource:0}: Error finding container f91951887d7d5796e9fbb5f92ef6d869aa876367cb29ba1a7a645b42ca21d6a1: Status 404 returned error can't find the container with id f91951887d7d5796e9fbb5f92ef6d869aa876367cb29ba1a7a645b42ca21d6a1
Oct 31 20:05:28.495122 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:28.495089    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:05:28.495202 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:28.495174    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:05:28.495240 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:28.495209    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:05:29.005087 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:29.005046    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:29.249351 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:29.242692    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:29.298414 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:29.297537    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd" event={"ID":"05f487dd-a849-4cb8-b5f0-1368f60518a3","Type":"ContainerStarted","Data":"4a4a5accd018c0747b0b44311c6383cba5cc15f19397a582160cab4f76526710"}
Oct 31 20:05:29.298414 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:29.297650    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd" event={"ID":"05f487dd-a849-4cb8-b5f0-1368f60518a3","Type":"ContainerStarted","Data":"f91951887d7d5796e9fbb5f92ef6d869aa876367cb29ba1a7a645b42ca21d6a1"}
Oct 31 20:05:29.351518 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:29.351452    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd" podStartSLOduration=2.351426746 podStartE2EDuration="2.351426746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:05:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:05:29.339819432 +0000 UTC m=+2759.913053565" watchObservedRunningTime="2026-10-31 20:05:29.351426746 +0000 UTC m=+2759.924660874"
Oct 31 20:05:29.396353 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:29.392073    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:29.596560 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:29.596463    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:29.794387 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:29.794350    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:29.948594 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:29.948548    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:30.071361 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:30.067895    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:30.136390 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:30.120294    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd"
Oct 31 20:05:30.140597 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:30.140566    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd"
Oct 31 20:05:30.307039 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:30.306957    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd"
Oct 31 20:05:30.308854 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:30.308827    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd"
Oct 31 20:05:30.457542 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:30.457512    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-c173d8-6d47cf995d-2bjft_c08e5b4b-a7da-4cfe-a48a-140363b68362/payload-processing/0.log"
Oct 31 20:05:30.610109 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:30.610018    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:30.789782 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:30.789755    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-76hjw_b14ee5b2-6bf8-4b41-a84d-d0a7dcc9b85a/payload-processing/0.log"
Oct 31 20:05:30.996269 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:30.996232    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc"]
Oct 31 20:05:31.002301 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:31.002273    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc"
Oct 31 20:05:31.045178 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:31.045106    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc"]
Oct 31 20:05:31.061705 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:31.061668    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d9801bc-b873-4f98-9395-0a19686d6b8a-istio-data\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc\" (UID: \"8d9801bc-b873-4f98-9395-0a19686d6b8a\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc"
Oct 31 20:05:31.061870 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:31.061773    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d9801bc-b873-4f98-9395-0a19686d6b8a-istio-token\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc\" (UID: \"8d9801bc-b873-4f98-9395-0a19686d6b8a\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc"
Oct 31 20:05:31.061870 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:31.061840    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d9801bc-b873-4f98-9395-0a19686d6b8a-credential-socket\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc\" (UID: \"8d9801bc-b873-4f98-9395-0a19686d6b8a\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc"
Oct 31 20:05:31.061962 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:31.061878    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d9801bc-b873-4f98-9395-0a19686d6b8a-istiod-ca-cert\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc\" (UID: \"8d9801bc-b873-4f98-9395-0a19686d6b8a\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc"
Oct 31 20:05:31.061962 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:31.061912    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d9801bc-b873-4f98-9395-0a19686d6b8a-istio-podinfo\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc\" (UID: \"8d9801bc-b873-4f98-9395-0a19686d6b8a\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc"
Oct 31 20:05:31.061962 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:31.061943    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d9801bc-b873-4f98-9395-0a19686d6b8a-workload-socket\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc\" (UID: \"8d9801bc-b873-4f98-9395-0a19686d6b8a\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc"
Oct 31 20:05:31.062088 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:31.061976    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-76hjw_b14ee5b2-6bf8-4b41-a84d-d0a7dcc9b85a/payload-processing/0.log"
Oct 31 20:05:31.062088 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:31.061979    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d9801bc-b873-4f98-9395-0a19686d6b8a-istio-envoy\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc\" (UID: \"8d9801bc-b873-4f98-9395-0a19686d6b8a\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc"
Oct 31 20:05:31.062170 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:31.062088    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz8db\" (UniqueName: \"kubernetes.io/projected/8d9801bc-b873-4f98-9395-0a19686d6b8a-kube-api-access-lz8db\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc\" (UID: \"8d9801bc-b873-4f98-9395-0a19686d6b8a\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc"
Oct 31 20:05:31.062170 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:31.062120    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d9801bc-b873-4f98-9395-0a19686d6b8a-workload-certs\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc\" (UID: \"8d9801bc-b873-4f98-9395-0a19686d6b8a\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc"
Oct 31 20:05:31.163240 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:31.163198    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d9801bc-b873-4f98-9395-0a19686d6b8a-credential-socket\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc\" (UID: \"8d9801bc-b873-4f98-9395-0a19686d6b8a\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc"
Oct 31 20:05:31.163462 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:31.163271    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d9801bc-b873-4f98-9395-0a19686d6b8a-istiod-ca-cert\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc\" (UID: \"8d9801bc-b873-4f98-9395-0a19686d6b8a\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc"
Oct 31 20:05:31.163462 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:31.163311    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d9801bc-b873-4f98-9395-0a19686d6b8a-istio-podinfo\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc\" (UID: \"8d9801bc-b873-4f98-9395-0a19686d6b8a\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc"
Oct 31 20:05:31.163462 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:31.163356    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d9801bc-b873-4f98-9395-0a19686d6b8a-workload-socket\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc\" (UID: \"8d9801bc-b873-4f98-9395-0a19686d6b8a\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc"
Oct 31 20:05:31.163462 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:31.163396    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d9801bc-b873-4f98-9395-0a19686d6b8a-istio-envoy\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc\" (UID: \"8d9801bc-b873-4f98-9395-0a19686d6b8a\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc"
Oct 31 20:05:31.163462 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:31.163439    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz8db\" (UniqueName: \"kubernetes.io/projected/8d9801bc-b873-4f98-9395-0a19686d6b8a-kube-api-access-lz8db\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc\" (UID: \"8d9801bc-b873-4f98-9395-0a19686d6b8a\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc"
Oct 31 20:05:31.163766 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:31.163472    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d9801bc-b873-4f98-9395-0a19686d6b8a-workload-certs\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc\" (UID: \"8d9801bc-b873-4f98-9395-0a19686d6b8a\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc"
Oct 31 20:05:31.163766 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:31.163693    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d9801bc-b873-4f98-9395-0a19686d6b8a-istio-data\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc\" (UID: \"8d9801bc-b873-4f98-9395-0a19686d6b8a\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc"
Oct 31 20:05:31.163996 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:31.163746    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d9801bc-b873-4f98-9395-0a19686d6b8a-istio-token\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc\" (UID: \"8d9801bc-b873-4f98-9395-0a19686d6b8a\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc"
Oct 31 20:05:31.164873 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:31.164818    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d9801bc-b873-4f98-9395-0a19686d6b8a-workload-socket\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc\" (UID: \"8d9801bc-b873-4f98-9395-0a19686d6b8a\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc"
Oct 31 20:05:31.164873 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:31.164846    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d9801bc-b873-4f98-9395-0a19686d6b8a-workload-certs\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc\" (UID: \"8d9801bc-b873-4f98-9395-0a19686d6b8a\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc"
Oct 31 20:05:31.165924 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:31.165656    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d9801bc-b873-4f98-9395-0a19686d6b8a-istiod-ca-cert\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc\" (UID: \"8d9801bc-b873-4f98-9395-0a19686d6b8a\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc"
Oct 31 20:05:31.165924 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:31.165767    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d9801bc-b873-4f98-9395-0a19686d6b8a-credential-socket\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc\" (UID: \"8d9801bc-b873-4f98-9395-0a19686d6b8a\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc"
Oct 31 20:05:31.166364 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:31.166324    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d9801bc-b873-4f98-9395-0a19686d6b8a-istio-data\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc\" (UID: \"8d9801bc-b873-4f98-9395-0a19686d6b8a\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc"
Oct 31 20:05:31.166877 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:31.166853    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d9801bc-b873-4f98-9395-0a19686d6b8a-istio-envoy\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc\" (UID: \"8d9801bc-b873-4f98-9395-0a19686d6b8a\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc"
Oct 31 20:05:31.166967 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:31.166946    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d9801bc-b873-4f98-9395-0a19686d6b8a-istio-podinfo\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc\" (UID: \"8d9801bc-b873-4f98-9395-0a19686d6b8a\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc"
Oct 31 20:05:31.183359 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:31.183305    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz8db\" (UniqueName: \"kubernetes.io/projected/8d9801bc-b873-4f98-9395-0a19686d6b8a-kube-api-access-lz8db\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc\" (UID: \"8d9801bc-b873-4f98-9395-0a19686d6b8a\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc"
Oct 31 20:05:31.183888 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:31.183861    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d9801bc-b873-4f98-9395-0a19686d6b8a-istio-token\") pod \"e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc\" (UID: \"8d9801bc-b873-4f98-9395-0a19686d6b8a\") " pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc"
Oct 31 20:05:31.211518 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:31.211481    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:31.230591 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:31.230557    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ae2e60-g9w27"]
Oct 31 20:05:31.235015 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:31.234974    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-ae2e60-g9w27"]
Oct 31 20:05:31.249896 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:31.249806    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-c9b9a1-f45479ffd-mqgll_d1e003dd-25ad-430d-bd3c-b7706d2788cd/payload-processing/0.log"
Oct 31 20:05:31.321589 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:31.320842    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc"
Oct 31 20:05:31.394939 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:31.394012    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:31.507763 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:31.507689    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-c9b9a1-f45479ffd-mqgll_d1e003dd-25ad-430d-bd3c-b7706d2788cd/payload-processing/0.log"
Oct 31 20:05:31.620952 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:31.620913    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de7f86dd-e3a8-4cf2-9b14-83f2a2e53dc3" path="/var/lib/kubelet/pods/de7f86dd-e3a8-4cf2-9b14-83f2a2e53dc3/volumes"
Oct 31 20:05:31.632179 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:05:31.632138    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d9801bc_b873_4f98_9395_0a19686d6b8a.slice/crio-6757c6067261b2a3a6c39d1db9ae1f6a55a95259617f981efac1e916525abfed WatchSource:0}: Error finding container 6757c6067261b2a3a6c39d1db9ae1f6a55a95259617f981efac1e916525abfed: Status 404 returned error can't find the container with id 6757c6067261b2a3a6c39d1db9ae1f6a55a95259617f981efac1e916525abfed
Oct 31 20:05:31.632535 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:31.632509    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc"]
Oct 31 20:05:31.635606 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:31.635563    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:05:31.635721 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:31.635658    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:05:31.635721 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:31.635698    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:05:31.808796 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:31.808712    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:31.945685 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:31.945653    4645 scope.go:122] "RemoveContainer" containerID="51e8b49226dda1ac9378769450aa1a85c3d80b6bd98a50e39643bf27b3d1d95a"
Oct 31 20:05:31.958623 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:31.958591    4645 scope.go:122] "RemoveContainer" containerID="78768d4731529b62a86b24591f9f089f7f4d08f762f395f752995d8a492dab1c"
Oct 31 20:05:31.973256 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:31.973225    4645 scope.go:122] "RemoveContainer" containerID="c7fd818880817cc4b98b12ccddc43c17eb4e4a4bd1b9342eda66520de9f279ee"
Oct 31 20:05:32.032113 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:32.032073    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:32.209773 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:32.209742    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:32.332889 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:32.332805    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc" event={"ID":"8d9801bc-b873-4f98-9395-0a19686d6b8a","Type":"ContainerStarted","Data":"b34933e0988ad6af10c3bb56ce43816fbd2140c76937a4014d6b6d176f4ffed7"}
Oct 31 20:05:32.332889 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:32.332870    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc" event={"ID":"8d9801bc-b873-4f98-9395-0a19686d6b8a","Type":"ContainerStarted","Data":"6757c6067261b2a3a6c39d1db9ae1f6a55a95259617f981efac1e916525abfed"}
Oct 31 20:05:32.366057 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:32.366001    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc" podStartSLOduration=2.365985718 podStartE2EDuration="2.365985718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:05:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:05:32.362619559 +0000 UTC m=+2762.935853687" watchObservedRunningTime="2026-10-31 20:05:32.365985718 +0000 UTC m=+2762.939219846"
Oct 31 20:05:32.496887 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:32.496845    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:33.321834 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:33.321796    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc"
Oct 31 20:05:33.323221 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:33.323194    4645 patch_prober.go:29] interesting pod/e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused" start-of-body=
Oct 31 20:05:33.323350 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:33.323262    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc" podUID="8d9801bc-b873-4f98-9395-0a19686d6b8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused"
Oct 31 20:05:33.485593 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:33.483395    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:33.800117 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:33.800082    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:33.946199 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:33.946109    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:34.223088 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:34.223046    4645 patch_prober.go:29] interesting pod/e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 20:05:34.223273 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:34.223113    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc" podUID="7309f62d-7bc0-4677-a81d-3b306f5eca82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 20:05:34.322166 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:34.322129    4645 patch_prober.go:29] interesting pod/e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused" start-of-body=
Oct 31 20:05:34.322600 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:34.322193    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc" podUID="8d9801bc-b873-4f98-9395-0a19686d6b8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused"
Oct 31 20:05:34.882069 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:34.881960    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:35.321499 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:35.321464    4645 patch_prober.go:29] interesting pod/e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused" start-of-body=
Oct 31 20:05:35.321783 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:35.321515    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc" podUID="8d9801bc-b873-4f98-9395-0a19686d6b8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused"
Oct 31 20:05:35.381354 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:35.378733    4645 generic.go:356] "Generic (PLEG): container finished" podID="7309f62d-7bc0-4677-a81d-3b306f5eca82" containerID="766a6cd8f5a7309349b6d7d5323cb51a83328f800aa67b77d9a7f37492975905" exitCode=0
Oct 31 20:05:35.381354 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:35.379022    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc" event={"ID":"7309f62d-7bc0-4677-a81d-3b306f5eca82","Type":"ContainerDied","Data":"766a6cd8f5a7309349b6d7d5323cb51a83328f800aa67b77d9a7f37492975905"}
Oct 31 20:05:35.381354 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:35.379601    4645 scope.go:122] "RemoveContainer" containerID="766a6cd8f5a7309349b6d7d5323cb51a83328f800aa67b77d9a7f37492975905"
Oct 31 20:05:35.382437 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:35.381879    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:05:35.382437 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:35.381950    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:05:35.382437 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:35.381982    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:05:35.801112 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:35.801066    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:36.254605 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:05:36.254566    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 20:05:36.322264 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:36.322225    4645 patch_prober.go:29] interesting pod/e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused" start-of-body=
Oct 31 20:05:36.322472 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:36.322292    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc" podUID="8d9801bc-b873-4f98-9395-0a19686d6b8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused"
Oct 31 20:05:36.411321 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:36.411280    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc" event={"ID":"7309f62d-7bc0-4677-a81d-3b306f5eca82","Type":"ContainerStarted","Data":"b9df2ec920c166a8700d224b7a0c44d5f31604992c3a02a9eccd41773f057ba6"}
Oct 31 20:05:37.009398 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:37.003582    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:37.322165 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:37.322079    4645 patch_prober.go:29] interesting pod/e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused" start-of-body=
Oct 31 20:05:37.322165 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:37.322129    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc" podUID="8d9801bc-b873-4f98-9395-0a19686d6b8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused"
Oct 31 20:05:37.329991 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:37.329926    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc"
Oct 31 20:05:37.330578 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:37.330552    4645 patch_prober.go:29] interesting pod/e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused" start-of-body=
Oct 31 20:05:37.330703 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:37.330598    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc" podUID="7309f62d-7bc0-4677-a81d-3b306f5eca82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused"
Oct 31 20:05:38.024206 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:38.024169    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:38.321897 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:38.321813    4645 patch_prober.go:29] interesting pod/e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused" start-of-body=
Oct 31 20:05:38.321897 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:38.321888    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc" podUID="8d9801bc-b873-4f98-9395-0a19686d6b8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused"
Oct 31 20:05:38.330307 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:38.330276    4645 patch_prober.go:29] interesting pod/e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused" start-of-body=
Oct 31 20:05:38.330469 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:38.330336    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc" podUID="7309f62d-7bc0-4677-a81d-3b306f5eca82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused"
Oct 31 20:05:38.790367 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:38.790310    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:38.914365 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:38.914301    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:39.134467 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:39.134380    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl"]
Oct 31 20:05:39.135307 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:39.135273    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl" podUID="92238ea2-f330-4873-ab87-37b37bb05f9a" containerName="istio-proxy" containerID="cri-o://2136df6ced5bec698141e9ae7567e6f1ac864a520bf01edcbbd4c907e8805305" gracePeriod=30
Oct 31 20:05:39.149152 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:39.149087    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:39.270358 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:39.267386    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:39.333382 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:39.332438    4645 patch_prober.go:29] interesting pod/e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused" start-of-body=
Oct 31 20:05:39.333382 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:39.332505    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc" podUID="7309f62d-7bc0-4677-a81d-3b306f5eca82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused"
Oct 31 20:05:39.401407 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:39.392654    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc"
Oct 31 20:05:39.401407 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:39.394676    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc"
Oct 31 20:05:39.401407 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:39.401358    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc"
Oct 31 20:05:39.680353 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:39.678221    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:39.814487 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:39.814434    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-996c32-88bf7fc6f-twmpc"]
Oct 31 20:05:39.814836 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:39.814793    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-996c32-88bf7fc6f-twmpc" podUID="f67e9a88-e1fd-4151-8023-8d44ce90e441" containerName="maas-api" containerID="cri-o://b29d029fe4bab602a01874f88ea7ac03b33b9c09fbe19dcf731fb5181b372450" gracePeriod=30
Oct 31 20:05:39.883554 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:39.883522    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8730d5-twrdd"]
Oct 31 20:05:39.887798 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:39.887780    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8730d5-twrdd"
Oct 31 20:05:39.892170 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:39.892151    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:39.904727 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:39.904704    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8730d5-twrdd"]
Oct 31 20:05:39.985784 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:39.985686    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 20:05:39.985784 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:39.985736    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:05:40.025354 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:40.025004    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26953ffd-3365-40e8-ad23-6b32cca6e57e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-8730d5-twrdd\" (UID: \"26953ffd-3365-40e8-ad23-6b32cca6e57e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8730d5-twrdd"
Oct 31 20:05:40.025354 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:40.025086    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpqc4\" (UniqueName: \"kubernetes.io/projected/26953ffd-3365-40e8-ad23-6b32cca6e57e-kube-api-access-cpqc4\") pod \"maas-api-revoke-keys-e2e-worker-w6-8730d5-twrdd\" (UID: \"26953ffd-3365-40e8-ad23-6b32cca6e57e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8730d5-twrdd"
Oct 31 20:05:40.025354 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:40.025273    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26953ffd-3365-40e8-ad23-6b32cca6e57e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-8730d5-twrdd\" (UID: \"26953ffd-3365-40e8-ad23-6b32cca6e57e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8730d5-twrdd"
Oct 31 20:05:40.125638 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:40.125609    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-996c32-88bf7fc6f-twmpc"
Oct 31 20:05:40.126512 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:40.126485    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26953ffd-3365-40e8-ad23-6b32cca6e57e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-8730d5-twrdd\" (UID: \"26953ffd-3365-40e8-ad23-6b32cca6e57e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8730d5-twrdd"
Oct 31 20:05:40.126635 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:40.126551    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpqc4\" (UniqueName: \"kubernetes.io/projected/26953ffd-3365-40e8-ad23-6b32cca6e57e-kube-api-access-cpqc4\") pod \"maas-api-revoke-keys-e2e-worker-w6-8730d5-twrdd\" (UID: \"26953ffd-3365-40e8-ad23-6b32cca6e57e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8730d5-twrdd"
Oct 31 20:05:40.126740 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:40.126726    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26953ffd-3365-40e8-ad23-6b32cca6e57e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-8730d5-twrdd\" (UID: \"26953ffd-3365-40e8-ad23-6b32cca6e57e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8730d5-twrdd"
Oct 31 20:05:40.127551 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:40.127266    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26953ffd-3365-40e8-ad23-6b32cca6e57e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-8730d5-twrdd\" (UID: \"26953ffd-3365-40e8-ad23-6b32cca6e57e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8730d5-twrdd"
Oct 31 20:05:40.129619 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:40.129594    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26953ffd-3365-40e8-ad23-6b32cca6e57e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-8730d5-twrdd\" (UID: \"26953ffd-3365-40e8-ad23-6b32cca6e57e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8730d5-twrdd"
Oct 31 20:05:40.264963 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:40.236859    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f67e9a88-e1fd-4151-8023-8d44ce90e441-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f67e9a88-e1fd-4151-8023-8d44ce90e441-metrics-certs\") pod \"f67e9a88-e1fd-4151-8023-8d44ce90e441\" (UID: \"f67e9a88-e1fd-4151-8023-8d44ce90e441\") "
Oct 31 20:05:40.264963 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:40.236928    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f67e9a88-e1fd-4151-8023-8d44ce90e441-kube-api-access-z5gzd\" (UniqueName: \"kubernetes.io/projected/f67e9a88-e1fd-4151-8023-8d44ce90e441-kube-api-access-z5gzd\") pod \"f67e9a88-e1fd-4151-8023-8d44ce90e441\" (UID: \"f67e9a88-e1fd-4151-8023-8d44ce90e441\") "
Oct 31 20:05:40.264963 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:40.236981    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f67e9a88-e1fd-4151-8023-8d44ce90e441-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f67e9a88-e1fd-4151-8023-8d44ce90e441-openshift-service-ca\") pod \"f67e9a88-e1fd-4151-8023-8d44ce90e441\" (UID: \"f67e9a88-e1fd-4151-8023-8d44ce90e441\") "
Oct 31 20:05:40.264963 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:40.237196    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f67e9a88-e1fd-4151-8023-8d44ce90e441-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f67e9a88-e1fd-4151-8023-8d44ce90e441-maas-api-tls\") pod \"f67e9a88-e1fd-4151-8023-8d44ce90e441\" (UID: \"f67e9a88-e1fd-4151-8023-8d44ce90e441\") "
Oct 31 20:05:40.264963 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:40.247993    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:40.264963 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:40.252905    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f67e9a88-e1fd-4151-8023-8d44ce90e441-openshift-service-ca" pod "f67e9a88-e1fd-4151-8023-8d44ce90e441" (UID: "f67e9a88-e1fd-4151-8023-8d44ce90e441"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:05:40.277351 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:40.275127    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpqc4\" (UniqueName: \"kubernetes.io/projected/26953ffd-3365-40e8-ad23-6b32cca6e57e-kube-api-access-cpqc4\") pod \"maas-api-revoke-keys-e2e-worker-w6-8730d5-twrdd\" (UID: \"26953ffd-3365-40e8-ad23-6b32cca6e57e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8730d5-twrdd"
Oct 31 20:05:40.350407 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:40.343031    4645 patch_prober.go:29] interesting pod/e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused" start-of-body=
Oct 31 20:05:40.350407 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:40.343091    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc" podUID="7309f62d-7bc0-4677-a81d-3b306f5eca82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused"
Oct 31 20:05:40.350407 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:40.344651    4645 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f67e9a88-e1fd-4151-8023-8d44ce90e441-openshift-service-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:40.388200 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:40.388147    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f67e9a88-e1fd-4151-8023-8d44ce90e441-maas-api-tls" pod "f67e9a88-e1fd-4151-8023-8d44ce90e441" (UID: "f67e9a88-e1fd-4151-8023-8d44ce90e441"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:05:40.388541 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:40.388513    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f67e9a88-e1fd-4151-8023-8d44ce90e441-kube-api-access-z5gzd" pod "f67e9a88-e1fd-4151-8023-8d44ce90e441" (UID: "f67e9a88-e1fd-4151-8023-8d44ce90e441"). InnerVolumeSpecName "kube-api-access-z5gzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:05:40.390804 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:40.390767    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f67e9a88-e1fd-4151-8023-8d44ce90e441-metrics-certs" pod "f67e9a88-e1fd-4151-8023-8d44ce90e441" (UID: "f67e9a88-e1fd-4151-8023-8d44ce90e441"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:05:40.395464 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:40.395438    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:40.442379 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:40.442355    4645 generic.go:356] "Generic (PLEG): container finished" podID="f67e9a88-e1fd-4151-8023-8d44ce90e441" containerID="b29d029fe4bab602a01874f88ea7ac03b33b9c09fbe19dcf731fb5181b372450" exitCode=0
Oct 31 20:05:40.442527 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:40.442384    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-996c32-88bf7fc6f-twmpc"
Oct 31 20:05:40.442527 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:40.442440    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-996c32-88bf7fc6f-twmpc" event={"ID":"f67e9a88-e1fd-4151-8023-8d44ce90e441","Type":"ContainerDied","Data":"b29d029fe4bab602a01874f88ea7ac03b33b9c09fbe19dcf731fb5181b372450"}
Oct 31 20:05:40.442527 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:40.442497    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-996c32-88bf7fc6f-twmpc" event={"ID":"f67e9a88-e1fd-4151-8023-8d44ce90e441","Type":"ContainerDied","Data":"935dc961630466be4b65fe2054ce65babd4d380b6002d7c2e3c444ff5157b179"}
Oct 31 20:05:40.442719 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:40.442527    4645 scope.go:122] "RemoveContainer" containerID="b29d029fe4bab602a01874f88ea7ac03b33b9c09fbe19dcf731fb5181b372450"
Oct 31 20:05:40.445480 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:40.445458    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f67e9a88-e1fd-4151-8023-8d44ce90e441-maas-api-tls\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:40.445480 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:40.445479    4645 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f67e9a88-e1fd-4151-8023-8d44ce90e441-metrics-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:40.445623 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:40.445496    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5gzd\" (UniqueName: \"kubernetes.io/projected/f67e9a88-e1fd-4151-8023-8d44ce90e441-kube-api-access-z5gzd\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:40.453150 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:40.453132    4645 scope.go:122] "RemoveContainer" containerID="b29d029fe4bab602a01874f88ea7ac03b33b9c09fbe19dcf731fb5181b372450"
Oct 31 20:05:40.453419 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:05:40.453396    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b29d029fe4bab602a01874f88ea7ac03b33b9c09fbe19dcf731fb5181b372450\": container with ID starting with b29d029fe4bab602a01874f88ea7ac03b33b9c09fbe19dcf731fb5181b372450 not found: ID does not exist" containerID="b29d029fe4bab602a01874f88ea7ac03b33b9c09fbe19dcf731fb5181b372450"
Oct 31 20:05:40.453490 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:40.453428    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b29d029fe4bab602a01874f88ea7ac03b33b9c09fbe19dcf731fb5181b372450"} err="failed to get container status \"b29d029fe4bab602a01874f88ea7ac03b33b9c09fbe19dcf731fb5181b372450\": rpc error: code = NotFound desc = could not find container \"b29d029fe4bab602a01874f88ea7ac03b33b9c09fbe19dcf731fb5181b372450\": container with ID starting with b29d029fe4bab602a01874f88ea7ac03b33b9c09fbe19dcf731fb5181b372450 not found: ID does not exist"
Oct 31 20:05:40.514354 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:40.514026    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8730d5-twrdd"
Oct 31 20:05:40.516287 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:40.516218    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:40.519221 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:40.519196    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-996c32-88bf7fc6f-twmpc"]
Oct 31 20:05:40.522110 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:40.522087    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-996c32-88bf7fc6f-twmpc"]
Oct 31 20:05:40.743100 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:40.743072    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:40.840742 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:40.840713    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8730d5-twrdd"]
Oct 31 20:05:40.842611 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:05:40.842588    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26953ffd_3365_40e8_ad23_6b32cca6e57e.slice/crio-1ad39898f29ee2b47ea506dcf34dcec51547436e0c9a2a36687a8bff46bcde86 WatchSource:0}: Error finding container 1ad39898f29ee2b47ea506dcf34dcec51547436e0c9a2a36687a8bff46bcde86: Status 404 returned error can't find the container with id 1ad39898f29ee2b47ea506dcf34dcec51547436e0c9a2a36687a8bff46bcde86
Oct 31 20:05:40.882163 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:40.882138    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:40.992792 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:40.992738    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:41.095734 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:41.095618    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk"]
Oct 31 20:05:41.095734 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:41.095705    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l"]
Oct 31 20:05:41.095991 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:41.095903    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk" podUID="22d8af28-f4e5-47ea-a25a-9076b9afbc11" containerName="istio-proxy" containerID="cri-o://9490917c1e492122036f9de619e640b0911012219d2c82f4dcf059fb9e3808f0" gracePeriod=30
Oct 31 20:05:41.103103 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:41.103072    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz"]
Oct 31 20:05:41.103321 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:41.103287    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz" podUID="0404b7e2-fd55-484a-bbec-3dd056db3e6b" containerName="istio-proxy" containerID="cri-o://aebbcc9192682a9b18b6a94a4f8f5ec8f3f244bdbb2bfb2a5db987db8094deb5" gracePeriod=30
Oct 31 20:05:41.121304 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:41.121161    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:41.329693 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:41.329658    4645 patch_prober.go:29] interesting pod/e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused" start-of-body=
Oct 31 20:05:41.330116 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:41.329706    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc" podUID="7309f62d-7bc0-4677-a81d-3b306f5eca82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused"
Oct 31 20:05:41.455235 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:41.455196    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8730d5-twrdd" event={"ID":"26953ffd-3365-40e8-ad23-6b32cca6e57e","Type":"ContainerStarted","Data":"299ffbbdb82d1ca774d0af57189f340b71c58ff362090599ba92d558e76e6e38"}
Oct 31 20:05:41.455453 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:41.455272    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8730d5-twrdd" event={"ID":"26953ffd-3365-40e8-ad23-6b32cca6e57e","Type":"ContainerStarted","Data":"1ad39898f29ee2b47ea506dcf34dcec51547436e0c9a2a36687a8bff46bcde86"}
Oct 31 20:05:41.455453 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:41.455354    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l" podUID="5cc42e0b-07f3-49f4-bb0b-8324751ea2e8" containerName="istio-proxy" containerID="cri-o://940af39b2e9946a3254be3fdd68594126537b9dc89696440c2a32857414513ef" gracePeriod=30
Oct 31 20:05:41.492354 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:41.492282    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8730d5-twrdd" podStartSLOduration=2.492266029 podStartE2EDuration="2.492266029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:05:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:05:41.489056436 +0000 UTC m=+2772.062290576" watchObservedRunningTime="2026-10-31 20:05:41.492266029 +0000 UTC m=+2772.065500162"
Oct 31 20:05:41.588050 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:41.588013    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:41.618967 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:41.618933    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f67e9a88-e1fd-4151-8023-8d44ce90e441" path="/var/lib/kubelet/pods/f67e9a88-e1fd-4151-8023-8d44ce90e441/volumes"
Oct 31 20:05:41.783353 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:41.774347    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:42.334045 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:42.333961    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc"
Oct 31 20:05:42.334512 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:42.334366    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc"
Oct 31 20:05:42.335201 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:42.335183    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc"
Oct 31 20:05:42.403570 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:42.403536    4645 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 20:05:42.403823 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:42.403774    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk" podUID="22d8af28-f4e5-47ea-a25a-9076b9afbc11" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 20:05:42.469859 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:42.469820    4645 generic.go:356] "Generic (PLEG): container finished" podID="26953ffd-3365-40e8-ad23-6b32cca6e57e" containerID="299ffbbdb82d1ca774d0af57189f340b71c58ff362090599ba92d558e76e6e38" exitCode=0
Oct 31 20:05:42.470082 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:42.470042    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8730d5-twrdd" event={"ID":"26953ffd-3365-40e8-ad23-6b32cca6e57e","Type":"ContainerDied","Data":"299ffbbdb82d1ca774d0af57189f340b71c58ff362090599ba92d558e76e6e38"}
Oct 31 20:05:42.899418 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:42.898094    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:43.437057 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:43.437017    4645 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 20:05:43.437485 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:43.437071    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz" podUID="0404b7e2-fd55-484a-bbec-3dd056db3e6b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 20:05:43.495028 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:43.494991    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9af3dc16-vxggv"]
Oct 31 20:05:43.495657 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:43.495639    4645 cpu_manager.go:414] "containerMap: removing container" podUID="f67e9a88-e1fd-4151-8023-8d44ce90e441" containerName="maas-api"
Oct 31 20:05:43.495657 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:43.495658    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f67e9a88-e1fd-4151-8023-8d44ce90e441" containerName="maas-api"
Oct 31 20:05:43.495792 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:43.495750    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="f67e9a88-e1fd-4151-8023-8d44ce90e441" containerName="maas-api"
Oct 31 20:05:43.499144 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:43.499125    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af3dc16-vxggv"
Oct 31 20:05:43.522699 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:43.522662    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9af3dc16-vxggv"]
Oct 31 20:05:43.527847 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:43.527810    4645 patch_prober.go:29] interesting pod/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 20:05:43.528101 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:43.527874    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l" podUID="5cc42e0b-07f3-49f4-bb0b-8324751ea2e8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 20:05:43.626978 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:43.626938    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fda00124-784f-4524-836b-ad9b3b9e23de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9af3dc16-vxggv\" (UID: \"fda00124-784f-4524-836b-ad9b3b9e23de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af3dc16-vxggv"
Oct 31 20:05:43.627244 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:43.627213    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd8t5\" (UniqueName: \"kubernetes.io/projected/fda00124-784f-4524-836b-ad9b3b9e23de-kube-api-access-qd8t5\") pod \"maas-api-revoke-sub-9af3dc16-vxggv\" (UID: \"fda00124-784f-4524-836b-ad9b3b9e23de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af3dc16-vxggv"
Oct 31 20:05:43.627360 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:43.627299    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fda00124-784f-4524-836b-ad9b3b9e23de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9af3dc16-vxggv\" (UID: \"fda00124-784f-4524-836b-ad9b3b9e23de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af3dc16-vxggv"
Oct 31 20:05:43.658651 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:43.658624    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8730d5-twrdd"
Oct 31 20:05:43.729462 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:43.729354    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fda00124-784f-4524-836b-ad9b3b9e23de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9af3dc16-vxggv\" (UID: \"fda00124-784f-4524-836b-ad9b3b9e23de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af3dc16-vxggv"
Oct 31 20:05:43.729638 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:43.729531    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fda00124-784f-4524-836b-ad9b3b9e23de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9af3dc16-vxggv\" (UID: \"fda00124-784f-4524-836b-ad9b3b9e23de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af3dc16-vxggv"
Oct 31 20:05:43.730163 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:43.730135    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fda00124-784f-4524-836b-ad9b3b9e23de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9af3dc16-vxggv\" (UID: \"fda00124-784f-4524-836b-ad9b3b9e23de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af3dc16-vxggv"
Oct 31 20:05:43.730288 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:43.730257    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd8t5\" (UniqueName: \"kubernetes.io/projected/fda00124-784f-4524-836b-ad9b3b9e23de-kube-api-access-qd8t5\") pod \"maas-api-revoke-sub-9af3dc16-vxggv\" (UID: \"fda00124-784f-4524-836b-ad9b3b9e23de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af3dc16-vxggv"
Oct 31 20:05:43.737014 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:43.736989    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fda00124-784f-4524-836b-ad9b3b9e23de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9af3dc16-vxggv\" (UID: \"fda00124-784f-4524-836b-ad9b3b9e23de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af3dc16-vxggv"
Oct 31 20:05:43.761478 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:43.761442    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd8t5\" (UniqueName: \"kubernetes.io/projected/fda00124-784f-4524-836b-ad9b3b9e23de-kube-api-access-qd8t5\") pod \"maas-api-revoke-sub-9af3dc16-vxggv\" (UID: \"fda00124-784f-4524-836b-ad9b3b9e23de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af3dc16-vxggv"
Oct 31 20:05:43.818929 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:43.818891    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af3dc16-vxggv"
Oct 31 20:05:43.820273 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:43.820245    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:43.831670 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:43.831642    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26953ffd-3365-40e8-ad23-6b32cca6e57e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26953ffd-3365-40e8-ad23-6b32cca6e57e-maas-api-xks-ca\") pod \"26953ffd-3365-40e8-ad23-6b32cca6e57e\" (UID: \"26953ffd-3365-40e8-ad23-6b32cca6e57e\") "
Oct 31 20:05:43.831778 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:43.831741    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26953ffd-3365-40e8-ad23-6b32cca6e57e-kube-api-access-cpqc4\" (UniqueName: \"kubernetes.io/projected/26953ffd-3365-40e8-ad23-6b32cca6e57e-kube-api-access-cpqc4\") pod \"26953ffd-3365-40e8-ad23-6b32cca6e57e\" (UID: \"26953ffd-3365-40e8-ad23-6b32cca6e57e\") "
Oct 31 20:05:43.831838 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:43.831782    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26953ffd-3365-40e8-ad23-6b32cca6e57e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26953ffd-3365-40e8-ad23-6b32cca6e57e-maas-api-openshift-ca\") pod \"26953ffd-3365-40e8-ad23-6b32cca6e57e\" (UID: \"26953ffd-3365-40e8-ad23-6b32cca6e57e\") "
Oct 31 20:05:43.832225 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:43.832195    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26953ffd-3365-40e8-ad23-6b32cca6e57e-maas-api-openshift-ca" pod "26953ffd-3365-40e8-ad23-6b32cca6e57e" (UID: "26953ffd-3365-40e8-ad23-6b32cca6e57e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:05:43.832364 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:43.832299    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26953ffd-3365-40e8-ad23-6b32cca6e57e-maas-api-openshift-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:43.833903 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:43.833880    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26953ffd-3365-40e8-ad23-6b32cca6e57e-kube-api-access-cpqc4" pod "26953ffd-3365-40e8-ad23-6b32cca6e57e" (UID: "26953ffd-3365-40e8-ad23-6b32cca6e57e"). InnerVolumeSpecName "kube-api-access-cpqc4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:05:43.834055 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:43.834029    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26953ffd-3365-40e8-ad23-6b32cca6e57e-maas-api-xks-ca" pod "26953ffd-3365-40e8-ad23-6b32cca6e57e" (UID: "26953ffd-3365-40e8-ad23-6b32cca6e57e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:05:43.931352 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:43.930410    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:43.935580 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:43.934608    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26953ffd-3365-40e8-ad23-6b32cca6e57e-maas-api-xks-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:43.935580 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:43.934646    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpqc4\" (UniqueName: \"kubernetes.io/projected/26953ffd-3365-40e8-ad23-6b32cca6e57e-kube-api-access-cpqc4\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:44.098160 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:44.098130    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9af3dc16-vxggv"]
Oct 31 20:05:44.099099 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:05:44.099073    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfda00124_784f_4524_836b_ad9b3b9e23de.slice/crio-35b994fa84a20be82e59d7e5652d1e6a02e827299d3f14b5719f792d141f0e66 WatchSource:0}: Error finding container 35b994fa84a20be82e59d7e5652d1e6a02e827299d3f14b5719f792d141f0e66: Status 404 returned error can't find the container with id 35b994fa84a20be82e59d7e5652d1e6a02e827299d3f14b5719f792d141f0e66
Oct 31 20:05:44.464363 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:44.461922    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl"
Oct 31 20:05:44.490200 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:44.490156    4645 generic.go:356] "Generic (PLEG): container finished" podID="92238ea2-f330-4873-ab87-37b37bb05f9a" containerID="2136df6ced5bec698141e9ae7567e6f1ac864a520bf01edcbbd4c907e8805305" exitCode=0
Oct 31 20:05:44.493297 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:44.490227    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl" event={"ID":"92238ea2-f330-4873-ab87-37b37bb05f9a","Type":"ContainerDied","Data":"2136df6ced5bec698141e9ae7567e6f1ac864a520bf01edcbbd4c907e8805305"}
Oct 31 20:05:44.493297 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:44.490405    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl" event={"ID":"92238ea2-f330-4873-ab87-37b37bb05f9a","Type":"ContainerDied","Data":"4f70efcb4979e4116b22b6fbc9b45643cd61a910bc882faf71654d4fb0aa2ef4"}
Oct 31 20:05:44.493297 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:44.490438    4645 scope.go:122] "RemoveContainer" containerID="2136df6ced5bec698141e9ae7567e6f1ac864a520bf01edcbbd4c907e8805305"
Oct 31 20:05:44.493297 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:44.490261    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl"
Oct 31 20:05:44.495231 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:44.493769    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af3dc16-vxggv" event={"ID":"fda00124-784f-4524-836b-ad9b3b9e23de","Type":"ContainerStarted","Data":"35b994fa84a20be82e59d7e5652d1e6a02e827299d3f14b5719f792d141f0e66"}
Oct 31 20:05:44.496123 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:44.496100    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8730d5-twrdd" event={"ID":"26953ffd-3365-40e8-ad23-6b32cca6e57e","Type":"ContainerDied","Data":"1ad39898f29ee2b47ea506dcf34dcec51547436e0c9a2a36687a8bff46bcde86"}
Oct 31 20:05:44.496591 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:44.496142    4645 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ad39898f29ee2b47ea506dcf34dcec51547436e0c9a2a36687a8bff46bcde86"
Oct 31 20:05:44.496591 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:44.496196    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8730d5-twrdd"
Oct 31 20:05:44.509810 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:44.509784    4645 scope.go:122] "RemoveContainer" containerID="809a98ee1208611517201f69b8c69c582c97c38049510312ccdfa1381982551d"
Oct 31 20:05:44.522392 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:44.522375    4645 scope.go:122] "RemoveContainer" containerID="2136df6ced5bec698141e9ae7567e6f1ac864a520bf01edcbbd4c907e8805305"
Oct 31 20:05:44.522631 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:05:44.522608    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2136df6ced5bec698141e9ae7567e6f1ac864a520bf01edcbbd4c907e8805305\": container with ID starting with 2136df6ced5bec698141e9ae7567e6f1ac864a520bf01edcbbd4c907e8805305 not found: ID does not exist" containerID="2136df6ced5bec698141e9ae7567e6f1ac864a520bf01edcbbd4c907e8805305"
Oct 31 20:05:44.522728 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:44.522636    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2136df6ced5bec698141e9ae7567e6f1ac864a520bf01edcbbd4c907e8805305"} err="failed to get container status \"2136df6ced5bec698141e9ae7567e6f1ac864a520bf01edcbbd4c907e8805305\": rpc error: code = NotFound desc = could not find container \"2136df6ced5bec698141e9ae7567e6f1ac864a520bf01edcbbd4c907e8805305\": container with ID starting with 2136df6ced5bec698141e9ae7567e6f1ac864a520bf01edcbbd4c907e8805305 not found: ID does not exist"
Oct 31 20:05:44.522728 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:44.522659    4645 scope.go:122] "RemoveContainer" containerID="809a98ee1208611517201f69b8c69c582c97c38049510312ccdfa1381982551d"
Oct 31 20:05:44.522909 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:05:44.522888    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"809a98ee1208611517201f69b8c69c582c97c38049510312ccdfa1381982551d\": container with ID starting with 809a98ee1208611517201f69b8c69c582c97c38049510312ccdfa1381982551d not found: ID does not exist" containerID="809a98ee1208611517201f69b8c69c582c97c38049510312ccdfa1381982551d"
Oct 31 20:05:44.522984 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:44.522913    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"809a98ee1208611517201f69b8c69c582c97c38049510312ccdfa1381982551d"} err="failed to get container status \"809a98ee1208611517201f69b8c69c582c97c38049510312ccdfa1381982551d\": rpc error: code = NotFound desc = could not find container \"809a98ee1208611517201f69b8c69c582c97c38049510312ccdfa1381982551d\": container with ID starting with 809a98ee1208611517201f69b8c69c582c97c38049510312ccdfa1381982551d not found: ID does not exist"
Oct 31 20:05:44.651846 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:44.651784    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/92238ea2-f330-4873-ab87-37b37bb05f9a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92238ea2-f330-4873-ab87-37b37bb05f9a-istio-podinfo\") pod \"92238ea2-f330-4873-ab87-37b37bb05f9a\" (UID: \"92238ea2-f330-4873-ab87-37b37bb05f9a\") "
Oct 31 20:05:44.652071 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:44.651886    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92238ea2-f330-4873-ab87-37b37bb05f9a-istio-token\" (UniqueName: \"kubernetes.io/projected/92238ea2-f330-4873-ab87-37b37bb05f9a-istio-token\") pod \"92238ea2-f330-4873-ab87-37b37bb05f9a\" (UID: \"92238ea2-f330-4873-ab87-37b37bb05f9a\") "
Oct 31 20:05:44.652071 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:44.651950    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92238ea2-f330-4873-ab87-37b37bb05f9a-kube-api-access-9z4rl\" (UniqueName: \"kubernetes.io/projected/92238ea2-f330-4873-ab87-37b37bb05f9a-kube-api-access-9z4rl\") pod \"92238ea2-f330-4873-ab87-37b37bb05f9a\" (UID: \"92238ea2-f330-4873-ab87-37b37bb05f9a\") "
Oct 31 20:05:44.652071 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:44.651996    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92238ea2-f330-4873-ab87-37b37bb05f9a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92238ea2-f330-4873-ab87-37b37bb05f9a-credential-socket\") pod \"92238ea2-f330-4873-ab87-37b37bb05f9a\" (UID: \"92238ea2-f330-4873-ab87-37b37bb05f9a\") "
Oct 31 20:05:44.653468 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:44.652125    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92238ea2-f330-4873-ab87-37b37bb05f9a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92238ea2-f330-4873-ab87-37b37bb05f9a-istio-data\") pod \"92238ea2-f330-4873-ab87-37b37bb05f9a\" (UID: \"92238ea2-f330-4873-ab87-37b37bb05f9a\") "
Oct 31 20:05:44.653468 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:44.652179    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92238ea2-f330-4873-ab87-37b37bb05f9a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92238ea2-f330-4873-ab87-37b37bb05f9a-istio-envoy\") pod \"92238ea2-f330-4873-ab87-37b37bb05f9a\" (UID: \"92238ea2-f330-4873-ab87-37b37bb05f9a\") "
Oct 31 20:05:44.653468 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:44.652209    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92238ea2-f330-4873-ab87-37b37bb05f9a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92238ea2-f330-4873-ab87-37b37bb05f9a-workload-certs\") pod \"92238ea2-f330-4873-ab87-37b37bb05f9a\" (UID: \"92238ea2-f330-4873-ab87-37b37bb05f9a\") "
Oct 31 20:05:44.653468 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:44.652273    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92238ea2-f330-4873-ab87-37b37bb05f9a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92238ea2-f330-4873-ab87-37b37bb05f9a-workload-socket\") pod \"92238ea2-f330-4873-ab87-37b37bb05f9a\" (UID: \"92238ea2-f330-4873-ab87-37b37bb05f9a\") "
Oct 31 20:05:44.653468 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:44.652348    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92238ea2-f330-4873-ab87-37b37bb05f9a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92238ea2-f330-4873-ab87-37b37bb05f9a-istiod-ca-cert\") pod \"92238ea2-f330-4873-ab87-37b37bb05f9a\" (UID: \"92238ea2-f330-4873-ab87-37b37bb05f9a\") "
Oct 31 20:05:44.653468 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:44.652597    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92238ea2-f330-4873-ab87-37b37bb05f9a-credential-socket" pod "92238ea2-f330-4873-ab87-37b37bb05f9a" (UID: "92238ea2-f330-4873-ab87-37b37bb05f9a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:44.653468 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:44.652684    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92238ea2-f330-4873-ab87-37b37bb05f9a-istio-data" pod "92238ea2-f330-4873-ab87-37b37bb05f9a" (UID: "92238ea2-f330-4873-ab87-37b37bb05f9a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:44.653468 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:44.652903    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92238ea2-f330-4873-ab87-37b37bb05f9a-workload-certs" pod "92238ea2-f330-4873-ab87-37b37bb05f9a" (UID: "92238ea2-f330-4873-ab87-37b37bb05f9a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:44.654027 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:44.653623    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92238ea2-f330-4873-ab87-37b37bb05f9a-istiod-ca-cert" pod "92238ea2-f330-4873-ab87-37b37bb05f9a" (UID: "92238ea2-f330-4873-ab87-37b37bb05f9a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:05:44.654027 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:44.653846    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92238ea2-f330-4873-ab87-37b37bb05f9a-workload-socket" pod "92238ea2-f330-4873-ab87-37b37bb05f9a" (UID: "92238ea2-f330-4873-ab87-37b37bb05f9a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:44.655287 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:44.654861    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92238ea2-f330-4873-ab87-37b37bb05f9a-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:44.655287 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:44.654888    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92238ea2-f330-4873-ab87-37b37bb05f9a-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:44.655287 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:44.654901    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92238ea2-f330-4873-ab87-37b37bb05f9a-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:44.655287 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:44.654914    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92238ea2-f330-4873-ab87-37b37bb05f9a-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:44.655287 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:44.654928    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92238ea2-f330-4873-ab87-37b37bb05f9a-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:44.658537 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:44.658512    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92238ea2-f330-4873-ab87-37b37bb05f9a-istio-token" pod "92238ea2-f330-4873-ab87-37b37bb05f9a" (UID: "92238ea2-f330-4873-ab87-37b37bb05f9a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:05:44.659869 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:44.659846    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/92238ea2-f330-4873-ab87-37b37bb05f9a-istio-podinfo" pod "92238ea2-f330-4873-ab87-37b37bb05f9a" (UID: "92238ea2-f330-4873-ab87-37b37bb05f9a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 20:05:44.659956 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:44.659910    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92238ea2-f330-4873-ab87-37b37bb05f9a-istio-envoy" pod "92238ea2-f330-4873-ab87-37b37bb05f9a" (UID: "92238ea2-f330-4873-ab87-37b37bb05f9a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:44.660187 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:44.660161    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92238ea2-f330-4873-ab87-37b37bb05f9a-kube-api-access-9z4rl" pod "92238ea2-f330-4873-ab87-37b37bb05f9a" (UID: "92238ea2-f330-4873-ab87-37b37bb05f9a"). InnerVolumeSpecName "kube-api-access-9z4rl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:05:44.728066 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:44.727955    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:44.756617 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:44.756578    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92238ea2-f330-4873-ab87-37b37bb05f9a-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:44.756617 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:44.756607    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92238ea2-f330-4873-ab87-37b37bb05f9a-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:44.756617 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:44.756621    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92238ea2-f330-4873-ab87-37b37bb05f9a-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:44.756858 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:44.756631    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9z4rl\" (UniqueName: \"kubernetes.io/projected/92238ea2-f330-4873-ab87-37b37bb05f9a-kube-api-access-9z4rl\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:44.824686 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:44.824639    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl"]
Oct 31 20:05:44.829268 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:44.829235    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e1d060-openshift-default-75d9d4df65-gzmxl"]
Oct 31 20:05:44.835528 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:44.835500    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:45.505472 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:45.505434    4645 generic.go:356] "Generic (PLEG): container finished" podID="fda00124-784f-4524-836b-ad9b3b9e23de" containerID="495811b6c6299ae635e112d62653cec9dc48321145f081ae586bbb2753759844" exitCode=0
Oct 31 20:05:45.505940 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:45.505508    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af3dc16-vxggv" event={"ID":"fda00124-784f-4524-836b-ad9b3b9e23de","Type":"ContainerDied","Data":"495811b6c6299ae635e112d62653cec9dc48321145f081ae586bbb2753759844"}
Oct 31 20:05:45.618174 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:45.618138    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92238ea2-f330-4873-ab87-37b37bb05f9a" path="/var/lib/kubelet/pods/92238ea2-f330-4873-ab87-37b37bb05f9a/volumes"
Oct 31 20:05:45.752159 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:45.752121    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:46.389557 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.389529    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz"
Oct 31 20:05:46.419280 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.419249    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk"
Oct 31 20:05:46.478577 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.478534    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0404b7e2-fd55-484a-bbec-3dd056db3e6b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0404b7e2-fd55-484a-bbec-3dd056db3e6b-credential-socket\") pod \"0404b7e2-fd55-484a-bbec-3dd056db3e6b\" (UID: \"0404b7e2-fd55-484a-bbec-3dd056db3e6b\") "
Oct 31 20:05:46.478760 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.478597    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0404b7e2-fd55-484a-bbec-3dd056db3e6b-kube-api-access-sxgcw\" (UniqueName: \"kubernetes.io/projected/0404b7e2-fd55-484a-bbec-3dd056db3e6b-kube-api-access-sxgcw\") pod \"0404b7e2-fd55-484a-bbec-3dd056db3e6b\" (UID: \"0404b7e2-fd55-484a-bbec-3dd056db3e6b\") "
Oct 31 20:05:46.478760 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.478635    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0404b7e2-fd55-484a-bbec-3dd056db3e6b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0404b7e2-fd55-484a-bbec-3dd056db3e6b-workload-certs\") pod \"0404b7e2-fd55-484a-bbec-3dd056db3e6b\" (UID: \"0404b7e2-fd55-484a-bbec-3dd056db3e6b\") "
Oct 31 20:05:46.478760 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.478668    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0404b7e2-fd55-484a-bbec-3dd056db3e6b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0404b7e2-fd55-484a-bbec-3dd056db3e6b-istio-envoy\") pod \"0404b7e2-fd55-484a-bbec-3dd056db3e6b\" (UID: \"0404b7e2-fd55-484a-bbec-3dd056db3e6b\") "
Oct 31 20:05:46.478957 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.478810    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0404b7e2-fd55-484a-bbec-3dd056db3e6b-credential-socket" pod "0404b7e2-fd55-484a-bbec-3dd056db3e6b" (UID: "0404b7e2-fd55-484a-bbec-3dd056db3e6b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:46.478957 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.478887    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22d8af28-f4e5-47ea-a25a-9076b9afbc11-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22d8af28-f4e5-47ea-a25a-9076b9afbc11-workload-certs\") pod \"22d8af28-f4e5-47ea-a25a-9076b9afbc11\" (UID: \"22d8af28-f4e5-47ea-a25a-9076b9afbc11\") "
Oct 31 20:05:46.478957 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.478923    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0404b7e2-fd55-484a-bbec-3dd056db3e6b-workload-certs" pod "0404b7e2-fd55-484a-bbec-3dd056db3e6b" (UID: "0404b7e2-fd55-484a-bbec-3dd056db3e6b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:46.479123 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.478975    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22d8af28-f4e5-47ea-a25a-9076b9afbc11-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22d8af28-f4e5-47ea-a25a-9076b9afbc11-istio-envoy\") pod \"22d8af28-f4e5-47ea-a25a-9076b9afbc11\" (UID: \"22d8af28-f4e5-47ea-a25a-9076b9afbc11\") "
Oct 31 20:05:46.479123 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.479012    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22d8af28-f4e5-47ea-a25a-9076b9afbc11-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22d8af28-f4e5-47ea-a25a-9076b9afbc11-workload-socket\") pod \"22d8af28-f4e5-47ea-a25a-9076b9afbc11\" (UID: \"22d8af28-f4e5-47ea-a25a-9076b9afbc11\") "
Oct 31 20:05:46.479123 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.479067    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22d8af28-f4e5-47ea-a25a-9076b9afbc11-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22d8af28-f4e5-47ea-a25a-9076b9afbc11-credential-socket\") pod \"22d8af28-f4e5-47ea-a25a-9076b9afbc11\" (UID: \"22d8af28-f4e5-47ea-a25a-9076b9afbc11\") "
Oct 31 20:05:46.479295 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.479115    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22d8af28-f4e5-47ea-a25a-9076b9afbc11-workload-certs" pod "22d8af28-f4e5-47ea-a25a-9076b9afbc11" (UID: "22d8af28-f4e5-47ea-a25a-9076b9afbc11"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:46.479295 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.479133    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22d8af28-f4e5-47ea-a25a-9076b9afbc11-istio-token\" (UniqueName: \"kubernetes.io/projected/22d8af28-f4e5-47ea-a25a-9076b9afbc11-istio-token\") pod \"22d8af28-f4e5-47ea-a25a-9076b9afbc11\" (UID: \"22d8af28-f4e5-47ea-a25a-9076b9afbc11\") "
Oct 31 20:05:46.479295 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.479158    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22d8af28-f4e5-47ea-a25a-9076b9afbc11-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22d8af28-f4e5-47ea-a25a-9076b9afbc11-istiod-ca-cert\") pod \"22d8af28-f4e5-47ea-a25a-9076b9afbc11\" (UID: \"22d8af28-f4e5-47ea-a25a-9076b9afbc11\") "
Oct 31 20:05:46.479295 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.479188    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/22d8af28-f4e5-47ea-a25a-9076b9afbc11-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22d8af28-f4e5-47ea-a25a-9076b9afbc11-istio-podinfo\") pod \"22d8af28-f4e5-47ea-a25a-9076b9afbc11\" (UID: \"22d8af28-f4e5-47ea-a25a-9076b9afbc11\") "
Oct 31 20:05:46.479295 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.479242    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0404b7e2-fd55-484a-bbec-3dd056db3e6b-istio-token\" (UniqueName: \"kubernetes.io/projected/0404b7e2-fd55-484a-bbec-3dd056db3e6b-istio-token\") pod \"0404b7e2-fd55-484a-bbec-3dd056db3e6b\" (UID: \"0404b7e2-fd55-484a-bbec-3dd056db3e6b\") "
Oct 31 20:05:46.481199 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.479276    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22d8af28-f4e5-47ea-a25a-9076b9afbc11-workload-socket" pod "22d8af28-f4e5-47ea-a25a-9076b9afbc11" (UID: "22d8af28-f4e5-47ea-a25a-9076b9afbc11"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:46.481199 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.479284    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0404b7e2-fd55-484a-bbec-3dd056db3e6b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0404b7e2-fd55-484a-bbec-3dd056db3e6b-istiod-ca-cert\") pod \"0404b7e2-fd55-484a-bbec-3dd056db3e6b\" (UID: \"0404b7e2-fd55-484a-bbec-3dd056db3e6b\") "
Oct 31 20:05:46.481199 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.479395    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22d8af28-f4e5-47ea-a25a-9076b9afbc11-kube-api-access-h62pb\" (UniqueName: \"kubernetes.io/projected/22d8af28-f4e5-47ea-a25a-9076b9afbc11-kube-api-access-h62pb\") pod \"22d8af28-f4e5-47ea-a25a-9076b9afbc11\" (UID: \"22d8af28-f4e5-47ea-a25a-9076b9afbc11\") "
Oct 31 20:05:46.481199 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.479427    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22d8af28-f4e5-47ea-a25a-9076b9afbc11-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22d8af28-f4e5-47ea-a25a-9076b9afbc11-istio-data\") pod \"22d8af28-f4e5-47ea-a25a-9076b9afbc11\" (UID: \"22d8af28-f4e5-47ea-a25a-9076b9afbc11\") "
Oct 31 20:05:46.481199 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.479470    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0404b7e2-fd55-484a-bbec-3dd056db3e6b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0404b7e2-fd55-484a-bbec-3dd056db3e6b-istio-podinfo\") pod \"0404b7e2-fd55-484a-bbec-3dd056db3e6b\" (UID: \"0404b7e2-fd55-484a-bbec-3dd056db3e6b\") "
Oct 31 20:05:46.481199 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.479511    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0404b7e2-fd55-484a-bbec-3dd056db3e6b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0404b7e2-fd55-484a-bbec-3dd056db3e6b-workload-socket\") pod \"0404b7e2-fd55-484a-bbec-3dd056db3e6b\" (UID: \"0404b7e2-fd55-484a-bbec-3dd056db3e6b\") "
Oct 31 20:05:46.481199 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.479657    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0404b7e2-fd55-484a-bbec-3dd056db3e6b-istiod-ca-cert" pod "0404b7e2-fd55-484a-bbec-3dd056db3e6b" (UID: "0404b7e2-fd55-484a-bbec-3dd056db3e6b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:05:46.481199 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.479717    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0404b7e2-fd55-484a-bbec-3dd056db3e6b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0404b7e2-fd55-484a-bbec-3dd056db3e6b-istio-data\") pod \"0404b7e2-fd55-484a-bbec-3dd056db3e6b\" (UID: \"0404b7e2-fd55-484a-bbec-3dd056db3e6b\") "
Oct 31 20:05:46.481199 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.479357    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22d8af28-f4e5-47ea-a25a-9076b9afbc11-credential-socket" pod "22d8af28-f4e5-47ea-a25a-9076b9afbc11" (UID: "22d8af28-f4e5-47ea-a25a-9076b9afbc11"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:46.481199 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.480149    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22d8af28-f4e5-47ea-a25a-9076b9afbc11-istio-data" pod "22d8af28-f4e5-47ea-a25a-9076b9afbc11" (UID: "22d8af28-f4e5-47ea-a25a-9076b9afbc11"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:46.481199 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.480532    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22d8af28-f4e5-47ea-a25a-9076b9afbc11-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:46.481199 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.480559    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22d8af28-f4e5-47ea-a25a-9076b9afbc11-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:46.481199 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.480574    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0404b7e2-fd55-484a-bbec-3dd056db3e6b-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:46.481199 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.480587    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22d8af28-f4e5-47ea-a25a-9076b9afbc11-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:46.481199 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.480601    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0404b7e2-fd55-484a-bbec-3dd056db3e6b-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:46.481199 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.480615    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0404b7e2-fd55-484a-bbec-3dd056db3e6b-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:46.481199 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.480628    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22d8af28-f4e5-47ea-a25a-9076b9afbc11-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:46.481199 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.480809    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22d8af28-f4e5-47ea-a25a-9076b9afbc11-istiod-ca-cert" pod "22d8af28-f4e5-47ea-a25a-9076b9afbc11" (UID: "22d8af28-f4e5-47ea-a25a-9076b9afbc11"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:05:46.481199 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.480890    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0404b7e2-fd55-484a-bbec-3dd056db3e6b-workload-socket" pod "0404b7e2-fd55-484a-bbec-3dd056db3e6b" (UID: "0404b7e2-fd55-484a-bbec-3dd056db3e6b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:46.481199 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.481086    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0404b7e2-fd55-484a-bbec-3dd056db3e6b-istio-data" pod "0404b7e2-fd55-484a-bbec-3dd056db3e6b" (UID: "0404b7e2-fd55-484a-bbec-3dd056db3e6b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:46.483214 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.483024    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0404b7e2-fd55-484a-bbec-3dd056db3e6b-istio-envoy" pod "0404b7e2-fd55-484a-bbec-3dd056db3e6b" (UID: "0404b7e2-fd55-484a-bbec-3dd056db3e6b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:46.483321 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.483258    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0404b7e2-fd55-484a-bbec-3dd056db3e6b-kube-api-access-sxgcw" pod "0404b7e2-fd55-484a-bbec-3dd056db3e6b" (UID: "0404b7e2-fd55-484a-bbec-3dd056db3e6b"). InnerVolumeSpecName "kube-api-access-sxgcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:05:46.483603 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.483578    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/22d8af28-f4e5-47ea-a25a-9076b9afbc11-istio-podinfo" pod "22d8af28-f4e5-47ea-a25a-9076b9afbc11" (UID: "22d8af28-f4e5-47ea-a25a-9076b9afbc11"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 20:05:46.484616 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.484592    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0404b7e2-fd55-484a-bbec-3dd056db3e6b-istio-token" pod "0404b7e2-fd55-484a-bbec-3dd056db3e6b" (UID: "0404b7e2-fd55-484a-bbec-3dd056db3e6b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:05:46.484861 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.484805    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22d8af28-f4e5-47ea-a25a-9076b9afbc11-kube-api-access-h62pb" pod "22d8af28-f4e5-47ea-a25a-9076b9afbc11" (UID: "22d8af28-f4e5-47ea-a25a-9076b9afbc11"). InnerVolumeSpecName "kube-api-access-h62pb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:05:46.484985 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.484915    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22d8af28-f4e5-47ea-a25a-9076b9afbc11-istio-token" pod "22d8af28-f4e5-47ea-a25a-9076b9afbc11" (UID: "22d8af28-f4e5-47ea-a25a-9076b9afbc11"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:05:46.485208 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.485186    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0404b7e2-fd55-484a-bbec-3dd056db3e6b-istio-podinfo" pod "0404b7e2-fd55-484a-bbec-3dd056db3e6b" (UID: "0404b7e2-fd55-484a-bbec-3dd056db3e6b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 20:05:46.485356 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.485310    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22d8af28-f4e5-47ea-a25a-9076b9afbc11-istio-envoy" pod "22d8af28-f4e5-47ea-a25a-9076b9afbc11" (UID: "22d8af28-f4e5-47ea-a25a-9076b9afbc11"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:46.515076 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.515046    4645 generic.go:356] "Generic (PLEG): container finished" podID="0404b7e2-fd55-484a-bbec-3dd056db3e6b" containerID="aebbcc9192682a9b18b6a94a4f8f5ec8f3f244bdbb2bfb2a5db987db8094deb5" exitCode=0
Oct 31 20:05:46.515472 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.515110    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz"
Oct 31 20:05:46.515472 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.515128    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz" event={"ID":"0404b7e2-fd55-484a-bbec-3dd056db3e6b","Type":"ContainerDied","Data":"aebbcc9192682a9b18b6a94a4f8f5ec8f3f244bdbb2bfb2a5db987db8094deb5"}
Oct 31 20:05:46.515472 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.515191    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz" event={"ID":"0404b7e2-fd55-484a-bbec-3dd056db3e6b","Type":"ContainerDied","Data":"f79ddf9dcee6156078a69efc81842aeed1dcf53b9b86dc98576e67a20138ab54"}
Oct 31 20:05:46.515472 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.515224    4645 scope.go:122] "RemoveContainer" containerID="aebbcc9192682a9b18b6a94a4f8f5ec8f3f244bdbb2bfb2a5db987db8094deb5"
Oct 31 20:05:46.516819 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.516800    4645 generic.go:356] "Generic (PLEG): container finished" podID="22d8af28-f4e5-47ea-a25a-9076b9afbc11" containerID="9490917c1e492122036f9de619e640b0911012219d2c82f4dcf059fb9e3808f0" exitCode=0
Oct 31 20:05:46.517021 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.517005    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk"
Oct 31 20:05:46.520120 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.519184    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk" event={"ID":"22d8af28-f4e5-47ea-a25a-9076b9afbc11","Type":"ContainerDied","Data":"9490917c1e492122036f9de619e640b0911012219d2c82f4dcf059fb9e3808f0"}
Oct 31 20:05:46.520120 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.519234    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk" event={"ID":"22d8af28-f4e5-47ea-a25a-9076b9afbc11","Type":"ContainerDied","Data":"3d54ceb5ab03e959f991cc3213f4c901ef9d688116a2c8f4626537e59cbfbbcc"}
Oct 31 20:05:46.568676 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.568550    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz"]
Oct 31 20:05:46.573426 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.572977    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw"]
Oct 31 20:05:46.582099 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.581883    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxgcw\" (UniqueName: \"kubernetes.io/projected/0404b7e2-fd55-484a-bbec-3dd056db3e6b-kube-api-access-sxgcw\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:46.582099 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.581917    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0404b7e2-fd55-484a-bbec-3dd056db3e6b-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:46.582099 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.581933    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22d8af28-f4e5-47ea-a25a-9076b9afbc11-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:46.582099 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.581947    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22d8af28-f4e5-47ea-a25a-9076b9afbc11-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:46.582099 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.581960    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22d8af28-f4e5-47ea-a25a-9076b9afbc11-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:46.582099 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.581973    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22d8af28-f4e5-47ea-a25a-9076b9afbc11-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:46.582099 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.581988    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0404b7e2-fd55-484a-bbec-3dd056db3e6b-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:46.582099 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.582034    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h62pb\" (UniqueName: \"kubernetes.io/projected/22d8af28-f4e5-47ea-a25a-9076b9afbc11-kube-api-access-h62pb\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:46.582099 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.582049    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0404b7e2-fd55-484a-bbec-3dd056db3e6b-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:46.582099 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.582063    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0404b7e2-fd55-484a-bbec-3dd056db3e6b-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:46.582099 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.582076    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0404b7e2-fd55-484a-bbec-3dd056db3e6b-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:46.593721 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.583122    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-s75qz"]
Oct 31 20:05:46.608962 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.608681    4645 scope.go:122] "RemoveContainer" containerID="aebbcc9192682a9b18b6a94a4f8f5ec8f3f244bdbb2bfb2a5db987db8094deb5"
Oct 31 20:05:46.609222 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:05:46.609099    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aebbcc9192682a9b18b6a94a4f8f5ec8f3f244bdbb2bfb2a5db987db8094deb5\": container with ID starting with aebbcc9192682a9b18b6a94a4f8f5ec8f3f244bdbb2bfb2a5db987db8094deb5 not found: ID does not exist" containerID="aebbcc9192682a9b18b6a94a4f8f5ec8f3f244bdbb2bfb2a5db987db8094deb5"
Oct 31 20:05:46.609222 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.609140    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aebbcc9192682a9b18b6a94a4f8f5ec8f3f244bdbb2bfb2a5db987db8094deb5"} err="failed to get container status \"aebbcc9192682a9b18b6a94a4f8f5ec8f3f244bdbb2bfb2a5db987db8094deb5\": rpc error: code = NotFound desc = could not find container \"aebbcc9192682a9b18b6a94a4f8f5ec8f3f244bdbb2bfb2a5db987db8094deb5\": container with ID starting with aebbcc9192682a9b18b6a94a4f8f5ec8f3f244bdbb2bfb2a5db987db8094deb5 not found: ID does not exist"
Oct 31 20:05:46.609222 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.609203    4645 scope.go:122] "RemoveContainer" containerID="9490917c1e492122036f9de619e640b0911012219d2c82f4dcf059fb9e3808f0"
Oct 31 20:05:46.621866 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.621843    4645 scope.go:122] "RemoveContainer" containerID="9490917c1e492122036f9de619e640b0911012219d2c82f4dcf059fb9e3808f0"
Oct 31 20:05:46.622357 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:05:46.622252    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9490917c1e492122036f9de619e640b0911012219d2c82f4dcf059fb9e3808f0\": container with ID starting with 9490917c1e492122036f9de619e640b0911012219d2c82f4dcf059fb9e3808f0 not found: ID does not exist" containerID="9490917c1e492122036f9de619e640b0911012219d2c82f4dcf059fb9e3808f0"
Oct 31 20:05:46.622357 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.622290    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9490917c1e492122036f9de619e640b0911012219d2c82f4dcf059fb9e3808f0"} err="failed to get container status \"9490917c1e492122036f9de619e640b0911012219d2c82f4dcf059fb9e3808f0\": rpc error: code = NotFound desc = could not find container \"9490917c1e492122036f9de619e640b0911012219d2c82f4dcf059fb9e3808f0\": container with ID starting with 9490917c1e492122036f9de619e640b0911012219d2c82f4dcf059fb9e3808f0 not found: ID does not exist"
Oct 31 20:05:46.646243 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.646216    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk"]
Oct 31 20:05:46.652578 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.652549    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-bqfpk"]
Oct 31 20:05:46.686022 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.685925    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:46.721835 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.721809    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af3dc16-vxggv"
Oct 31 20:05:46.832460 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.832431    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l"
Oct 31 20:05:46.854914 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.854887    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:46.925763 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.925726    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cc42e0b-07f3-49f4-bb0b-8324751ea2e8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cc42e0b-07f3-49f4-bb0b-8324751ea2e8-istiod-ca-cert\") pod \"5cc42e0b-07f3-49f4-bb0b-8324751ea2e8\" (UID: \"5cc42e0b-07f3-49f4-bb0b-8324751ea2e8\") "
Oct 31 20:05:46.925941 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.925785    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cc42e0b-07f3-49f4-bb0b-8324751ea2e8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cc42e0b-07f3-49f4-bb0b-8324751ea2e8-workload-socket\") pod \"5cc42e0b-07f3-49f4-bb0b-8324751ea2e8\" (UID: \"5cc42e0b-07f3-49f4-bb0b-8324751ea2e8\") "
Oct 31 20:05:46.925941 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.925863    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cc42e0b-07f3-49f4-bb0b-8324751ea2e8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cc42e0b-07f3-49f4-bb0b-8324751ea2e8-istio-data\") pod \"5cc42e0b-07f3-49f4-bb0b-8324751ea2e8\" (UID: \"5cc42e0b-07f3-49f4-bb0b-8324751ea2e8\") "
Oct 31 20:05:46.925941 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.925915    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cc42e0b-07f3-49f4-bb0b-8324751ea2e8-kube-api-access-9s4qf\" (UniqueName: \"kubernetes.io/projected/5cc42e0b-07f3-49f4-bb0b-8324751ea2e8-kube-api-access-9s4qf\") pod \"5cc42e0b-07f3-49f4-bb0b-8324751ea2e8\" (UID: \"5cc42e0b-07f3-49f4-bb0b-8324751ea2e8\") "
Oct 31 20:05:46.926128 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.925985    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cc42e0b-07f3-49f4-bb0b-8324751ea2e8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cc42e0b-07f3-49f4-bb0b-8324751ea2e8-istio-envoy\") pod \"5cc42e0b-07f3-49f4-bb0b-8324751ea2e8\" (UID: \"5cc42e0b-07f3-49f4-bb0b-8324751ea2e8\") "
Oct 31 20:05:46.926128 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.926024    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fda00124-784f-4524-836b-ad9b3b9e23de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fda00124-784f-4524-836b-ad9b3b9e23de-maas-api-xks-ca\") pod \"fda00124-784f-4524-836b-ad9b3b9e23de\" (UID: \"fda00124-784f-4524-836b-ad9b3b9e23de\") "
Oct 31 20:05:46.926128 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.926061    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cc42e0b-07f3-49f4-bb0b-8324751ea2e8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cc42e0b-07f3-49f4-bb0b-8324751ea2e8-credential-socket\") pod \"5cc42e0b-07f3-49f4-bb0b-8324751ea2e8\" (UID: \"5cc42e0b-07f3-49f4-bb0b-8324751ea2e8\") "
Oct 31 20:05:46.926128 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.926113    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5cc42e0b-07f3-49f4-bb0b-8324751ea2e8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cc42e0b-07f3-49f4-bb0b-8324751ea2e8-istio-podinfo\") pod \"5cc42e0b-07f3-49f4-bb0b-8324751ea2e8\" (UID: \"5cc42e0b-07f3-49f4-bb0b-8324751ea2e8\") "
Oct 31 20:05:46.926366 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.926119    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cc42e0b-07f3-49f4-bb0b-8324751ea2e8-workload-socket" pod "5cc42e0b-07f3-49f4-bb0b-8324751ea2e8" (UID: "5cc42e0b-07f3-49f4-bb0b-8324751ea2e8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:46.926366 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.926159    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fda00124-784f-4524-836b-ad9b3b9e23de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fda00124-784f-4524-836b-ad9b3b9e23de-maas-api-openshift-ca\") pod \"fda00124-784f-4524-836b-ad9b3b9e23de\" (UID: \"fda00124-784f-4524-836b-ad9b3b9e23de\") "
Oct 31 20:05:46.926366 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.926207    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fda00124-784f-4524-836b-ad9b3b9e23de-kube-api-access-qd8t5\" (UniqueName: \"kubernetes.io/projected/fda00124-784f-4524-836b-ad9b3b9e23de-kube-api-access-qd8t5\") pod \"fda00124-784f-4524-836b-ad9b3b9e23de\" (UID: \"fda00124-784f-4524-836b-ad9b3b9e23de\") "
Oct 31 20:05:46.926366 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.926254    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cc42e0b-07f3-49f4-bb0b-8324751ea2e8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cc42e0b-07f3-49f4-bb0b-8324751ea2e8-workload-certs\") pod \"5cc42e0b-07f3-49f4-bb0b-8324751ea2e8\" (UID: \"5cc42e0b-07f3-49f4-bb0b-8324751ea2e8\") "
Oct 31 20:05:46.926366 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.926268    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cc42e0b-07f3-49f4-bb0b-8324751ea2e8-istio-data" pod "5cc42e0b-07f3-49f4-bb0b-8324751ea2e8" (UID: "5cc42e0b-07f3-49f4-bb0b-8324751ea2e8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:46.926783 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.926366    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cc42e0b-07f3-49f4-bb0b-8324751ea2e8-istio-token\" (UniqueName: \"kubernetes.io/projected/5cc42e0b-07f3-49f4-bb0b-8324751ea2e8-istio-token\") pod \"5cc42e0b-07f3-49f4-bb0b-8324751ea2e8\" (UID: \"5cc42e0b-07f3-49f4-bb0b-8324751ea2e8\") "
Oct 31 20:05:46.926783 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.926319    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cc42e0b-07f3-49f4-bb0b-8324751ea2e8-istiod-ca-cert" pod "5cc42e0b-07f3-49f4-bb0b-8324751ea2e8" (UID: "5cc42e0b-07f3-49f4-bb0b-8324751ea2e8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:05:46.926783 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.926435    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cc42e0b-07f3-49f4-bb0b-8324751ea2e8-credential-socket" pod "5cc42e0b-07f3-49f4-bb0b-8324751ea2e8" (UID: "5cc42e0b-07f3-49f4-bb0b-8324751ea2e8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:46.926783 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.926720    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cc42e0b-07f3-49f4-bb0b-8324751ea2e8-workload-certs" pod "5cc42e0b-07f3-49f4-bb0b-8324751ea2e8" (UID: "5cc42e0b-07f3-49f4-bb0b-8324751ea2e8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:46.926783 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.926735    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fda00124-784f-4524-836b-ad9b3b9e23de-maas-api-openshift-ca" pod "fda00124-784f-4524-836b-ad9b3b9e23de" (UID: "fda00124-784f-4524-836b-ad9b3b9e23de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:05:46.927092 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.926983    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cc42e0b-07f3-49f4-bb0b-8324751ea2e8-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:46.927092 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.927006    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fda00124-784f-4524-836b-ad9b3b9e23de-maas-api-openshift-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:46.927092 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.927023    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cc42e0b-07f3-49f4-bb0b-8324751ea2e8-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:46.927092 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.927038    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cc42e0b-07f3-49f4-bb0b-8324751ea2e8-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:46.927092 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.927053    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cc42e0b-07f3-49f4-bb0b-8324751ea2e8-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:46.927092 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.927067    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cc42e0b-07f3-49f4-bb0b-8324751ea2e8-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:46.928875 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.928846    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5cc42e0b-07f3-49f4-bb0b-8324751ea2e8-istio-podinfo" pod "5cc42e0b-07f3-49f4-bb0b-8324751ea2e8" (UID: "5cc42e0b-07f3-49f4-bb0b-8324751ea2e8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 20:05:46.929008 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.928853    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cc42e0b-07f3-49f4-bb0b-8324751ea2e8-istio-envoy" pod "5cc42e0b-07f3-49f4-bb0b-8324751ea2e8" (UID: "5cc42e0b-07f3-49f4-bb0b-8324751ea2e8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:05:46.929226 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.929202    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cc42e0b-07f3-49f4-bb0b-8324751ea2e8-kube-api-access-9s4qf" pod "5cc42e0b-07f3-49f4-bb0b-8324751ea2e8" (UID: "5cc42e0b-07f3-49f4-bb0b-8324751ea2e8"). InnerVolumeSpecName "kube-api-access-9s4qf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:05:46.929834 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.929809    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fda00124-784f-4524-836b-ad9b3b9e23de-kube-api-access-qd8t5" pod "fda00124-784f-4524-836b-ad9b3b9e23de" (UID: "fda00124-784f-4524-836b-ad9b3b9e23de"). InnerVolumeSpecName "kube-api-access-qd8t5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:05:46.929919 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.929839    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cc42e0b-07f3-49f4-bb0b-8324751ea2e8-istio-token" pod "5cc42e0b-07f3-49f4-bb0b-8324751ea2e8" (UID: "5cc42e0b-07f3-49f4-bb0b-8324751ea2e8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:05:46.929969 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:46.929941    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fda00124-784f-4524-836b-ad9b3b9e23de-maas-api-xks-ca" pod "fda00124-784f-4524-836b-ad9b3b9e23de" (UID: "fda00124-784f-4524-836b-ad9b3b9e23de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:05:47.027773 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:47.027741    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9s4qf\" (UniqueName: \"kubernetes.io/projected/5cc42e0b-07f3-49f4-bb0b-8324751ea2e8-kube-api-access-9s4qf\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:47.027773 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:47.027770    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cc42e0b-07f3-49f4-bb0b-8324751ea2e8-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:47.027950 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:47.027785    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fda00124-784f-4524-836b-ad9b3b9e23de-maas-api-xks-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:47.027950 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:47.027797    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cc42e0b-07f3-49f4-bb0b-8324751ea2e8-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:47.027950 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:47.027812    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qd8t5\" (UniqueName: \"kubernetes.io/projected/fda00124-784f-4524-836b-ad9b3b9e23de-kube-api-access-qd8t5\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:47.027950 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:47.027827    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cc42e0b-07f3-49f4-bb0b-8324751ea2e8-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:47.527020 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:47.526985    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af3dc16-vxggv"
Oct 31 20:05:47.527461 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:47.526983    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9af3dc16-vxggv" event={"ID":"fda00124-784f-4524-836b-ad9b3b9e23de","Type":"ContainerDied","Data":"35b994fa84a20be82e59d7e5652d1e6a02e827299d3f14b5719f792d141f0e66"}
Oct 31 20:05:47.527461 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:47.527138    4645 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35b994fa84a20be82e59d7e5652d1e6a02e827299d3f14b5719f792d141f0e66"
Oct 31 20:05:47.528512 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:47.528483    4645 generic.go:356] "Generic (PLEG): container finished" podID="5cc42e0b-07f3-49f4-bb0b-8324751ea2e8" containerID="940af39b2e9946a3254be3fdd68594126537b9dc89696440c2a32857414513ef" exitCode=0
Oct 31 20:05:47.528658 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:47.528529    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l"
Oct 31 20:05:47.528658 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:47.528561    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l" event={"ID":"5cc42e0b-07f3-49f4-bb0b-8324751ea2e8","Type":"ContainerDied","Data":"940af39b2e9946a3254be3fdd68594126537b9dc89696440c2a32857414513ef"}
Oct 31 20:05:47.528658 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:47.528609    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l" event={"ID":"5cc42e0b-07f3-49f4-bb0b-8324751ea2e8","Type":"ContainerDied","Data":"6bb672ef664c1120f0014e25204e8d28c146cacdd873deb091174fb52a7979b4"}
Oct 31 20:05:47.528658 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:47.528638    4645 scope.go:122] "RemoveContainer" containerID="940af39b2e9946a3254be3fdd68594126537b9dc89696440c2a32857414513ef"
Oct 31 20:05:47.539426 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:47.539394    4645 scope.go:122] "RemoveContainer" containerID="940af39b2e9946a3254be3fdd68594126537b9dc89696440c2a32857414513ef"
Oct 31 20:05:47.539697 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:05:47.539675    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"940af39b2e9946a3254be3fdd68594126537b9dc89696440c2a32857414513ef\": container with ID starting with 940af39b2e9946a3254be3fdd68594126537b9dc89696440c2a32857414513ef not found: ID does not exist" containerID="940af39b2e9946a3254be3fdd68594126537b9dc89696440c2a32857414513ef"
Oct 31 20:05:47.539757 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:47.539704    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"940af39b2e9946a3254be3fdd68594126537b9dc89696440c2a32857414513ef"} err="failed to get container status \"940af39b2e9946a3254be3fdd68594126537b9dc89696440c2a32857414513ef\": rpc error: code = NotFound desc = could not find container \"940af39b2e9946a3254be3fdd68594126537b9dc89696440c2a32857414513ef\": container with ID starting with 940af39b2e9946a3254be3fdd68594126537b9dc89696440c2a32857414513ef not found: ID does not exist"
Oct 31 20:05:47.558750 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:47.558721    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l"]
Oct 31 20:05:47.564085 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:47.564062    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-ae2e60-openshift-default-5787499c47-v7d6l"]
Oct 31 20:05:47.617942 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:47.617914    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0404b7e2-fd55-484a-bbec-3dd056db3e6b" path="/var/lib/kubelet/pods/0404b7e2-fd55-484a-bbec-3dd056db3e6b/volumes"
Oct 31 20:05:47.618475 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:47.618459    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22d8af28-f4e5-47ea-a25a-9076b9afbc11" path="/var/lib/kubelet/pods/22d8af28-f4e5-47ea-a25a-9076b9afbc11/volumes"
Oct 31 20:05:47.618918 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:47.618904    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cc42e0b-07f3-49f4-bb0b-8324751ea2e8" path="/var/lib/kubelet/pods/5cc42e0b-07f3-49f4-bb0b-8324751ea2e8/volumes"
Oct 31 20:05:47.821796 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:47.818350    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:48.471376 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:48.471323    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:48.576216 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:48.576180    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:48.718374 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:48.718310    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:49.040310 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:49.040271    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:49.147980 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:49.147950    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:49.438571 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:49.438536    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:49.841463 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:49.841371    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:50.191187 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:50.191142    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:50.329588 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:50.329554    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:50.589986 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:50.589887    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:51.558901 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:51.558865    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:51.767996 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:51.767949    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:51.994024 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:51.993989    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:52.107714 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:52.107662    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:52.461313 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:52.461280    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dc93a5b-l8q72"]
Oct 31 20:05:52.461832 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:52.461818    4645 cpu_manager.go:414] "containerMap: removing container" podUID="5cc42e0b-07f3-49f4-bb0b-8324751ea2e8" containerName="istio-proxy"
Oct 31 20:05:52.461892 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:52.461833    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cc42e0b-07f3-49f4-bb0b-8324751ea2e8" containerName="istio-proxy"
Oct 31 20:05:52.461892 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:52.461855    4645 cpu_manager.go:414] "containerMap: removing container" podUID="0404b7e2-fd55-484a-bbec-3dd056db3e6b" containerName="istio-proxy"
Oct 31 20:05:52.461892 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:52.461861    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0404b7e2-fd55-484a-bbec-3dd056db3e6b" containerName="istio-proxy"
Oct 31 20:05:52.461892 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:52.461873    4645 cpu_manager.go:414] "containerMap: removing container" podUID="26953ffd-3365-40e8-ad23-6b32cca6e57e" containerName="revoke-keys"
Oct 31 20:05:52.461892 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:52.461878    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26953ffd-3365-40e8-ad23-6b32cca6e57e" containerName="revoke-keys"
Oct 31 20:05:52.461892 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:52.461889    4645 cpu_manager.go:414] "containerMap: removing container" podUID="22d8af28-f4e5-47ea-a25a-9076b9afbc11" containerName="istio-proxy"
Oct 31 20:05:52.461892 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:52.461894    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22d8af28-f4e5-47ea-a25a-9076b9afbc11" containerName="istio-proxy"
Oct 31 20:05:52.462162 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:52.461905    4645 cpu_manager.go:414] "containerMap: removing container" podUID="92238ea2-f330-4873-ab87-37b37bb05f9a" containerName="istio-proxy"
Oct 31 20:05:52.462162 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:52.461911    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92238ea2-f330-4873-ab87-37b37bb05f9a" containerName="istio-proxy"
Oct 31 20:05:52.462162 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:52.461919    4645 cpu_manager.go:414] "containerMap: removing container" podUID="92238ea2-f330-4873-ab87-37b37bb05f9a" containerName="istio-proxy"
Oct 31 20:05:52.462162 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:52.461924    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92238ea2-f330-4873-ab87-37b37bb05f9a" containerName="istio-proxy"
Oct 31 20:05:52.462162 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:52.461935    4645 cpu_manager.go:414] "containerMap: removing container" podUID="fda00124-784f-4524-836b-ad9b3b9e23de" containerName="revoke-keys"
Oct 31 20:05:52.462162 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:52.461939    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fda00124-784f-4524-836b-ad9b3b9e23de" containerName="revoke-keys"
Oct 31 20:05:52.462162 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:52.462007    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="0404b7e2-fd55-484a-bbec-3dd056db3e6b" containerName="istio-proxy"
Oct 31 20:05:52.462162 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:52.462019    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="26953ffd-3365-40e8-ad23-6b32cca6e57e" containerName="revoke-keys"
Oct 31 20:05:52.462162 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:52.462027    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="22d8af28-f4e5-47ea-a25a-9076b9afbc11" containerName="istio-proxy"
Oct 31 20:05:52.462162 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:52.462035    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="92238ea2-f330-4873-ab87-37b37bb05f9a" containerName="istio-proxy"
Oct 31 20:05:52.462162 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:52.462040    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="92238ea2-f330-4873-ab87-37b37bb05f9a" containerName="istio-proxy"
Oct 31 20:05:52.462162 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:52.462046    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="fda00124-784f-4524-836b-ad9b3b9e23de" containerName="revoke-keys"
Oct 31 20:05:52.462162 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:52.462054    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cc42e0b-07f3-49f4-bb0b-8324751ea2e8" containerName="istio-proxy"
Oct 31 20:05:52.467009 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:52.466990    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dc93a5b-l8q72"
Oct 31 20:05:52.476592 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:52.476570    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dc93a5b-l8q72"]
Oct 31 20:05:52.477033 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:52.476989    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:05:52.477309 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:52.477288    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 20:05:52.556023 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:52.555307    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48365630-1b94-419c-a248-39fd40ffb067-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1dc93a5b-l8q72\" (UID: \"48365630-1b94-419c-a248-39fd40ffb067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dc93a5b-l8q72"
Oct 31 20:05:52.556181 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:52.556099    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48365630-1b94-419c-a248-39fd40ffb067-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1dc93a5b-l8q72\" (UID: \"48365630-1b94-419c-a248-39fd40ffb067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dc93a5b-l8q72"
Oct 31 20:05:52.556260 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:52.556246    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gzks\" (UniqueName: \"kubernetes.io/projected/48365630-1b94-419c-a248-39fd40ffb067-kube-api-access-6gzks\") pod \"maas-api-revoke-sub-1dc93a5b-l8q72\" (UID: \"48365630-1b94-419c-a248-39fd40ffb067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dc93a5b-l8q72"
Oct 31 20:05:52.656992 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:52.656960    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gzks\" (UniqueName: \"kubernetes.io/projected/48365630-1b94-419c-a248-39fd40ffb067-kube-api-access-6gzks\") pod \"maas-api-revoke-sub-1dc93a5b-l8q72\" (UID: \"48365630-1b94-419c-a248-39fd40ffb067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dc93a5b-l8q72"
Oct 31 20:05:52.657377 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:52.657075    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48365630-1b94-419c-a248-39fd40ffb067-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1dc93a5b-l8q72\" (UID: \"48365630-1b94-419c-a248-39fd40ffb067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dc93a5b-l8q72"
Oct 31 20:05:52.657377 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:52.657100    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48365630-1b94-419c-a248-39fd40ffb067-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1dc93a5b-l8q72\" (UID: \"48365630-1b94-419c-a248-39fd40ffb067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dc93a5b-l8q72"
Oct 31 20:05:52.657799 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:52.657770    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48365630-1b94-419c-a248-39fd40ffb067-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1dc93a5b-l8q72\" (UID: \"48365630-1b94-419c-a248-39fd40ffb067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dc93a5b-l8q72"
Oct 31 20:05:52.659571 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:52.659550    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48365630-1b94-419c-a248-39fd40ffb067-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1dc93a5b-l8q72\" (UID: \"48365630-1b94-419c-a248-39fd40ffb067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dc93a5b-l8q72"
Oct 31 20:05:52.667983 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:52.667958    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gzks\" (UniqueName: \"kubernetes.io/projected/48365630-1b94-419c-a248-39fd40ffb067-kube-api-access-6gzks\") pod \"maas-api-revoke-sub-1dc93a5b-l8q72\" (UID: \"48365630-1b94-419c-a248-39fd40ffb067\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dc93a5b-l8q72"
Oct 31 20:05:52.776565 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:52.776492    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:52.783243 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:52.783217    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dc93a5b-l8q72"
Oct 31 20:05:53.095678 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:53.095652    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dc93a5b-l8q72"]
Oct 31 20:05:53.097233 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:05:53.097205    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48365630_1b94_419c_a248_39fd40ffb067.slice/crio-42846eabcc482c42351108590ab03a3590fb4149ed95b56d4df20482455f7e2e WatchSource:0}: Error finding container 42846eabcc482c42351108590ab03a3590fb4149ed95b56d4df20482455f7e2e: Status 404 returned error can't find the container with id 42846eabcc482c42351108590ab03a3590fb4149ed95b56d4df20482455f7e2e
Oct 31 20:05:53.617548 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:53.617456    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dc93a5b-l8q72" event={"ID":"48365630-1b94-419c-a248-39fd40ffb067","Type":"ContainerStarted","Data":"7db09618b7ad28e7963ffd93be055a9b7dac17ee390545e461fb36616f853b71"}
Oct 31 20:05:53.617548 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:53.617522    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dc93a5b-l8q72" event={"ID":"48365630-1b94-419c-a248-39fd40ffb067","Type":"ContainerStarted","Data":"42846eabcc482c42351108590ab03a3590fb4149ed95b56d4df20482455f7e2e"}
Oct 31 20:05:53.640459 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:53.640404    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dc93a5b-l8q72" podStartSLOduration=1.640390666 podStartE2EDuration="1.640390666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:05:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:05:53.63640969 +0000 UTC m=+2784.209643816" watchObservedRunningTime="2026-10-31 20:05:53.640390666 +0000 UTC m=+2784.213624795"
Oct 31 20:05:53.760301 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:53.760267    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:54.571488 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:54.571454    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:54.623064 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:54.623031    4645 generic.go:356] "Generic (PLEG): container finished" podID="48365630-1b94-419c-a248-39fd40ffb067" containerID="7db09618b7ad28e7963ffd93be055a9b7dac17ee390545e461fb36616f853b71" exitCode=0
Oct 31 20:05:54.623236 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:54.623108    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dc93a5b-l8q72" event={"ID":"48365630-1b94-419c-a248-39fd40ffb067","Type":"ContainerDied","Data":"7db09618b7ad28e7963ffd93be055a9b7dac17ee390545e461fb36616f853b71"}
Oct 31 20:05:55.427077 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:55.427035    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:55.793061 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:55.793037    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dc93a5b-l8q72"
Oct 31 20:05:55.824709 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:55.824679    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48365630-1b94-419c-a248-39fd40ffb067-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48365630-1b94-419c-a248-39fd40ffb067-maas-api-openshift-ca\") pod \"48365630-1b94-419c-a248-39fd40ffb067\" (UID: \"48365630-1b94-419c-a248-39fd40ffb067\") "
Oct 31 20:05:55.824861 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:55.824826    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48365630-1b94-419c-a248-39fd40ffb067-kube-api-access-6gzks\" (UniqueName: \"kubernetes.io/projected/48365630-1b94-419c-a248-39fd40ffb067-kube-api-access-6gzks\") pod \"48365630-1b94-419c-a248-39fd40ffb067\" (UID: \"48365630-1b94-419c-a248-39fd40ffb067\") "
Oct 31 20:05:55.824919 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:55.824880    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48365630-1b94-419c-a248-39fd40ffb067-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48365630-1b94-419c-a248-39fd40ffb067-maas-api-xks-ca\") pod \"48365630-1b94-419c-a248-39fd40ffb067\" (UID: \"48365630-1b94-419c-a248-39fd40ffb067\") "
Oct 31 20:05:55.825038 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:55.825016    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48365630-1b94-419c-a248-39fd40ffb067-maas-api-openshift-ca" pod "48365630-1b94-419c-a248-39fd40ffb067" (UID: "48365630-1b94-419c-a248-39fd40ffb067"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:05:55.825221 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:55.825201    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48365630-1b94-419c-a248-39fd40ffb067-maas-api-openshift-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:55.827053 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:55.827026    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48365630-1b94-419c-a248-39fd40ffb067-maas-api-xks-ca" pod "48365630-1b94-419c-a248-39fd40ffb067" (UID: "48365630-1b94-419c-a248-39fd40ffb067"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:05:55.827168 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:55.827144    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48365630-1b94-419c-a248-39fd40ffb067-kube-api-access-6gzks" pod "48365630-1b94-419c-a248-39fd40ffb067" (UID: "48365630-1b94-419c-a248-39fd40ffb067"). InnerVolumeSpecName "kube-api-access-6gzks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:05:55.926177 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:55.926149    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gzks\" (UniqueName: \"kubernetes.io/projected/48365630-1b94-419c-a248-39fd40ffb067-kube-api-access-6gzks\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:55.926177 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:55.926175    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48365630-1b94-419c-a248-39fd40ffb067-maas-api-xks-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:05:56.636745 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:56.636716    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dc93a5b-l8q72"
Oct 31 20:05:56.637139 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:56.636724    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dc93a5b-l8q72" event={"ID":"48365630-1b94-419c-a248-39fd40ffb067","Type":"ContainerDied","Data":"42846eabcc482c42351108590ab03a3590fb4149ed95b56d4df20482455f7e2e"}
Oct 31 20:05:56.637139 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:56.636838    4645 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42846eabcc482c42351108590ab03a3590fb4149ed95b56d4df20482455f7e2e"
Oct 31 20:05:56.776214 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:56.776177    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:57.426128 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:57.426096    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:57.606120 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:57.606080    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:58.025563 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:58.025526    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:58.965737 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:58.965700    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:59.546264 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:59.546219    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:59.779371 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:59.779317    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c173d8-6d47cf995d-2bjft"]
Oct 31 20:05:59.779608 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:59.779581    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-c173d8-6d47cf995d-2bjft" podUID="c08e5b4b-a7da-4cfe-a48a-140363b68362" containerName="payload-processing" containerID="cri-o://9f9c5c0ed0aef45da1b4adcdf968a16c2e934079bd18c95b77ac8cd8bd2602b7" gracePeriod=30
Oct 31 20:05:59.798995 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:59.798876    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c173d8-7dbb76c547-zwgd7"]
Oct 31 20:05:59.800176 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:59.799806    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c173d8-7dbb76c547-zwgd7" podUID="0121438a-b3ff-4d68-91c9-ab91f97fa181" containerName="payload-pre-processing" containerID="cri-o://ef809c72a50a92bd9176e11c1ec8dafa95845f170ea12a9e75902ab1a8906e73" gracePeriod=30
Oct 31 20:05:59.836113 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:59.836048    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:05:59.972648 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:05:59.972619    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:00.203047 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:00.203014    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:00.211951 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:00.211923    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c173d8-6d47cf995d-2bjft"
Oct 31 20:06:00.292149 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:00.292097    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c08e5b4b-a7da-4cfe-a48a-140363b68362-kube-api-access-9bjm6\" (UniqueName: \"kubernetes.io/projected/c08e5b4b-a7da-4cfe-a48a-140363b68362-kube-api-access-9bjm6\") pod \"c08e5b4b-a7da-4cfe-a48a-140363b68362\" (UID: \"c08e5b4b-a7da-4cfe-a48a-140363b68362\") "
Oct 31 20:06:00.292309 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:00.292155    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c08e5b4b-a7da-4cfe-a48a-140363b68362-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c08e5b4b-a7da-4cfe-a48a-140363b68362-plugins-config-volume\") pod \"c08e5b4b-a7da-4cfe-a48a-140363b68362\" (UID: \"c08e5b4b-a7da-4cfe-a48a-140363b68362\") "
Oct 31 20:06:00.292561 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:00.292539    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c08e5b4b-a7da-4cfe-a48a-140363b68362-plugins-config-volume" pod "c08e5b4b-a7da-4cfe-a48a-140363b68362" (UID: "c08e5b4b-a7da-4cfe-a48a-140363b68362"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:06:00.292778 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:00.292757    4645 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c08e5b4b-a7da-4cfe-a48a-140363b68362-plugins-config-volume\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:00.294949 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:00.294928    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c08e5b4b-a7da-4cfe-a48a-140363b68362-kube-api-access-9bjm6" pod "c08e5b4b-a7da-4cfe-a48a-140363b68362" (UID: "c08e5b4b-a7da-4cfe-a48a-140363b68362"). InnerVolumeSpecName "kube-api-access-9bjm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:06:00.299627 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:00.299608    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c173d8-7dbb76c547-zwgd7"
Oct 31 20:06:00.393317 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:00.393282    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0121438a-b3ff-4d68-91c9-ab91f97fa181-kube-api-access-mfgzh\" (UniqueName: \"kubernetes.io/projected/0121438a-b3ff-4d68-91c9-ab91f97fa181-kube-api-access-mfgzh\") pod \"0121438a-b3ff-4d68-91c9-ab91f97fa181\" (UID: \"0121438a-b3ff-4d68-91c9-ab91f97fa181\") "
Oct 31 20:06:00.393508 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:00.393395    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0121438a-b3ff-4d68-91c9-ab91f97fa181-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0121438a-b3ff-4d68-91c9-ab91f97fa181-plugins-config-volume\") pod \"0121438a-b3ff-4d68-91c9-ab91f97fa181\" (UID: \"0121438a-b3ff-4d68-91c9-ab91f97fa181\") "
Oct 31 20:06:00.393766 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:00.393743    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0121438a-b3ff-4d68-91c9-ab91f97fa181-plugins-config-volume" pod "0121438a-b3ff-4d68-91c9-ab91f97fa181" (UID: "0121438a-b3ff-4d68-91c9-ab91f97fa181"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:06:00.393930 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:00.393915    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bjm6\" (UniqueName: \"kubernetes.io/projected/c08e5b4b-a7da-4cfe-a48a-140363b68362-kube-api-access-9bjm6\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:00.393989 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:00.393932    4645 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0121438a-b3ff-4d68-91c9-ab91f97fa181-plugins-config-volume\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:00.395436 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:00.395414    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0121438a-b3ff-4d68-91c9-ab91f97fa181-kube-api-access-mfgzh" pod "0121438a-b3ff-4d68-91c9-ab91f97fa181" (UID: "0121438a-b3ff-4d68-91c9-ab91f97fa181"). InnerVolumeSpecName "kube-api-access-mfgzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:06:00.495700 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:00.495607    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfgzh\" (UniqueName: \"kubernetes.io/projected/0121438a-b3ff-4d68-91c9-ab91f97fa181-kube-api-access-mfgzh\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:00.533696 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:00.533667    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:00.662763 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:00.662724    4645 generic.go:356] "Generic (PLEG): container finished" podID="c08e5b4b-a7da-4cfe-a48a-140363b68362" containerID="9f9c5c0ed0aef45da1b4adcdf968a16c2e934079bd18c95b77ac8cd8bd2602b7" exitCode=0
Oct 31 20:06:00.663247 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:00.662779    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c173d8-6d47cf995d-2bjft"
Oct 31 20:06:00.663247 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:00.662812    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c173d8-6d47cf995d-2bjft" event={"ID":"c08e5b4b-a7da-4cfe-a48a-140363b68362","Type":"ContainerDied","Data":"9f9c5c0ed0aef45da1b4adcdf968a16c2e934079bd18c95b77ac8cd8bd2602b7"}
Oct 31 20:06:00.663247 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:00.662886    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c173d8-6d47cf995d-2bjft" event={"ID":"c08e5b4b-a7da-4cfe-a48a-140363b68362","Type":"ContainerDied","Data":"b4d0225aa84b54b39d6fe7460fa9a37e50e24ebb743ee72add2ad4ea82c45cdf"}
Oct 31 20:06:00.663247 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:00.662923    4645 scope.go:122] "RemoveContainer" containerID="9f9c5c0ed0aef45da1b4adcdf968a16c2e934079bd18c95b77ac8cd8bd2602b7"
Oct 31 20:06:00.664423 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:00.664397    4645 generic.go:356] "Generic (PLEG): container finished" podID="0121438a-b3ff-4d68-91c9-ab91f97fa181" containerID="ef809c72a50a92bd9176e11c1ec8dafa95845f170ea12a9e75902ab1a8906e73" exitCode=0
Oct 31 20:06:00.664537 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:00.664433    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c173d8-7dbb76c547-zwgd7"
Oct 31 20:06:00.664537 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:00.664440    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c173d8-7dbb76c547-zwgd7" event={"ID":"0121438a-b3ff-4d68-91c9-ab91f97fa181","Type":"ContainerDied","Data":"ef809c72a50a92bd9176e11c1ec8dafa95845f170ea12a9e75902ab1a8906e73"}
Oct 31 20:06:00.664537 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:00.664478    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c173d8-7dbb76c547-zwgd7" event={"ID":"0121438a-b3ff-4d68-91c9-ab91f97fa181","Type":"ContainerDied","Data":"e37e2f4376bea81108df8326f99699a41f288ee64927e40157f85abece1ab742"}
Oct 31 20:06:00.674359 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:00.674308    4645 scope.go:122] "RemoveContainer" containerID="9f9c5c0ed0aef45da1b4adcdf968a16c2e934079bd18c95b77ac8cd8bd2602b7"
Oct 31 20:06:00.674658 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:06:00.674632    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f9c5c0ed0aef45da1b4adcdf968a16c2e934079bd18c95b77ac8cd8bd2602b7\": container with ID starting with 9f9c5c0ed0aef45da1b4adcdf968a16c2e934079bd18c95b77ac8cd8bd2602b7 not found: ID does not exist" containerID="9f9c5c0ed0aef45da1b4adcdf968a16c2e934079bd18c95b77ac8cd8bd2602b7"
Oct 31 20:06:00.674788 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:00.674668    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f9c5c0ed0aef45da1b4adcdf968a16c2e934079bd18c95b77ac8cd8bd2602b7"} err="failed to get container status \"9f9c5c0ed0aef45da1b4adcdf968a16c2e934079bd18c95b77ac8cd8bd2602b7\": rpc error: code = NotFound desc = could not find container \"9f9c5c0ed0aef45da1b4adcdf968a16c2e934079bd18c95b77ac8cd8bd2602b7\": container with ID starting with 9f9c5c0ed0aef45da1b4adcdf968a16c2e934079bd18c95b77ac8cd8bd2602b7 not found: ID does not exist"
Oct 31 20:06:00.674788 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:00.674691    4645 scope.go:122] "RemoveContainer" containerID="ef809c72a50a92bd9176e11c1ec8dafa95845f170ea12a9e75902ab1a8906e73"
Oct 31 20:06:00.686101 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:00.686080    4645 scope.go:122] "RemoveContainer" containerID="ef809c72a50a92bd9176e11c1ec8dafa95845f170ea12a9e75902ab1a8906e73"
Oct 31 20:06:00.686460 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:06:00.686434    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef809c72a50a92bd9176e11c1ec8dafa95845f170ea12a9e75902ab1a8906e73\": container with ID starting with ef809c72a50a92bd9176e11c1ec8dafa95845f170ea12a9e75902ab1a8906e73 not found: ID does not exist" containerID="ef809c72a50a92bd9176e11c1ec8dafa95845f170ea12a9e75902ab1a8906e73"
Oct 31 20:06:00.686523 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:00.686470    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef809c72a50a92bd9176e11c1ec8dafa95845f170ea12a9e75902ab1a8906e73"} err="failed to get container status \"ef809c72a50a92bd9176e11c1ec8dafa95845f170ea12a9e75902ab1a8906e73\": rpc error: code = NotFound desc = could not find container \"ef809c72a50a92bd9176e11c1ec8dafa95845f170ea12a9e75902ab1a8906e73\": container with ID starting with ef809c72a50a92bd9176e11c1ec8dafa95845f170ea12a9e75902ab1a8906e73 not found: ID does not exist"
Oct 31 20:06:00.698082 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:00.697963    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c173d8-6d47cf995d-2bjft"]
Oct 31 20:06:00.702710 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:00.702685    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c173d8-6d47cf995d-2bjft"]
Oct 31 20:06:00.708827 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:00.708802    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c173d8-7dbb76c547-zwgd7"]
Oct 31 20:06:00.713755 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:00.713728    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c173d8-7dbb76c547-zwgd7"]
Oct 31 20:06:00.815889 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:00.815856    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:01.553123 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:01.553093    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44ea86bf-bx7k4"]
Oct 31 20:06:01.553788 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:01.553766    4645 cpu_manager.go:414] "containerMap: removing container" podUID="48365630-1b94-419c-a248-39fd40ffb067" containerName="revoke-keys"
Oct 31 20:06:01.553788 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:01.553788    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48365630-1b94-419c-a248-39fd40ffb067" containerName="revoke-keys"
Oct 31 20:06:01.553917 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:01.553825    4645 cpu_manager.go:414] "containerMap: removing container" podUID="0121438a-b3ff-4d68-91c9-ab91f97fa181" containerName="payload-pre-processing"
Oct 31 20:06:01.553917 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:01.553832    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0121438a-b3ff-4d68-91c9-ab91f97fa181" containerName="payload-pre-processing"
Oct 31 20:06:01.553917 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:01.553843    4645 cpu_manager.go:414] "containerMap: removing container" podUID="c08e5b4b-a7da-4cfe-a48a-140363b68362" containerName="payload-processing"
Oct 31 20:06:01.553917 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:01.553848    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c08e5b4b-a7da-4cfe-a48a-140363b68362" containerName="payload-processing"
Oct 31 20:06:01.554035 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:01.553930    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="48365630-1b94-419c-a248-39fd40ffb067" containerName="revoke-keys"
Oct 31 20:06:01.554035 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:01.553943    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="0121438a-b3ff-4d68-91c9-ab91f97fa181" containerName="payload-pre-processing"
Oct 31 20:06:01.554035 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:01.553950    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="c08e5b4b-a7da-4cfe-a48a-140363b68362" containerName="payload-processing"
Oct 31 20:06:01.558265 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:01.558241    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44ea86bf-bx7k4"
Oct 31 20:06:01.572685 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:01.572638    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44ea86bf-bx7k4"]
Oct 31 20:06:01.608607 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:01.608577    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/994c4584-538a-4750-9fc8-242816cfadf8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44ea86bf-bx7k4\" (UID: \"994c4584-538a-4750-9fc8-242816cfadf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44ea86bf-bx7k4"
Oct 31 20:06:01.608790 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:01.608768    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwxbw\" (UniqueName: \"kubernetes.io/projected/994c4584-538a-4750-9fc8-242816cfadf8-kube-api-access-wwxbw\") pod \"maas-api-revoke-sub-44ea86bf-bx7k4\" (UID: \"994c4584-538a-4750-9fc8-242816cfadf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44ea86bf-bx7k4"
Oct 31 20:06:01.608861 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:01.608822    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/994c4584-538a-4750-9fc8-242816cfadf8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44ea86bf-bx7k4\" (UID: \"994c4584-538a-4750-9fc8-242816cfadf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44ea86bf-bx7k4"
Oct 31 20:06:01.610543 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:01.610512    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw" podUID="d504b2d3-c26d-4dcf-a7eb-efe20732ae4e" containerName="main" containerID="cri-o://a58c7aab57f48cf169ca6488f1bcb792f0a6180f3da1d7e066201d05a8684cf5" gracePeriod=45
Oct 31 20:06:01.622169 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:01.616841    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:06:01.622169 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:01.617559    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 20:06:01.628468 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:01.628430    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0121438a-b3ff-4d68-91c9-ab91f97fa181" path="/var/lib/kubelet/pods/0121438a-b3ff-4d68-91c9-ab91f97fa181/volumes"
Oct 31 20:06:01.629157 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:01.629138    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c08e5b4b-a7da-4cfe-a48a-140363b68362" path="/var/lib/kubelet/pods/c08e5b4b-a7da-4cfe-a48a-140363b68362/volumes"
Oct 31 20:06:01.710507 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:01.710477    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwxbw\" (UniqueName: \"kubernetes.io/projected/994c4584-538a-4750-9fc8-242816cfadf8-kube-api-access-wwxbw\") pod \"maas-api-revoke-sub-44ea86bf-bx7k4\" (UID: \"994c4584-538a-4750-9fc8-242816cfadf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44ea86bf-bx7k4"
Oct 31 20:06:01.710921 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:01.710524    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/994c4584-538a-4750-9fc8-242816cfadf8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44ea86bf-bx7k4\" (UID: \"994c4584-538a-4750-9fc8-242816cfadf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44ea86bf-bx7k4"
Oct 31 20:06:01.710921 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:01.710647    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/994c4584-538a-4750-9fc8-242816cfadf8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44ea86bf-bx7k4\" (UID: \"994c4584-538a-4750-9fc8-242816cfadf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44ea86bf-bx7k4"
Oct 31 20:06:01.711437 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:01.711408    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/994c4584-538a-4750-9fc8-242816cfadf8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44ea86bf-bx7k4\" (UID: \"994c4584-538a-4750-9fc8-242816cfadf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44ea86bf-bx7k4"
Oct 31 20:06:01.713196 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:01.713169    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/994c4584-538a-4750-9fc8-242816cfadf8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44ea86bf-bx7k4\" (UID: \"994c4584-538a-4750-9fc8-242816cfadf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44ea86bf-bx7k4"
Oct 31 20:06:01.725828 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:01.725801    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwxbw\" (UniqueName: \"kubernetes.io/projected/994c4584-538a-4750-9fc8-242816cfadf8-kube-api-access-wwxbw\") pod \"maas-api-revoke-sub-44ea86bf-bx7k4\" (UID: \"994c4584-538a-4750-9fc8-242816cfadf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44ea86bf-bx7k4"
Oct 31 20:06:01.842094 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:01.841776    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:01.871871 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:01.871591    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44ea86bf-bx7k4"
Oct 31 20:06:01.925071 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:01.924182    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw"
Oct 31 20:06:02.025459 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:02.025420    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d504b2d3-c26d-4dcf-a7eb-efe20732ae4e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d504b2d3-c26d-4dcf-a7eb-efe20732ae4e-model-cache\") pod \"d504b2d3-c26d-4dcf-a7eb-efe20732ae4e\" (UID: \"d504b2d3-c26d-4dcf-a7eb-efe20732ae4e\") "
Oct 31 20:06:02.025666 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:02.025543    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d504b2d3-c26d-4dcf-a7eb-efe20732ae4e-kube-api-access-zbz44\" (UniqueName: \"kubernetes.io/projected/d504b2d3-c26d-4dcf-a7eb-efe20732ae4e-kube-api-access-zbz44\") pod \"d504b2d3-c26d-4dcf-a7eb-efe20732ae4e\" (UID: \"d504b2d3-c26d-4dcf-a7eb-efe20732ae4e\") "
Oct 31 20:06:02.025666 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:02.025609    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d504b2d3-c26d-4dcf-a7eb-efe20732ae4e-home\" (UniqueName: \"kubernetes.io/empty-dir/d504b2d3-c26d-4dcf-a7eb-efe20732ae4e-home\") pod \"d504b2d3-c26d-4dcf-a7eb-efe20732ae4e\" (UID: \"d504b2d3-c26d-4dcf-a7eb-efe20732ae4e\") "
Oct 31 20:06:02.025804 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:02.025759    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d504b2d3-c26d-4dcf-a7eb-efe20732ae4e-model-cache" pod "d504b2d3-c26d-4dcf-a7eb-efe20732ae4e" (UID: "d504b2d3-c26d-4dcf-a7eb-efe20732ae4e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:06:02.025908 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:02.025882    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d504b2d3-c26d-4dcf-a7eb-efe20732ae4e-home" pod "d504b2d3-c26d-4dcf-a7eb-efe20732ae4e" (UID: "d504b2d3-c26d-4dcf-a7eb-efe20732ae4e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:06:02.025995 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:02.025887    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d504b2d3-c26d-4dcf-a7eb-efe20732ae4e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d504b2d3-c26d-4dcf-a7eb-efe20732ae4e-tmp-dir\") pod \"d504b2d3-c26d-4dcf-a7eb-efe20732ae4e\" (UID: \"d504b2d3-c26d-4dcf-a7eb-efe20732ae4e\") "
Oct 31 20:06:02.025995 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:02.025968    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d504b2d3-c26d-4dcf-a7eb-efe20732ae4e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d504b2d3-c26d-4dcf-a7eb-efe20732ae4e-dshm\") pod \"d504b2d3-c26d-4dcf-a7eb-efe20732ae4e\" (UID: \"d504b2d3-c26d-4dcf-a7eb-efe20732ae4e\") "
Oct 31 20:06:02.026122 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:02.026075    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d504b2d3-c26d-4dcf-a7eb-efe20732ae4e-tls-certs\" (UniqueName: \"kubernetes.io/secret/d504b2d3-c26d-4dcf-a7eb-efe20732ae4e-tls-certs\") pod \"d504b2d3-c26d-4dcf-a7eb-efe20732ae4e\" (UID: \"d504b2d3-c26d-4dcf-a7eb-efe20732ae4e\") "
Oct 31 20:06:02.026122 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:02.026077    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d504b2d3-c26d-4dcf-a7eb-efe20732ae4e-tmp-dir" pod "d504b2d3-c26d-4dcf-a7eb-efe20732ae4e" (UID: "d504b2d3-c26d-4dcf-a7eb-efe20732ae4e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:06:02.026635 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:02.026608    4645 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d504b2d3-c26d-4dcf-a7eb-efe20732ae4e-tmp-dir\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:02.026740 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:02.026638    4645 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d504b2d3-c26d-4dcf-a7eb-efe20732ae4e-model-cache\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:02.026740 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:02.026652    4645 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d504b2d3-c26d-4dcf-a7eb-efe20732ae4e-home\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:02.029265 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:02.029206    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d504b2d3-c26d-4dcf-a7eb-efe20732ae4e-tls-certs" pod "d504b2d3-c26d-4dcf-a7eb-efe20732ae4e" (UID: "d504b2d3-c26d-4dcf-a7eb-efe20732ae4e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:06:02.030417 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:02.030368    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d504b2d3-c26d-4dcf-a7eb-efe20732ae4e-kube-api-access-zbz44" pod "d504b2d3-c26d-4dcf-a7eb-efe20732ae4e" (UID: "d504b2d3-c26d-4dcf-a7eb-efe20732ae4e"). InnerVolumeSpecName "kube-api-access-zbz44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:06:02.030915 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:02.030873    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d504b2d3-c26d-4dcf-a7eb-efe20732ae4e-dshm" pod "d504b2d3-c26d-4dcf-a7eb-efe20732ae4e" (UID: "d504b2d3-c26d-4dcf-a7eb-efe20732ae4e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:06:02.127557 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:02.127470    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbz44\" (UniqueName: \"kubernetes.io/projected/d504b2d3-c26d-4dcf-a7eb-efe20732ae4e-kube-api-access-zbz44\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:02.127557 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:02.127501    4645 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d504b2d3-c26d-4dcf-a7eb-efe20732ae4e-dshm\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:02.127557 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:02.127511    4645 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d504b2d3-c26d-4dcf-a7eb-efe20732ae4e-tls-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:02.170455 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:02.170418    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44ea86bf-bx7k4"]
Oct 31 20:06:02.174075 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:06:02.174032    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod994c4584_538a_4750_9fc8_242816cfadf8.slice/crio-d93ae1110ffa86432c9e4f51d7621bf397853161f37f973eaad10b0592b288da WatchSource:0}: Error finding container d93ae1110ffa86432c9e4f51d7621bf397853161f37f973eaad10b0592b288da: Status 404 returned error can't find the container with id d93ae1110ffa86432c9e4f51d7621bf397853161f37f973eaad10b0592b288da
Oct 31 20:06:02.686079 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:02.686030    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44ea86bf-bx7k4" event={"ID":"994c4584-538a-4750-9fc8-242816cfadf8","Type":"ContainerStarted","Data":"a67caed1dc85c660a3e02882942cc443b16f2c20b156cc17b4a742dded2bccc0"}
Oct 31 20:06:02.686382 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:02.686357    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44ea86bf-bx7k4" event={"ID":"994c4584-538a-4750-9fc8-242816cfadf8","Type":"ContainerStarted","Data":"d93ae1110ffa86432c9e4f51d7621bf397853161f37f973eaad10b0592b288da"}
Oct 31 20:06:02.688131 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:02.688081    4645 generic.go:356] "Generic (PLEG): container finished" podID="d504b2d3-c26d-4dcf-a7eb-efe20732ae4e" containerID="a58c7aab57f48cf169ca6488f1bcb792f0a6180f3da1d7e066201d05a8684cf5" exitCode=0
Oct 31 20:06:02.688131 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:02.688122    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw"
Oct 31 20:06:02.688342 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:02.688174    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw" event={"ID":"d504b2d3-c26d-4dcf-a7eb-efe20732ae4e","Type":"ContainerDied","Data":"a58c7aab57f48cf169ca6488f1bcb792f0a6180f3da1d7e066201d05a8684cf5"}
Oct 31 20:06:02.688342 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:02.688214    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw" event={"ID":"d504b2d3-c26d-4dcf-a7eb-efe20732ae4e","Type":"ContainerDied","Data":"2a0a51a1702babeea1f099e644d328bc2ce72e19807a543de340ac2012fa7297"}
Oct 31 20:06:02.688342 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:02.688242    4645 scope.go:122] "RemoveContainer" containerID="a58c7aab57f48cf169ca6488f1bcb792f0a6180f3da1d7e066201d05a8684cf5"
Oct 31 20:06:02.712813 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:02.712794    4645 scope.go:122] "RemoveContainer" containerID="a58c7aab57f48cf169ca6488f1bcb792f0a6180f3da1d7e066201d05a8684cf5"
Oct 31 20:06:02.713147 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:06:02.713124    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a58c7aab57f48cf169ca6488f1bcb792f0a6180f3da1d7e066201d05a8684cf5\": container with ID starting with a58c7aab57f48cf169ca6488f1bcb792f0a6180f3da1d7e066201d05a8684cf5 not found: ID does not exist" containerID="a58c7aab57f48cf169ca6488f1bcb792f0a6180f3da1d7e066201d05a8684cf5"
Oct 31 20:06:02.713217 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:02.713162    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a58c7aab57f48cf169ca6488f1bcb792f0a6180f3da1d7e066201d05a8684cf5"} err="failed to get container status \"a58c7aab57f48cf169ca6488f1bcb792f0a6180f3da1d7e066201d05a8684cf5\": rpc error: code = NotFound desc = could not find container \"a58c7aab57f48cf169ca6488f1bcb792f0a6180f3da1d7e066201d05a8684cf5\": container with ID starting with a58c7aab57f48cf169ca6488f1bcb792f0a6180f3da1d7e066201d05a8684cf5 not found: ID does not exist"
Oct 31 20:06:02.717846 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:02.717763    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44ea86bf-bx7k4" podStartSLOduration=1.7177504909999999 podStartE2EDuration="1.717750491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:06:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:06:02.714558014 +0000 UTC m=+2793.287792144" watchObservedRunningTime="2026-10-31 20:06:02.717750491 +0000 UTC m=+2793.290984619"
Oct 31 20:06:02.728987 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:02.728963    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw"]
Oct 31 20:06:02.733019 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:02.732999    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-aeb292/simulator-w2-aeb292-kserve-77f5bfb5c8-8jzdw"]
Oct 31 20:06:03.140399 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:03.140364    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:03.617629 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:03.617586    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d504b2d3-c26d-4dcf-a7eb-efe20732ae4e" path="/var/lib/kubelet/pods/d504b2d3-c26d-4dcf-a7eb-efe20732ae4e/volumes"
Oct 31 20:06:03.696890 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:03.696856    4645 generic.go:356] "Generic (PLEG): container finished" podID="994c4584-538a-4750-9fc8-242816cfadf8" containerID="a67caed1dc85c660a3e02882942cc443b16f2c20b156cc17b4a742dded2bccc0" exitCode=0
Oct 31 20:06:03.697061 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:03.696939    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44ea86bf-bx7k4" event={"ID":"994c4584-538a-4750-9fc8-242816cfadf8","Type":"ContainerDied","Data":"a67caed1dc85c660a3e02882942cc443b16f2c20b156cc17b4a742dded2bccc0"}
Oct 31 20:06:04.041483 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:04.041450    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:04.388143 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:04.388068    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:04.588148 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:04.588107    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:04.863190 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:04.863168    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44ea86bf-bx7k4"
Oct 31 20:06:04.972637 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:04.972559    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/994c4584-538a-4750-9fc8-242816cfadf8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/994c4584-538a-4750-9fc8-242816cfadf8-maas-api-openshift-ca\") pod \"994c4584-538a-4750-9fc8-242816cfadf8\" (UID: \"994c4584-538a-4750-9fc8-242816cfadf8\") "
Oct 31 20:06:04.972637 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:04.972615    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/994c4584-538a-4750-9fc8-242816cfadf8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/994c4584-538a-4750-9fc8-242816cfadf8-maas-api-xks-ca\") pod \"994c4584-538a-4750-9fc8-242816cfadf8\" (UID: \"994c4584-538a-4750-9fc8-242816cfadf8\") "
Oct 31 20:06:04.972809 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:04.972735    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/994c4584-538a-4750-9fc8-242816cfadf8-kube-api-access-wwxbw\" (UniqueName: \"kubernetes.io/projected/994c4584-538a-4750-9fc8-242816cfadf8-kube-api-access-wwxbw\") pod \"994c4584-538a-4750-9fc8-242816cfadf8\" (UID: \"994c4584-538a-4750-9fc8-242816cfadf8\") "
Oct 31 20:06:04.972998 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:04.972972    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/994c4584-538a-4750-9fc8-242816cfadf8-maas-api-openshift-ca" pod "994c4584-538a-4750-9fc8-242816cfadf8" (UID: "994c4584-538a-4750-9fc8-242816cfadf8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:06:04.973186 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:04.973160    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/994c4584-538a-4750-9fc8-242816cfadf8-maas-api-openshift-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:04.974901 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:04.974877    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/994c4584-538a-4750-9fc8-242816cfadf8-kube-api-access-wwxbw" pod "994c4584-538a-4750-9fc8-242816cfadf8" (UID: "994c4584-538a-4750-9fc8-242816cfadf8"). InnerVolumeSpecName "kube-api-access-wwxbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:06:04.975282 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:04.975263    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/994c4584-538a-4750-9fc8-242816cfadf8-maas-api-xks-ca" pod "994c4584-538a-4750-9fc8-242816cfadf8" (UID: "994c4584-538a-4750-9fc8-242816cfadf8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:06:05.001555 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:05.001532    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:05.074747 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:05.074725    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/994c4584-538a-4750-9fc8-242816cfadf8-maas-api-xks-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:05.075028 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:05.074748    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwxbw\" (UniqueName: \"kubernetes.io/projected/994c4584-538a-4750-9fc8-242816cfadf8-kube-api-access-wwxbw\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:05.587600 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:05.587561    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:05.714825 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:05.714787    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44ea86bf-bx7k4" event={"ID":"994c4584-538a-4750-9fc8-242816cfadf8","Type":"ContainerDied","Data":"d93ae1110ffa86432c9e4f51d7621bf397853161f37f973eaad10b0592b288da"}
Oct 31 20:06:05.714825 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:05.714814    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44ea86bf-bx7k4"
Oct 31 20:06:05.715022 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:05.714846    4645 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d93ae1110ffa86432c9e4f51d7621bf397853161f37f973eaad10b0592b288da"
Oct 31 20:06:05.879767 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:05.879688    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-c173d8/ipp-route-c173d8-kserve-776fb486f7-zjnkc"]
Oct 31 20:06:06.025976 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:06.025941    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:06.245663 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:06:06.245629    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 20:06:06.766518 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:06.766484    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:07.390269 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:07.390235    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:07.517055 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:07.517014    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:07.953557 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:07.953520    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:08.669152 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:08.669116    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:09.428628 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:09.428588    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:10.156411 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:10.156378    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:10.847303 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:10.847270    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:11.757501 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:11.757468    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:12.310422 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:12.310387    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:12.433876 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:12.433834    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:12.658060 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:12.657979    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln"]
Oct 31 20:06:12.658245 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:12.658222    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln" podUID="f3b22b1e-9d49-4a54-b016-bc504f03390c" containerName="istio-proxy" containerID="cri-o://7948823885e27f58d3ed14f621d4c87fc5c250551f48ef8cc354a49116d4ef10" gracePeriod=30
Oct 31 20:06:12.707310 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:12.707280    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:12.844352 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:12.843615    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:13.685098 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:13.685062    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:14.936646 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:14.936600    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:15.882132 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:15.882092    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:17.017460 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:17.017232    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:17.782384 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:17.782351    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:17.902806 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:17.902772    4645 generic.go:356] "Generic (PLEG): container finished" podID="f3b22b1e-9d49-4a54-b016-bc504f03390c" containerID="7948823885e27f58d3ed14f621d4c87fc5c250551f48ef8cc354a49116d4ef10" exitCode=0
Oct 31 20:06:17.902992 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:17.902849    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln" event={"ID":"f3b22b1e-9d49-4a54-b016-bc504f03390c","Type":"ContainerDied","Data":"7948823885e27f58d3ed14f621d4c87fc5c250551f48ef8cc354a49116d4ef10"}
Oct 31 20:06:17.902992 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:17.902915    4645 scope.go:122] "RemoveContainer" containerID="8429f3ed823bbf3a7a32a9203f1e0fd2704ca88b7be4e1ac071810c4fd8e5711"
Oct 31 20:06:18.385041 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:18.384983    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:18.397606 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:18.397572    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln"
Oct 31 20:06:18.502959 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:18.502865    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3b22b1e-9d49-4a54-b016-bc504f03390c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3b22b1e-9d49-4a54-b016-bc504f03390c-istio-envoy\") pod \"f3b22b1e-9d49-4a54-b016-bc504f03390c\" (UID: \"f3b22b1e-9d49-4a54-b016-bc504f03390c\") "
Oct 31 20:06:18.503156 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:18.503002    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3b22b1e-9d49-4a54-b016-bc504f03390c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3b22b1e-9d49-4a54-b016-bc504f03390c-workload-certs\") pod \"f3b22b1e-9d49-4a54-b016-bc504f03390c\" (UID: \"f3b22b1e-9d49-4a54-b016-bc504f03390c\") "
Oct 31 20:06:18.503156 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:18.503070    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f3b22b1e-9d49-4a54-b016-bc504f03390c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3b22b1e-9d49-4a54-b016-bc504f03390c-istio-podinfo\") pod \"f3b22b1e-9d49-4a54-b016-bc504f03390c\" (UID: \"f3b22b1e-9d49-4a54-b016-bc504f03390c\") "
Oct 31 20:06:18.503156 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:18.503144    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3b22b1e-9d49-4a54-b016-bc504f03390c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3b22b1e-9d49-4a54-b016-bc504f03390c-workload-socket\") pod \"f3b22b1e-9d49-4a54-b016-bc504f03390c\" (UID: \"f3b22b1e-9d49-4a54-b016-bc504f03390c\") "
Oct 31 20:06:18.503365 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:18.503173    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3b22b1e-9d49-4a54-b016-bc504f03390c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3b22b1e-9d49-4a54-b016-bc504f03390c-istiod-ca-cert\") pod \"f3b22b1e-9d49-4a54-b016-bc504f03390c\" (UID: \"f3b22b1e-9d49-4a54-b016-bc504f03390c\") "
Oct 31 20:06:18.503365 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:18.503254    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3b22b1e-9d49-4a54-b016-bc504f03390c-kube-api-access-b6t77\" (UniqueName: \"kubernetes.io/projected/f3b22b1e-9d49-4a54-b016-bc504f03390c-kube-api-access-b6t77\") pod \"f3b22b1e-9d49-4a54-b016-bc504f03390c\" (UID: \"f3b22b1e-9d49-4a54-b016-bc504f03390c\") "
Oct 31 20:06:18.503365 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:18.503296    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3b22b1e-9d49-4a54-b016-bc504f03390c-istio-token\" (UniqueName: \"kubernetes.io/projected/f3b22b1e-9d49-4a54-b016-bc504f03390c-istio-token\") pod \"f3b22b1e-9d49-4a54-b016-bc504f03390c\" (UID: \"f3b22b1e-9d49-4a54-b016-bc504f03390c\") "
Oct 31 20:06:18.503365 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:18.503324    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3b22b1e-9d49-4a54-b016-bc504f03390c-workload-certs" pod "f3b22b1e-9d49-4a54-b016-bc504f03390c" (UID: "f3b22b1e-9d49-4a54-b016-bc504f03390c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:06:18.503604 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:18.503470    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3b22b1e-9d49-4a54-b016-bc504f03390c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3b22b1e-9d49-4a54-b016-bc504f03390c-credential-socket\") pod \"f3b22b1e-9d49-4a54-b016-bc504f03390c\" (UID: \"f3b22b1e-9d49-4a54-b016-bc504f03390c\") "
Oct 31 20:06:18.503604 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:18.503512    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3b22b1e-9d49-4a54-b016-bc504f03390c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3b22b1e-9d49-4a54-b016-bc504f03390c-istio-data\") pod \"f3b22b1e-9d49-4a54-b016-bc504f03390c\" (UID: \"f3b22b1e-9d49-4a54-b016-bc504f03390c\") "
Oct 31 20:06:18.504237 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:18.503508    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3b22b1e-9d49-4a54-b016-bc504f03390c-workload-socket" pod "f3b22b1e-9d49-4a54-b016-bc504f03390c" (UID: "f3b22b1e-9d49-4a54-b016-bc504f03390c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:06:18.504237 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:18.503597    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3b22b1e-9d49-4a54-b016-bc504f03390c-istiod-ca-cert" pod "f3b22b1e-9d49-4a54-b016-bc504f03390c" (UID: "f3b22b1e-9d49-4a54-b016-bc504f03390c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:06:18.504237 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:18.503763    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3b22b1e-9d49-4a54-b016-bc504f03390c-credential-socket" pod "f3b22b1e-9d49-4a54-b016-bc504f03390c" (UID: "f3b22b1e-9d49-4a54-b016-bc504f03390c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:06:18.504237 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:18.503857    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3b22b1e-9d49-4a54-b016-bc504f03390c-istio-data" pod "f3b22b1e-9d49-4a54-b016-bc504f03390c" (UID: "f3b22b1e-9d49-4a54-b016-bc504f03390c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:06:18.504237 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:18.504135    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3b22b1e-9d49-4a54-b016-bc504f03390c-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:18.504237 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:18.504162    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3b22b1e-9d49-4a54-b016-bc504f03390c-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:18.504237 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:18.504176    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3b22b1e-9d49-4a54-b016-bc504f03390c-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:18.504237 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:18.504191    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3b22b1e-9d49-4a54-b016-bc504f03390c-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:18.504237 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:18.504205    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3b22b1e-9d49-4a54-b016-bc504f03390c-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:18.505928 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:18.505899    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3b22b1e-9d49-4a54-b016-bc504f03390c-kube-api-access-b6t77" pod "f3b22b1e-9d49-4a54-b016-bc504f03390c" (UID: "f3b22b1e-9d49-4a54-b016-bc504f03390c"). InnerVolumeSpecName "kube-api-access-b6t77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:06:18.506737 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:18.506714    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3b22b1e-9d49-4a54-b016-bc504f03390c-istio-envoy" pod "f3b22b1e-9d49-4a54-b016-bc504f03390c" (UID: "f3b22b1e-9d49-4a54-b016-bc504f03390c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:06:18.506814 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:18.506751    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f3b22b1e-9d49-4a54-b016-bc504f03390c-istio-podinfo" pod "f3b22b1e-9d49-4a54-b016-bc504f03390c" (UID: "f3b22b1e-9d49-4a54-b016-bc504f03390c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 20:06:18.506814 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:18.506768    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3b22b1e-9d49-4a54-b016-bc504f03390c-istio-token" pod "f3b22b1e-9d49-4a54-b016-bc504f03390c" (UID: "f3b22b1e-9d49-4a54-b016-bc504f03390c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:06:18.605340 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:18.605287    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6t77\" (UniqueName: \"kubernetes.io/projected/f3b22b1e-9d49-4a54-b016-bc504f03390c-kube-api-access-b6t77\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:18.605501 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:18.605346    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3b22b1e-9d49-4a54-b016-bc504f03390c-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:18.605501 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:18.605365    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3b22b1e-9d49-4a54-b016-bc504f03390c-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:18.605501 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:18.605381    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3b22b1e-9d49-4a54-b016-bc504f03390c-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:18.660652 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:18.659934    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:18.912792 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:18.912761    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln"
Oct 31 20:06:18.912994 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:18.912811    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln" event={"ID":"f3b22b1e-9d49-4a54-b016-bc504f03390c","Type":"ContainerDied","Data":"51aa454ee4f0e4362566b1f245e32ce6510814ba54e6f8551f9cf6d9e63c7195"}
Oct 31 20:06:18.912994 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:18.912882    4645 scope.go:122] "RemoveContainer" containerID="7948823885e27f58d3ed14f621d4c87fc5c250551f48ef8cc354a49116d4ef10"
Oct 31 20:06:18.936607 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:18.936581    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln"]
Oct 31 20:06:18.942772 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:18.942751    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-aeb292-openshift-default-58cb568b9f-vzbln"]
Oct 31 20:06:19.381821 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:19.381729    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:19.496170 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:19.496138    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:19.618099 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:19.618065    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3b22b1e-9d49-4a54-b016-bc504f03390c" path="/var/lib/kubelet/pods/f3b22b1e-9d49-4a54-b016-bc504f03390c/volumes"
Oct 31 20:06:19.741379 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:19.741319    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:20.121767 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:20.121689    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:20.425842 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:20.425798    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:20.899917 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:20.899821    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-c173d8/ipp-route-c173d8-kserve-776fb486f7-zjnkc" podUID="2b4334dd-204f-4794-830d-a375db82fa39" containerName="main" containerID="cri-o://f9dfe0177f43228dc26fa6c35b8657b6d3ea35ffbce66589884ee686b8d5f4b9" gracePeriod=45
Oct 31 20:06:21.221168 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:21.221103    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:21.261098 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:21.261065    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-c173d8/ipp-route-c173d8-kserve-776fb486f7-zjnkc"
Oct 31 20:06:21.449472 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:21.449426    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b4334dd-204f-4794-830d-a375db82fa39-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b4334dd-204f-4794-830d-a375db82fa39-tmp-dir\") pod \"2b4334dd-204f-4794-830d-a375db82fa39\" (UID: \"2b4334dd-204f-4794-830d-a375db82fa39\") "
Oct 31 20:06:21.449689 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:21.449532    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b4334dd-204f-4794-830d-a375db82fa39-tls-certs\" (UniqueName: \"kubernetes.io/secret/2b4334dd-204f-4794-830d-a375db82fa39-tls-certs\") pod \"2b4334dd-204f-4794-830d-a375db82fa39\" (UID: \"2b4334dd-204f-4794-830d-a375db82fa39\") "
Oct 31 20:06:21.449762 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:21.449729    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b4334dd-204f-4794-830d-a375db82fa39-tmp-dir" pod "2b4334dd-204f-4794-830d-a375db82fa39" (UID: "2b4334dd-204f-4794-830d-a375db82fa39"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:06:21.449762 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:21.449735    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b4334dd-204f-4794-830d-a375db82fa39-kube-api-access-ddrxj\" (UniqueName: \"kubernetes.io/projected/2b4334dd-204f-4794-830d-a375db82fa39-kube-api-access-ddrxj\") pod \"2b4334dd-204f-4794-830d-a375db82fa39\" (UID: \"2b4334dd-204f-4794-830d-a375db82fa39\") "
Oct 31 20:06:21.449887 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:21.449844    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b4334dd-204f-4794-830d-a375db82fa39-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b4334dd-204f-4794-830d-a375db82fa39-dshm\") pod \"2b4334dd-204f-4794-830d-a375db82fa39\" (UID: \"2b4334dd-204f-4794-830d-a375db82fa39\") "
Oct 31 20:06:21.449961 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:21.449886    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b4334dd-204f-4794-830d-a375db82fa39-home\" (UniqueName: \"kubernetes.io/empty-dir/2b4334dd-204f-4794-830d-a375db82fa39-home\") pod \"2b4334dd-204f-4794-830d-a375db82fa39\" (UID: \"2b4334dd-204f-4794-830d-a375db82fa39\") "
Oct 31 20:06:21.450023 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:21.449961    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b4334dd-204f-4794-830d-a375db82fa39-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b4334dd-204f-4794-830d-a375db82fa39-model-cache\") pod \"2b4334dd-204f-4794-830d-a375db82fa39\" (UID: \"2b4334dd-204f-4794-830d-a375db82fa39\") "
Oct 31 20:06:21.450227 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:21.450180    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b4334dd-204f-4794-830d-a375db82fa39-home" pod "2b4334dd-204f-4794-830d-a375db82fa39" (UID: "2b4334dd-204f-4794-830d-a375db82fa39"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:06:21.450318 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:21.450293    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b4334dd-204f-4794-830d-a375db82fa39-model-cache" pod "2b4334dd-204f-4794-830d-a375db82fa39" (UID: "2b4334dd-204f-4794-830d-a375db82fa39"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:06:21.450607 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:21.450587    4645 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b4334dd-204f-4794-830d-a375db82fa39-home\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:21.450706 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:21.450612    4645 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b4334dd-204f-4794-830d-a375db82fa39-model-cache\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:21.450706 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:21.450626    4645 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b4334dd-204f-4794-830d-a375db82fa39-tmp-dir\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:21.452185 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:21.452159    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b4334dd-204f-4794-830d-a375db82fa39-tls-certs" pod "2b4334dd-204f-4794-830d-a375db82fa39" (UID: "2b4334dd-204f-4794-830d-a375db82fa39"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:06:21.452308 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:21.452196    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b4334dd-204f-4794-830d-a375db82fa39-kube-api-access-ddrxj" pod "2b4334dd-204f-4794-830d-a375db82fa39" (UID: "2b4334dd-204f-4794-830d-a375db82fa39"). InnerVolumeSpecName "kube-api-access-ddrxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:06:21.452404 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:21.452391    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b4334dd-204f-4794-830d-a375db82fa39-dshm" pod "2b4334dd-204f-4794-830d-a375db82fa39" (UID: "2b4334dd-204f-4794-830d-a375db82fa39"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:06:21.551770 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:21.551691    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddrxj\" (UniqueName: \"kubernetes.io/projected/2b4334dd-204f-4794-830d-a375db82fa39-kube-api-access-ddrxj\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:21.551770 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:21.551721    4645 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b4334dd-204f-4794-830d-a375db82fa39-dshm\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:21.551770 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:21.551731    4645 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b4334dd-204f-4794-830d-a375db82fa39-tls-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:21.936590 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:21.936549    4645 generic.go:356] "Generic (PLEG): container finished" podID="2b4334dd-204f-4794-830d-a375db82fa39" containerID="f9dfe0177f43228dc26fa6c35b8657b6d3ea35ffbce66589884ee686b8d5f4b9" exitCode=0
Oct 31 20:06:21.937081 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:21.936601    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-c173d8/ipp-route-c173d8-kserve-776fb486f7-zjnkc"
Oct 31 20:06:21.937081 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:21.936633    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-c173d8/ipp-route-c173d8-kserve-776fb486f7-zjnkc" event={"ID":"2b4334dd-204f-4794-830d-a375db82fa39","Type":"ContainerDied","Data":"f9dfe0177f43228dc26fa6c35b8657b6d3ea35ffbce66589884ee686b8d5f4b9"}
Oct 31 20:06:21.937081 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:21.936688    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-c173d8/ipp-route-c173d8-kserve-776fb486f7-zjnkc" event={"ID":"2b4334dd-204f-4794-830d-a375db82fa39","Type":"ContainerDied","Data":"ac74accec21373d9de3cb978c2f2bbc9b42b21fac56a2d271fd0ed4b9c2fbe75"}
Oct 31 20:06:21.937081 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:21.936712    4645 scope.go:122] "RemoveContainer" containerID="f9dfe0177f43228dc26fa6c35b8657b6d3ea35ffbce66589884ee686b8d5f4b9"
Oct 31 20:06:21.947606 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:21.947585    4645 scope.go:122] "RemoveContainer" containerID="f9dfe0177f43228dc26fa6c35b8657b6d3ea35ffbce66589884ee686b8d5f4b9"
Oct 31 20:06:21.947912 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:06:21.947892    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9dfe0177f43228dc26fa6c35b8657b6d3ea35ffbce66589884ee686b8d5f4b9\": container with ID starting with f9dfe0177f43228dc26fa6c35b8657b6d3ea35ffbce66589884ee686b8d5f4b9 not found: ID does not exist" containerID="f9dfe0177f43228dc26fa6c35b8657b6d3ea35ffbce66589884ee686b8d5f4b9"
Oct 31 20:06:21.947957 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:21.947923    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9dfe0177f43228dc26fa6c35b8657b6d3ea35ffbce66589884ee686b8d5f4b9"} err="failed to get container status \"f9dfe0177f43228dc26fa6c35b8657b6d3ea35ffbce66589884ee686b8d5f4b9\": rpc error: code = NotFound desc = could not find container \"f9dfe0177f43228dc26fa6c35b8657b6d3ea35ffbce66589884ee686b8d5f4b9\": container with ID starting with f9dfe0177f43228dc26fa6c35b8657b6d3ea35ffbce66589884ee686b8d5f4b9 not found: ID does not exist"
Oct 31 20:06:21.998425 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:21.998390    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-c173d8/ipp-route-c173d8-kserve-776fb486f7-zjnkc"]
Oct 31 20:06:22.000104 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:22.000073    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-c173d8/ipp-route-c173d8-kserve-776fb486f7-zjnkc"]
Oct 31 20:06:22.499512 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:22.499468    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:22.959138 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:22.959103    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:23.094100 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:23.094018    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c53dcfe0-kxlzm"]
Oct 31 20:06:23.102760 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:23.102730    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c53dcfe0-kxlzm"]
Oct 31 20:06:23.243126 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:23.243090    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:23.387098 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:23.387012    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:23.530597 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:23.530561    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:23.622272 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:23.622234    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b4334dd-204f-4794-830d-a375db82fa39" path="/var/lib/kubelet/pods/2b4334dd-204f-4794-830d-a375db82fa39/volumes"
Oct 31 20:06:23.622949 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:23.622927    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="647f9975-219b-4b00-aa2a-93adf2ef16ab" path="/var/lib/kubelet/pods/647f9975-219b-4b00-aa2a-93adf2ef16ab/volumes"
Oct 31 20:06:23.733429 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:23.733384    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:24.683227 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:24.683159    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:24.961278 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:24.961195    4645 generic.go:356] "Generic (PLEG): container finished" podID="05f487dd-a849-4cb8-b5f0-1368f60518a3" containerID="4a4a5accd018c0747b0b44311c6383cba5cc15f19397a582160cab4f76526710" exitCode=0
Oct 31 20:06:24.961457 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:24.961270    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd" event={"ID":"05f487dd-a849-4cb8-b5f0-1368f60518a3","Type":"ContainerDied","Data":"4a4a5accd018c0747b0b44311c6383cba5cc15f19397a582160cab4f76526710"}
Oct 31 20:06:24.961709 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:24.961694    4645 scope.go:122] "RemoveContainer" containerID="4a4a5accd018c0747b0b44311c6383cba5cc15f19397a582160cab4f76526710"
Oct 31 20:06:24.963135 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:24.963108    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:06:24.963193 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:24.963175    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:06:24.963229 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:24.963208    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:06:25.377678 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:25.377576    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:25.581365 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:25.581294    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:25.699348 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:25.699172    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:25.985904 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:25.985813    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd" event={"ID":"05f487dd-a849-4cb8-b5f0-1368f60518a3","Type":"ContainerStarted","Data":"5cfe23f787aa493389c65a1b194fcd5b49746d2129b6a0b277c99c8a81f6fd8c"}
Oct 31 20:06:26.118963 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:26.118931    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd"
Oct 31 20:06:26.120227 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:26.120202    4645 patch_prober.go:29] interesting pod/e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.49:15021/healthz/ready\": dial tcp 10.133.1.49:15021: connect: connection refused" start-of-body=
Oct 31 20:06:26.120289 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:26.120256    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd" podUID="05f487dd-a849-4cb8-b5f0-1368f60518a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.49:15021/healthz/ready\": dial tcp 10.133.1.49:15021: connect: connection refused"
Oct 31 20:06:26.180451 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:26.180408    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:26.529654 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:26.529613    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-8730d5-549774fbc4-wrjg5"]
Oct 31 20:06:26.529888 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:26.529861    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-8730d5-549774fbc4-wrjg5" podUID="a08b47e2-773b-4f08-9b57-957c735fb0f3" containerName="maas-api" containerID="cri-o://4203006cdc9454ad518df90cb980fa826b20f148d5c8d25c17df5fba9298fb72" gracePeriod=30
Oct 31 20:06:26.588501 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:26.588466    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:26.722488 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:26.722456    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-76hjw_b14ee5b2-6bf8-4b41-a84d-d0a7dcc9b85a/payload-processing/0.log"
Oct 31 20:06:26.791723 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:26.791564    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:26.874460 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:26.874386    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8730d5-twrdd"]
Oct 31 20:06:26.885395 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:26.884562    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-8730d5-twrdd"]
Oct 31 20:06:26.918271 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:26.918181    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:26.979627 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:26.979591    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-76hjw_b14ee5b2-6bf8-4b41-a84d-d0a7dcc9b85a/payload-processing/0.log"
Oct 31 20:06:26.999390 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:26.999360    4645 generic.go:356] "Generic (PLEG): container finished" podID="a08b47e2-773b-4f08-9b57-957c735fb0f3" containerID="4203006cdc9454ad518df90cb980fa826b20f148d5c8d25c17df5fba9298fb72" exitCode=0
Oct 31 20:06:26.999566 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:26.999417    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-8730d5-549774fbc4-wrjg5" event={"ID":"a08b47e2-773b-4f08-9b57-957c735fb0f3","Type":"ContainerDied","Data":"4203006cdc9454ad518df90cb980fa826b20f148d5c8d25c17df5fba9298fb72"}
Oct 31 20:06:27.039721 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:27.039696    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-8730d5-549774fbc4-wrjg5"
Oct 31 20:06:27.118914 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:27.118833    4645 patch_prober.go:29] interesting pod/e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.49:15021/healthz/ready\": dial tcp 10.133.1.49:15021: connect: connection refused" start-of-body=
Oct 31 20:06:27.118914 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:27.118890    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd" podUID="05f487dd-a849-4cb8-b5f0-1368f60518a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.49:15021/healthz/ready\": dial tcp 10.133.1.49:15021: connect: connection refused"
Oct 31 20:06:27.184048 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:27.184011    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a08b47e2-773b-4f08-9b57-957c735fb0f3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a08b47e2-773b-4f08-9b57-957c735fb0f3-openshift-service-ca\") pod \"a08b47e2-773b-4f08-9b57-957c735fb0f3\" (UID: \"a08b47e2-773b-4f08-9b57-957c735fb0f3\") "
Oct 31 20:06:27.184048 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:27.184069    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a08b47e2-773b-4f08-9b57-957c735fb0f3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a08b47e2-773b-4f08-9b57-957c735fb0f3-maas-api-tls\") pod \"a08b47e2-773b-4f08-9b57-957c735fb0f3\" (UID: \"a08b47e2-773b-4f08-9b57-957c735fb0f3\") "
Oct 31 20:06:27.184399 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:27.184101    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a08b47e2-773b-4f08-9b57-957c735fb0f3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a08b47e2-773b-4f08-9b57-957c735fb0f3-metrics-certs\") pod \"a08b47e2-773b-4f08-9b57-957c735fb0f3\" (UID: \"a08b47e2-773b-4f08-9b57-957c735fb0f3\") "
Oct 31 20:06:27.184399 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:27.184281    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a08b47e2-773b-4f08-9b57-957c735fb0f3-kube-api-access-g7slt\" (UniqueName: \"kubernetes.io/projected/a08b47e2-773b-4f08-9b57-957c735fb0f3-kube-api-access-g7slt\") pod \"a08b47e2-773b-4f08-9b57-957c735fb0f3\" (UID: \"a08b47e2-773b-4f08-9b57-957c735fb0f3\") "
Oct 31 20:06:27.184537 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:27.184518    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a08b47e2-773b-4f08-9b57-957c735fb0f3-openshift-service-ca" pod "a08b47e2-773b-4f08-9b57-957c735fb0f3" (UID: "a08b47e2-773b-4f08-9b57-957c735fb0f3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:06:27.184870 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:27.184846    4645 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a08b47e2-773b-4f08-9b57-957c735fb0f3-openshift-service-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:27.187016 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:27.186976    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a08b47e2-773b-4f08-9b57-957c735fb0f3-maas-api-tls" pod "a08b47e2-773b-4f08-9b57-957c735fb0f3" (UID: "a08b47e2-773b-4f08-9b57-957c735fb0f3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:06:27.187669 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:27.187648    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a08b47e2-773b-4f08-9b57-957c735fb0f3-metrics-certs" pod "a08b47e2-773b-4f08-9b57-957c735fb0f3" (UID: "a08b47e2-773b-4f08-9b57-957c735fb0f3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:06:27.188051 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:27.188028    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a08b47e2-773b-4f08-9b57-957c735fb0f3-kube-api-access-g7slt" pod "a08b47e2-773b-4f08-9b57-957c735fb0f3" (UID: "a08b47e2-773b-4f08-9b57-957c735fb0f3"). InnerVolumeSpecName "kube-api-access-g7slt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:06:27.286211 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:27.286169    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7slt\" (UniqueName: \"kubernetes.io/projected/a08b47e2-773b-4f08-9b57-957c735fb0f3-kube-api-access-g7slt\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:27.286211 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:27.286206    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a08b47e2-773b-4f08-9b57-957c735fb0f3-maas-api-tls\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:27.286211 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:27.286216    4645 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a08b47e2-773b-4f08-9b57-957c735fb0f3-metrics-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:27.376621 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:27.376525    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:27.503814 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:27.503774    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:27.617706 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:27.617671    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26953ffd-3365-40e8-ad23-6b32cca6e57e" path="/var/lib/kubelet/pods/26953ffd-3365-40e8-ad23-6b32cca6e57e/volumes"
Oct 31 20:06:27.834897 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:27.834860    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:27.955683 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:27.955650    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f28d48-grgf8"]
Oct 31 20:06:27.963111 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:27.963076    4645 cpu_manager.go:414] "containerMap: removing container" podUID="994c4584-538a-4750-9fc8-242816cfadf8" containerName="revoke-keys"
Oct 31 20:06:27.963312 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:27.963295    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="994c4584-538a-4750-9fc8-242816cfadf8" containerName="revoke-keys"
Oct 31 20:06:27.963453 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:27.963438    4645 cpu_manager.go:414] "containerMap: removing container" podUID="f3b22b1e-9d49-4a54-b016-bc504f03390c" containerName="istio-proxy"
Oct 31 20:06:27.963551 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:27.963539    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3b22b1e-9d49-4a54-b016-bc504f03390c" containerName="istio-proxy"
Oct 31 20:06:27.963655 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:27.963643    4645 cpu_manager.go:414] "containerMap: removing container" podUID="a08b47e2-773b-4f08-9b57-957c735fb0f3" containerName="maas-api"
Oct 31 20:06:27.963741 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:27.963729    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a08b47e2-773b-4f08-9b57-957c735fb0f3" containerName="maas-api"
Oct 31 20:06:27.963838 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:27.963827    4645 cpu_manager.go:414] "containerMap: removing container" podUID="2b4334dd-204f-4794-830d-a375db82fa39" containerName="main"
Oct 31 20:06:27.963925 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:27.963912    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b4334dd-204f-4794-830d-a375db82fa39" containerName="main"
Oct 31 20:06:27.964040 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:27.964029    4645 cpu_manager.go:414] "containerMap: removing container" podUID="d504b2d3-c26d-4dcf-a7eb-efe20732ae4e" containerName="main"
Oct 31 20:06:27.964124 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:27.964113    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d504b2d3-c26d-4dcf-a7eb-efe20732ae4e" containerName="main"
Oct 31 20:06:27.964215 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:27.964205    4645 cpu_manager.go:414] "containerMap: removing container" podUID="f3b22b1e-9d49-4a54-b016-bc504f03390c" containerName="istio-proxy"
Oct 31 20:06:27.964309 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:27.964297    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3b22b1e-9d49-4a54-b016-bc504f03390c" containerName="istio-proxy"
Oct 31 20:06:27.964577 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:27.964558    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="a08b47e2-773b-4f08-9b57-957c735fb0f3" containerName="maas-api"
Oct 31 20:06:27.964698 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:27.964687    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b4334dd-204f-4794-830d-a375db82fa39" containerName="main"
Oct 31 20:06:27.964800 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:27.964790    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="d504b2d3-c26d-4dcf-a7eb-efe20732ae4e" containerName="main"
Oct 31 20:06:27.964887 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:27.964877    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3b22b1e-9d49-4a54-b016-bc504f03390c" containerName="istio-proxy"
Oct 31 20:06:27.964982 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:27.964972    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="994c4584-538a-4750-9fc8-242816cfadf8" containerName="revoke-keys"
Oct 31 20:06:27.988686 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:27.987445    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f28d48-grgf8"]
Oct 31 20:06:27.988686 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:27.987542    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f28d48-grgf8"
Oct 31 20:06:28.009863 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:28.009839    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-8730d5-549774fbc4-wrjg5"
Oct 31 20:06:28.010007 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:28.009862    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-8730d5-549774fbc4-wrjg5" event={"ID":"a08b47e2-773b-4f08-9b57-957c735fb0f3","Type":"ContainerDied","Data":"6b001455e69cc5591b9c6c3d07c014af234dc4d728253a5812dcea49a49c2a6a"}
Oct 31 20:06:28.010007 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:28.009916    4645 scope.go:122] "RemoveContainer" containerID="4203006cdc9454ad518df90cb980fa826b20f148d5c8d25c17df5fba9298fb72"
Oct 31 20:06:28.023678 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:28.023651    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:06:28.023798 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:28.023729    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 20:06:28.028948 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:28.028927    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-8730d5-549774fbc4-wrjg5"]
Oct 31 20:06:28.032716 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:28.032695    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-8730d5-549774fbc4-wrjg5"]
Oct 31 20:06:28.100918 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:28.100837    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17a5f4e3-1aba-4e82-aa9a-c50c4710d782-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f28d48-grgf8\" (UID: \"17a5f4e3-1aba-4e82-aa9a-c50c4710d782\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f28d48-grgf8"
Oct 31 20:06:28.101072 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:28.100968    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17a5f4e3-1aba-4e82-aa9a-c50c4710d782-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f28d48-grgf8\" (UID: \"17a5f4e3-1aba-4e82-aa9a-c50c4710d782\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f28d48-grgf8"
Oct 31 20:06:28.101072 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:28.101047    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jldxd\" (UniqueName: \"kubernetes.io/projected/17a5f4e3-1aba-4e82-aa9a-c50c4710d782-kube-api-access-jldxd\") pod \"maas-api-revoke-keys-e2e-ipp-a-f28d48-grgf8\" (UID: \"17a5f4e3-1aba-4e82-aa9a-c50c4710d782\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f28d48-grgf8"
Oct 31 20:06:28.118815 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:28.118787    4645 patch_prober.go:29] interesting pod/e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.49:15021/healthz/ready\": dial tcp 10.133.1.49:15021: connect: connection refused" start-of-body=
Oct 31 20:06:28.118916 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:28.118832    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd" podUID="05f487dd-a849-4cb8-b5f0-1368f60518a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.49:15021/healthz/ready\": dial tcp 10.133.1.49:15021: connect: connection refused"
Oct 31 20:06:28.202119 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:28.202084    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17a5f4e3-1aba-4e82-aa9a-c50c4710d782-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f28d48-grgf8\" (UID: \"17a5f4e3-1aba-4e82-aa9a-c50c4710d782\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f28d48-grgf8"
Oct 31 20:06:28.202293 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:28.202172    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17a5f4e3-1aba-4e82-aa9a-c50c4710d782-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f28d48-grgf8\" (UID: \"17a5f4e3-1aba-4e82-aa9a-c50c4710d782\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f28d48-grgf8"
Oct 31 20:06:28.202293 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:28.202209    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jldxd\" (UniqueName: \"kubernetes.io/projected/17a5f4e3-1aba-4e82-aa9a-c50c4710d782-kube-api-access-jldxd\") pod \"maas-api-revoke-keys-e2e-ipp-a-f28d48-grgf8\" (UID: \"17a5f4e3-1aba-4e82-aa9a-c50c4710d782\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f28d48-grgf8"
Oct 31 20:06:28.202849 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:28.202827    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17a5f4e3-1aba-4e82-aa9a-c50c4710d782-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f28d48-grgf8\" (UID: \"17a5f4e3-1aba-4e82-aa9a-c50c4710d782\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f28d48-grgf8"
Oct 31 20:06:28.204529 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:28.204503    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17a5f4e3-1aba-4e82-aa9a-c50c4710d782-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f28d48-grgf8\" (UID: \"17a5f4e3-1aba-4e82-aa9a-c50c4710d782\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f28d48-grgf8"
Oct 31 20:06:28.213601 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:28.213577    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jldxd\" (UniqueName: \"kubernetes.io/projected/17a5f4e3-1aba-4e82-aa9a-c50c4710d782-kube-api-access-jldxd\") pod \"maas-api-revoke-keys-e2e-ipp-a-f28d48-grgf8\" (UID: \"17a5f4e3-1aba-4e82-aa9a-c50c4710d782\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f28d48-grgf8"
Oct 31 20:06:28.304200 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:28.304163    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f28d48-grgf8"
Oct 31 20:06:28.619072 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:28.618973    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f28d48-grgf8"]
Oct 31 20:06:28.622470 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:06:28.622443    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17a5f4e3_1aba_4e82_aa9a_c50c4710d782.slice/crio-95999fd831a3d1a0d98ac1fb33fc78f6a158f3c1a281d3c8291d411d3962393d WatchSource:0}: Error finding container 95999fd831a3d1a0d98ac1fb33fc78f6a158f3c1a281d3c8291d411d3962393d: Status 404 returned error can't find the container with id 95999fd831a3d1a0d98ac1fb33fc78f6a158f3c1a281d3c8291d411d3962393d
Oct 31 20:06:28.775359 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:28.771909    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:29.018179 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:29.018143    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f28d48-grgf8" event={"ID":"17a5f4e3-1aba-4e82-aa9a-c50c4710d782","Type":"ContainerStarted","Data":"f9f5f837be77eaf90db1bab308d1e9f4e4ee098280845b2a7a76bbacc97972f2"}
Oct 31 20:06:29.018645 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:29.018206    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f28d48-grgf8" event={"ID":"17a5f4e3-1aba-4e82-aa9a-c50c4710d782","Type":"ContainerStarted","Data":"95999fd831a3d1a0d98ac1fb33fc78f6a158f3c1a281d3c8291d411d3962393d"}
Oct 31 20:06:29.044501 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:29.044454    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f28d48-grgf8" podStartSLOduration=2.044441918 podStartE2EDuration="2.044441918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:06:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:06:29.034299911 +0000 UTC m=+2819.607534040" watchObservedRunningTime="2026-10-31 20:06:29.044441918 +0000 UTC m=+2819.617676047"
Oct 31 20:06:29.118830 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:29.118796    4645 patch_prober.go:29] interesting pod/e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.49:15021/healthz/ready\": dial tcp 10.133.1.49:15021: connect: connection refused" start-of-body=
Oct 31 20:06:29.118983 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:29.118842    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd" podUID="05f487dd-a849-4cb8-b5f0-1368f60518a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.49:15021/healthz/ready\": dial tcp 10.133.1.49:15021: connect: connection refused"
Oct 31 20:06:29.618448 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:29.618414    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a08b47e2-773b-4f08-9b57-957c735fb0f3" path="/var/lib/kubelet/pods/a08b47e2-773b-4f08-9b57-957c735fb0f3/volumes"
Oct 31 20:06:29.684938 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:29.684908    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:30.027489 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:30.027456    4645 generic.go:356] "Generic (PLEG): container finished" podID="17a5f4e3-1aba-4e82-aa9a-c50c4710d782" containerID="f9f5f837be77eaf90db1bab308d1e9f4e4ee098280845b2a7a76bbacc97972f2" exitCode=0
Oct 31 20:06:30.027908 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:30.027532    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f28d48-grgf8" event={"ID":"17a5f4e3-1aba-4e82-aa9a-c50c4710d782","Type":"ContainerDied","Data":"f9f5f837be77eaf90db1bab308d1e9f4e4ee098280845b2a7a76bbacc97972f2"}
Oct 31 20:06:30.118492 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:30.118459    4645 patch_prober.go:29] interesting pod/e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.49:15021/healthz/ready\": dial tcp 10.133.1.49:15021: connect: connection refused" start-of-body=
Oct 31 20:06:30.118687 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:30.118523    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd" podUID="05f487dd-a849-4cb8-b5f0-1368f60518a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.49:15021/healthz/ready\": dial tcp 10.133.1.49:15021: connect: connection refused"
Oct 31 20:06:30.307898 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:30.307827    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd"
Oct 31 20:06:30.574828 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:30.574694    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:31.189708 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:31.189673    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f28d48-grgf8"
Oct 31 20:06:31.198610 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:31.198430    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd"
Oct 31 20:06:31.200249 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:31.200217    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd"
Oct 31 20:06:31.346091 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:31.346006    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17a5f4e3-1aba-4e82-aa9a-c50c4710d782-kube-api-access-jldxd\" (UniqueName: \"kubernetes.io/projected/17a5f4e3-1aba-4e82-aa9a-c50c4710d782-kube-api-access-jldxd\") pod \"17a5f4e3-1aba-4e82-aa9a-c50c4710d782\" (UID: \"17a5f4e3-1aba-4e82-aa9a-c50c4710d782\") "
Oct 31 20:06:31.346091 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:31.346055    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17a5f4e3-1aba-4e82-aa9a-c50c4710d782-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17a5f4e3-1aba-4e82-aa9a-c50c4710d782-maas-api-openshift-ca\") pod \"17a5f4e3-1aba-4e82-aa9a-c50c4710d782\" (UID: \"17a5f4e3-1aba-4e82-aa9a-c50c4710d782\") "
Oct 31 20:06:31.346455 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:31.346125    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17a5f4e3-1aba-4e82-aa9a-c50c4710d782-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17a5f4e3-1aba-4e82-aa9a-c50c4710d782-maas-api-xks-ca\") pod \"17a5f4e3-1aba-4e82-aa9a-c50c4710d782\" (UID: \"17a5f4e3-1aba-4e82-aa9a-c50c4710d782\") "
Oct 31 20:06:31.346548 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:31.346522    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17a5f4e3-1aba-4e82-aa9a-c50c4710d782-maas-api-openshift-ca" pod "17a5f4e3-1aba-4e82-aa9a-c50c4710d782" (UID: "17a5f4e3-1aba-4e82-aa9a-c50c4710d782"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:06:31.346744 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:31.346719    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17a5f4e3-1aba-4e82-aa9a-c50c4710d782-maas-api-openshift-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:31.348299 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:31.348272    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17a5f4e3-1aba-4e82-aa9a-c50c4710d782-maas-api-xks-ca" pod "17a5f4e3-1aba-4e82-aa9a-c50c4710d782" (UID: "17a5f4e3-1aba-4e82-aa9a-c50c4710d782"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:06:31.348600 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:31.348577    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17a5f4e3-1aba-4e82-aa9a-c50c4710d782-kube-api-access-jldxd" pod "17a5f4e3-1aba-4e82-aa9a-c50c4710d782" (UID: "17a5f4e3-1aba-4e82-aa9a-c50c4710d782"). InnerVolumeSpecName "kube-api-access-jldxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:06:31.447855 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:31.447828    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jldxd\" (UniqueName: \"kubernetes.io/projected/17a5f4e3-1aba-4e82-aa9a-c50c4710d782-kube-api-access-jldxd\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:31.447855 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:31.447856    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17a5f4e3-1aba-4e82-aa9a-c50c4710d782-maas-api-xks-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:31.554629 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:31.554594    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:32.041818 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:32.041789    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f28d48-grgf8"
Oct 31 20:06:32.041987 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:32.041819    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f28d48-grgf8" event={"ID":"17a5f4e3-1aba-4e82-aa9a-c50c4710d782","Type":"ContainerDied","Data":"95999fd831a3d1a0d98ac1fb33fc78f6a158f3c1a281d3c8291d411d3962393d"}
Oct 31 20:06:32.041987 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:32.041864    4645 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95999fd831a3d1a0d98ac1fb33fc78f6a158f3c1a281d3c8291d411d3962393d"
Oct 31 20:06:32.179141 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:32.179099    4645 scope.go:122] "RemoveContainer" containerID="16875b9e9e74219d6de1c3dda4aedcc4945145f5b8df793486eab95cf93a33b7"
Oct 31 20:06:32.189270 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:32.189251    4645 scope.go:122] "RemoveContainer" containerID="d703025bf065d3c1c3e0fe60c888c9a997be647b03c85ad9537a93260de88d8a"
Oct 31 20:06:32.197759 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:32.197737    4645 scope.go:122] "RemoveContainer" containerID="fae17cbac3f49413a152b889ed3b03a9aae46d411281dea981aff1511abd7be4"
Oct 31 20:06:32.761545 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:32.761501    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:33.708803 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:33.708773    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:34.480172 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:34.480134    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:34.701804 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:34.701774    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:35.493000 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:35.492957    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:36.022685 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:36.022612    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:36.225714 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:36.225688    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:36.246446 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:06:36.246417    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 20:06:36.344270 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:36.344166    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:36.481382 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:36.481345    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:36.561319 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:36.561283    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm"]
Oct 31 20:06:36.561809 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:36.561545    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm" podUID="d78f81fc-d267-407f-8bd7-51ac2d787720" containerName="istio-proxy" containerID="cri-o://2cf9ad422be8e2ec90162aa8aebe6302089535b2a06dd8da963a84db064ea742" gracePeriod=30
Oct 31 20:06:36.608120 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:36.608027    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:37.538896 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:37.538856    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:38.330967 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:38.330933    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:38.504753 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:38.504716    4645 patch_prober.go:29] interesting pod/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 20:06:38.504924 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:38.504797    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm" podUID="d78f81fc-d267-407f-8bd7-51ac2d787720" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 20:06:39.056244 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:39.056193    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:39.791136 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:39.791099    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:39.810929 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:39.810887    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f28d48-6fc75df778-4x22w"]
Oct 31 20:06:39.811115 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:39.811094    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-f28d48-6fc75df778-4x22w" podUID="2dd3d751-d8d8-4fa7-b1bf-5c2da531f07d" containerName="payload-processing" containerID="cri-o://44ff49ee1dabc8669a574a379e718960648e053ef65a77745bdb00971fbb3adf" gracePeriod=30
Oct 31 20:06:39.827982 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:39.827953    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f28d48-5bbcbb6f5-8jbqq"]
Oct 31 20:06:39.828231 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:39.828194    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f28d48-5bbcbb6f5-8jbqq" podUID="7599e792-03f8-41bf-9cda-aca650a9022b" containerName="payload-pre-processing" containerID="cri-o://76d7cc3b1c1bd8a8e68b5626855048a60998412d230e7c0b8a4a74ed307019f1" gracePeriod=30
Oct 31 20:06:39.938038 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:39.937991    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:40.076061 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:40.076028    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:40.104091 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:40.104063    4645 generic.go:356] "Generic (PLEG): container finished" podID="7599e792-03f8-41bf-9cda-aca650a9022b" containerID="76d7cc3b1c1bd8a8e68b5626855048a60998412d230e7c0b8a4a74ed307019f1" exitCode=0
Oct 31 20:06:40.104212 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:40.104091    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f28d48-5bbcbb6f5-8jbqq" event={"ID":"7599e792-03f8-41bf-9cda-aca650a9022b","Type":"ContainerDied","Data":"76d7cc3b1c1bd8a8e68b5626855048a60998412d230e7c0b8a4a74ed307019f1"}
Oct 31 20:06:40.105656 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:40.105635    4645 generic.go:356] "Generic (PLEG): container finished" podID="2dd3d751-d8d8-4fa7-b1bf-5c2da531f07d" containerID="44ff49ee1dabc8669a574a379e718960648e053ef65a77745bdb00971fbb3adf" exitCode=0
Oct 31 20:06:40.105770 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:40.105708    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f28d48-6fc75df778-4x22w" event={"ID":"2dd3d751-d8d8-4fa7-b1bf-5c2da531f07d","Type":"ContainerDied","Data":"44ff49ee1dabc8669a574a379e718960648e053ef65a77745bdb00971fbb3adf"}
Oct 31 20:06:40.105770 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:40.105760    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f28d48-6fc75df778-4x22w" event={"ID":"2dd3d751-d8d8-4fa7-b1bf-5c2da531f07d","Type":"ContainerDied","Data":"fb972368091a8a5c896c348e417cf9aa6bcfe7f61ae687132d1c924818609ec5"}
Oct 31 20:06:40.105892 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:40.105784    4645 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb972368091a8a5c896c348e417cf9aa6bcfe7f61ae687132d1c924818609ec5"
Oct 31 20:06:40.109909 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:40.109892    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f28d48-6fc75df778-4x22w"
Oct 31 20:06:40.141187 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:40.141164    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f28d48-5bbcbb6f5-8jbqq"
Oct 31 20:06:40.141306 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:40.141211    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dd3d751-d8d8-4fa7-b1bf-5c2da531f07d-kube-api-access-v4swf\" (UniqueName: \"kubernetes.io/projected/2dd3d751-d8d8-4fa7-b1bf-5c2da531f07d-kube-api-access-v4swf\") pod \"2dd3d751-d8d8-4fa7-b1bf-5c2da531f07d\" (UID: \"2dd3d751-d8d8-4fa7-b1bf-5c2da531f07d\") "
Oct 31 20:06:40.141432 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:40.141415    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2dd3d751-d8d8-4fa7-b1bf-5c2da531f07d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dd3d751-d8d8-4fa7-b1bf-5c2da531f07d-plugins-config-volume\") pod \"2dd3d751-d8d8-4fa7-b1bf-5c2da531f07d\" (UID: \"2dd3d751-d8d8-4fa7-b1bf-5c2da531f07d\") "
Oct 31 20:06:40.141999 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:40.141973    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dd3d751-d8d8-4fa7-b1bf-5c2da531f07d-plugins-config-volume" pod "2dd3d751-d8d8-4fa7-b1bf-5c2da531f07d" (UID: "2dd3d751-d8d8-4fa7-b1bf-5c2da531f07d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:06:40.143657 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:40.143631    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dd3d751-d8d8-4fa7-b1bf-5c2da531f07d-kube-api-access-v4swf" pod "2dd3d751-d8d8-4fa7-b1bf-5c2da531f07d" (UID: "2dd3d751-d8d8-4fa7-b1bf-5c2da531f07d"). InnerVolumeSpecName "kube-api-access-v4swf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:06:40.149995 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:40.149940    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f28d48-grgf8"]
Oct 31 20:06:40.184365 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:40.169072    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f28d48-grgf8"]
Oct 31 20:06:40.221365 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:40.210909    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:40.242621 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:40.242586    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7599e792-03f8-41bf-9cda-aca650a9022b-kube-api-access-tnpd8\" (UniqueName: \"kubernetes.io/projected/7599e792-03f8-41bf-9cda-aca650a9022b-kube-api-access-tnpd8\") pod \"7599e792-03f8-41bf-9cda-aca650a9022b\" (UID: \"7599e792-03f8-41bf-9cda-aca650a9022b\") "
Oct 31 20:06:40.242840 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:40.242805    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7599e792-03f8-41bf-9cda-aca650a9022b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7599e792-03f8-41bf-9cda-aca650a9022b-plugins-config-volume\") pod \"7599e792-03f8-41bf-9cda-aca650a9022b\" (UID: \"7599e792-03f8-41bf-9cda-aca650a9022b\") "
Oct 31 20:06:40.243262 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:40.243233    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7599e792-03f8-41bf-9cda-aca650a9022b-plugins-config-volume" pod "7599e792-03f8-41bf-9cda-aca650a9022b" (UID: "7599e792-03f8-41bf-9cda-aca650a9022b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:06:40.246140 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:40.246108    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7599e792-03f8-41bf-9cda-aca650a9022b-kube-api-access-tnpd8" pod "7599e792-03f8-41bf-9cda-aca650a9022b" (UID: "7599e792-03f8-41bf-9cda-aca650a9022b"). InnerVolumeSpecName "kube-api-access-tnpd8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:06:40.246559 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:40.246539    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnpd8\" (UniqueName: \"kubernetes.io/projected/7599e792-03f8-41bf-9cda-aca650a9022b-kube-api-access-tnpd8\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:40.246646 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:40.246562    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4swf\" (UniqueName: \"kubernetes.io/projected/2dd3d751-d8d8-4fa7-b1bf-5c2da531f07d-kube-api-access-v4swf\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:40.246646 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:40.246577    4645 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7599e792-03f8-41bf-9cda-aca650a9022b-plugins-config-volume\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:40.246646 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:40.246591    4645 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2dd3d751-d8d8-4fa7-b1bf-5c2da531f07d-plugins-config-volume\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:40.323556 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:40.323468    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:40.715756 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:40.715724    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:40.862835 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:40.862806    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:40.972421 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:40.972344    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:41.112745 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:41.112724    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f28d48-5bbcbb6f5-8jbqq"
Oct 31 20:06:41.112882 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:41.112766    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f28d48-5bbcbb6f5-8jbqq" event={"ID":"7599e792-03f8-41bf-9cda-aca650a9022b","Type":"ContainerDied","Data":"eed3ccf5717df7b26c9deeeed230fa5a7c4f2ed8ea2243b585443a9bb475cbd5"}
Oct 31 20:06:41.112882 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:41.112826    4645 scope.go:122] "RemoveContainer" containerID="76d7cc3b1c1bd8a8e68b5626855048a60998412d230e7c0b8a4a74ed307019f1"
Oct 31 20:06:41.113009 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:41.112944    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f28d48-6fc75df778-4x22w"
Oct 31 20:06:41.149269 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:41.149235    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f28d48-5bbcbb6f5-8jbqq"]
Oct 31 20:06:41.157779 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:41.157752    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f28d48-5bbcbb6f5-8jbqq"]
Oct 31 20:06:41.171382 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:41.171357    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f28d48-6fc75df778-4x22w"]
Oct 31 20:06:41.178977 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:41.178955    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f28d48-6fc75df778-4x22w"]
Oct 31 20:06:41.475527 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:41.475498    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:41.617815 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:41.617783    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17a5f4e3-1aba-4e82-aa9a-c50c4710d782" path="/var/lib/kubelet/pods/17a5f4e3-1aba-4e82-aa9a-c50c4710d782/volumes"
Oct 31 20:06:41.618211 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:41.618197    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2dd3d751-d8d8-4fa7-b1bf-5c2da531f07d" path="/var/lib/kubelet/pods/2dd3d751-d8d8-4fa7-b1bf-5c2da531f07d/volumes"
Oct 31 20:06:41.618625 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:41.618611    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7599e792-03f8-41bf-9cda-aca650a9022b" path="/var/lib/kubelet/pods/7599e792-03f8-41bf-9cda-aca650a9022b/volumes"
Oct 31 20:06:41.821764 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:41.821722    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm"
Oct 31 20:06:41.863423 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:41.863391    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d78f81fc-d267-407f-8bd7-51ac2d787720-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d78f81fc-d267-407f-8bd7-51ac2d787720-istio-podinfo\") pod \"d78f81fc-d267-407f-8bd7-51ac2d787720\" (UID: \"d78f81fc-d267-407f-8bd7-51ac2d787720\") "
Oct 31 20:06:41.863858 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:41.863444    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d78f81fc-d267-407f-8bd7-51ac2d787720-kube-api-access-wpr6x\" (UniqueName: \"kubernetes.io/projected/d78f81fc-d267-407f-8bd7-51ac2d787720-kube-api-access-wpr6x\") pod \"d78f81fc-d267-407f-8bd7-51ac2d787720\" (UID: \"d78f81fc-d267-407f-8bd7-51ac2d787720\") "
Oct 31 20:06:41.863858 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:41.863472    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d78f81fc-d267-407f-8bd7-51ac2d787720-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d78f81fc-d267-407f-8bd7-51ac2d787720-workload-certs\") pod \"d78f81fc-d267-407f-8bd7-51ac2d787720\" (UID: \"d78f81fc-d267-407f-8bd7-51ac2d787720\") "
Oct 31 20:06:41.863858 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:41.863529    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d78f81fc-d267-407f-8bd7-51ac2d787720-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d78f81fc-d267-407f-8bd7-51ac2d787720-workload-socket\") pod \"d78f81fc-d267-407f-8bd7-51ac2d787720\" (UID: \"d78f81fc-d267-407f-8bd7-51ac2d787720\") "
Oct 31 20:06:41.863858 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:41.863592    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d78f81fc-d267-407f-8bd7-51ac2d787720-istio-token\" (UniqueName: \"kubernetes.io/projected/d78f81fc-d267-407f-8bd7-51ac2d787720-istio-token\") pod \"d78f81fc-d267-407f-8bd7-51ac2d787720\" (UID: \"d78f81fc-d267-407f-8bd7-51ac2d787720\") "
Oct 31 20:06:41.863858 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:41.863656    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d78f81fc-d267-407f-8bd7-51ac2d787720-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d78f81fc-d267-407f-8bd7-51ac2d787720-credential-socket\") pod \"d78f81fc-d267-407f-8bd7-51ac2d787720\" (UID: \"d78f81fc-d267-407f-8bd7-51ac2d787720\") "
Oct 31 20:06:41.863858 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:41.863694    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d78f81fc-d267-407f-8bd7-51ac2d787720-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d78f81fc-d267-407f-8bd7-51ac2d787720-istio-envoy\") pod \"d78f81fc-d267-407f-8bd7-51ac2d787720\" (UID: \"d78f81fc-d267-407f-8bd7-51ac2d787720\") "
Oct 31 20:06:41.863858 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:41.863729    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d78f81fc-d267-407f-8bd7-51ac2d787720-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d78f81fc-d267-407f-8bd7-51ac2d787720-istio-data\") pod \"d78f81fc-d267-407f-8bd7-51ac2d787720\" (UID: \"d78f81fc-d267-407f-8bd7-51ac2d787720\") "
Oct 31 20:06:41.863858 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:41.863772    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d78f81fc-d267-407f-8bd7-51ac2d787720-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d78f81fc-d267-407f-8bd7-51ac2d787720-istiod-ca-cert\") pod \"d78f81fc-d267-407f-8bd7-51ac2d787720\" (UID: \"d78f81fc-d267-407f-8bd7-51ac2d787720\") "
Oct 31 20:06:41.863858 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:41.863810    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d78f81fc-d267-407f-8bd7-51ac2d787720-workload-certs" pod "d78f81fc-d267-407f-8bd7-51ac2d787720" (UID: "d78f81fc-d267-407f-8bd7-51ac2d787720"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:06:41.864519 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:41.864003    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d78f81fc-d267-407f-8bd7-51ac2d787720-credential-socket" pod "d78f81fc-d267-407f-8bd7-51ac2d787720" (UID: "d78f81fc-d267-407f-8bd7-51ac2d787720"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:06:41.864519 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:41.864272    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d78f81fc-d267-407f-8bd7-51ac2d787720-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:41.864519 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:41.864295    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d78f81fc-d267-407f-8bd7-51ac2d787720-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:41.864519 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:41.864365    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d78f81fc-d267-407f-8bd7-51ac2d787720-istiod-ca-cert" pod "d78f81fc-d267-407f-8bd7-51ac2d787720" (UID: "d78f81fc-d267-407f-8bd7-51ac2d787720"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:06:41.864519 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:41.864445    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d78f81fc-d267-407f-8bd7-51ac2d787720-istio-data" pod "d78f81fc-d267-407f-8bd7-51ac2d787720" (UID: "d78f81fc-d267-407f-8bd7-51ac2d787720"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:06:41.864519 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:41.864512    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d78f81fc-d267-407f-8bd7-51ac2d787720-workload-socket" pod "d78f81fc-d267-407f-8bd7-51ac2d787720" (UID: "d78f81fc-d267-407f-8bd7-51ac2d787720"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:06:41.866057 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:41.866031    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d78f81fc-d267-407f-8bd7-51ac2d787720-kube-api-access-wpr6x" pod "d78f81fc-d267-407f-8bd7-51ac2d787720" (UID: "d78f81fc-d267-407f-8bd7-51ac2d787720"). InnerVolumeSpecName "kube-api-access-wpr6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:06:41.866269 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:41.866251    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d78f81fc-d267-407f-8bd7-51ac2d787720-istio-envoy" pod "d78f81fc-d267-407f-8bd7-51ac2d787720" (UID: "d78f81fc-d267-407f-8bd7-51ac2d787720"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:06:41.866381 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:41.866263    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d78f81fc-d267-407f-8bd7-51ac2d787720-istio-podinfo" pod "d78f81fc-d267-407f-8bd7-51ac2d787720" (UID: "d78f81fc-d267-407f-8bd7-51ac2d787720"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 20:06:41.866715 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:41.866696    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d78f81fc-d267-407f-8bd7-51ac2d787720-istio-token" pod "d78f81fc-d267-407f-8bd7-51ac2d787720" (UID: "d78f81fc-d267-407f-8bd7-51ac2d787720"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:06:41.965948 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:41.965876    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d78f81fc-d267-407f-8bd7-51ac2d787720-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:41.965948 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:41.965913    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d78f81fc-d267-407f-8bd7-51ac2d787720-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:41.965948 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:41.965930    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d78f81fc-d267-407f-8bd7-51ac2d787720-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:41.965948 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:41.965944    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d78f81fc-d267-407f-8bd7-51ac2d787720-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:41.966194 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:41.965956    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d78f81fc-d267-407f-8bd7-51ac2d787720-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:41.966194 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:41.965970    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d78f81fc-d267-407f-8bd7-51ac2d787720-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:41.966194 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:41.965982    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpr6x\" (UniqueName: \"kubernetes.io/projected/d78f81fc-d267-407f-8bd7-51ac2d787720-kube-api-access-wpr6x\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:42.122760 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:42.122681    4645 generic.go:356] "Generic (PLEG): container finished" podID="d78f81fc-d267-407f-8bd7-51ac2d787720" containerID="2cf9ad422be8e2ec90162aa8aebe6302089535b2a06dd8da963a84db064ea742" exitCode=0
Oct 31 20:06:42.122760 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:42.122739    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm"
Oct 31 20:06:42.122930 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:42.122761    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm" event={"ID":"d78f81fc-d267-407f-8bd7-51ac2d787720","Type":"ContainerDied","Data":"2cf9ad422be8e2ec90162aa8aebe6302089535b2a06dd8da963a84db064ea742"}
Oct 31 20:06:42.122930 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:42.122812    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm" event={"ID":"d78f81fc-d267-407f-8bd7-51ac2d787720","Type":"ContainerDied","Data":"a270d828d57a88d096c122a801235f869a7a5a825044fd3081eef3ea34f1b946"}
Oct 31 20:06:42.122930 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:42.122841    4645 scope.go:122] "RemoveContainer" containerID="2cf9ad422be8e2ec90162aa8aebe6302089535b2a06dd8da963a84db064ea742"
Oct 31 20:06:42.133665 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:42.133648    4645 scope.go:122] "RemoveContainer" containerID="2471599d7086c3806a504ba57ca06e314c6c6f0d2a66edd6d38f44acd7e05b97"
Oct 31 20:06:42.143169 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:42.143149    4645 scope.go:122] "RemoveContainer" containerID="2cf9ad422be8e2ec90162aa8aebe6302089535b2a06dd8da963a84db064ea742"
Oct 31 20:06:42.143446 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:06:42.143420    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cf9ad422be8e2ec90162aa8aebe6302089535b2a06dd8da963a84db064ea742\": container with ID starting with 2cf9ad422be8e2ec90162aa8aebe6302089535b2a06dd8da963a84db064ea742 not found: ID does not exist" containerID="2cf9ad422be8e2ec90162aa8aebe6302089535b2a06dd8da963a84db064ea742"
Oct 31 20:06:42.143507 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:42.143453    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cf9ad422be8e2ec90162aa8aebe6302089535b2a06dd8da963a84db064ea742"} err="failed to get container status \"2cf9ad422be8e2ec90162aa8aebe6302089535b2a06dd8da963a84db064ea742\": rpc error: code = NotFound desc = could not find container \"2cf9ad422be8e2ec90162aa8aebe6302089535b2a06dd8da963a84db064ea742\": container with ID starting with 2cf9ad422be8e2ec90162aa8aebe6302089535b2a06dd8da963a84db064ea742 not found: ID does not exist"
Oct 31 20:06:42.143507 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:42.143470    4645 scope.go:122] "RemoveContainer" containerID="2471599d7086c3806a504ba57ca06e314c6c6f0d2a66edd6d38f44acd7e05b97"
Oct 31 20:06:42.143727 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:06:42.143708    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2471599d7086c3806a504ba57ca06e314c6c6f0d2a66edd6d38f44acd7e05b97\": container with ID starting with 2471599d7086c3806a504ba57ca06e314c6c6f0d2a66edd6d38f44acd7e05b97 not found: ID does not exist" containerID="2471599d7086c3806a504ba57ca06e314c6c6f0d2a66edd6d38f44acd7e05b97"
Oct 31 20:06:42.143791 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:42.143737    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2471599d7086c3806a504ba57ca06e314c6c6f0d2a66edd6d38f44acd7e05b97"} err="failed to get container status \"2471599d7086c3806a504ba57ca06e314c6c6f0d2a66edd6d38f44acd7e05b97\": rpc error: code = NotFound desc = could not find container \"2471599d7086c3806a504ba57ca06e314c6c6f0d2a66edd6d38f44acd7e05b97\": container with ID starting with 2471599d7086c3806a504ba57ca06e314c6c6f0d2a66edd6d38f44acd7e05b97 not found: ID does not exist"
Oct 31 20:06:42.147837 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:42.147814    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm"]
Oct 31 20:06:42.154459 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:42.154437    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-8730d5-openshift-default-59d8dd4587-9jcjm"]
Oct 31 20:06:42.356593 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:42.356561    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:43.220209 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:43.220171    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:43.618178 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:43.618102    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d78f81fc-d267-407f-8bd7-51ac2d787720" path="/var/lib/kubelet/pods/d78f81fc-d267-407f-8bd7-51ac2d787720/volumes"
Oct 31 20:06:44.056657 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:44.056616    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:44.780383 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:44.780322    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:45.052376 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:45.052266    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:45.827764 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:45.827734    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:46.082959 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:46.082702    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:46.954281 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:46.954243    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:48.210258 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:48.210183    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:49.237384 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:49.236407    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:49.506055 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:49.505976    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:49.550943 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:49.550911    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc"]
Oct 31 20:06:49.551168 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:49.551124    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc" podUID="7309f62d-7bc0-4677-a81d-3b306f5eca82" containerName="istio-proxy" containerID="cri-o://b9df2ec920c166a8700d224b7a0c44d5f31604992c3a02a9eccd41773f057ba6" gracePeriod=30
Oct 31 20:06:49.574555 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:49.574526    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc"]
Oct 31 20:06:49.574708 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:49.574690    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc" podUID="8d9801bc-b873-4f98-9395-0a19686d6b8a" containerName="istio-proxy" containerID="cri-o://b34933e0988ad6af10c3bb56ce43816fbd2140c76937a4014d6b6d176f4ffed7" gracePeriod=30
Oct 31 20:06:49.712236 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:49.712179    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:50.086549 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:50.086510    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7fe845-kstgm"]
Oct 31 20:06:50.087000 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:50.086984    4645 cpu_manager.go:414] "containerMap: removing container" podUID="17a5f4e3-1aba-4e82-aa9a-c50c4710d782" containerName="revoke-keys"
Oct 31 20:06:50.087056 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:50.087001    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17a5f4e3-1aba-4e82-aa9a-c50c4710d782" containerName="revoke-keys"
Oct 31 20:06:50.087056 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:50.087023    4645 cpu_manager.go:414] "containerMap: removing container" podUID="d78f81fc-d267-407f-8bd7-51ac2d787720" containerName="istio-proxy"
Oct 31 20:06:50.087056 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:50.087028    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d78f81fc-d267-407f-8bd7-51ac2d787720" containerName="istio-proxy"
Oct 31 20:06:50.087056 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:50.087042    4645 cpu_manager.go:414] "containerMap: removing container" podUID="d78f81fc-d267-407f-8bd7-51ac2d787720" containerName="istio-proxy"
Oct 31 20:06:50.087056 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:50.087048    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d78f81fc-d267-407f-8bd7-51ac2d787720" containerName="istio-proxy"
Oct 31 20:06:50.087056 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:50.087055    4645 cpu_manager.go:414] "containerMap: removing container" podUID="2dd3d751-d8d8-4fa7-b1bf-5c2da531f07d" containerName="payload-processing"
Oct 31 20:06:50.087253 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:50.087060    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dd3d751-d8d8-4fa7-b1bf-5c2da531f07d" containerName="payload-processing"
Oct 31 20:06:50.087253 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:50.087070    4645 cpu_manager.go:414] "containerMap: removing container" podUID="7599e792-03f8-41bf-9cda-aca650a9022b" containerName="payload-pre-processing"
Oct 31 20:06:50.087253 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:50.087075    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7599e792-03f8-41bf-9cda-aca650a9022b" containerName="payload-pre-processing"
Oct 31 20:06:50.087253 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:50.087141    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="17a5f4e3-1aba-4e82-aa9a-c50c4710d782" containerName="revoke-keys"
Oct 31 20:06:50.087253 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:50.087154    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="d78f81fc-d267-407f-8bd7-51ac2d787720" containerName="istio-proxy"
Oct 31 20:06:50.087253 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:50.087163    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3b22b1e-9d49-4a54-b016-bc504f03390c" containerName="istio-proxy"
Oct 31 20:06:50.087253 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:50.087169    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="d78f81fc-d267-407f-8bd7-51ac2d787720" containerName="istio-proxy"
Oct 31 20:06:50.087253 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:50.087175    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dd3d751-d8d8-4fa7-b1bf-5c2da531f07d" containerName="payload-processing"
Oct 31 20:06:50.087253 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:50.087183    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="7599e792-03f8-41bf-9cda-aca650a9022b" containerName="payload-pre-processing"
Oct 31 20:06:50.091780 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:50.091761    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7fe845-kstgm"
Oct 31 20:06:50.099596 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:50.099579    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:06:50.106870 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:50.106842    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7fe845-kstgm"]
Oct 31 20:06:50.153852 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:50.153813    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eff0a26e-843b-41eb-9125-6273720e3329-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-7fe845-kstgm\" (UID: \"eff0a26e-843b-41eb-9125-6273720e3329\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7fe845-kstgm"
Oct 31 20:06:50.154039 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:50.153988    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eff0a26e-843b-41eb-9125-6273720e3329-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-7fe845-kstgm\" (UID: \"eff0a26e-843b-41eb-9125-6273720e3329\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7fe845-kstgm"
Oct 31 20:06:50.154039 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:50.154023    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppv98\" (UniqueName: \"kubernetes.io/projected/eff0a26e-843b-41eb-9125-6273720e3329-kube-api-access-ppv98\") pod \"maas-api-revoke-keys-e2e-ipp-b-7fe845-kstgm\" (UID: \"eff0a26e-843b-41eb-9125-6273720e3329\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7fe845-kstgm"
Oct 31 20:06:50.199240 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:50.199206    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 20:06:50.255373 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:50.255302    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eff0a26e-843b-41eb-9125-6273720e3329-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-7fe845-kstgm\" (UID: \"eff0a26e-843b-41eb-9125-6273720e3329\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7fe845-kstgm"
Oct 31 20:06:50.255850 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:50.255434    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eff0a26e-843b-41eb-9125-6273720e3329-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-7fe845-kstgm\" (UID: \"eff0a26e-843b-41eb-9125-6273720e3329\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7fe845-kstgm"
Oct 31 20:06:50.255850 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:50.255458    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppv98\" (UniqueName: \"kubernetes.io/projected/eff0a26e-843b-41eb-9125-6273720e3329-kube-api-access-ppv98\") pod \"maas-api-revoke-keys-e2e-ipp-b-7fe845-kstgm\" (UID: \"eff0a26e-843b-41eb-9125-6273720e3329\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7fe845-kstgm"
Oct 31 20:06:50.256051 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:50.256026    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eff0a26e-843b-41eb-9125-6273720e3329-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-7fe845-kstgm\" (UID: \"eff0a26e-843b-41eb-9125-6273720e3329\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7fe845-kstgm"
Oct 31 20:06:50.257904 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:50.257878    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eff0a26e-843b-41eb-9125-6273720e3329-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-7fe845-kstgm\" (UID: \"eff0a26e-843b-41eb-9125-6273720e3329\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7fe845-kstgm"
Oct 31 20:06:50.279785 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:50.279754    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppv98\" (UniqueName: \"kubernetes.io/projected/eff0a26e-843b-41eb-9125-6273720e3329-kube-api-access-ppv98\") pod \"maas-api-revoke-keys-e2e-ipp-b-7fe845-kstgm\" (UID: \"eff0a26e-843b-41eb-9125-6273720e3329\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7fe845-kstgm"
Oct 31 20:06:50.405726 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:50.405635    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7fe845-kstgm"
Oct 31 20:06:50.686659 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:50.686622    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7fe845-kstgm"]
Oct 31 20:06:50.690116 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:06:50.690086    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeff0a26e_843b_41eb_9125_6273720e3329.slice/crio-9f55f625ed901a190a610e75c78f69fb356767ce441e5376cf29962dc2c16d12 WatchSource:0}: Error finding container 9f55f625ed901a190a610e75c78f69fb356767ce441e5376cf29962dc2c16d12: Status 404 returned error can't find the container with id 9f55f625ed901a190a610e75c78f69fb356767ce441e5376cf29962dc2c16d12
Oct 31 20:06:50.857186 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:50.857143    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:51.226777 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:51.226698    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7fe845-kstgm" event={"ID":"eff0a26e-843b-41eb-9125-6273720e3329","Type":"ContainerStarted","Data":"48c237d2655d1c08acf6f1f7ce2d5b20f8319f4fe37800f7d1a250d107a20a1b"}
Oct 31 20:06:51.226777 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:51.226748    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7fe845-kstgm" event={"ID":"eff0a26e-843b-41eb-9125-6273720e3329","Type":"ContainerStarted","Data":"9f55f625ed901a190a610e75c78f69fb356767ce441e5376cf29962dc2c16d12"}
Oct 31 20:06:51.255707 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:51.255654    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7fe845-kstgm" podStartSLOduration=1.2556396969999999 podStartE2EDuration="1.255639697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:06:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:06:51.252323643 +0000 UTC m=+2841.825557771" watchObservedRunningTime="2026-10-31 20:06:51.255639697 +0000 UTC m=+2841.828873888"
Oct 31 20:06:51.642068 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:51.641997    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:52.234824 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:52.234786    4645 generic.go:356] "Generic (PLEG): container finished" podID="eff0a26e-843b-41eb-9125-6273720e3329" containerID="48c237d2655d1c08acf6f1f7ce2d5b20f8319f4fe37800f7d1a250d107a20a1b" exitCode=0
Oct 31 20:06:52.234991 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:52.234837    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7fe845-kstgm" event={"ID":"eff0a26e-843b-41eb-9125-6273720e3329","Type":"ContainerDied","Data":"48c237d2655d1c08acf6f1f7ce2d5b20f8319f4fe37800f7d1a250d107a20a1b"}
Oct 31 20:06:52.480016 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:52.479978    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:53.272856 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:53.272820    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:53.379986 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:53.379962    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7fe845-kstgm"
Oct 31 20:06:53.447765 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:53.447726    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:53.504041 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:53.503949    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eff0a26e-843b-41eb-9125-6273720e3329-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eff0a26e-843b-41eb-9125-6273720e3329-maas-api-xks-ca\") pod \"eff0a26e-843b-41eb-9125-6273720e3329\" (UID: \"eff0a26e-843b-41eb-9125-6273720e3329\") "
Oct 31 20:06:53.504468 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:53.504104    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eff0a26e-843b-41eb-9125-6273720e3329-kube-api-access-ppv98\" (UniqueName: \"kubernetes.io/projected/eff0a26e-843b-41eb-9125-6273720e3329-kube-api-access-ppv98\") pod \"eff0a26e-843b-41eb-9125-6273720e3329\" (UID: \"eff0a26e-843b-41eb-9125-6273720e3329\") "
Oct 31 20:06:53.504468 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:53.504139    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eff0a26e-843b-41eb-9125-6273720e3329-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eff0a26e-843b-41eb-9125-6273720e3329-maas-api-openshift-ca\") pod \"eff0a26e-843b-41eb-9125-6273720e3329\" (UID: \"eff0a26e-843b-41eb-9125-6273720e3329\") "
Oct 31 20:06:53.504597 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:53.504568    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eff0a26e-843b-41eb-9125-6273720e3329-maas-api-openshift-ca" pod "eff0a26e-843b-41eb-9125-6273720e3329" (UID: "eff0a26e-843b-41eb-9125-6273720e3329"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:06:53.506085 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:53.506062    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eff0a26e-843b-41eb-9125-6273720e3329-maas-api-xks-ca" pod "eff0a26e-843b-41eb-9125-6273720e3329" (UID: "eff0a26e-843b-41eb-9125-6273720e3329"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:06:53.506255 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:53.506231    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eff0a26e-843b-41eb-9125-6273720e3329-kube-api-access-ppv98" pod "eff0a26e-843b-41eb-9125-6273720e3329" (UID: "eff0a26e-843b-41eb-9125-6273720e3329"). InnerVolumeSpecName "kube-api-access-ppv98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:06:53.600658 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:53.600618    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b"]
Oct 31 20:06:53.600865 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:53.600837    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b" podUID="710e3436-3828-45fb-b090-e28d1147cbfe" containerName="istio-proxy" containerID="cri-o://02bcde84b1170dd9c98e7e4e3557752013a43e5911a91514dbbd3ac1076bf67f" gracePeriod=30
Oct 31 20:06:53.605298 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:53.605268    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eff0a26e-843b-41eb-9125-6273720e3329-maas-api-xks-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:53.605489 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:53.605301    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppv98\" (UniqueName: \"kubernetes.io/projected/eff0a26e-843b-41eb-9125-6273720e3329-kube-api-access-ppv98\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:53.605489 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:53.605318    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eff0a26e-843b-41eb-9125-6273720e3329-maas-api-openshift-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:53.751361 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:53.750052    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:54.191206 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:54.191167    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c9b9a1-klfx5"]
Oct 31 20:06:54.191657 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:54.191641    4645 cpu_manager.go:414] "containerMap: removing container" podUID="eff0a26e-843b-41eb-9125-6273720e3329" containerName="revoke-keys"
Oct 31 20:06:54.191715 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:54.191658    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eff0a26e-843b-41eb-9125-6273720e3329" containerName="revoke-keys"
Oct 31 20:06:54.191753 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:54.191743    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="eff0a26e-843b-41eb-9125-6273720e3329" containerName="revoke-keys"
Oct 31 20:06:54.195019 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:54.195000    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c9b9a1-klfx5"
Oct 31 20:06:54.210072 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:54.210045    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c9b9a1-klfx5"]
Oct 31 20:06:54.250035 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:54.250005    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7fe845-kstgm"
Oct 31 20:06:54.250244 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:54.250043    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7fe845-kstgm" event={"ID":"eff0a26e-843b-41eb-9125-6273720e3329","Type":"ContainerDied","Data":"9f55f625ed901a190a610e75c78f69fb356767ce441e5376cf29962dc2c16d12"}
Oct 31 20:06:54.250244 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:54.250106    4645 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f55f625ed901a190a610e75c78f69fb356767ce441e5376cf29962dc2c16d12"
Oct 31 20:06:54.346839 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:54.346804    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a8637ab-6147-4718-8be4-6af8f696a40d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c9b9a1-klfx5\" (UID: \"6a8637ab-6147-4718-8be4-6af8f696a40d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c9b9a1-klfx5"
Oct 31 20:06:54.347015 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:54.346864    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a8637ab-6147-4718-8be4-6af8f696a40d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c9b9a1-klfx5\" (UID: \"6a8637ab-6147-4718-8be4-6af8f696a40d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c9b9a1-klfx5"
Oct 31 20:06:54.347015 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:54.346984    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwxhz\" (UniqueName: \"kubernetes.io/projected/6a8637ab-6147-4718-8be4-6af8f696a40d-kube-api-access-fwxhz\") pod \"maas-api-revoke-keys-e2e-ipp-b-c9b9a1-klfx5\" (UID: \"6a8637ab-6147-4718-8be4-6af8f696a40d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c9b9a1-klfx5"
Oct 31 20:06:54.397014 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:54.396975    4645 patch_prober.go:29] interesting pod/e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 20:06:54.397189 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:54.397039    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc" podUID="8d9801bc-b873-4f98-9395-0a19686d6b8a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 20:06:54.447896 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:54.447810    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwxhz\" (UniqueName: \"kubernetes.io/projected/6a8637ab-6147-4718-8be4-6af8f696a40d-kube-api-access-fwxhz\") pod \"maas-api-revoke-keys-e2e-ipp-b-c9b9a1-klfx5\" (UID: \"6a8637ab-6147-4718-8be4-6af8f696a40d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c9b9a1-klfx5"
Oct 31 20:06:54.447896 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:54.447876    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a8637ab-6147-4718-8be4-6af8f696a40d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c9b9a1-klfx5\" (UID: \"6a8637ab-6147-4718-8be4-6af8f696a40d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c9b9a1-klfx5"
Oct 31 20:06:54.448113 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:54.447938    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a8637ab-6147-4718-8be4-6af8f696a40d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c9b9a1-klfx5\" (UID: \"6a8637ab-6147-4718-8be4-6af8f696a40d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c9b9a1-klfx5"
Oct 31 20:06:54.448673 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:54.448647    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a8637ab-6147-4718-8be4-6af8f696a40d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c9b9a1-klfx5\" (UID: \"6a8637ab-6147-4718-8be4-6af8f696a40d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c9b9a1-klfx5"
Oct 31 20:06:54.450237 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:54.450214    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a8637ab-6147-4718-8be4-6af8f696a40d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c9b9a1-klfx5\" (UID: \"6a8637ab-6147-4718-8be4-6af8f696a40d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c9b9a1-klfx5"
Oct 31 20:06:54.461549 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:54.461521    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwxhz\" (UniqueName: \"kubernetes.io/projected/6a8637ab-6147-4718-8be4-6af8f696a40d-kube-api-access-fwxhz\") pod \"maas-api-revoke-keys-e2e-ipp-b-c9b9a1-klfx5\" (UID: \"6a8637ab-6147-4718-8be4-6af8f696a40d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c9b9a1-klfx5"
Oct 31 20:06:54.509585 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:54.509558    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c9b9a1-klfx5"
Oct 31 20:06:54.797429 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:54.797394    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c9b9a1-klfx5"]
Oct 31 20:06:54.799229 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:06:54.799205    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a8637ab_6147_4718_8be4_6af8f696a40d.slice/crio-19071a8718150c1e6c4e9ada1bb0511b3cf25ea628b525e1c84bff90228f5dee WatchSource:0}: Error finding container 19071a8718150c1e6c4e9ada1bb0511b3cf25ea628b525e1c84bff90228f5dee: Status 404 returned error can't find the container with id 19071a8718150c1e6c4e9ada1bb0511b3cf25ea628b525e1c84bff90228f5dee
Oct 31 20:06:54.802557 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:54.802536    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc"
Oct 31 20:06:54.839849 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:54.839825    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc"
Oct 31 20:06:54.954958 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:54.954922    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7309f62d-7bc0-4677-a81d-3b306f5eca82-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7309f62d-7bc0-4677-a81d-3b306f5eca82-istiod-ca-cert\") pod \"7309f62d-7bc0-4677-a81d-3b306f5eca82\" (UID: \"7309f62d-7bc0-4677-a81d-3b306f5eca82\") "
Oct 31 20:06:54.954958 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:54.954959    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7309f62d-7bc0-4677-a81d-3b306f5eca82-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7309f62d-7bc0-4677-a81d-3b306f5eca82-istio-podinfo\") pod \"7309f62d-7bc0-4677-a81d-3b306f5eca82\" (UID: \"7309f62d-7bc0-4677-a81d-3b306f5eca82\") "
Oct 31 20:06:54.955185 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:54.954982    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d9801bc-b873-4f98-9395-0a19686d6b8a-istio-token\" (UniqueName: \"kubernetes.io/projected/8d9801bc-b873-4f98-9395-0a19686d6b8a-istio-token\") pod \"8d9801bc-b873-4f98-9395-0a19686d6b8a\" (UID: \"8d9801bc-b873-4f98-9395-0a19686d6b8a\") "
Oct 31 20:06:54.955185 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:54.955007    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d9801bc-b873-4f98-9395-0a19686d6b8a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d9801bc-b873-4f98-9395-0a19686d6b8a-istio-envoy\") pod \"8d9801bc-b873-4f98-9395-0a19686d6b8a\" (UID: \"8d9801bc-b873-4f98-9395-0a19686d6b8a\") "
Oct 31 20:06:54.955185 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:54.955139    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d9801bc-b873-4f98-9395-0a19686d6b8a-kube-api-access-lz8db\" (UniqueName: \"kubernetes.io/projected/8d9801bc-b873-4f98-9395-0a19686d6b8a-kube-api-access-lz8db\") pod \"8d9801bc-b873-4f98-9395-0a19686d6b8a\" (UID: \"8d9801bc-b873-4f98-9395-0a19686d6b8a\") "
Oct 31 20:06:54.955398 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:54.955259    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7309f62d-7bc0-4677-a81d-3b306f5eca82-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7309f62d-7bc0-4677-a81d-3b306f5eca82-workload-socket\") pod \"7309f62d-7bc0-4677-a81d-3b306f5eca82\" (UID: \"7309f62d-7bc0-4677-a81d-3b306f5eca82\") "
Oct 31 20:06:54.955398 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:54.955376    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7309f62d-7bc0-4677-a81d-3b306f5eca82-istiod-ca-cert" pod "7309f62d-7bc0-4677-a81d-3b306f5eca82" (UID: "7309f62d-7bc0-4677-a81d-3b306f5eca82"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:06:54.955565 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:54.955292    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d9801bc-b873-4f98-9395-0a19686d6b8a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d9801bc-b873-4f98-9395-0a19686d6b8a-credential-socket\") pod \"8d9801bc-b873-4f98-9395-0a19686d6b8a\" (UID: \"8d9801bc-b873-4f98-9395-0a19686d6b8a\") "
Oct 31 20:06:54.955644 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:54.955621    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7309f62d-7bc0-4677-a81d-3b306f5eca82-istio-token\" (UniqueName: \"kubernetes.io/projected/7309f62d-7bc0-4677-a81d-3b306f5eca82-istio-token\") pod \"7309f62d-7bc0-4677-a81d-3b306f5eca82\" (UID: \"7309f62d-7bc0-4677-a81d-3b306f5eca82\") "
Oct 31 20:06:54.955706 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:54.955552    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7309f62d-7bc0-4677-a81d-3b306f5eca82-workload-socket" pod "7309f62d-7bc0-4677-a81d-3b306f5eca82" (UID: "7309f62d-7bc0-4677-a81d-3b306f5eca82"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:06:54.955706 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:54.955561    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d9801bc-b873-4f98-9395-0a19686d6b8a-credential-socket" pod "8d9801bc-b873-4f98-9395-0a19686d6b8a" (UID: "8d9801bc-b873-4f98-9395-0a19686d6b8a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:06:54.955875 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:54.955659    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7309f62d-7bc0-4677-a81d-3b306f5eca82-kube-api-access-bhsdt\" (UniqueName: \"kubernetes.io/projected/7309f62d-7bc0-4677-a81d-3b306f5eca82-kube-api-access-bhsdt\") pod \"7309f62d-7bc0-4677-a81d-3b306f5eca82\" (UID: \"7309f62d-7bc0-4677-a81d-3b306f5eca82\") "
Oct 31 20:06:54.956015 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:54.955997    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d9801bc-b873-4f98-9395-0a19686d6b8a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d9801bc-b873-4f98-9395-0a19686d6b8a-istiod-ca-cert\") pod \"8d9801bc-b873-4f98-9395-0a19686d6b8a\" (UID: \"8d9801bc-b873-4f98-9395-0a19686d6b8a\") "
Oct 31 20:06:54.956556 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:54.956537    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d9801bc-b873-4f98-9395-0a19686d6b8a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d9801bc-b873-4f98-9395-0a19686d6b8a-workload-certs\") pod \"8d9801bc-b873-4f98-9395-0a19686d6b8a\" (UID: \"8d9801bc-b873-4f98-9395-0a19686d6b8a\") "
Oct 31 20:06:54.956832 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:54.956443    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d9801bc-b873-4f98-9395-0a19686d6b8a-istiod-ca-cert" pod "8d9801bc-b873-4f98-9395-0a19686d6b8a" (UID: "8d9801bc-b873-4f98-9395-0a19686d6b8a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:06:54.956925 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:54.956809    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7309f62d-7bc0-4677-a81d-3b306f5eca82-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7309f62d-7bc0-4677-a81d-3b306f5eca82-workload-certs\") pod \"7309f62d-7bc0-4677-a81d-3b306f5eca82\" (UID: \"7309f62d-7bc0-4677-a81d-3b306f5eca82\") "
Oct 31 20:06:54.956925 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:54.956909    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7309f62d-7bc0-4677-a81d-3b306f5eca82-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7309f62d-7bc0-4677-a81d-3b306f5eca82-credential-socket\") pod \"7309f62d-7bc0-4677-a81d-3b306f5eca82\" (UID: \"7309f62d-7bc0-4677-a81d-3b306f5eca82\") "
Oct 31 20:06:54.957043 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:54.956946    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d9801bc-b873-4f98-9395-0a19686d6b8a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d9801bc-b873-4f98-9395-0a19686d6b8a-workload-socket\") pod \"8d9801bc-b873-4f98-9395-0a19686d6b8a\" (UID: \"8d9801bc-b873-4f98-9395-0a19686d6b8a\") "
Oct 31 20:06:54.957043 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:54.956981    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7309f62d-7bc0-4677-a81d-3b306f5eca82-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7309f62d-7bc0-4677-a81d-3b306f5eca82-istio-data\") pod \"7309f62d-7bc0-4677-a81d-3b306f5eca82\" (UID: \"7309f62d-7bc0-4677-a81d-3b306f5eca82\") "
Oct 31 20:06:54.957043 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:54.957025    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d9801bc-b873-4f98-9395-0a19686d6b8a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d9801bc-b873-4f98-9395-0a19686d6b8a-istio-data\") pod \"8d9801bc-b873-4f98-9395-0a19686d6b8a\" (UID: \"8d9801bc-b873-4f98-9395-0a19686d6b8a\") "
Oct 31 20:06:54.957308 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:54.957061    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7309f62d-7bc0-4677-a81d-3b306f5eca82-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7309f62d-7bc0-4677-a81d-3b306f5eca82-istio-envoy\") pod \"7309f62d-7bc0-4677-a81d-3b306f5eca82\" (UID: \"7309f62d-7bc0-4677-a81d-3b306f5eca82\") "
Oct 31 20:06:54.957308 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:54.957152    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8d9801bc-b873-4f98-9395-0a19686d6b8a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d9801bc-b873-4f98-9395-0a19686d6b8a-istio-podinfo\") pod \"8d9801bc-b873-4f98-9395-0a19686d6b8a\" (UID: \"8d9801bc-b873-4f98-9395-0a19686d6b8a\") "
Oct 31 20:06:54.957308 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:54.957280    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d9801bc-b873-4f98-9395-0a19686d6b8a-workload-certs" pod "8d9801bc-b873-4f98-9395-0a19686d6b8a" (UID: "8d9801bc-b873-4f98-9395-0a19686d6b8a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:06:54.957607 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:54.957578    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7309f62d-7bc0-4677-a81d-3b306f5eca82-istio-data" pod "7309f62d-7bc0-4677-a81d-3b306f5eca82" (UID: "7309f62d-7bc0-4677-a81d-3b306f5eca82"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:06:54.957607 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:54.957596    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7309f62d-7bc0-4677-a81d-3b306f5eca82-credential-socket" pod "7309f62d-7bc0-4677-a81d-3b306f5eca82" (UID: "7309f62d-7bc0-4677-a81d-3b306f5eca82"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:06:54.957761 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:54.957736    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7309f62d-7bc0-4677-a81d-3b306f5eca82-workload-certs" pod "7309f62d-7bc0-4677-a81d-3b306f5eca82" (UID: "7309f62d-7bc0-4677-a81d-3b306f5eca82"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:06:54.957846 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:54.957825    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d9801bc-b873-4f98-9395-0a19686d6b8a-workload-socket" pod "8d9801bc-b873-4f98-9395-0a19686d6b8a" (UID: "8d9801bc-b873-4f98-9395-0a19686d6b8a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:06:54.957922 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:54.957827    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d9801bc-b873-4f98-9395-0a19686d6b8a-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:54.957922 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:54.957872    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d9801bc-b873-4f98-9395-0a19686d6b8a-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:54.957922 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:54.957914    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7309f62d-7bc0-4677-a81d-3b306f5eca82-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:54.958104 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:54.957930    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7309f62d-7bc0-4677-a81d-3b306f5eca82-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:54.958104 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:54.957945    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7309f62d-7bc0-4677-a81d-3b306f5eca82-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:54.958104 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:54.957960    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7309f62d-7bc0-4677-a81d-3b306f5eca82-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:54.958104 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:54.957963    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d9801bc-b873-4f98-9395-0a19686d6b8a-istio-data" pod "8d9801bc-b873-4f98-9395-0a19686d6b8a" (UID: "8d9801bc-b873-4f98-9395-0a19686d6b8a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:06:54.958104 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:54.957975    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d9801bc-b873-4f98-9395-0a19686d6b8a-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:54.958446 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:54.958416    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7309f62d-7bc0-4677-a81d-3b306f5eca82-istio-podinfo" pod "7309f62d-7bc0-4677-a81d-3b306f5eca82" (UID: "7309f62d-7bc0-4677-a81d-3b306f5eca82"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 20:06:54.958511 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:54.958496    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d9801bc-b873-4f98-9395-0a19686d6b8a-istio-token" pod "8d9801bc-b873-4f98-9395-0a19686d6b8a" (UID: "8d9801bc-b873-4f98-9395-0a19686d6b8a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:06:54.960169 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:54.960035    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7309f62d-7bc0-4677-a81d-3b306f5eca82-istio-envoy" pod "7309f62d-7bc0-4677-a81d-3b306f5eca82" (UID: "7309f62d-7bc0-4677-a81d-3b306f5eca82"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:06:54.960169 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:54.960091    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7309f62d-7bc0-4677-a81d-3b306f5eca82-istio-token" pod "7309f62d-7bc0-4677-a81d-3b306f5eca82" (UID: "7309f62d-7bc0-4677-a81d-3b306f5eca82"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:06:54.960169 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:54.960119    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7309f62d-7bc0-4677-a81d-3b306f5eca82-kube-api-access-bhsdt" pod "7309f62d-7bc0-4677-a81d-3b306f5eca82" (UID: "7309f62d-7bc0-4677-a81d-3b306f5eca82"). InnerVolumeSpecName "kube-api-access-bhsdt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:06:54.960169 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:54.960142    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d9801bc-b873-4f98-9395-0a19686d6b8a-istio-envoy" pod "8d9801bc-b873-4f98-9395-0a19686d6b8a" (UID: "8d9801bc-b873-4f98-9395-0a19686d6b8a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:06:54.960169 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:54.960151    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d9801bc-b873-4f98-9395-0a19686d6b8a-kube-api-access-lz8db" pod "8d9801bc-b873-4f98-9395-0a19686d6b8a" (UID: "8d9801bc-b873-4f98-9395-0a19686d6b8a"). InnerVolumeSpecName "kube-api-access-lz8db". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:06:54.960609 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:54.960587    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d9801bc-b873-4f98-9395-0a19686d6b8a-istio-podinfo" pod "8d9801bc-b873-4f98-9395-0a19686d6b8a" (UID: "8d9801bc-b873-4f98-9395-0a19686d6b8a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 20:06:55.059575 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:55.059514    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7309f62d-7bc0-4677-a81d-3b306f5eca82-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:55.059575 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:55.059545    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhsdt\" (UniqueName: \"kubernetes.io/projected/7309f62d-7bc0-4677-a81d-3b306f5eca82-kube-api-access-bhsdt\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:55.059575 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:55.059558    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7309f62d-7bc0-4677-a81d-3b306f5eca82-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:55.059575 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:55.059567    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d9801bc-b873-4f98-9395-0a19686d6b8a-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:55.059575 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:55.059576    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d9801bc-b873-4f98-9395-0a19686d6b8a-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:55.059839 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:55.059585    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7309f62d-7bc0-4677-a81d-3b306f5eca82-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:55.059839 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:55.059593    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d9801bc-b873-4f98-9395-0a19686d6b8a-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:55.059839 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:55.059601    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7309f62d-7bc0-4677-a81d-3b306f5eca82-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:55.059839 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:55.059612    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d9801bc-b873-4f98-9395-0a19686d6b8a-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:55.059839 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:55.059624    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d9801bc-b873-4f98-9395-0a19686d6b8a-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:55.059839 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:55.059633    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lz8db\" (UniqueName: \"kubernetes.io/projected/8d9801bc-b873-4f98-9395-0a19686d6b8a-kube-api-access-lz8db\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:55.091052 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:55.090543    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:55.259142 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:55.259108    4645 generic.go:356] "Generic (PLEG): container finished" podID="7309f62d-7bc0-4677-a81d-3b306f5eca82" containerID="b9df2ec920c166a8700d224b7a0c44d5f31604992c3a02a9eccd41773f057ba6" exitCode=0
Oct 31 20:06:55.259351 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:55.259201    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc"
Oct 31 20:06:55.259351 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:55.259197    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc" event={"ID":"7309f62d-7bc0-4677-a81d-3b306f5eca82","Type":"ContainerDied","Data":"b9df2ec920c166a8700d224b7a0c44d5f31604992c3a02a9eccd41773f057ba6"}
Oct 31 20:06:55.259449 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:55.259388    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc" event={"ID":"7309f62d-7bc0-4677-a81d-3b306f5eca82","Type":"ContainerDied","Data":"be87b0bfca961aa4010714e627f42dfd4973491edfc3e5716f2a57bb6707fd1a"}
Oct 31 20:06:55.259449 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:55.259428    4645 scope.go:122] "RemoveContainer" containerID="b9df2ec920c166a8700d224b7a0c44d5f31604992c3a02a9eccd41773f057ba6"
Oct 31 20:06:55.261089 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:55.261060    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c9b9a1-klfx5" event={"ID":"6a8637ab-6147-4718-8be4-6af8f696a40d","Type":"ContainerStarted","Data":"17663dd659909937b7fed7b2d9abd901cf512b5dd21e352024b6e27abef64b5b"}
Oct 31 20:06:55.261246 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:55.261112    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c9b9a1-klfx5" event={"ID":"6a8637ab-6147-4718-8be4-6af8f696a40d","Type":"ContainerStarted","Data":"19071a8718150c1e6c4e9ada1bb0511b3cf25ea628b525e1c84bff90228f5dee"}
Oct 31 20:06:55.262579 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:55.262553    4645 generic.go:356] "Generic (PLEG): container finished" podID="8d9801bc-b873-4f98-9395-0a19686d6b8a" containerID="b34933e0988ad6af10c3bb56ce43816fbd2140c76937a4014d6b6d176f4ffed7" exitCode=0
Oct 31 20:06:55.262694 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:55.262628    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc"
Oct 31 20:06:55.262764 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:55.262628    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc" event={"ID":"8d9801bc-b873-4f98-9395-0a19686d6b8a","Type":"ContainerDied","Data":"b34933e0988ad6af10c3bb56ce43816fbd2140c76937a4014d6b6d176f4ffed7"}
Oct 31 20:06:55.262764 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:55.262752    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc" event={"ID":"8d9801bc-b873-4f98-9395-0a19686d6b8a","Type":"ContainerDied","Data":"6757c6067261b2a3a6c39d1db9ae1f6a55a95259617f981efac1e916525abfed"}
Oct 31 20:06:55.274819 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:55.274802    4645 scope.go:122] "RemoveContainer" containerID="766a6cd8f5a7309349b6d7d5323cb51a83328f800aa67b77d9a7f37492975905"
Oct 31 20:06:55.284611 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:55.284591    4645 scope.go:122] "RemoveContainer" containerID="b9df2ec920c166a8700d224b7a0c44d5f31604992c3a02a9eccd41773f057ba6"
Oct 31 20:06:55.284888 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:06:55.284866    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9df2ec920c166a8700d224b7a0c44d5f31604992c3a02a9eccd41773f057ba6\": container with ID starting with b9df2ec920c166a8700d224b7a0c44d5f31604992c3a02a9eccd41773f057ba6 not found: ID does not exist" containerID="b9df2ec920c166a8700d224b7a0c44d5f31604992c3a02a9eccd41773f057ba6"
Oct 31 20:06:55.284944 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:55.284897    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9df2ec920c166a8700d224b7a0c44d5f31604992c3a02a9eccd41773f057ba6"} err="failed to get container status \"b9df2ec920c166a8700d224b7a0c44d5f31604992c3a02a9eccd41773f057ba6\": rpc error: code = NotFound desc = could not find container \"b9df2ec920c166a8700d224b7a0c44d5f31604992c3a02a9eccd41773f057ba6\": container with ID starting with b9df2ec920c166a8700d224b7a0c44d5f31604992c3a02a9eccd41773f057ba6 not found: ID does not exist"
Oct 31 20:06:55.284944 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:55.284913    4645 scope.go:122] "RemoveContainer" containerID="766a6cd8f5a7309349b6d7d5323cb51a83328f800aa67b77d9a7f37492975905"
Oct 31 20:06:55.285130 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:06:55.285113    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"766a6cd8f5a7309349b6d7d5323cb51a83328f800aa67b77d9a7f37492975905\": container with ID starting with 766a6cd8f5a7309349b6d7d5323cb51a83328f800aa67b77d9a7f37492975905 not found: ID does not exist" containerID="766a6cd8f5a7309349b6d7d5323cb51a83328f800aa67b77d9a7f37492975905"
Oct 31 20:06:55.285179 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:55.285133    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"766a6cd8f5a7309349b6d7d5323cb51a83328f800aa67b77d9a7f37492975905"} err="failed to get container status \"766a6cd8f5a7309349b6d7d5323cb51a83328f800aa67b77d9a7f37492975905\": rpc error: code = NotFound desc = could not find container \"766a6cd8f5a7309349b6d7d5323cb51a83328f800aa67b77d9a7f37492975905\": container with ID starting with 766a6cd8f5a7309349b6d7d5323cb51a83328f800aa67b77d9a7f37492975905 not found: ID does not exist"
Oct 31 20:06:55.285179 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:55.285148    4645 scope.go:122] "RemoveContainer" containerID="b34933e0988ad6af10c3bb56ce43816fbd2140c76937a4014d6b6d176f4ffed7"
Oct 31 20:06:55.293744 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:55.293728    4645 scope.go:122] "RemoveContainer" containerID="b34933e0988ad6af10c3bb56ce43816fbd2140c76937a4014d6b6d176f4ffed7"
Oct 31 20:06:55.294007 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:06:55.293988    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b34933e0988ad6af10c3bb56ce43816fbd2140c76937a4014d6b6d176f4ffed7\": container with ID starting with b34933e0988ad6af10c3bb56ce43816fbd2140c76937a4014d6b6d176f4ffed7 not found: ID does not exist" containerID="b34933e0988ad6af10c3bb56ce43816fbd2140c76937a4014d6b6d176f4ffed7"
Oct 31 20:06:55.294117 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:55.294012    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b34933e0988ad6af10c3bb56ce43816fbd2140c76937a4014d6b6d176f4ffed7"} err="failed to get container status \"b34933e0988ad6af10c3bb56ce43816fbd2140c76937a4014d6b6d176f4ffed7\": rpc error: code = NotFound desc = could not find container \"b34933e0988ad6af10c3bb56ce43816fbd2140c76937a4014d6b6d176f4ffed7\": container with ID starting with b34933e0988ad6af10c3bb56ce43816fbd2140c76937a4014d6b6d176f4ffed7 not found: ID does not exist"
Oct 31 20:06:55.298006 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:55.297966    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c9b9a1-klfx5" podStartSLOduration=1.2979509089999999 podStartE2EDuration="1.297950909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:06:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:06:55.296323521 +0000 UTC m=+2845.869557650" watchObservedRunningTime="2026-10-31 20:06:55.297950909 +0000 UTC m=+2845.871185038"
Oct 31 20:06:55.312491 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:55.312443    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc"]
Oct 31 20:06:55.325362 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:55.325340    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-8gsbc"]
Oct 31 20:06:55.339433 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:55.339407    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc"]
Oct 31 20:06:55.349922 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:55.349900    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-f28d48-openshift-default-c89647f6c-lqwbc"]
Oct 31 20:06:55.617383 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:55.617289    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7309f62d-7bc0-4677-a81d-3b306f5eca82" path="/var/lib/kubelet/pods/7309f62d-7bc0-4677-a81d-3b306f5eca82/volumes"
Oct 31 20:06:55.617792 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:55.617776    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d9801bc-b873-4f98-9395-0a19686d6b8a" path="/var/lib/kubelet/pods/8d9801bc-b873-4f98-9395-0a19686d6b8a/volumes"
Oct 31 20:06:55.934063 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:55.934021    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:56.021870 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:56.021831    4645 patch_prober.go:29] interesting pod/e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 20:06:56.022050 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:56.021888    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b" podUID="710e3436-3828-45fb-b090-e28d1147cbfe" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 20:06:56.270950 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:56.270859    4645 generic.go:356] "Generic (PLEG): container finished" podID="6a8637ab-6147-4718-8be4-6af8f696a40d" containerID="17663dd659909937b7fed7b2d9abd901cf512b5dd21e352024b6e27abef64b5b" exitCode=0
Oct 31 20:06:56.271114 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:56.270948    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c9b9a1-klfx5" event={"ID":"6a8637ab-6147-4718-8be4-6af8f696a40d","Type":"ContainerDied","Data":"17663dd659909937b7fed7b2d9abd901cf512b5dd21e352024b6e27abef64b5b"}
Oct 31 20:06:56.826519 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:56.826481    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:57.421142 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:57.421116    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c9b9a1-klfx5"
Oct 31 20:06:57.585168 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:57.585066    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a8637ab-6147-4718-8be4-6af8f696a40d-kube-api-access-fwxhz\" (UniqueName: \"kubernetes.io/projected/6a8637ab-6147-4718-8be4-6af8f696a40d-kube-api-access-fwxhz\") pod \"6a8637ab-6147-4718-8be4-6af8f696a40d\" (UID: \"6a8637ab-6147-4718-8be4-6af8f696a40d\") "
Oct 31 20:06:57.585168 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:57.585153    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a8637ab-6147-4718-8be4-6af8f696a40d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a8637ab-6147-4718-8be4-6af8f696a40d-maas-api-openshift-ca\") pod \"6a8637ab-6147-4718-8be4-6af8f696a40d\" (UID: \"6a8637ab-6147-4718-8be4-6af8f696a40d\") "
Oct 31 20:06:57.585457 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:57.585227    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a8637ab-6147-4718-8be4-6af8f696a40d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a8637ab-6147-4718-8be4-6af8f696a40d-maas-api-xks-ca\") pod \"6a8637ab-6147-4718-8be4-6af8f696a40d\" (UID: \"6a8637ab-6147-4718-8be4-6af8f696a40d\") "
Oct 31 20:06:57.585534 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:57.585513    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a8637ab-6147-4718-8be4-6af8f696a40d-maas-api-openshift-ca" pod "6a8637ab-6147-4718-8be4-6af8f696a40d" (UID: "6a8637ab-6147-4718-8be4-6af8f696a40d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:06:57.585655 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:57.585634    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a8637ab-6147-4718-8be4-6af8f696a40d-maas-api-openshift-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:57.587310 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:57.587288    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a8637ab-6147-4718-8be4-6af8f696a40d-maas-api-xks-ca" pod "6a8637ab-6147-4718-8be4-6af8f696a40d" (UID: "6a8637ab-6147-4718-8be4-6af8f696a40d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:06:57.587528 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:57.587509    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a8637ab-6147-4718-8be4-6af8f696a40d-kube-api-access-fwxhz" pod "6a8637ab-6147-4718-8be4-6af8f696a40d" (UID: "6a8637ab-6147-4718-8be4-6af8f696a40d"). InnerVolumeSpecName "kube-api-access-fwxhz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:06:57.650545 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:57.650506    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:57.699423 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:57.696564    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwxhz\" (UniqueName: \"kubernetes.io/projected/6a8637ab-6147-4718-8be4-6af8f696a40d-kube-api-access-fwxhz\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:57.699423 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:57.696603    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a8637ab-6147-4718-8be4-6af8f696a40d-maas-api-xks-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:58.288784 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:58.288745    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c9b9a1-klfx5" event={"ID":"6a8637ab-6147-4718-8be4-6af8f696a40d","Type":"ContainerDied","Data":"19071a8718150c1e6c4e9ada1bb0511b3cf25ea628b525e1c84bff90228f5dee"}
Oct 31 20:06:58.288784 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:58.288775    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c9b9a1-klfx5"
Oct 31 20:06:58.289233 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:58.288807    4645 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19071a8718150c1e6c4e9ada1bb0511b3cf25ea628b525e1c84bff90228f5dee"
Oct 31 20:06:58.601416 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:58.600226    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:58.871706 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:58.871673    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b"
Oct 31 20:06:58.910054 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:58.910019    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/710e3436-3828-45fb-b090-e28d1147cbfe-istio-token\" (UniqueName: \"kubernetes.io/projected/710e3436-3828-45fb-b090-e28d1147cbfe-istio-token\") pod \"710e3436-3828-45fb-b090-e28d1147cbfe\" (UID: \"710e3436-3828-45fb-b090-e28d1147cbfe\") "
Oct 31 20:06:58.910195 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:58.910075    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/710e3436-3828-45fb-b090-e28d1147cbfe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/710e3436-3828-45fb-b090-e28d1147cbfe-istio-envoy\") pod \"710e3436-3828-45fb-b090-e28d1147cbfe\" (UID: \"710e3436-3828-45fb-b090-e28d1147cbfe\") "
Oct 31 20:06:58.910195 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:58.910148    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/710e3436-3828-45fb-b090-e28d1147cbfe-kube-api-access-ldsq7\" (UniqueName: \"kubernetes.io/projected/710e3436-3828-45fb-b090-e28d1147cbfe-kube-api-access-ldsq7\") pod \"710e3436-3828-45fb-b090-e28d1147cbfe\" (UID: \"710e3436-3828-45fb-b090-e28d1147cbfe\") "
Oct 31 20:06:58.910195 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:58.910177    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/710e3436-3828-45fb-b090-e28d1147cbfe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/710e3436-3828-45fb-b090-e28d1147cbfe-workload-socket\") pod \"710e3436-3828-45fb-b090-e28d1147cbfe\" (UID: \"710e3436-3828-45fb-b090-e28d1147cbfe\") "
Oct 31 20:06:58.910385 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:58.910222    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/710e3436-3828-45fb-b090-e28d1147cbfe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/710e3436-3828-45fb-b090-e28d1147cbfe-workload-certs\") pod \"710e3436-3828-45fb-b090-e28d1147cbfe\" (UID: \"710e3436-3828-45fb-b090-e28d1147cbfe\") "
Oct 31 20:06:58.910385 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:58.910288    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/710e3436-3828-45fb-b090-e28d1147cbfe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/710e3436-3828-45fb-b090-e28d1147cbfe-istio-data\") pod \"710e3436-3828-45fb-b090-e28d1147cbfe\" (UID: \"710e3436-3828-45fb-b090-e28d1147cbfe\") "
Oct 31 20:06:58.910569 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:58.910420    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/710e3436-3828-45fb-b090-e28d1147cbfe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/710e3436-3828-45fb-b090-e28d1147cbfe-istio-podinfo\") pod \"710e3436-3828-45fb-b090-e28d1147cbfe\" (UID: \"710e3436-3828-45fb-b090-e28d1147cbfe\") "
Oct 31 20:06:58.910569 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:58.910487    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/710e3436-3828-45fb-b090-e28d1147cbfe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/710e3436-3828-45fb-b090-e28d1147cbfe-credential-socket\") pod \"710e3436-3828-45fb-b090-e28d1147cbfe\" (UID: \"710e3436-3828-45fb-b090-e28d1147cbfe\") "
Oct 31 20:06:58.910695 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:58.910613    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/710e3436-3828-45fb-b090-e28d1147cbfe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/710e3436-3828-45fb-b090-e28d1147cbfe-istiod-ca-cert\") pod \"710e3436-3828-45fb-b090-e28d1147cbfe\" (UID: \"710e3436-3828-45fb-b090-e28d1147cbfe\") "
Oct 31 20:06:58.910695 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:58.910652    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/710e3436-3828-45fb-b090-e28d1147cbfe-istio-data" pod "710e3436-3828-45fb-b090-e28d1147cbfe" (UID: "710e3436-3828-45fb-b090-e28d1147cbfe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:06:58.910807 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:58.910762    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/710e3436-3828-45fb-b090-e28d1147cbfe-workload-certs" pod "710e3436-3828-45fb-b090-e28d1147cbfe" (UID: "710e3436-3828-45fb-b090-e28d1147cbfe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:06:58.911150 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:58.911122    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/710e3436-3828-45fb-b090-e28d1147cbfe-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:58.911150 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:58.911151    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/710e3436-3828-45fb-b090-e28d1147cbfe-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:58.911382 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:58.911192    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/710e3436-3828-45fb-b090-e28d1147cbfe-workload-socket" pod "710e3436-3828-45fb-b090-e28d1147cbfe" (UID: "710e3436-3828-45fb-b090-e28d1147cbfe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:06:58.911610 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:58.911585    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/710e3436-3828-45fb-b090-e28d1147cbfe-credential-socket" pod "710e3436-3828-45fb-b090-e28d1147cbfe" (UID: "710e3436-3828-45fb-b090-e28d1147cbfe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:06:58.911765 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:58.911744    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/710e3436-3828-45fb-b090-e28d1147cbfe-istiod-ca-cert" pod "710e3436-3828-45fb-b090-e28d1147cbfe" (UID: "710e3436-3828-45fb-b090-e28d1147cbfe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:06:58.912529 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:58.912490    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/710e3436-3828-45fb-b090-e28d1147cbfe-istio-envoy" pod "710e3436-3828-45fb-b090-e28d1147cbfe" (UID: "710e3436-3828-45fb-b090-e28d1147cbfe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:06:58.913258 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:58.913231    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/710e3436-3828-45fb-b090-e28d1147cbfe-istio-token" pod "710e3436-3828-45fb-b090-e28d1147cbfe" (UID: "710e3436-3828-45fb-b090-e28d1147cbfe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:06:58.913422 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:58.913400    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/710e3436-3828-45fb-b090-e28d1147cbfe-istio-podinfo" pod "710e3436-3828-45fb-b090-e28d1147cbfe" (UID: "710e3436-3828-45fb-b090-e28d1147cbfe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 20:06:58.913572 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:58.913551    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/710e3436-3828-45fb-b090-e28d1147cbfe-kube-api-access-ldsq7" pod "710e3436-3828-45fb-b090-e28d1147cbfe" (UID: "710e3436-3828-45fb-b090-e28d1147cbfe"). InnerVolumeSpecName "kube-api-access-ldsq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:06:59.011914 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:59.011880    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/710e3436-3828-45fb-b090-e28d1147cbfe-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:59.011914 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:59.011908    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/710e3436-3828-45fb-b090-e28d1147cbfe-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:59.012086 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:59.011923    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/710e3436-3828-45fb-b090-e28d1147cbfe-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:59.012086 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:59.011936    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldsq7\" (UniqueName: \"kubernetes.io/projected/710e3436-3828-45fb-b090-e28d1147cbfe-kube-api-access-ldsq7\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:59.012086 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:59.011949    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/710e3436-3828-45fb-b090-e28d1147cbfe-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:59.012086 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:59.011963    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/710e3436-3828-45fb-b090-e28d1147cbfe-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:59.012086 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:59.011976    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/710e3436-3828-45fb-b090-e28d1147cbfe-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:06:59.296927 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:59.296890    4645 generic.go:356] "Generic (PLEG): container finished" podID="710e3436-3828-45fb-b090-e28d1147cbfe" containerID="02bcde84b1170dd9c98e7e4e3557752013a43e5911a91514dbbd3ac1076bf67f" exitCode=0
Oct 31 20:06:59.297380 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:59.296939    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b"
Oct 31 20:06:59.297380 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:59.296969    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b" event={"ID":"710e3436-3828-45fb-b090-e28d1147cbfe","Type":"ContainerDied","Data":"02bcde84b1170dd9c98e7e4e3557752013a43e5911a91514dbbd3ac1076bf67f"}
Oct 31 20:06:59.297380 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:59.297043    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b" event={"ID":"710e3436-3828-45fb-b090-e28d1147cbfe","Type":"ContainerDied","Data":"4e21813536ba9b6b815b6943f43ff7def10cd87aef5801e707d3b12b2806ecdd"}
Oct 31 20:06:59.297380 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:59.297067    4645 scope.go:122] "RemoveContainer" containerID="02bcde84b1170dd9c98e7e4e3557752013a43e5911a91514dbbd3ac1076bf67f"
Oct 31 20:06:59.308932 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:59.308909    4645 scope.go:122] "RemoveContainer" containerID="0af901541f97b5801a6cbce660bb1a1c5085864c764964cb4ce45babf30fafa0"
Oct 31 20:06:59.317592 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:59.317572    4645 scope.go:122] "RemoveContainer" containerID="02bcde84b1170dd9c98e7e4e3557752013a43e5911a91514dbbd3ac1076bf67f"
Oct 31 20:06:59.317886 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:06:59.317867    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02bcde84b1170dd9c98e7e4e3557752013a43e5911a91514dbbd3ac1076bf67f\": container with ID starting with 02bcde84b1170dd9c98e7e4e3557752013a43e5911a91514dbbd3ac1076bf67f not found: ID does not exist" containerID="02bcde84b1170dd9c98e7e4e3557752013a43e5911a91514dbbd3ac1076bf67f"
Oct 31 20:06:59.317988 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:59.317897    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02bcde84b1170dd9c98e7e4e3557752013a43e5911a91514dbbd3ac1076bf67f"} err="failed to get container status \"02bcde84b1170dd9c98e7e4e3557752013a43e5911a91514dbbd3ac1076bf67f\": rpc error: code = NotFound desc = could not find container \"02bcde84b1170dd9c98e7e4e3557752013a43e5911a91514dbbd3ac1076bf67f\": container with ID starting with 02bcde84b1170dd9c98e7e4e3557752013a43e5911a91514dbbd3ac1076bf67f not found: ID does not exist"
Oct 31 20:06:59.317988 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:59.317920    4645 scope.go:122] "RemoveContainer" containerID="0af901541f97b5801a6cbce660bb1a1c5085864c764964cb4ce45babf30fafa0"
Oct 31 20:06:59.318222 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:06:59.318202    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0af901541f97b5801a6cbce660bb1a1c5085864c764964cb4ce45babf30fafa0\": container with ID starting with 0af901541f97b5801a6cbce660bb1a1c5085864c764964cb4ce45babf30fafa0 not found: ID does not exist" containerID="0af901541f97b5801a6cbce660bb1a1c5085864c764964cb4ce45babf30fafa0"
Oct 31 20:06:59.318277 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:59.318235    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0af901541f97b5801a6cbce660bb1a1c5085864c764964cb4ce45babf30fafa0"} err="failed to get container status \"0af901541f97b5801a6cbce660bb1a1c5085864c764964cb4ce45babf30fafa0\": rpc error: code = NotFound desc = could not find container \"0af901541f97b5801a6cbce660bb1a1c5085864c764964cb4ce45babf30fafa0\": container with ID starting with 0af901541f97b5801a6cbce660bb1a1c5085864c764964cb4ce45babf30fafa0 not found: ID does not exist"
Oct 31 20:06:59.321446 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:59.321421    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b"]
Oct 31 20:06:59.327948 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:59.327926    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-c173d8-openshift-default-6945698f95-6fj9b"]
Oct 31 20:06:59.525438 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:59.525406    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:06:59.617835 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:06:59.617756    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="710e3436-3828-45fb-b090-e28d1147cbfe" path="/var/lib/kubelet/pods/710e3436-3828-45fb-b090-e28d1147cbfe/volumes"
Oct 31 20:07:00.376613 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:00.376575    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:01.048088 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:01.046698    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:01.856070 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:01.856030    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:02.031210 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:02.031180    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:02.276555 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:02.276522    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:02.320387 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:02.320355    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7fe845-kstgm"]
Oct 31 20:07:02.322984 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:02.322958    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7fe845-kstgm"]
Oct 31 20:07:02.393777 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:02.393734    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:02.518069 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:02.518036    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:02.843152 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:02.843118    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:03.399909 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:03.399875    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:03.617187 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:03.617149    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eff0a26e-843b-41eb-9125-6273720e3329" path="/var/lib/kubelet/pods/eff0a26e-843b-41eb-9125-6273720e3329/volumes"
Oct 31 20:07:04.497965 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:04.497928    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:05.279240 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:05.279207    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:06.084880 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:06.084847    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:06.120882 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:06.120843    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c9b9a1-f45479ffd-mqgll"]
Oct 31 20:07:06.121144 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:06.121090    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-c9b9a1-f45479ffd-mqgll" podUID="d1e003dd-25ad-430d-bd3c-b7706d2788cd" containerName="payload-processing" containerID="cri-o://7f26d871198a41675d7b35d9e637e7b44e6536ecb8c9b3bcd8f4a8225357d31a" gracePeriod=30
Oct 31 20:07:06.135483 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:06.135456    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c9b9a1-666b4798c-b9svq"]
Oct 31 20:07:06.135721 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:06.135693    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c9b9a1-666b4798c-b9svq" podUID="c9e1a3d6-8031-4e60-a4ab-c53aa827d93a" containerName="payload-pre-processing" containerID="cri-o://774356287b9d6786753329d8d3d7b8acb8d87fd058dd4d770845e0eff37a5472" gracePeriod=30
Oct 31 20:07:06.243807 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:06.243777    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:06.254841 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:07:06.254810    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 20:07:06.351456 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:06.350244    4645 generic.go:356] "Generic (PLEG): container finished" podID="d1e003dd-25ad-430d-bd3c-b7706d2788cd" containerID="7f26d871198a41675d7b35d9e637e7b44e6536ecb8c9b3bcd8f4a8225357d31a" exitCode=0
Oct 31 20:07:06.351456 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:06.350362    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c9b9a1-f45479ffd-mqgll" event={"ID":"d1e003dd-25ad-430d-bd3c-b7706d2788cd","Type":"ContainerDied","Data":"7f26d871198a41675d7b35d9e637e7b44e6536ecb8c9b3bcd8f4a8225357d31a"}
Oct 31 20:07:06.352047 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:06.352025    4645 generic.go:356] "Generic (PLEG): container finished" podID="c9e1a3d6-8031-4e60-a4ab-c53aa827d93a" containerID="774356287b9d6786753329d8d3d7b8acb8d87fd058dd4d770845e0eff37a5472" exitCode=0
Oct 31 20:07:06.352190 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:06.352108    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c9b9a1-666b4798c-b9svq" event={"ID":"c9e1a3d6-8031-4e60-a4ab-c53aa827d93a","Type":"ContainerDied","Data":"774356287b9d6786753329d8d3d7b8acb8d87fd058dd4d770845e0eff37a5472"}
Oct 31 20:07:06.391661 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:06.391630    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:06.417289 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:06.417268    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c9b9a1-f45479ffd-mqgll"
Oct 31 20:07:06.443776 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:06.443753    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c9b9a1-666b4798c-b9svq"
Oct 31 20:07:06.485778 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:06.485747    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1e003dd-25ad-430d-bd3c-b7706d2788cd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1e003dd-25ad-430d-bd3c-b7706d2788cd-plugins-config-volume\") pod \"d1e003dd-25ad-430d-bd3c-b7706d2788cd\" (UID: \"d1e003dd-25ad-430d-bd3c-b7706d2788cd\") "
Oct 31 20:07:06.485914 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:06.485867    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9e1a3d6-8031-4e60-a4ab-c53aa827d93a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9e1a3d6-8031-4e60-a4ab-c53aa827d93a-plugins-config-volume\") pod \"c9e1a3d6-8031-4e60-a4ab-c53aa827d93a\" (UID: \"c9e1a3d6-8031-4e60-a4ab-c53aa827d93a\") "
Oct 31 20:07:06.485984 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:06.485958    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1e003dd-25ad-430d-bd3c-b7706d2788cd-kube-api-access-xb8k8\" (UniqueName: \"kubernetes.io/projected/d1e003dd-25ad-430d-bd3c-b7706d2788cd-kube-api-access-xb8k8\") pod \"d1e003dd-25ad-430d-bd3c-b7706d2788cd\" (UID: \"d1e003dd-25ad-430d-bd3c-b7706d2788cd\") "
Oct 31 20:07:06.486050 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:06.486034    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9e1a3d6-8031-4e60-a4ab-c53aa827d93a-kube-api-access-l8h9x\" (UniqueName: \"kubernetes.io/projected/c9e1a3d6-8031-4e60-a4ab-c53aa827d93a-kube-api-access-l8h9x\") pod \"c9e1a3d6-8031-4e60-a4ab-c53aa827d93a\" (UID: \"c9e1a3d6-8031-4e60-a4ab-c53aa827d93a\") "
Oct 31 20:07:06.486183 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:06.486160    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1e003dd-25ad-430d-bd3c-b7706d2788cd-plugins-config-volume" pod "d1e003dd-25ad-430d-bd3c-b7706d2788cd" (UID: "d1e003dd-25ad-430d-bd3c-b7706d2788cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:07:06.486267 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:06.486243    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9e1a3d6-8031-4e60-a4ab-c53aa827d93a-plugins-config-volume" pod "c9e1a3d6-8031-4e60-a4ab-c53aa827d93a" (UID: "c9e1a3d6-8031-4e60-a4ab-c53aa827d93a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:07:06.486442 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:06.486425    4645 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1e003dd-25ad-430d-bd3c-b7706d2788cd-plugins-config-volume\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:06.486515 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:06.486446    4645 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9e1a3d6-8031-4e60-a4ab-c53aa827d93a-plugins-config-volume\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:06.488186 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:06.488162    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1e003dd-25ad-430d-bd3c-b7706d2788cd-kube-api-access-xb8k8" pod "d1e003dd-25ad-430d-bd3c-b7706d2788cd" (UID: "d1e003dd-25ad-430d-bd3c-b7706d2788cd"). InnerVolumeSpecName "kube-api-access-xb8k8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:07:06.488284 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:06.488202    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9e1a3d6-8031-4e60-a4ab-c53aa827d93a-kube-api-access-l8h9x" pod "c9e1a3d6-8031-4e60-a4ab-c53aa827d93a" (UID: "c9e1a3d6-8031-4e60-a4ab-c53aa827d93a"). InnerVolumeSpecName "kube-api-access-l8h9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:07:06.516781 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:06.516758    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c9b9a1-klfx5"]
Oct 31 20:07:06.520100 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:06.520080    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c9b9a1-klfx5"]
Oct 31 20:07:06.537958 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:06.537931    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:06.587046 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:06.586997    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8h9x\" (UniqueName: \"kubernetes.io/projected/c9e1a3d6-8031-4e60-a4ab-c53aa827d93a-kube-api-access-l8h9x\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:06.587046 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:06.587019    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xb8k8\" (UniqueName: \"kubernetes.io/projected/d1e003dd-25ad-430d-bd3c-b7706d2788cd-kube-api-access-xb8k8\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:06.819569 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:06.819536    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:07.090291 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:07.090254    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:07.360376 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:07.360277    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c9b9a1-f45479ffd-mqgll"
Oct 31 20:07:07.360376 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:07.360314    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c9b9a1-f45479ffd-mqgll" event={"ID":"d1e003dd-25ad-430d-bd3c-b7706d2788cd","Type":"ContainerDied","Data":"ead99044d01033549695e63b22d1453b0750d364c1a63f9e0d16b57b06f63d44"}
Oct 31 20:07:07.360588 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:07.360400    4645 scope.go:122] "RemoveContainer" containerID="7f26d871198a41675d7b35d9e637e7b44e6536ecb8c9b3bcd8f4a8225357d31a"
Oct 31 20:07:07.361872 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:07.361852    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c9b9a1-666b4798c-b9svq" event={"ID":"c9e1a3d6-8031-4e60-a4ab-c53aa827d93a","Type":"ContainerDied","Data":"73a018603afcba29b2c3c70656a2557d45d382cf1e1fc694361b3dda4ac7e588"}
Oct 31 20:07:07.361872 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:07.361864    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c9b9a1-666b4798c-b9svq"
Oct 31 20:07:07.371700 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:07.371678    4645 scope.go:122] "RemoveContainer" containerID="774356287b9d6786753329d8d3d7b8acb8d87fd058dd4d770845e0eff37a5472"
Oct 31 20:07:07.393501 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:07.393477    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c9b9a1-f45479ffd-mqgll"]
Oct 31 20:07:07.402504 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:07.402477    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c9b9a1-f45479ffd-mqgll"]
Oct 31 20:07:07.415319 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:07.415290    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c9b9a1-666b4798c-b9svq"]
Oct 31 20:07:07.424973 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:07.424950    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c9b9a1-666b4798c-b9svq"]
Oct 31 20:07:07.617952 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:07.617893    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a8637ab-6147-4718-8be4-6af8f696a40d" path="/var/lib/kubelet/pods/6a8637ab-6147-4718-8be4-6af8f696a40d/volumes"
Oct 31 20:07:07.618301 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:07.618286    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9e1a3d6-8031-4e60-a4ab-c53aa827d93a" path="/var/lib/kubelet/pods/c9e1a3d6-8031-4e60-a4ab-c53aa827d93a/volumes"
Oct 31 20:07:07.618678 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:07.618664    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1e003dd-25ad-430d-bd3c-b7706d2788cd" path="/var/lib/kubelet/pods/d1e003dd-25ad-430d-bd3c-b7706d2788cd/volumes"
Oct 31 20:07:07.886528 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:07.886446    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:09.016528 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:09.016485    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:09.943800 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:09.943764    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:10.753012 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:10.752967    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:11.120086 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:11.120009    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:11.275967 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:11.275932    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:11.439727 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:11.439690    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd"]
Oct 31 20:07:11.439959 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:11.439931    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd" podUID="05f487dd-a849-4cb8-b5f0-1368f60518a3" containerName="istio-proxy" containerID="cri-o://5cfe23f787aa493389c65a1b194fcd5b49746d2129b6a0b277c99c8a81f6fd8c" gracePeriod=30
Oct 31 20:07:11.480032 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:11.480003    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:11.647585 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:11.647547    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:11.977798 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:11.977769    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cd9729-jgv7h"]
Oct 31 20:07:11.978193 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:11.978156    4645 cpu_manager.go:414] "containerMap: removing container" podUID="d1e003dd-25ad-430d-bd3c-b7706d2788cd" containerName="payload-processing"
Oct 31 20:07:11.978193 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:11.978167    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1e003dd-25ad-430d-bd3c-b7706d2788cd" containerName="payload-processing"
Oct 31 20:07:11.978193 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:11.978176    4645 cpu_manager.go:414] "containerMap: removing container" podUID="710e3436-3828-45fb-b090-e28d1147cbfe" containerName="istio-proxy"
Oct 31 20:07:11.978193 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:11.978181    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="710e3436-3828-45fb-b090-e28d1147cbfe" containerName="istio-proxy"
Oct 31 20:07:11.978474 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:11.978198    4645 cpu_manager.go:414] "containerMap: removing container" podUID="8d9801bc-b873-4f98-9395-0a19686d6b8a" containerName="istio-proxy"
Oct 31 20:07:11.978474 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:11.978204    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d9801bc-b873-4f98-9395-0a19686d6b8a" containerName="istio-proxy"
Oct 31 20:07:11.978474 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:11.978222    4645 cpu_manager.go:414] "containerMap: removing container" podUID="7309f62d-7bc0-4677-a81d-3b306f5eca82" containerName="istio-proxy"
Oct 31 20:07:11.978474 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:11.978228    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7309f62d-7bc0-4677-a81d-3b306f5eca82" containerName="istio-proxy"
Oct 31 20:07:11.978474 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:11.978242    4645 cpu_manager.go:414] "containerMap: removing container" podUID="c9e1a3d6-8031-4e60-a4ab-c53aa827d93a" containerName="payload-pre-processing"
Oct 31 20:07:11.978474 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:11.978250    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9e1a3d6-8031-4e60-a4ab-c53aa827d93a" containerName="payload-pre-processing"
Oct 31 20:07:11.978474 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:11.978261    4645 cpu_manager.go:414] "containerMap: removing container" podUID="6a8637ab-6147-4718-8be4-6af8f696a40d" containerName="revoke-keys"
Oct 31 20:07:11.978474 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:11.978266    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a8637ab-6147-4718-8be4-6af8f696a40d" containerName="revoke-keys"
Oct 31 20:07:11.978474 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:11.978352    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1e003dd-25ad-430d-bd3c-b7706d2788cd" containerName="payload-processing"
Oct 31 20:07:11.978474 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:11.978364    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="710e3436-3828-45fb-b090-e28d1147cbfe" containerName="istio-proxy"
Oct 31 20:07:11.978474 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:11.978374    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d9801bc-b873-4f98-9395-0a19686d6b8a" containerName="istio-proxy"
Oct 31 20:07:11.978474 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:11.978383    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="7309f62d-7bc0-4677-a81d-3b306f5eca82" containerName="istio-proxy"
Oct 31 20:07:11.978474 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:11.978388    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9e1a3d6-8031-4e60-a4ab-c53aa827d93a" containerName="payload-pre-processing"
Oct 31 20:07:11.978474 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:11.978395    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a8637ab-6147-4718-8be4-6af8f696a40d" containerName="revoke-keys"
Oct 31 20:07:11.983058 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:11.983041    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cd9729-jgv7h"
Oct 31 20:07:11.996089 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:11.996068    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:07:11.996386 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:11.996365    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 20:07:11.999675 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:11.999650    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cd9729-jgv7h"]
Oct 31 20:07:12.050216 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:12.050195    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5246e7a8-48b9-4996-b107-ab0d97855022-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-cd9729-jgv7h\" (UID: \"5246e7a8-48b9-4996-b107-ab0d97855022\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cd9729-jgv7h"
Oct 31 20:07:12.050351 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:12.050226    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcjkk\" (UniqueName: \"kubernetes.io/projected/5246e7a8-48b9-4996-b107-ab0d97855022-kube-api-access-kcjkk\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-cd9729-jgv7h\" (UID: \"5246e7a8-48b9-4996-b107-ab0d97855022\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cd9729-jgv7h"
Oct 31 20:07:12.050351 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:12.050254    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5246e7a8-48b9-4996-b107-ab0d97855022-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-cd9729-jgv7h\" (UID: \"5246e7a8-48b9-4996-b107-ab0d97855022\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cd9729-jgv7h"
Oct 31 20:07:12.151009 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:12.150982    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5246e7a8-48b9-4996-b107-ab0d97855022-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-cd9729-jgv7h\" (UID: \"5246e7a8-48b9-4996-b107-ab0d97855022\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cd9729-jgv7h"
Oct 31 20:07:12.151113 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:12.151019    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcjkk\" (UniqueName: \"kubernetes.io/projected/5246e7a8-48b9-4996-b107-ab0d97855022-kube-api-access-kcjkk\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-cd9729-jgv7h\" (UID: \"5246e7a8-48b9-4996-b107-ab0d97855022\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cd9729-jgv7h"
Oct 31 20:07:12.151113 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:12.151059    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5246e7a8-48b9-4996-b107-ab0d97855022-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-cd9729-jgv7h\" (UID: \"5246e7a8-48b9-4996-b107-ab0d97855022\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cd9729-jgv7h"
Oct 31 20:07:12.151656 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:12.151637    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5246e7a8-48b9-4996-b107-ab0d97855022-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-cd9729-jgv7h\" (UID: \"5246e7a8-48b9-4996-b107-ab0d97855022\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cd9729-jgv7h"
Oct 31 20:07:12.153459 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:12.153442    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5246e7a8-48b9-4996-b107-ab0d97855022-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-cd9729-jgv7h\" (UID: \"5246e7a8-48b9-4996-b107-ab0d97855022\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cd9729-jgv7h"
Oct 31 20:07:12.163874 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:12.163854    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcjkk\" (UniqueName: \"kubernetes.io/projected/5246e7a8-48b9-4996-b107-ab0d97855022-kube-api-access-kcjkk\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-cd9729-jgv7h\" (UID: \"5246e7a8-48b9-4996-b107-ab0d97855022\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cd9729-jgv7h"
Oct 31 20:07:12.295223 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:12.295168    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cd9729-jgv7h"
Oct 31 20:07:12.631242 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:12.631214    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cd9729-jgv7h"]
Oct 31 20:07:12.804909 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:12.804876    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:13.402372 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:13.402266    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cd9729-jgv7h" event={"ID":"5246e7a8-48b9-4996-b107-ab0d97855022","Type":"ContainerStarted","Data":"427309ca680a4c09425cac5edded8608be0b096985faf4485934154c7384cbb9"}
Oct 31 20:07:13.402372 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:13.402350    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cd9729-jgv7h" event={"ID":"5246e7a8-48b9-4996-b107-ab0d97855022","Type":"ContainerStarted","Data":"60286d841568e6165382676921ccaab6324aa48caa64b6de234e1b9e02fe10af"}
Oct 31 20:07:13.424993 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:13.424945    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cd9729-jgv7h" podStartSLOduration=2.4249313900000002 podStartE2EDuration="2.42493139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:07:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:07:13.42225533 +0000 UTC m=+2863.995489461" watchObservedRunningTime="2026-10-31 20:07:13.42493139 +0000 UTC m=+2863.998165519"
Oct 31 20:07:13.896411 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:13.896373    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:14.410296 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:14.410261    4645 generic.go:356] "Generic (PLEG): container finished" podID="5246e7a8-48b9-4996-b107-ab0d97855022" containerID="427309ca680a4c09425cac5edded8608be0b096985faf4485934154c7384cbb9" exitCode=0
Oct 31 20:07:14.410753 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:14.410353    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cd9729-jgv7h" event={"ID":"5246e7a8-48b9-4996-b107-ab0d97855022","Type":"ContainerDied","Data":"427309ca680a4c09425cac5edded8608be0b096985faf4485934154c7384cbb9"}
Oct 31 20:07:14.650070 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:14.650034    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:14.813387 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:14.813281    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:14.967843 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:14.967809    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:15.166144 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:15.166113    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8"]
Oct 31 20:07:15.166393 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:15.166366    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8" podUID="af802480-762e-446b-95d5-7754f4bad230" containerName="istio-proxy" containerID="cri-o://6c231135b1e2990ef65b1e6c444157573558442406d304dd10c2f9ef664c14dc" gracePeriod=30
Oct 31 20:07:15.285185 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:15.285149    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:15.553514 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:15.553489    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cd9729-jgv7h"
Oct 31 20:07:15.636310 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:15.636277    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:15.687607 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:15.687540    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5246e7a8-48b9-4996-b107-ab0d97855022-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5246e7a8-48b9-4996-b107-ab0d97855022-maas-api-xks-ca\") pod \"5246e7a8-48b9-4996-b107-ab0d97855022\" (UID: \"5246e7a8-48b9-4996-b107-ab0d97855022\") "
Oct 31 20:07:15.687607 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:15.687599    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5246e7a8-48b9-4996-b107-ab0d97855022-kube-api-access-kcjkk\" (UniqueName: \"kubernetes.io/projected/5246e7a8-48b9-4996-b107-ab0d97855022-kube-api-access-kcjkk\") pod \"5246e7a8-48b9-4996-b107-ab0d97855022\" (UID: \"5246e7a8-48b9-4996-b107-ab0d97855022\") "
Oct 31 20:07:15.687800 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:15.687715    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5246e7a8-48b9-4996-b107-ab0d97855022-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5246e7a8-48b9-4996-b107-ab0d97855022-maas-api-openshift-ca\") pod \"5246e7a8-48b9-4996-b107-ab0d97855022\" (UID: \"5246e7a8-48b9-4996-b107-ab0d97855022\") "
Oct 31 20:07:15.688082 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:15.688058    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5246e7a8-48b9-4996-b107-ab0d97855022-maas-api-openshift-ca" pod "5246e7a8-48b9-4996-b107-ab0d97855022" (UID: "5246e7a8-48b9-4996-b107-ab0d97855022"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:07:15.689677 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:15.689656    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5246e7a8-48b9-4996-b107-ab0d97855022-maas-api-xks-ca" pod "5246e7a8-48b9-4996-b107-ab0d97855022" (UID: "5246e7a8-48b9-4996-b107-ab0d97855022"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:07:15.689746 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:15.689715    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5246e7a8-48b9-4996-b107-ab0d97855022-kube-api-access-kcjkk" pod "5246e7a8-48b9-4996-b107-ab0d97855022" (UID: "5246e7a8-48b9-4996-b107-ab0d97855022"). InnerVolumeSpecName "kube-api-access-kcjkk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:07:15.709279 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:15.709256    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c55cf1-rwdhp"]
Oct 31 20:07:15.709845 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:15.709824    4645 cpu_manager.go:414] "containerMap: removing container" podUID="710e3436-3828-45fb-b090-e28d1147cbfe" containerName="istio-proxy"
Oct 31 20:07:15.709845 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:15.709845    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="710e3436-3828-45fb-b090-e28d1147cbfe" containerName="istio-proxy"
Oct 31 20:07:15.710003 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:15.709888    4645 cpu_manager.go:414] "containerMap: removing container" podUID="7309f62d-7bc0-4677-a81d-3b306f5eca82" containerName="istio-proxy"
Oct 31 20:07:15.710003 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:15.709896    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7309f62d-7bc0-4677-a81d-3b306f5eca82" containerName="istio-proxy"
Oct 31 20:07:15.710003 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:15.709916    4645 cpu_manager.go:414] "containerMap: removing container" podUID="5246e7a8-48b9-4996-b107-ab0d97855022" containerName="revoke-keys"
Oct 31 20:07:15.710003 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:15.709924    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5246e7a8-48b9-4996-b107-ab0d97855022" containerName="revoke-keys"
Oct 31 20:07:15.710155 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:15.710014    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="5246e7a8-48b9-4996-b107-ab0d97855022" containerName="revoke-keys"
Oct 31 20:07:15.710155 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:15.710029    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="710e3436-3828-45fb-b090-e28d1147cbfe" containerName="istio-proxy"
Oct 31 20:07:15.710155 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:15.710043    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="7309f62d-7bc0-4677-a81d-3b306f5eca82" containerName="istio-proxy"
Oct 31 20:07:15.713498 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:15.713479    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c55cf1-rwdhp"
Oct 31 20:07:15.728947 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:15.728922    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c55cf1-rwdhp"]
Oct 31 20:07:15.789499 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:15.789469    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57bd507f-90c5-45eb-9688-6e7e4daa0c68-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-c55cf1-rwdhp\" (UID: \"57bd507f-90c5-45eb-9688-6e7e4daa0c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c55cf1-rwdhp"
Oct 31 20:07:15.789626 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:15.789576    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jltlf\" (UniqueName: \"kubernetes.io/projected/57bd507f-90c5-45eb-9688-6e7e4daa0c68-kube-api-access-jltlf\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-c55cf1-rwdhp\" (UID: \"57bd507f-90c5-45eb-9688-6e7e4daa0c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c55cf1-rwdhp"
Oct 31 20:07:15.789699 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:15.789674    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57bd507f-90c5-45eb-9688-6e7e4daa0c68-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-c55cf1-rwdhp\" (UID: \"57bd507f-90c5-45eb-9688-6e7e4daa0c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c55cf1-rwdhp"
Oct 31 20:07:15.789754 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:15.789740    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcjkk\" (UniqueName: \"kubernetes.io/projected/5246e7a8-48b9-4996-b107-ab0d97855022-kube-api-access-kcjkk\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:15.789790 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:15.789753    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5246e7a8-48b9-4996-b107-ab0d97855022-maas-api-openshift-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:15.789790 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:15.789763    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5246e7a8-48b9-4996-b107-ab0d97855022-maas-api-xks-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:15.890661 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:15.890625    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jltlf\" (UniqueName: \"kubernetes.io/projected/57bd507f-90c5-45eb-9688-6e7e4daa0c68-kube-api-access-jltlf\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-c55cf1-rwdhp\" (UID: \"57bd507f-90c5-45eb-9688-6e7e4daa0c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c55cf1-rwdhp"
Oct 31 20:07:15.890774 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:15.890708    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57bd507f-90c5-45eb-9688-6e7e4daa0c68-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-c55cf1-rwdhp\" (UID: \"57bd507f-90c5-45eb-9688-6e7e4daa0c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c55cf1-rwdhp"
Oct 31 20:07:15.890774 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:15.890735    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57bd507f-90c5-45eb-9688-6e7e4daa0c68-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-c55cf1-rwdhp\" (UID: \"57bd507f-90c5-45eb-9688-6e7e4daa0c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c55cf1-rwdhp"
Oct 31 20:07:15.891360 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:15.891317    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57bd507f-90c5-45eb-9688-6e7e4daa0c68-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-c55cf1-rwdhp\" (UID: \"57bd507f-90c5-45eb-9688-6e7e4daa0c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c55cf1-rwdhp"
Oct 31 20:07:15.892984 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:15.892966    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57bd507f-90c5-45eb-9688-6e7e4daa0c68-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-c55cf1-rwdhp\" (UID: \"57bd507f-90c5-45eb-9688-6e7e4daa0c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c55cf1-rwdhp"
Oct 31 20:07:15.903995 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:15.903973    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jltlf\" (UniqueName: \"kubernetes.io/projected/57bd507f-90c5-45eb-9688-6e7e4daa0c68-kube-api-access-jltlf\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-c55cf1-rwdhp\" (UID: \"57bd507f-90c5-45eb-9688-6e7e4daa0c68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c55cf1-rwdhp"
Oct 31 20:07:16.026891 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:16.026839    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c55cf1-rwdhp"
Oct 31 20:07:16.200566 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:16.200535    4645 patch_prober.go:29] interesting pod/e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 20:07:16.200708 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:16.200592    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd" podUID="05f487dd-a849-4cb8-b5f0-1368f60518a3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 20:07:16.305738 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:16.305711    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c55cf1-rwdhp"]
Oct 31 20:07:16.307122 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:07:16.307091    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57bd507f_90c5_45eb_9688_6e7e4daa0c68.slice/crio-c59efa75cf4546591f54be5fef78b4413a95810f4c093c3b3d07aef542522565 WatchSource:0}: Error finding container c59efa75cf4546591f54be5fef78b4413a95810f4c093c3b3d07aef542522565: Status 404 returned error can't find the container with id c59efa75cf4546591f54be5fef78b4413a95810f4c093c3b3d07aef542522565
Oct 31 20:07:16.424085 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:16.424065    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cd9729-jgv7h"
Oct 31 20:07:16.424215 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:16.424104    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cd9729-jgv7h" event={"ID":"5246e7a8-48b9-4996-b107-ab0d97855022","Type":"ContainerDied","Data":"60286d841568e6165382676921ccaab6324aa48caa64b6de234e1b9e02fe10af"}
Oct 31 20:07:16.424215 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:16.424155    4645 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60286d841568e6165382676921ccaab6324aa48caa64b6de234e1b9e02fe10af"
Oct 31 20:07:16.425507 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:16.425482    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c55cf1-rwdhp" event={"ID":"57bd507f-90c5-45eb-9688-6e7e4daa0c68","Type":"ContainerStarted","Data":"c59efa75cf4546591f54be5fef78b4413a95810f4c093c3b3d07aef542522565"}
Oct 31 20:07:16.520161 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:16.520137    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:16.710422 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:16.710399    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd"
Oct 31 20:07:16.799818 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:16.799794    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05f487dd-a849-4cb8-b5f0-1368f60518a3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05f487dd-a849-4cb8-b5f0-1368f60518a3-istiod-ca-cert\") pod \"05f487dd-a849-4cb8-b5f0-1368f60518a3\" (UID: \"05f487dd-a849-4cb8-b5f0-1368f60518a3\") "
Oct 31 20:07:16.799948 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:16.799826    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05f487dd-a849-4cb8-b5f0-1368f60518a3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05f487dd-a849-4cb8-b5f0-1368f60518a3-istio-envoy\") pod \"05f487dd-a849-4cb8-b5f0-1368f60518a3\" (UID: \"05f487dd-a849-4cb8-b5f0-1368f60518a3\") "
Oct 31 20:07:16.799948 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:16.799871    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05f487dd-a849-4cb8-b5f0-1368f60518a3-istio-token\" (UniqueName: \"kubernetes.io/projected/05f487dd-a849-4cb8-b5f0-1368f60518a3-istio-token\") pod \"05f487dd-a849-4cb8-b5f0-1368f60518a3\" (UID: \"05f487dd-a849-4cb8-b5f0-1368f60518a3\") "
Oct 31 20:07:16.799948 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:16.799894    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/05f487dd-a849-4cb8-b5f0-1368f60518a3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05f487dd-a849-4cb8-b5f0-1368f60518a3-istio-podinfo\") pod \"05f487dd-a849-4cb8-b5f0-1368f60518a3\" (UID: \"05f487dd-a849-4cb8-b5f0-1368f60518a3\") "
Oct 31 20:07:16.799948 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:16.799924    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05f487dd-a849-4cb8-b5f0-1368f60518a3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05f487dd-a849-4cb8-b5f0-1368f60518a3-istio-data\") pod \"05f487dd-a849-4cb8-b5f0-1368f60518a3\" (UID: \"05f487dd-a849-4cb8-b5f0-1368f60518a3\") "
Oct 31 20:07:16.800181 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:16.799962    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05f487dd-a849-4cb8-b5f0-1368f60518a3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05f487dd-a849-4cb8-b5f0-1368f60518a3-workload-certs\") pod \"05f487dd-a849-4cb8-b5f0-1368f60518a3\" (UID: \"05f487dd-a849-4cb8-b5f0-1368f60518a3\") "
Oct 31 20:07:16.800181 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:16.800037    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05f487dd-a849-4cb8-b5f0-1368f60518a3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f487dd-a849-4cb8-b5f0-1368f60518a3-workload-socket\") pod \"05f487dd-a849-4cb8-b5f0-1368f60518a3\" (UID: \"05f487dd-a849-4cb8-b5f0-1368f60518a3\") "
Oct 31 20:07:16.800181 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:16.800087    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05f487dd-a849-4cb8-b5f0-1368f60518a3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f487dd-a849-4cb8-b5f0-1368f60518a3-credential-socket\") pod \"05f487dd-a849-4cb8-b5f0-1368f60518a3\" (UID: \"05f487dd-a849-4cb8-b5f0-1368f60518a3\") "
Oct 31 20:07:16.800181 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:16.800166    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05f487dd-a849-4cb8-b5f0-1368f60518a3-istiod-ca-cert" pod "05f487dd-a849-4cb8-b5f0-1368f60518a3" (UID: "05f487dd-a849-4cb8-b5f0-1368f60518a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:07:16.800416 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:16.800189    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05f487dd-a849-4cb8-b5f0-1368f60518a3-kube-api-access-2pftr\" (UniqueName: \"kubernetes.io/projected/05f487dd-a849-4cb8-b5f0-1368f60518a3-kube-api-access-2pftr\") pod \"05f487dd-a849-4cb8-b5f0-1368f60518a3\" (UID: \"05f487dd-a849-4cb8-b5f0-1368f60518a3\") "
Oct 31 20:07:16.800416 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:16.800371    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05f487dd-a849-4cb8-b5f0-1368f60518a3-workload-socket" pod "05f487dd-a849-4cb8-b5f0-1368f60518a3" (UID: "05f487dd-a849-4cb8-b5f0-1368f60518a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:07:16.800416 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:16.800386    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05f487dd-a849-4cb8-b5f0-1368f60518a3-istio-data" pod "05f487dd-a849-4cb8-b5f0-1368f60518a3" (UID: "05f487dd-a849-4cb8-b5f0-1368f60518a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:07:16.800565 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:16.800502    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05f487dd-a849-4cb8-b5f0-1368f60518a3-credential-socket" pod "05f487dd-a849-4cb8-b5f0-1368f60518a3" (UID: "05f487dd-a849-4cb8-b5f0-1368f60518a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:07:16.800709 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:16.800680    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05f487dd-a849-4cb8-b5f0-1368f60518a3-workload-certs" pod "05f487dd-a849-4cb8-b5f0-1368f60518a3" (UID: "05f487dd-a849-4cb8-b5f0-1368f60518a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:07:16.800814 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:16.800734    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f487dd-a849-4cb8-b5f0-1368f60518a3-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:16.800814 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:16.800754    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f487dd-a849-4cb8-b5f0-1368f60518a3-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:16.800814 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:16.800769    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05f487dd-a849-4cb8-b5f0-1368f60518a3-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:16.800814 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:16.800778    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05f487dd-a849-4cb8-b5f0-1368f60518a3-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:16.802445 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:16.802422    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05f487dd-a849-4cb8-b5f0-1368f60518a3-kube-api-access-2pftr" pod "05f487dd-a849-4cb8-b5f0-1368f60518a3" (UID: "05f487dd-a849-4cb8-b5f0-1368f60518a3"). InnerVolumeSpecName "kube-api-access-2pftr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:07:16.802581 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:16.802460    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05f487dd-a849-4cb8-b5f0-1368f60518a3-istio-envoy" pod "05f487dd-a849-4cb8-b5f0-1368f60518a3" (UID: "05f487dd-a849-4cb8-b5f0-1368f60518a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:07:16.802685 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:16.802668    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/05f487dd-a849-4cb8-b5f0-1368f60518a3-istio-podinfo" pod "05f487dd-a849-4cb8-b5f0-1368f60518a3" (UID: "05f487dd-a849-4cb8-b5f0-1368f60518a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 20:07:16.802747 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:16.802689    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05f487dd-a849-4cb8-b5f0-1368f60518a3-istio-token" pod "05f487dd-a849-4cb8-b5f0-1368f60518a3" (UID: "05f487dd-a849-4cb8-b5f0-1368f60518a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:07:16.901620 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:16.901596    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05f487dd-a849-4cb8-b5f0-1368f60518a3-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:16.901620 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:16.901618    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05f487dd-a849-4cb8-b5f0-1368f60518a3-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:16.901754 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:16.901630    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05f487dd-a849-4cb8-b5f0-1368f60518a3-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:16.901754 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:16.901639    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pftr\" (UniqueName: \"kubernetes.io/projected/05f487dd-a849-4cb8-b5f0-1368f60518a3-kube-api-access-2pftr\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:16.901754 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:16.901649    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05f487dd-a849-4cb8-b5f0-1368f60518a3-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:17.434051 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:17.434020    4645 generic.go:356] "Generic (PLEG): container finished" podID="57bd507f-90c5-45eb-9688-6e7e4daa0c68" containerID="f016a15c329eaf18f51b90201d0343d124a87418f3448d853ef51b4b11fc1822" exitCode=0
Oct 31 20:07:17.434198 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:17.434098    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c55cf1-rwdhp" event={"ID":"57bd507f-90c5-45eb-9688-6e7e4daa0c68","Type":"ContainerDied","Data":"f016a15c329eaf18f51b90201d0343d124a87418f3448d853ef51b4b11fc1822"}
Oct 31 20:07:17.435924 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:17.435904    4645 generic.go:356] "Generic (PLEG): container finished" podID="05f487dd-a849-4cb8-b5f0-1368f60518a3" containerID="5cfe23f787aa493389c65a1b194fcd5b49746d2129b6a0b277c99c8a81f6fd8c" exitCode=0
Oct 31 20:07:17.436035 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:17.435949    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd"
Oct 31 20:07:17.436035 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:17.435961    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd" event={"ID":"05f487dd-a849-4cb8-b5f0-1368f60518a3","Type":"ContainerDied","Data":"5cfe23f787aa493389c65a1b194fcd5b49746d2129b6a0b277c99c8a81f6fd8c"}
Oct 31 20:07:17.436035 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:17.435997    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd" event={"ID":"05f487dd-a849-4cb8-b5f0-1368f60518a3","Type":"ContainerDied","Data":"f91951887d7d5796e9fbb5f92ef6d869aa876367cb29ba1a7a645b42ca21d6a1"}
Oct 31 20:07:17.436035 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:17.436019    4645 scope.go:122] "RemoveContainer" containerID="5cfe23f787aa493389c65a1b194fcd5b49746d2129b6a0b277c99c8a81f6fd8c"
Oct 31 20:07:17.446227 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:17.446208    4645 scope.go:122] "RemoveContainer" containerID="4a4a5accd018c0747b0b44311c6383cba5cc15f19397a582160cab4f76526710"
Oct 31 20:07:17.455364 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:17.455345    4645 scope.go:122] "RemoveContainer" containerID="5cfe23f787aa493389c65a1b194fcd5b49746d2129b6a0b277c99c8a81f6fd8c"
Oct 31 20:07:17.455622 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:07:17.455601    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cfe23f787aa493389c65a1b194fcd5b49746d2129b6a0b277c99c8a81f6fd8c\": container with ID starting with 5cfe23f787aa493389c65a1b194fcd5b49746d2129b6a0b277c99c8a81f6fd8c not found: ID does not exist" containerID="5cfe23f787aa493389c65a1b194fcd5b49746d2129b6a0b277c99c8a81f6fd8c"
Oct 31 20:07:17.455698 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:17.455631    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cfe23f787aa493389c65a1b194fcd5b49746d2129b6a0b277c99c8a81f6fd8c"} err="failed to get container status \"5cfe23f787aa493389c65a1b194fcd5b49746d2129b6a0b277c99c8a81f6fd8c\": rpc error: code = NotFound desc = could not find container \"5cfe23f787aa493389c65a1b194fcd5b49746d2129b6a0b277c99c8a81f6fd8c\": container with ID starting with 5cfe23f787aa493389c65a1b194fcd5b49746d2129b6a0b277c99c8a81f6fd8c not found: ID does not exist"
Oct 31 20:07:17.455698 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:17.455648    4645 scope.go:122] "RemoveContainer" containerID="4a4a5accd018c0747b0b44311c6383cba5cc15f19397a582160cab4f76526710"
Oct 31 20:07:17.455914 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:07:17.455894    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a4a5accd018c0747b0b44311c6383cba5cc15f19397a582160cab4f76526710\": container with ID starting with 4a4a5accd018c0747b0b44311c6383cba5cc15f19397a582160cab4f76526710 not found: ID does not exist" containerID="4a4a5accd018c0747b0b44311c6383cba5cc15f19397a582160cab4f76526710"
Oct 31 20:07:17.455957 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:17.455919    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a4a5accd018c0747b0b44311c6383cba5cc15f19397a582160cab4f76526710"} err="failed to get container status \"4a4a5accd018c0747b0b44311c6383cba5cc15f19397a582160cab4f76526710\": rpc error: code = NotFound desc = could not find container \"4a4a5accd018c0747b0b44311c6383cba5cc15f19397a582160cab4f76526710\": container with ID starting with 4a4a5accd018c0747b0b44311c6383cba5cc15f19397a582160cab4f76526710 not found: ID does not exist"
Oct 31 20:07:17.469483 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:17.469461    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:17.515968 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:17.515931    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd"]
Oct 31 20:07:17.519535 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:17.519508    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-7fe845-openshift-default-5fdf5fd855-br6xd"]
Oct 31 20:07:17.616619 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:17.616587    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05f487dd-a849-4cb8-b5f0-1368f60518a3" path="/var/lib/kubelet/pods/05f487dd-a849-4cb8-b5f0-1368f60518a3/volumes"
Oct 31 20:07:18.326384 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:18.326343    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:18.483703 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:18.483665    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:18.577490 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:18.577433    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c55cf1-rwdhp"
Oct 31 20:07:18.727975 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:18.727946    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57bd507f-90c5-45eb-9688-6e7e4daa0c68-kube-api-access-jltlf\" (UniqueName: \"kubernetes.io/projected/57bd507f-90c5-45eb-9688-6e7e4daa0c68-kube-api-access-jltlf\") pod \"57bd507f-90c5-45eb-9688-6e7e4daa0c68\" (UID: \"57bd507f-90c5-45eb-9688-6e7e4daa0c68\") "
Oct 31 20:07:18.728140 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:18.728005    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57bd507f-90c5-45eb-9688-6e7e4daa0c68-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57bd507f-90c5-45eb-9688-6e7e4daa0c68-maas-api-openshift-ca\") pod \"57bd507f-90c5-45eb-9688-6e7e4daa0c68\" (UID: \"57bd507f-90c5-45eb-9688-6e7e4daa0c68\") "
Oct 31 20:07:18.728195 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:18.728172    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57bd507f-90c5-45eb-9688-6e7e4daa0c68-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57bd507f-90c5-45eb-9688-6e7e4daa0c68-maas-api-xks-ca\") pod \"57bd507f-90c5-45eb-9688-6e7e4daa0c68\" (UID: \"57bd507f-90c5-45eb-9688-6e7e4daa0c68\") "
Oct 31 20:07:18.728420 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:18.728398    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57bd507f-90c5-45eb-9688-6e7e4daa0c68-maas-api-openshift-ca" pod "57bd507f-90c5-45eb-9688-6e7e4daa0c68" (UID: "57bd507f-90c5-45eb-9688-6e7e4daa0c68"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:07:18.728644 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:18.728624    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57bd507f-90c5-45eb-9688-6e7e4daa0c68-maas-api-openshift-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:18.730106 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:18.730077    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57bd507f-90c5-45eb-9688-6e7e4daa0c68-kube-api-access-jltlf" pod "57bd507f-90c5-45eb-9688-6e7e4daa0c68" (UID: "57bd507f-90c5-45eb-9688-6e7e4daa0c68"). InnerVolumeSpecName "kube-api-access-jltlf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:07:18.730197 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:18.730160    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57bd507f-90c5-45eb-9688-6e7e4daa0c68-maas-api-xks-ca" pod "57bd507f-90c5-45eb-9688-6e7e4daa0c68" (UID: "57bd507f-90c5-45eb-9688-6e7e4daa0c68"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:07:18.829362 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:18.829288    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57bd507f-90c5-45eb-9688-6e7e4daa0c68-maas-api-xks-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:18.829362 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:18.829312    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jltlf\" (UniqueName: \"kubernetes.io/projected/57bd507f-90c5-45eb-9688-6e7e4daa0c68-kube-api-access-jltlf\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:19.210902 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:19.210869    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:19.436412 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:19.436379    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:19.449673 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:19.449653    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c55cf1-rwdhp"
Oct 31 20:07:19.449673 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:19.449663    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c55cf1-rwdhp" event={"ID":"57bd507f-90c5-45eb-9688-6e7e4daa0c68","Type":"ContainerDied","Data":"c59efa75cf4546591f54be5fef78b4413a95810f4c093c3b3d07aef542522565"}
Oct 31 20:07:19.449846 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:19.449704    4645 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c59efa75cf4546591f54be5fef78b4413a95810f4c093c3b3d07aef542522565"
Oct 31 20:07:19.951182 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:19.951121    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:20.421461 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:20.421435    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8"
Oct 31 20:07:20.458591 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:20.458562    4645 generic.go:356] "Generic (PLEG): container finished" podID="af802480-762e-446b-95d5-7754f4bad230" containerID="6c231135b1e2990ef65b1e6c444157573558442406d304dd10c2f9ef664c14dc" exitCode=0
Oct 31 20:07:20.459012 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:20.458610    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8"
Oct 31 20:07:20.459012 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:20.458759    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8" event={"ID":"af802480-762e-446b-95d5-7754f4bad230","Type":"ContainerDied","Data":"6c231135b1e2990ef65b1e6c444157573558442406d304dd10c2f9ef664c14dc"}
Oct 31 20:07:20.459012 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:20.458797    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8" event={"ID":"af802480-762e-446b-95d5-7754f4bad230","Type":"ContainerDied","Data":"82320e1b9e68cf114bcc959d6692b7a5b9394a9f9f6c370ea205c077d6991210"}
Oct 31 20:07:20.459012 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:20.458821    4645 scope.go:122] "RemoveContainer" containerID="6c231135b1e2990ef65b1e6c444157573558442406d304dd10c2f9ef664c14dc"
Oct 31 20:07:20.468611 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:20.468591    4645 scope.go:122] "RemoveContainer" containerID="56b8550b6909ce6e6c49e17a78b45605af8219a8eb57b4f2cb4aacb62527450e"
Oct 31 20:07:20.479544 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:20.479515    4645 scope.go:122] "RemoveContainer" containerID="6c231135b1e2990ef65b1e6c444157573558442406d304dd10c2f9ef664c14dc"
Oct 31 20:07:20.479797 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:07:20.479780    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c231135b1e2990ef65b1e6c444157573558442406d304dd10c2f9ef664c14dc\": container with ID starting with 6c231135b1e2990ef65b1e6c444157573558442406d304dd10c2f9ef664c14dc not found: ID does not exist" containerID="6c231135b1e2990ef65b1e6c444157573558442406d304dd10c2f9ef664c14dc"
Oct 31 20:07:20.479859 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:20.479805    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c231135b1e2990ef65b1e6c444157573558442406d304dd10c2f9ef664c14dc"} err="failed to get container status \"6c231135b1e2990ef65b1e6c444157573558442406d304dd10c2f9ef664c14dc\": rpc error: code = NotFound desc = could not find container \"6c231135b1e2990ef65b1e6c444157573558442406d304dd10c2f9ef664c14dc\": container with ID starting with 6c231135b1e2990ef65b1e6c444157573558442406d304dd10c2f9ef664c14dc not found: ID does not exist"
Oct 31 20:07:20.479859 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:20.479820    4645 scope.go:122] "RemoveContainer" containerID="56b8550b6909ce6e6c49e17a78b45605af8219a8eb57b4f2cb4aacb62527450e"
Oct 31 20:07:20.480085 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:07:20.480063    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56b8550b6909ce6e6c49e17a78b45605af8219a8eb57b4f2cb4aacb62527450e\": container with ID starting with 56b8550b6909ce6e6c49e17a78b45605af8219a8eb57b4f2cb4aacb62527450e not found: ID does not exist" containerID="56b8550b6909ce6e6c49e17a78b45605af8219a8eb57b4f2cb4aacb62527450e"
Oct 31 20:07:20.480150 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:20.480096    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56b8550b6909ce6e6c49e17a78b45605af8219a8eb57b4f2cb4aacb62527450e"} err="failed to get container status \"56b8550b6909ce6e6c49e17a78b45605af8219a8eb57b4f2cb4aacb62527450e\": rpc error: code = NotFound desc = could not find container \"56b8550b6909ce6e6c49e17a78b45605af8219a8eb57b4f2cb4aacb62527450e\": container with ID starting with 56b8550b6909ce6e6c49e17a78b45605af8219a8eb57b4f2cb4aacb62527450e not found: ID does not exist"
Oct 31 20:07:20.496928 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:20.496902    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:20.545245 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:20.545197    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/af802480-762e-446b-95d5-7754f4bad230-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af802480-762e-446b-95d5-7754f4bad230-istio-podinfo\") pod \"af802480-762e-446b-95d5-7754f4bad230\" (UID: \"af802480-762e-446b-95d5-7754f4bad230\") "
Oct 31 20:07:20.545363 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:20.545244    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af802480-762e-446b-95d5-7754f4bad230-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af802480-762e-446b-95d5-7754f4bad230-workload-socket\") pod \"af802480-762e-446b-95d5-7754f4bad230\" (UID: \"af802480-762e-446b-95d5-7754f4bad230\") "
Oct 31 20:07:20.545363 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:20.545277    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af802480-762e-446b-95d5-7754f4bad230-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af802480-762e-446b-95d5-7754f4bad230-istio-data\") pod \"af802480-762e-446b-95d5-7754f4bad230\" (UID: \"af802480-762e-446b-95d5-7754f4bad230\") "
Oct 31 20:07:20.545363 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:20.545294    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af802480-762e-446b-95d5-7754f4bad230-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af802480-762e-446b-95d5-7754f4bad230-istio-envoy\") pod \"af802480-762e-446b-95d5-7754f4bad230\" (UID: \"af802480-762e-446b-95d5-7754f4bad230\") "
Oct 31 20:07:20.545363 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:20.545322    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af802480-762e-446b-95d5-7754f4bad230-istio-token\" (UniqueName: \"kubernetes.io/projected/af802480-762e-446b-95d5-7754f4bad230-istio-token\") pod \"af802480-762e-446b-95d5-7754f4bad230\" (UID: \"af802480-762e-446b-95d5-7754f4bad230\") "
Oct 31 20:07:20.545363 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:20.545357    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af802480-762e-446b-95d5-7754f4bad230-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af802480-762e-446b-95d5-7754f4bad230-istiod-ca-cert\") pod \"af802480-762e-446b-95d5-7754f4bad230\" (UID: \"af802480-762e-446b-95d5-7754f4bad230\") "
Oct 31 20:07:20.545662 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:20.545503    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af802480-762e-446b-95d5-7754f4bad230-kube-api-access-fw44d\" (UniqueName: \"kubernetes.io/projected/af802480-762e-446b-95d5-7754f4bad230-kube-api-access-fw44d\") pod \"af802480-762e-446b-95d5-7754f4bad230\" (UID: \"af802480-762e-446b-95d5-7754f4bad230\") "
Oct 31 20:07:20.545662 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:20.545538    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af802480-762e-446b-95d5-7754f4bad230-workload-socket" pod "af802480-762e-446b-95d5-7754f4bad230" (UID: "af802480-762e-446b-95d5-7754f4bad230"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:07:20.545662 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:20.545582    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af802480-762e-446b-95d5-7754f4bad230-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af802480-762e-446b-95d5-7754f4bad230-workload-certs\") pod \"af802480-762e-446b-95d5-7754f4bad230\" (UID: \"af802480-762e-446b-95d5-7754f4bad230\") "
Oct 31 20:07:20.545662 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:20.545614    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af802480-762e-446b-95d5-7754f4bad230-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af802480-762e-446b-95d5-7754f4bad230-credential-socket\") pod \"af802480-762e-446b-95d5-7754f4bad230\" (UID: \"af802480-762e-446b-95d5-7754f4bad230\") "
Oct 31 20:07:20.545968 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:20.545865    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af802480-762e-446b-95d5-7754f4bad230-istiod-ca-cert" pod "af802480-762e-446b-95d5-7754f4bad230" (UID: "af802480-762e-446b-95d5-7754f4bad230"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:07:20.545968 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:20.545940    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af802480-762e-446b-95d5-7754f4bad230-workload-certs" pod "af802480-762e-446b-95d5-7754f4bad230" (UID: "af802480-762e-446b-95d5-7754f4bad230"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:07:20.546127 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:20.546107    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af802480-762e-446b-95d5-7754f4bad230-credential-socket" pod "af802480-762e-446b-95d5-7754f4bad230" (UID: "af802480-762e-446b-95d5-7754f4bad230"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:07:20.546177 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:20.546126    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af802480-762e-446b-95d5-7754f4bad230-istio-data" pod "af802480-762e-446b-95d5-7754f4bad230" (UID: "af802480-762e-446b-95d5-7754f4bad230"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:07:20.546177 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:20.546137    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af802480-762e-446b-95d5-7754f4bad230-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:20.546177 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:20.546160    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af802480-762e-446b-95d5-7754f4bad230-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:20.546177 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:20.546174    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af802480-762e-446b-95d5-7754f4bad230-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:20.547813 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:20.547793    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af802480-762e-446b-95d5-7754f4bad230-istio-podinfo" pod "af802480-762e-446b-95d5-7754f4bad230" (UID: "af802480-762e-446b-95d5-7754f4bad230"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 20:07:20.548120 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:20.548093    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af802480-762e-446b-95d5-7754f4bad230-istio-token" pod "af802480-762e-446b-95d5-7754f4bad230" (UID: "af802480-762e-446b-95d5-7754f4bad230"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:07:20.548221 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:20.548129    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af802480-762e-446b-95d5-7754f4bad230-kube-api-access-fw44d" pod "af802480-762e-446b-95d5-7754f4bad230" (UID: "af802480-762e-446b-95d5-7754f4bad230"). InnerVolumeSpecName "kube-api-access-fw44d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:07:20.548221 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:20.548190    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af802480-762e-446b-95d5-7754f4bad230-istio-envoy" pod "af802480-762e-446b-95d5-7754f4bad230" (UID: "af802480-762e-446b-95d5-7754f4bad230"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:07:20.646996 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:20.646972    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fw44d\" (UniqueName: \"kubernetes.io/projected/af802480-762e-446b-95d5-7754f4bad230-kube-api-access-fw44d\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:20.646996 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:20.646995    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af802480-762e-446b-95d5-7754f4bad230-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:20.647133 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:20.647005    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af802480-762e-446b-95d5-7754f4bad230-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:20.647133 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:20.647014    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af802480-762e-446b-95d5-7754f4bad230-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:20.647133 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:20.647024    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af802480-762e-446b-95d5-7754f4bad230-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:20.647133 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:20.647038    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af802480-762e-446b-95d5-7754f4bad230-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:20.705706 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:20.705682    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:20.787864 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:20.787832    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8"]
Oct 31 20:07:20.801643 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:20.801577    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-c9b9a1-openshift-default-bb66f5c95-wgvv8"]
Oct 31 20:07:21.619071 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:21.619032    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af802480-762e-446b-95d5-7754f4bad230" path="/var/lib/kubelet/pods/af802480-762e-446b-95d5-7754f4bad230/volumes"
Oct 31 20:07:21.646322 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:21.646292    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:22.273728 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:22.273690    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:22.518725 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:22.518691    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:22.755208 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:22.755158    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:23.415390 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:23.415351    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:23.630201 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:23.630165    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:23.716027 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:23.715949    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cd9729-jgv7h"]
Oct 31 20:07:23.719838 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:23.719813    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-cd9729-jgv7h"]
Oct 31 20:07:23.760004 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:23.759969    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:23.921958 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:23.921907    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:24.263956 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:24.263926    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:24.747536 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:24.747500    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:25.524968 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:25.524925    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:25.616434 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:25.616401    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5246e7a8-48b9-4996-b107-ab0d97855022" path="/var/lib/kubelet/pods/5246e7a8-48b9-4996-b107-ab0d97855022/volumes"
Oct 31 20:07:25.955808 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:25.955775    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:26.063115 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:26.063081    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:26.453176 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:26.453147    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:27.257759 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:27.257715    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:28.100425 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:28.100387    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:28.962205 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:28.962170    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:29.622768 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:29.622736    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:29.810768 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:29.810727    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:29.901795 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:29.901720    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c55cf1-rwdhp"]
Oct 31 20:07:29.905850 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:29.905822    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-c55cf1-rwdhp"]
Oct 31 20:07:29.931529 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:29.931500    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:30.241627 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:30.241597    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:30.473758 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:30.473718    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:30.958629 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:30.958594    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:31.617015 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:31.616983    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57bd507f-90c5-45eb-9688-6e7e4daa0c68" path="/var/lib/kubelet/pods/57bd507f-90c5-45eb-9688-6e7e4daa0c68/volumes"
Oct 31 20:07:31.773767 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:31.773732    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:32.172614 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:32.172579    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:32.282691 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:32.282662    4645 scope.go:122] "RemoveContainer" containerID="da58cf3b3eff3e47fed59ff37bade58a3b7232285727b4c4e92f99e620fc1fa2"
Oct 31 20:07:32.296128 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:32.296095    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:32.488718 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:32.488640    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll"]
Oct 31 20:07:32.488916 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:32.488890    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll" podUID="bffa80e8-d986-43ac-a6d5-8d828ca2a35f" containerName="istio-proxy" containerID="cri-o://d8925cf40a6b02dee387da91b1f80712d42494f63a6af9daafb8f91075ebe447" gracePeriod=30
Oct 31 20:07:32.541468 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:32.541432    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:32.652670 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:32.652630    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:33.012303 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:33.012270    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-435728-jbs48"]
Oct 31 20:07:33.012680 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:33.012665    4645 cpu_manager.go:414] "containerMap: removing container" podUID="05f487dd-a849-4cb8-b5f0-1368f60518a3" containerName="istio-proxy"
Oct 31 20:07:33.012726 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:33.012681    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05f487dd-a849-4cb8-b5f0-1368f60518a3" containerName="istio-proxy"
Oct 31 20:07:33.012726 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:33.012702    4645 cpu_manager.go:414] "containerMap: removing container" podUID="af802480-762e-446b-95d5-7754f4bad230" containerName="istio-proxy"
Oct 31 20:07:33.012726 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:33.012707    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af802480-762e-446b-95d5-7754f4bad230" containerName="istio-proxy"
Oct 31 20:07:33.012726 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:33.012718    4645 cpu_manager.go:414] "containerMap: removing container" podUID="05f487dd-a849-4cb8-b5f0-1368f60518a3" containerName="istio-proxy"
Oct 31 20:07:33.012726 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:33.012724    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05f487dd-a849-4cb8-b5f0-1368f60518a3" containerName="istio-proxy"
Oct 31 20:07:33.012896 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:33.012738    4645 cpu_manager.go:414] "containerMap: removing container" podUID="57bd507f-90c5-45eb-9688-6e7e4daa0c68" containerName="revoke-keys"
Oct 31 20:07:33.012896 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:33.012743    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57bd507f-90c5-45eb-9688-6e7e4daa0c68" containerName="revoke-keys"
Oct 31 20:07:33.012896 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:33.012750    4645 cpu_manager.go:414] "containerMap: removing container" podUID="af802480-762e-446b-95d5-7754f4bad230" containerName="istio-proxy"
Oct 31 20:07:33.012896 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:33.012755    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af802480-762e-446b-95d5-7754f4bad230" containerName="istio-proxy"
Oct 31 20:07:33.012896 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:33.012818    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="af802480-762e-446b-95d5-7754f4bad230" containerName="istio-proxy"
Oct 31 20:07:33.012896 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:33.012826    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="05f487dd-a849-4cb8-b5f0-1368f60518a3" containerName="istio-proxy"
Oct 31 20:07:33.012896 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:33.012833    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="57bd507f-90c5-45eb-9688-6e7e4daa0c68" containerName="revoke-keys"
Oct 31 20:07:33.012896 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:33.012838    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="af802480-762e-446b-95d5-7754f4bad230" containerName="istio-proxy"
Oct 31 20:07:33.012896 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:33.012846    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="05f487dd-a849-4cb8-b5f0-1368f60518a3" containerName="istio-proxy"
Oct 31 20:07:33.017834 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:33.017811    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-435728-jbs48"
Oct 31 20:07:33.023716 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:33.023695    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:07:33.024131 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:33.024109    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 20:07:33.027303 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:33.027281    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-435728-jbs48"]
Oct 31 20:07:33.072391 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:33.072358    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5038c2c6-211f-4760-8cd1-ed799d8cecc3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-435728-jbs48\" (UID: \"5038c2c6-211f-4760-8cd1-ed799d8cecc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-435728-jbs48"
Oct 31 20:07:33.072560 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:33.072499    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5038c2c6-211f-4760-8cd1-ed799d8cecc3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-435728-jbs48\" (UID: \"5038c2c6-211f-4760-8cd1-ed799d8cecc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-435728-jbs48"
Oct 31 20:07:33.072560 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:33.072536    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t55gx\" (UniqueName: \"kubernetes.io/projected/5038c2c6-211f-4760-8cd1-ed799d8cecc3-kube-api-access-t55gx\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-435728-jbs48\" (UID: \"5038c2c6-211f-4760-8cd1-ed799d8cecc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-435728-jbs48"
Oct 31 20:07:33.173729 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:33.173687    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5038c2c6-211f-4760-8cd1-ed799d8cecc3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-435728-jbs48\" (UID: \"5038c2c6-211f-4760-8cd1-ed799d8cecc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-435728-jbs48"
Oct 31 20:07:33.173945 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:33.173771    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5038c2c6-211f-4760-8cd1-ed799d8cecc3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-435728-jbs48\" (UID: \"5038c2c6-211f-4760-8cd1-ed799d8cecc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-435728-jbs48"
Oct 31 20:07:33.173945 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:33.173795    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t55gx\" (UniqueName: \"kubernetes.io/projected/5038c2c6-211f-4760-8cd1-ed799d8cecc3-kube-api-access-t55gx\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-435728-jbs48\" (UID: \"5038c2c6-211f-4760-8cd1-ed799d8cecc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-435728-jbs48"
Oct 31 20:07:33.174437 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:33.174400    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5038c2c6-211f-4760-8cd1-ed799d8cecc3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-435728-jbs48\" (UID: \"5038c2c6-211f-4760-8cd1-ed799d8cecc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-435728-jbs48"
Oct 31 20:07:33.176276 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:33.176252    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5038c2c6-211f-4760-8cd1-ed799d8cecc3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-435728-jbs48\" (UID: \"5038c2c6-211f-4760-8cd1-ed799d8cecc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-435728-jbs48"
Oct 31 20:07:33.185606 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:33.185583    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t55gx\" (UniqueName: \"kubernetes.io/projected/5038c2c6-211f-4760-8cd1-ed799d8cecc3-kube-api-access-t55gx\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-435728-jbs48\" (UID: \"5038c2c6-211f-4760-8cd1-ed799d8cecc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-435728-jbs48"
Oct 31 20:07:33.332690 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:33.332638    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-435728-jbs48"
Oct 31 20:07:33.472580 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:33.472544    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:33.624482 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:33.624220    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-435728-jbs48"]
Oct 31 20:07:33.627056 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:07:33.627028    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5038c2c6_211f_4760_8cd1_ed799d8cecc3.slice/crio-3105de22fbf911c545baa498b7ce66397c0d2bbff079477d71805d6584a4a425 WatchSource:0}: Error finding container 3105de22fbf911c545baa498b7ce66397c0d2bbff079477d71805d6584a4a425: Status 404 returned error can't find the container with id 3105de22fbf911c545baa498b7ce66397c0d2bbff079477d71805d6584a4a425
Oct 31 20:07:34.467135 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:34.467094    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:34.574952 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:34.574920    4645 generic.go:356] "Generic (PLEG): container finished" podID="5038c2c6-211f-4760-8cd1-ed799d8cecc3" containerID="98ee8c6535ecb67493125bb363ae7a74df109356eec6f975f8a83d1a80b38373" exitCode=0
Oct 31 20:07:34.575086 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:34.574981    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-435728-jbs48" event={"ID":"5038c2c6-211f-4760-8cd1-ed799d8cecc3","Type":"ContainerDied","Data":"98ee8c6535ecb67493125bb363ae7a74df109356eec6f975f8a83d1a80b38373"}
Oct 31 20:07:34.575086 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:34.575019    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-435728-jbs48" event={"ID":"5038c2c6-211f-4760-8cd1-ed799d8cecc3","Type":"ContainerStarted","Data":"3105de22fbf911c545baa498b7ce66397c0d2bbff079477d71805d6584a4a425"}
Oct 31 20:07:35.292141 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:35.292112    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:35.723007 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:35.722983    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-435728-jbs48"
Oct 31 20:07:35.809011 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:35.808984    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5038c2c6-211f-4760-8cd1-ed799d8cecc3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5038c2c6-211f-4760-8cd1-ed799d8cecc3-maas-api-openshift-ca\") pod \"5038c2c6-211f-4760-8cd1-ed799d8cecc3\" (UID: \"5038c2c6-211f-4760-8cd1-ed799d8cecc3\") "
Oct 31 20:07:35.809152 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:35.809041    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5038c2c6-211f-4760-8cd1-ed799d8cecc3-kube-api-access-t55gx\" (UniqueName: \"kubernetes.io/projected/5038c2c6-211f-4760-8cd1-ed799d8cecc3-kube-api-access-t55gx\") pod \"5038c2c6-211f-4760-8cd1-ed799d8cecc3\" (UID: \"5038c2c6-211f-4760-8cd1-ed799d8cecc3\") "
Oct 31 20:07:35.809223 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:35.809175    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5038c2c6-211f-4760-8cd1-ed799d8cecc3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5038c2c6-211f-4760-8cd1-ed799d8cecc3-maas-api-xks-ca\") pod \"5038c2c6-211f-4760-8cd1-ed799d8cecc3\" (UID: \"5038c2c6-211f-4760-8cd1-ed799d8cecc3\") "
Oct 31 20:07:35.809445 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:35.809416    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5038c2c6-211f-4760-8cd1-ed799d8cecc3-maas-api-openshift-ca" pod "5038c2c6-211f-4760-8cd1-ed799d8cecc3" (UID: "5038c2c6-211f-4760-8cd1-ed799d8cecc3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:07:35.809552 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:35.809524    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5038c2c6-211f-4760-8cd1-ed799d8cecc3-maas-api-openshift-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:35.811232 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:35.811201    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5038c2c6-211f-4760-8cd1-ed799d8cecc3-maas-api-xks-ca" pod "5038c2c6-211f-4760-8cd1-ed799d8cecc3" (UID: "5038c2c6-211f-4760-8cd1-ed799d8cecc3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:07:35.811547 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:35.811527    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5038c2c6-211f-4760-8cd1-ed799d8cecc3-kube-api-access-t55gx" pod "5038c2c6-211f-4760-8cd1-ed799d8cecc3" (UID: "5038c2c6-211f-4760-8cd1-ed799d8cecc3"). InnerVolumeSpecName "kube-api-access-t55gx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:07:35.910601 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:35.910580    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5038c2c6-211f-4760-8cd1-ed799d8cecc3-maas-api-xks-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:35.910601 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:35.910601    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t55gx\" (UniqueName: \"kubernetes.io/projected/5038c2c6-211f-4760-8cd1-ed799d8cecc3-kube-api-access-t55gx\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:36.246546 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:07:36.246436    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 20:07:36.391594 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:36.391553    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:36.589047 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:36.588959    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-435728-jbs48"
Oct 31 20:07:36.589047 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:36.588994    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-435728-jbs48" event={"ID":"5038c2c6-211f-4760-8cd1-ed799d8cecc3","Type":"ContainerDied","Data":"3105de22fbf911c545baa498b7ce66397c0d2bbff079477d71805d6584a4a425"}
Oct 31 20:07:36.589047 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:36.589041    4645 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3105de22fbf911c545baa498b7ce66397c0d2bbff079477d71805d6584a4a425"
Oct 31 20:07:37.314623 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:37.314590    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:37.751195 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:37.751171    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll"
Oct 31 20:07:37.829769 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:37.829745    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bffa80e8-d986-43ac-a6d5-8d828ca2a35f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bffa80e8-d986-43ac-a6d5-8d828ca2a35f-workload-socket\") pod \"bffa80e8-d986-43ac-a6d5-8d828ca2a35f\" (UID: \"bffa80e8-d986-43ac-a6d5-8d828ca2a35f\") "
Oct 31 20:07:37.829899 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:37.829790    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bffa80e8-d986-43ac-a6d5-8d828ca2a35f-kube-api-access-rnr7j\" (UniqueName: \"kubernetes.io/projected/bffa80e8-d986-43ac-a6d5-8d828ca2a35f-kube-api-access-rnr7j\") pod \"bffa80e8-d986-43ac-a6d5-8d828ca2a35f\" (UID: \"bffa80e8-d986-43ac-a6d5-8d828ca2a35f\") "
Oct 31 20:07:37.829899 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:37.829822    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bffa80e8-d986-43ac-a6d5-8d828ca2a35f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bffa80e8-d986-43ac-a6d5-8d828ca2a35f-credential-socket\") pod \"bffa80e8-d986-43ac-a6d5-8d828ca2a35f\" (UID: \"bffa80e8-d986-43ac-a6d5-8d828ca2a35f\") "
Oct 31 20:07:37.829899 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:37.829850    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bffa80e8-d986-43ac-a6d5-8d828ca2a35f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bffa80e8-d986-43ac-a6d5-8d828ca2a35f-workload-certs\") pod \"bffa80e8-d986-43ac-a6d5-8d828ca2a35f\" (UID: \"bffa80e8-d986-43ac-a6d5-8d828ca2a35f\") "
Oct 31 20:07:37.829899 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:37.829888    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bffa80e8-d986-43ac-a6d5-8d828ca2a35f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bffa80e8-d986-43ac-a6d5-8d828ca2a35f-istio-data\") pod \"bffa80e8-d986-43ac-a6d5-8d828ca2a35f\" (UID: \"bffa80e8-d986-43ac-a6d5-8d828ca2a35f\") "
Oct 31 20:07:37.830104 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:37.829956    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bffa80e8-d986-43ac-a6d5-8d828ca2a35f-istio-token\" (UniqueName: \"kubernetes.io/projected/bffa80e8-d986-43ac-a6d5-8d828ca2a35f-istio-token\") pod \"bffa80e8-d986-43ac-a6d5-8d828ca2a35f\" (UID: \"bffa80e8-d986-43ac-a6d5-8d828ca2a35f\") "
Oct 31 20:07:37.830104 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:37.829987    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bffa80e8-d986-43ac-a6d5-8d828ca2a35f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bffa80e8-d986-43ac-a6d5-8d828ca2a35f-istiod-ca-cert\") pod \"bffa80e8-d986-43ac-a6d5-8d828ca2a35f\" (UID: \"bffa80e8-d986-43ac-a6d5-8d828ca2a35f\") "
Oct 31 20:07:37.830104 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:37.830031    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bffa80e8-d986-43ac-a6d5-8d828ca2a35f-workload-socket" pod "bffa80e8-d986-43ac-a6d5-8d828ca2a35f" (UID: "bffa80e8-d986-43ac-a6d5-8d828ca2a35f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:07:37.830104 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:37.830081    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bffa80e8-d986-43ac-a6d5-8d828ca2a35f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bffa80e8-d986-43ac-a6d5-8d828ca2a35f-istio-envoy\") pod \"bffa80e8-d986-43ac-a6d5-8d828ca2a35f\" (UID: \"bffa80e8-d986-43ac-a6d5-8d828ca2a35f\") "
Oct 31 20:07:37.830402 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:37.830118    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bffa80e8-d986-43ac-a6d5-8d828ca2a35f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bffa80e8-d986-43ac-a6d5-8d828ca2a35f-istio-podinfo\") pod \"bffa80e8-d986-43ac-a6d5-8d828ca2a35f\" (UID: \"bffa80e8-d986-43ac-a6d5-8d828ca2a35f\") "
Oct 31 20:07:37.830402 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:37.830161    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bffa80e8-d986-43ac-a6d5-8d828ca2a35f-workload-certs" pod "bffa80e8-d986-43ac-a6d5-8d828ca2a35f" (UID: "bffa80e8-d986-43ac-a6d5-8d828ca2a35f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:07:37.830402 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:37.830177    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bffa80e8-d986-43ac-a6d5-8d828ca2a35f-credential-socket" pod "bffa80e8-d986-43ac-a6d5-8d828ca2a35f" (UID: "bffa80e8-d986-43ac-a6d5-8d828ca2a35f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:07:37.830402 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:37.830270    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bffa80e8-d986-43ac-a6d5-8d828ca2a35f-istio-data" pod "bffa80e8-d986-43ac-a6d5-8d828ca2a35f" (UID: "bffa80e8-d986-43ac-a6d5-8d828ca2a35f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:07:37.830589 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:37.830450    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bffa80e8-d986-43ac-a6d5-8d828ca2a35f-istiod-ca-cert" pod "bffa80e8-d986-43ac-a6d5-8d828ca2a35f" (UID: "bffa80e8-d986-43ac-a6d5-8d828ca2a35f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:07:37.830632 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:37.830603    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bffa80e8-d986-43ac-a6d5-8d828ca2a35f-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:37.830632 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:37.830625    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bffa80e8-d986-43ac-a6d5-8d828ca2a35f-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:37.830717 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:37.830639    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bffa80e8-d986-43ac-a6d5-8d828ca2a35f-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:37.830717 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:37.830652    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bffa80e8-d986-43ac-a6d5-8d828ca2a35f-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:37.830717 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:37.830665    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bffa80e8-d986-43ac-a6d5-8d828ca2a35f-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:37.832110 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:37.832083    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bffa80e8-d986-43ac-a6d5-8d828ca2a35f-kube-api-access-rnr7j" pod "bffa80e8-d986-43ac-a6d5-8d828ca2a35f" (UID: "bffa80e8-d986-43ac-a6d5-8d828ca2a35f"). InnerVolumeSpecName "kube-api-access-rnr7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:07:37.832218 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:37.832113    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bffa80e8-d986-43ac-a6d5-8d828ca2a35f-istio-token" pod "bffa80e8-d986-43ac-a6d5-8d828ca2a35f" (UID: "bffa80e8-d986-43ac-a6d5-8d828ca2a35f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:07:37.832543 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:37.832520    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bffa80e8-d986-43ac-a6d5-8d828ca2a35f-istio-envoy" pod "bffa80e8-d986-43ac-a6d5-8d828ca2a35f" (UID: "bffa80e8-d986-43ac-a6d5-8d828ca2a35f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:07:37.832543 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:37.832526    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bffa80e8-d986-43ac-a6d5-8d828ca2a35f-istio-podinfo" pod "bffa80e8-d986-43ac-a6d5-8d828ca2a35f" (UID: "bffa80e8-d986-43ac-a6d5-8d828ca2a35f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 20:07:37.931763 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:37.931741    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bffa80e8-d986-43ac-a6d5-8d828ca2a35f-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:37.931860 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:37.931764    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnr7j\" (UniqueName: \"kubernetes.io/projected/bffa80e8-d986-43ac-a6d5-8d828ca2a35f-kube-api-access-rnr7j\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:37.931860 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:37.931775    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bffa80e8-d986-43ac-a6d5-8d828ca2a35f-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:37.931860 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:37.931784    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bffa80e8-d986-43ac-a6d5-8d828ca2a35f-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:38.067627 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:38.067594    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:38.464188 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:38.464155    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:38.603501 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:38.603469    4645 generic.go:356] "Generic (PLEG): container finished" podID="bffa80e8-d986-43ac-a6d5-8d828ca2a35f" containerID="d8925cf40a6b02dee387da91b1f80712d42494f63a6af9daafb8f91075ebe447" exitCode=0
Oct 31 20:07:38.603678 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:38.603520    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll"
Oct 31 20:07:38.603678 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:38.603534    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll" event={"ID":"bffa80e8-d986-43ac-a6d5-8d828ca2a35f","Type":"ContainerDied","Data":"d8925cf40a6b02dee387da91b1f80712d42494f63a6af9daafb8f91075ebe447"}
Oct 31 20:07:38.603678 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:38.603578    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll" event={"ID":"bffa80e8-d986-43ac-a6d5-8d828ca2a35f","Type":"ContainerDied","Data":"62f5e7f9faaf6364a2cad595b5246dd5bd4af467ea2b063b745ed95cdd4e3ba1"}
Oct 31 20:07:38.603678 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:38.603600    4645 scope.go:122] "RemoveContainer" containerID="d8925cf40a6b02dee387da91b1f80712d42494f63a6af9daafb8f91075ebe447"
Oct 31 20:07:38.614711 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:38.614533    4645 scope.go:122] "RemoveContainer" containerID="6c96f35c698866060dfbbb3dc18ffe4cc41c8ddd5e22cf2bc628827917956371"
Oct 31 20:07:38.623469 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:38.623450    4645 scope.go:122] "RemoveContainer" containerID="d8925cf40a6b02dee387da91b1f80712d42494f63a6af9daafb8f91075ebe447"
Oct 31 20:07:38.623734 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:07:38.623714    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8925cf40a6b02dee387da91b1f80712d42494f63a6af9daafb8f91075ebe447\": container with ID starting with d8925cf40a6b02dee387da91b1f80712d42494f63a6af9daafb8f91075ebe447 not found: ID does not exist" containerID="d8925cf40a6b02dee387da91b1f80712d42494f63a6af9daafb8f91075ebe447"
Oct 31 20:07:38.623783 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:38.623742    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8925cf40a6b02dee387da91b1f80712d42494f63a6af9daafb8f91075ebe447"} err="failed to get container status \"d8925cf40a6b02dee387da91b1f80712d42494f63a6af9daafb8f91075ebe447\": rpc error: code = NotFound desc = could not find container \"d8925cf40a6b02dee387da91b1f80712d42494f63a6af9daafb8f91075ebe447\": container with ID starting with d8925cf40a6b02dee387da91b1f80712d42494f63a6af9daafb8f91075ebe447 not found: ID does not exist"
Oct 31 20:07:38.623783 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:38.623758    4645 scope.go:122] "RemoveContainer" containerID="6c96f35c698866060dfbbb3dc18ffe4cc41c8ddd5e22cf2bc628827917956371"
Oct 31 20:07:38.624021 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:07:38.624003    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c96f35c698866060dfbbb3dc18ffe4cc41c8ddd5e22cf2bc628827917956371\": container with ID starting with 6c96f35c698866060dfbbb3dc18ffe4cc41c8ddd5e22cf2bc628827917956371 not found: ID does not exist" containerID="6c96f35c698866060dfbbb3dc18ffe4cc41c8ddd5e22cf2bc628827917956371"
Oct 31 20:07:38.624065 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:38.624028    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c96f35c698866060dfbbb3dc18ffe4cc41c8ddd5e22cf2bc628827917956371"} err="failed to get container status \"6c96f35c698866060dfbbb3dc18ffe4cc41c8ddd5e22cf2bc628827917956371\": rpc error: code = NotFound desc = could not find container \"6c96f35c698866060dfbbb3dc18ffe4cc41c8ddd5e22cf2bc628827917956371\": container with ID starting with 6c96f35c698866060dfbbb3dc18ffe4cc41c8ddd5e22cf2bc628827917956371 not found: ID does not exist"
Oct 31 20:07:38.633191 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:38.633170    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll"]
Oct 31 20:07:38.634929 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:38.634904    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:38.638720 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:38.638698    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-cd9729-openshift-default-6cdbfbb869-wxqll"]
Oct 31 20:07:38.834256 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:38.834185    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25"]
Oct 31 20:07:38.834441 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:38.834414    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25" podUID="f2be79e8-0166-4066-8e48-fad5a7e85684" containerName="istio-proxy" containerID="cri-o://4170efc426099cb7ba25f4eef3e13a78a9d5c05bb5a28a1033bff39c14611b04" gracePeriod=30
Oct 31 20:07:38.893355 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:38.893293    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:39.033358 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:39.033308    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:39.358086 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:39.358055    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae4ce2-xhcdk"]
Oct 31 20:07:39.358452 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:39.358439    4645 cpu_manager.go:414] "containerMap: removing container" podUID="bffa80e8-d986-43ac-a6d5-8d828ca2a35f" containerName="istio-proxy"
Oct 31 20:07:39.358497 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:39.358453    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bffa80e8-d986-43ac-a6d5-8d828ca2a35f" containerName="istio-proxy"
Oct 31 20:07:39.358497 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:39.358469    4645 cpu_manager.go:414] "containerMap: removing container" podUID="bffa80e8-d986-43ac-a6d5-8d828ca2a35f" containerName="istio-proxy"
Oct 31 20:07:39.358497 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:39.358474    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bffa80e8-d986-43ac-a6d5-8d828ca2a35f" containerName="istio-proxy"
Oct 31 20:07:39.358606 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:39.358506    4645 cpu_manager.go:414] "containerMap: removing container" podUID="5038c2c6-211f-4760-8cd1-ed799d8cecc3" containerName="revoke-keys"
Oct 31 20:07:39.358606 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:39.358511    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5038c2c6-211f-4760-8cd1-ed799d8cecc3" containerName="revoke-keys"
Oct 31 20:07:39.358606 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:39.358564    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="5038c2c6-211f-4760-8cd1-ed799d8cecc3" containerName="revoke-keys"
Oct 31 20:07:39.358606 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:39.358572    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="bffa80e8-d986-43ac-a6d5-8d828ca2a35f" containerName="istio-proxy"
Oct 31 20:07:39.358606 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:39.358579    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="bffa80e8-d986-43ac-a6d5-8d828ca2a35f" containerName="istio-proxy"
Oct 31 20:07:39.362890 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:39.362874    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae4ce2-xhcdk"
Oct 31 20:07:39.371086 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:39.371049    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:07:39.371278 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:39.371112    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 20:07:39.373543 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:39.373519    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae4ce2-xhcdk"]
Oct 31 20:07:39.445139 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:39.445116    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrnnn\" (UniqueName: \"kubernetes.io/projected/e95b1cd3-9e02-41af-a69a-13270b64ec87-kube-api-access-zrnnn\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ae4ce2-xhcdk\" (UID: \"e95b1cd3-9e02-41af-a69a-13270b64ec87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae4ce2-xhcdk"
Oct 31 20:07:39.445286 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:39.445157    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e95b1cd3-9e02-41af-a69a-13270b64ec87-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ae4ce2-xhcdk\" (UID: \"e95b1cd3-9e02-41af-a69a-13270b64ec87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae4ce2-xhcdk"
Oct 31 20:07:39.445286 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:39.445181    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e95b1cd3-9e02-41af-a69a-13270b64ec87-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ae4ce2-xhcdk\" (UID: \"e95b1cd3-9e02-41af-a69a-13270b64ec87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae4ce2-xhcdk"
Oct 31 20:07:39.546759 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:39.546729    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e95b1cd3-9e02-41af-a69a-13270b64ec87-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ae4ce2-xhcdk\" (UID: \"e95b1cd3-9e02-41af-a69a-13270b64ec87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae4ce2-xhcdk"
Oct 31 20:07:39.547141 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:39.546773    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e95b1cd3-9e02-41af-a69a-13270b64ec87-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ae4ce2-xhcdk\" (UID: \"e95b1cd3-9e02-41af-a69a-13270b64ec87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae4ce2-xhcdk"
Oct 31 20:07:39.547141 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:39.546904    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrnnn\" (UniqueName: \"kubernetes.io/projected/e95b1cd3-9e02-41af-a69a-13270b64ec87-kube-api-access-zrnnn\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ae4ce2-xhcdk\" (UID: \"e95b1cd3-9e02-41af-a69a-13270b64ec87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae4ce2-xhcdk"
Oct 31 20:07:39.547505 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:39.547482    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e95b1cd3-9e02-41af-a69a-13270b64ec87-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ae4ce2-xhcdk\" (UID: \"e95b1cd3-9e02-41af-a69a-13270b64ec87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae4ce2-xhcdk"
Oct 31 20:07:39.549258 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:39.549241    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e95b1cd3-9e02-41af-a69a-13270b64ec87-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ae4ce2-xhcdk\" (UID: \"e95b1cd3-9e02-41af-a69a-13270b64ec87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae4ce2-xhcdk"
Oct 31 20:07:39.558791 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:39.558767    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrnnn\" (UniqueName: \"kubernetes.io/projected/e95b1cd3-9e02-41af-a69a-13270b64ec87-kube-api-access-zrnnn\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ae4ce2-xhcdk\" (UID: \"e95b1cd3-9e02-41af-a69a-13270b64ec87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae4ce2-xhcdk"
Oct 31 20:07:39.616551 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:39.616495    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bffa80e8-d986-43ac-a6d5-8d828ca2a35f" path="/var/lib/kubelet/pods/bffa80e8-d986-43ac-a6d5-8d828ca2a35f/volumes"
Oct 31 20:07:39.676953 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:39.676930    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae4ce2-xhcdk"
Oct 31 20:07:39.958103 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:39.958071    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:39.959564 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:39.959537    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae4ce2-xhcdk"]
Oct 31 20:07:39.961408 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:07:39.961384    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode95b1cd3_9e02_41af_a69a_13270b64ec87.slice/crio-e944340bbf9da8bd413e3e6231a9a1cf58a2211dbf203f4e28d20364d4581c8c WatchSource:0}: Error finding container e944340bbf9da8bd413e3e6231a9a1cf58a2211dbf203f4e28d20364d4581c8c: Status 404 returned error can't find the container with id e944340bbf9da8bd413e3e6231a9a1cf58a2211dbf203f4e28d20364d4581c8c
Oct 31 20:07:40.625926 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:40.625853    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae4ce2-xhcdk" event={"ID":"e95b1cd3-9e02-41af-a69a-13270b64ec87","Type":"ContainerStarted","Data":"6525006b443ea29fb4b87648c87bd574857e233bbb4317f5108db94d110d1d75"}
Oct 31 20:07:40.626290 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:40.625929    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae4ce2-xhcdk" event={"ID":"e95b1cd3-9e02-41af-a69a-13270b64ec87","Type":"ContainerStarted","Data":"e944340bbf9da8bd413e3e6231a9a1cf58a2211dbf203f4e28d20364d4581c8c"}
Oct 31 20:07:40.650294 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:40.650250    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae4ce2-xhcdk" podStartSLOduration=1.650238781 podStartE2EDuration="1.650238781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:07:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:07:40.650202591 +0000 UTC m=+2891.223436719" watchObservedRunningTime="2026-10-31 20:07:40.650238781 +0000 UTC m=+2891.223472910"
Oct 31 20:07:40.863638 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:40.863610    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:41.632134 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:41.632092    4645 generic.go:356] "Generic (PLEG): container finished" podID="e95b1cd3-9e02-41af-a69a-13270b64ec87" containerID="6525006b443ea29fb4b87648c87bd574857e233bbb4317f5108db94d110d1d75" exitCode=0
Oct 31 20:07:41.632613 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:41.632130    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae4ce2-xhcdk" event={"ID":"e95b1cd3-9e02-41af-a69a-13270b64ec87","Type":"ContainerDied","Data":"6525006b443ea29fb4b87648c87bd574857e233bbb4317f5108db94d110d1d75"}
Oct 31 20:07:41.641166 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:41.641143    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:42.472824 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:42.472791    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:42.771369 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:42.771343    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae4ce2-xhcdk"
Oct 31 20:07:42.878141 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:42.878114    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e95b1cd3-9e02-41af-a69a-13270b64ec87-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e95b1cd3-9e02-41af-a69a-13270b64ec87-maas-api-xks-ca\") pod \"e95b1cd3-9e02-41af-a69a-13270b64ec87\" (UID: \"e95b1cd3-9e02-41af-a69a-13270b64ec87\") "
Oct 31 20:07:42.878281 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:42.878172    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e95b1cd3-9e02-41af-a69a-13270b64ec87-kube-api-access-zrnnn\" (UniqueName: \"kubernetes.io/projected/e95b1cd3-9e02-41af-a69a-13270b64ec87-kube-api-access-zrnnn\") pod \"e95b1cd3-9e02-41af-a69a-13270b64ec87\" (UID: \"e95b1cd3-9e02-41af-a69a-13270b64ec87\") "
Oct 31 20:07:42.878281 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:42.878219    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e95b1cd3-9e02-41af-a69a-13270b64ec87-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e95b1cd3-9e02-41af-a69a-13270b64ec87-maas-api-openshift-ca\") pod \"e95b1cd3-9e02-41af-a69a-13270b64ec87\" (UID: \"e95b1cd3-9e02-41af-a69a-13270b64ec87\") "
Oct 31 20:07:42.878624 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:42.878602    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e95b1cd3-9e02-41af-a69a-13270b64ec87-maas-api-openshift-ca" pod "e95b1cd3-9e02-41af-a69a-13270b64ec87" (UID: "e95b1cd3-9e02-41af-a69a-13270b64ec87"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:07:42.880263 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:42.880241    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e95b1cd3-9e02-41af-a69a-13270b64ec87-maas-api-xks-ca" pod "e95b1cd3-9e02-41af-a69a-13270b64ec87" (UID: "e95b1cd3-9e02-41af-a69a-13270b64ec87"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:07:42.880263 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:42.880252    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e95b1cd3-9e02-41af-a69a-13270b64ec87-kube-api-access-zrnnn" pod "e95b1cd3-9e02-41af-a69a-13270b64ec87" (UID: "e95b1cd3-9e02-41af-a69a-13270b64ec87"). InnerVolumeSpecName "kube-api-access-zrnnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:07:42.979500 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:42.979477    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e95b1cd3-9e02-41af-a69a-13270b64ec87-maas-api-openshift-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:42.979500 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:42.979497    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e95b1cd3-9e02-41af-a69a-13270b64ec87-maas-api-xks-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:42.979630 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:42.979507    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrnnn\" (UniqueName: \"kubernetes.io/projected/e95b1cd3-9e02-41af-a69a-13270b64ec87-kube-api-access-zrnnn\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:43.268442 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:43.268409    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:43.428937 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:43.428889    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:43.644262 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:43.644224    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae4ce2-xhcdk"
Oct 31 20:07:43.644262 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:43.644250    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae4ce2-xhcdk" event={"ID":"e95b1cd3-9e02-41af-a69a-13270b64ec87","Type":"ContainerDied","Data":"e944340bbf9da8bd413e3e6231a9a1cf58a2211dbf203f4e28d20364d4581c8c"}
Oct 31 20:07:43.644505 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:43.644293    4645 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e944340bbf9da8bd413e3e6231a9a1cf58a2211dbf203f4e28d20364d4581c8c"
Oct 31 20:07:44.051271 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:44.051225    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:44.087968 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:44.087941    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25"
Oct 31 20:07:44.189305 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:44.189276    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2be79e8-0166-4066-8e48-fad5a7e85684-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2be79e8-0166-4066-8e48-fad5a7e85684-workload-socket\") pod \"f2be79e8-0166-4066-8e48-fad5a7e85684\" (UID: \"f2be79e8-0166-4066-8e48-fad5a7e85684\") "
Oct 31 20:07:44.189503 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:44.189314    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2be79e8-0166-4066-8e48-fad5a7e85684-kube-api-access-xpqkf\" (UniqueName: \"kubernetes.io/projected/f2be79e8-0166-4066-8e48-fad5a7e85684-kube-api-access-xpqkf\") pod \"f2be79e8-0166-4066-8e48-fad5a7e85684\" (UID: \"f2be79e8-0166-4066-8e48-fad5a7e85684\") "
Oct 31 20:07:44.189503 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:44.189360    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2be79e8-0166-4066-8e48-fad5a7e85684-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2be79e8-0166-4066-8e48-fad5a7e85684-istiod-ca-cert\") pod \"f2be79e8-0166-4066-8e48-fad5a7e85684\" (UID: \"f2be79e8-0166-4066-8e48-fad5a7e85684\") "
Oct 31 20:07:44.189503 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:44.189392    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2be79e8-0166-4066-8e48-fad5a7e85684-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2be79e8-0166-4066-8e48-fad5a7e85684-istio-envoy\") pod \"f2be79e8-0166-4066-8e48-fad5a7e85684\" (UID: \"f2be79e8-0166-4066-8e48-fad5a7e85684\") "
Oct 31 20:07:44.189693 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:44.189564    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2be79e8-0166-4066-8e48-fad5a7e85684-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2be79e8-0166-4066-8e48-fad5a7e85684-istio-data\") pod \"f2be79e8-0166-4066-8e48-fad5a7e85684\" (UID: \"f2be79e8-0166-4066-8e48-fad5a7e85684\") "
Oct 31 20:07:44.189693 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:44.189600    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f2be79e8-0166-4066-8e48-fad5a7e85684-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2be79e8-0166-4066-8e48-fad5a7e85684-istio-podinfo\") pod \"f2be79e8-0166-4066-8e48-fad5a7e85684\" (UID: \"f2be79e8-0166-4066-8e48-fad5a7e85684\") "
Oct 31 20:07:44.189693 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:44.189641    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2be79e8-0166-4066-8e48-fad5a7e85684-workload-socket" pod "f2be79e8-0166-4066-8e48-fad5a7e85684" (UID: "f2be79e8-0166-4066-8e48-fad5a7e85684"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:07:44.189875 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:44.189763    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2be79e8-0166-4066-8e48-fad5a7e85684-istio-token\" (UniqueName: \"kubernetes.io/projected/f2be79e8-0166-4066-8e48-fad5a7e85684-istio-token\") pod \"f2be79e8-0166-4066-8e48-fad5a7e85684\" (UID: \"f2be79e8-0166-4066-8e48-fad5a7e85684\") "
Oct 31 20:07:44.189875 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:44.189802    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2be79e8-0166-4066-8e48-fad5a7e85684-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2be79e8-0166-4066-8e48-fad5a7e85684-workload-certs\") pod \"f2be79e8-0166-4066-8e48-fad5a7e85684\" (UID: \"f2be79e8-0166-4066-8e48-fad5a7e85684\") "
Oct 31 20:07:44.189980 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:44.189889    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2be79e8-0166-4066-8e48-fad5a7e85684-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2be79e8-0166-4066-8e48-fad5a7e85684-credential-socket\") pod \"f2be79e8-0166-4066-8e48-fad5a7e85684\" (UID: \"f2be79e8-0166-4066-8e48-fad5a7e85684\") "
Oct 31 20:07:44.189980 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:44.189889    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2be79e8-0166-4066-8e48-fad5a7e85684-istiod-ca-cert" pod "f2be79e8-0166-4066-8e48-fad5a7e85684" (UID: "f2be79e8-0166-4066-8e48-fad5a7e85684"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:07:44.190144 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:44.190119    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2be79e8-0166-4066-8e48-fad5a7e85684-istio-data" pod "f2be79e8-0166-4066-8e48-fad5a7e85684" (UID: "f2be79e8-0166-4066-8e48-fad5a7e85684"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:07:44.190376 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:44.190317    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2be79e8-0166-4066-8e48-fad5a7e85684-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:44.190376 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:44.190368    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2be79e8-0166-4066-8e48-fad5a7e85684-credential-socket" pod "f2be79e8-0166-4066-8e48-fad5a7e85684" (UID: "f2be79e8-0166-4066-8e48-fad5a7e85684"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:07:44.190548 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:44.190374    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2be79e8-0166-4066-8e48-fad5a7e85684-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:44.190548 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:44.190410    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2be79e8-0166-4066-8e48-fad5a7e85684-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:44.190750 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:44.190716    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2be79e8-0166-4066-8e48-fad5a7e85684-workload-certs" pod "f2be79e8-0166-4066-8e48-fad5a7e85684" (UID: "f2be79e8-0166-4066-8e48-fad5a7e85684"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:07:44.192162 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:44.192136    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2be79e8-0166-4066-8e48-fad5a7e85684-kube-api-access-xpqkf" pod "f2be79e8-0166-4066-8e48-fad5a7e85684" (UID: "f2be79e8-0166-4066-8e48-fad5a7e85684"). InnerVolumeSpecName "kube-api-access-xpqkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:07:44.192265 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:44.192184    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f2be79e8-0166-4066-8e48-fad5a7e85684-istio-podinfo" pod "f2be79e8-0166-4066-8e48-fad5a7e85684" (UID: "f2be79e8-0166-4066-8e48-fad5a7e85684"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 20:07:44.192535 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:44.192511    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2be79e8-0166-4066-8e48-fad5a7e85684-istio-envoy" pod "f2be79e8-0166-4066-8e48-fad5a7e85684" (UID: "f2be79e8-0166-4066-8e48-fad5a7e85684"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:07:44.192605 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:44.192582    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2be79e8-0166-4066-8e48-fad5a7e85684-istio-token" pod "f2be79e8-0166-4066-8e48-fad5a7e85684" (UID: "f2be79e8-0166-4066-8e48-fad5a7e85684"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:07:44.290951 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:44.290891    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2be79e8-0166-4066-8e48-fad5a7e85684-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:44.290951 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:44.290923    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2be79e8-0166-4066-8e48-fad5a7e85684-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:44.290951 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:44.290935    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2be79e8-0166-4066-8e48-fad5a7e85684-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:44.290951 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:44.290943    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpqkf\" (UniqueName: \"kubernetes.io/projected/f2be79e8-0166-4066-8e48-fad5a7e85684-kube-api-access-xpqkf\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:44.290951 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:44.290952    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2be79e8-0166-4066-8e48-fad5a7e85684-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:44.290951 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:44.290961    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2be79e8-0166-4066-8e48-fad5a7e85684-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:07:44.652674 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:44.652602    4645 generic.go:356] "Generic (PLEG): container finished" podID="f2be79e8-0166-4066-8e48-fad5a7e85684" containerID="4170efc426099cb7ba25f4eef3e13a78a9d5c05bb5a28a1033bff39c14611b04" exitCode=0
Oct 31 20:07:44.652674 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:44.652645    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25"
Oct 31 20:07:44.652674 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:44.652663    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25" event={"ID":"f2be79e8-0166-4066-8e48-fad5a7e85684","Type":"ContainerDied","Data":"4170efc426099cb7ba25f4eef3e13a78a9d5c05bb5a28a1033bff39c14611b04"}
Oct 31 20:07:44.652887 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:44.652704    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25" event={"ID":"f2be79e8-0166-4066-8e48-fad5a7e85684","Type":"ContainerDied","Data":"76cfcafb84b0fa5fde104f0f1659a97b9c062279b3b167dbc969742401c49f7b"}
Oct 31 20:07:44.652887 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:44.652727    4645 scope.go:122] "RemoveContainer" containerID="4170efc426099cb7ba25f4eef3e13a78a9d5c05bb5a28a1033bff39c14611b04"
Oct 31 20:07:44.663029 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:44.663013    4645 scope.go:122] "RemoveContainer" containerID="a3220b1028914edc39c544970b4bcd369fad5406860f38d28edd234b289d22d1"
Oct 31 20:07:44.672149 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:44.672134    4645 scope.go:122] "RemoveContainer" containerID="4170efc426099cb7ba25f4eef3e13a78a9d5c05bb5a28a1033bff39c14611b04"
Oct 31 20:07:44.672507 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:07:44.672475    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4170efc426099cb7ba25f4eef3e13a78a9d5c05bb5a28a1033bff39c14611b04\": container with ID starting with 4170efc426099cb7ba25f4eef3e13a78a9d5c05bb5a28a1033bff39c14611b04 not found: ID does not exist" containerID="4170efc426099cb7ba25f4eef3e13a78a9d5c05bb5a28a1033bff39c14611b04"
Oct 31 20:07:44.672664 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:44.672513    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4170efc426099cb7ba25f4eef3e13a78a9d5c05bb5a28a1033bff39c14611b04"} err="failed to get container status \"4170efc426099cb7ba25f4eef3e13a78a9d5c05bb5a28a1033bff39c14611b04\": rpc error: code = NotFound desc = could not find container \"4170efc426099cb7ba25f4eef3e13a78a9d5c05bb5a28a1033bff39c14611b04\": container with ID starting with 4170efc426099cb7ba25f4eef3e13a78a9d5c05bb5a28a1033bff39c14611b04 not found: ID does not exist"
Oct 31 20:07:44.672664 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:44.672530    4645 scope.go:122] "RemoveContainer" containerID="a3220b1028914edc39c544970b4bcd369fad5406860f38d28edd234b289d22d1"
Oct 31 20:07:44.672917 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:07:44.672845    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3220b1028914edc39c544970b4bcd369fad5406860f38d28edd234b289d22d1\": container with ID starting with a3220b1028914edc39c544970b4bcd369fad5406860f38d28edd234b289d22d1 not found: ID does not exist" containerID="a3220b1028914edc39c544970b4bcd369fad5406860f38d28edd234b289d22d1"
Oct 31 20:07:44.672917 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:44.672877    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3220b1028914edc39c544970b4bcd369fad5406860f38d28edd234b289d22d1"} err="failed to get container status \"a3220b1028914edc39c544970b4bcd369fad5406860f38d28edd234b289d22d1\": rpc error: code = NotFound desc = could not find container \"a3220b1028914edc39c544970b4bcd369fad5406860f38d28edd234b289d22d1\": container with ID starting with a3220b1028914edc39c544970b4bcd369fad5406860f38d28edd234b289d22d1 not found: ID does not exist"
Oct 31 20:07:44.675205 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:44.675180    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25"]
Oct 31 20:07:44.680124 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:44.680104    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-c55cf1-openshift-default-5c956b6f87-m5f25"]
Oct 31 20:07:44.843004 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:44.842981    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:45.616480 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:45.616440    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2be79e8-0166-4066-8e48-fad5a7e85684" path="/var/lib/kubelet/pods/f2be79e8-0166-4066-8e48-fad5a7e85684/volumes"
Oct 31 20:07:45.756525 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:45.756490    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:46.689283 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:46.689250    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:47.253831 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:47.253796    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:47.504132 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:47.504048    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:47.601007 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:47.600972    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-435728-jbs48"]
Oct 31 20:07:47.607234 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:47.607202    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-435728-jbs48"]
Oct 31 20:07:47.618921 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:47.618889    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5038c2c6-211f-4760-8cd1-ed799d8cecc3" path="/var/lib/kubelet/pods/5038c2c6-211f-4760-8cd1-ed799d8cecc3/volumes"
Oct 31 20:07:47.657726 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:47.657697    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:47.798278 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:47.798184    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:48.141358 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:48.141269    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:48.613742 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:48.613701    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:49.660684 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:49.660646    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:50.092148 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:50.092070    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:50.658655 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:50.658622    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:51.316571 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:51.316532    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:51.500696 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:51.500662    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:51.568964 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:51.568886    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae4ce2-xhcdk"]
Oct 31 20:07:51.580801 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:51.580769    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae4ce2-xhcdk"]
Oct 31 20:07:51.616389 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:51.616359    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e95b1cd3-9e02-41af-a69a-13270b64ec87" path="/var/lib/kubelet/pods/e95b1cd3-9e02-41af-a69a-13270b64ec87/volumes"
Oct 31 20:07:51.620932 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:51.620902    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:51.751675 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:51.751636    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:52.094176 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:52.094141    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:52.465437 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:52.465406    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:53.349785 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:53.349754    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:54.306243 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:54.306181    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:55.145473 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:55.145318    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:56.084353 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:56.084308    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:56.255207 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:56.255168    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:56.410467 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:56.410430    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:56.615080 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:56.615051    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5"]
Oct 31 20:07:56.615278 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:56.615230    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5" podUID="401797c4-b8ea-463e-acca-b35f26b5643f" containerName="istio-proxy" containerID="cri-o://271ce49aa8f7f10b0663d40744450f0f5c499fd51ac960453264139275f114a8" gracePeriod=30
Oct 31 20:07:56.741678 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:56.741599    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:57.147648 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:57.147574    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:57.311530 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:57.311496    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:58.354202 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:58.354166    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:07:59.273380 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:07:59.273338    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:08:00.409010 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:00.408950    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:08:00.904013 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:00.903976    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:08:01.051569 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:01.051533    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:08:01.212659 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:01.212630    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4"]
Oct 31 20:08:01.212873 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:01.212853    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4" podUID="5c010664-b8c0-4a2c-91d7-3f66e006b4ec" containerName="istio-proxy" containerID="cri-o://262b687feec098c344273a668a9b3207ee0dafc35a0ef0dc7a36fc538dd8a3bd" gracePeriod=30
Oct 31 20:08:01.263921 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:01.263887    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:08:01.551865 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:01.551774    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:08:01.755764 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:01.755715    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:08:01.765246 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:01.765217    4645 generic.go:356] "Generic (PLEG): container finished" podID="401797c4-b8ea-463e-acca-b35f26b5643f" containerID="271ce49aa8f7f10b0663d40744450f0f5c499fd51ac960453264139275f114a8" exitCode=0
Oct 31 20:08:01.765400 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:01.765292    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5" event={"ID":"401797c4-b8ea-463e-acca-b35f26b5643f","Type":"ContainerDied","Data":"271ce49aa8f7f10b0663d40744450f0f5c499fd51ac960453264139275f114a8"}
Oct 31 20:08:01.765400 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:01.765381    4645 scope.go:122] "RemoveContainer" containerID="a9d11f90740c9cbd633cb93e292437b3c040d227e42a288daaa29fc5df8ff904"
Oct 31 20:08:01.870744 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:01.870718    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5"
Oct 31 20:08:01.964747 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:01.964718    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/401797c4-b8ea-463e-acca-b35f26b5643f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/401797c4-b8ea-463e-acca-b35f26b5643f-istio-podinfo\") pod \"401797c4-b8ea-463e-acca-b35f26b5643f\" (UID: \"401797c4-b8ea-463e-acca-b35f26b5643f\") "
Oct 31 20:08:01.964904 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:01.964754    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/401797c4-b8ea-463e-acca-b35f26b5643f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/401797c4-b8ea-463e-acca-b35f26b5643f-istiod-ca-cert\") pod \"401797c4-b8ea-463e-acca-b35f26b5643f\" (UID: \"401797c4-b8ea-463e-acca-b35f26b5643f\") "
Oct 31 20:08:01.964904 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:01.964790    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/401797c4-b8ea-463e-acca-b35f26b5643f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/401797c4-b8ea-463e-acca-b35f26b5643f-istio-envoy\") pod \"401797c4-b8ea-463e-acca-b35f26b5643f\" (UID: \"401797c4-b8ea-463e-acca-b35f26b5643f\") "
Oct 31 20:08:01.964904 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:01.964814    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/401797c4-b8ea-463e-acca-b35f26b5643f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/401797c4-b8ea-463e-acca-b35f26b5643f-istio-data\") pod \"401797c4-b8ea-463e-acca-b35f26b5643f\" (UID: \"401797c4-b8ea-463e-acca-b35f26b5643f\") "
Oct 31 20:08:01.964904 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:01.964846    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/401797c4-b8ea-463e-acca-b35f26b5643f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/401797c4-b8ea-463e-acca-b35f26b5643f-credential-socket\") pod \"401797c4-b8ea-463e-acca-b35f26b5643f\" (UID: \"401797c4-b8ea-463e-acca-b35f26b5643f\") "
Oct 31 20:08:01.964904 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:01.964869    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/401797c4-b8ea-463e-acca-b35f26b5643f-istio-token\" (UniqueName: \"kubernetes.io/projected/401797c4-b8ea-463e-acca-b35f26b5643f-istio-token\") pod \"401797c4-b8ea-463e-acca-b35f26b5643f\" (UID: \"401797c4-b8ea-463e-acca-b35f26b5643f\") "
Oct 31 20:08:01.964904 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:01.964897    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/401797c4-b8ea-463e-acca-b35f26b5643f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/401797c4-b8ea-463e-acca-b35f26b5643f-workload-socket\") pod \"401797c4-b8ea-463e-acca-b35f26b5643f\" (UID: \"401797c4-b8ea-463e-acca-b35f26b5643f\") "
Oct 31 20:08:01.965533 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:01.964929    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/401797c4-b8ea-463e-acca-b35f26b5643f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/401797c4-b8ea-463e-acca-b35f26b5643f-workload-certs\") pod \"401797c4-b8ea-463e-acca-b35f26b5643f\" (UID: \"401797c4-b8ea-463e-acca-b35f26b5643f\") "
Oct 31 20:08:01.965533 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:01.965039    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/401797c4-b8ea-463e-acca-b35f26b5643f-kube-api-access-5lwfp\" (UniqueName: \"kubernetes.io/projected/401797c4-b8ea-463e-acca-b35f26b5643f-kube-api-access-5lwfp\") pod \"401797c4-b8ea-463e-acca-b35f26b5643f\" (UID: \"401797c4-b8ea-463e-acca-b35f26b5643f\") "
Oct 31 20:08:01.965533 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:01.965186    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/401797c4-b8ea-463e-acca-b35f26b5643f-istio-data" pod "401797c4-b8ea-463e-acca-b35f26b5643f" (UID: "401797c4-b8ea-463e-acca-b35f26b5643f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:08:01.965533 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:01.965212    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/401797c4-b8ea-463e-acca-b35f26b5643f-credential-socket" pod "401797c4-b8ea-463e-acca-b35f26b5643f" (UID: "401797c4-b8ea-463e-acca-b35f26b5643f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:08:01.965533 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:01.965224    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/401797c4-b8ea-463e-acca-b35f26b5643f-istiod-ca-cert" pod "401797c4-b8ea-463e-acca-b35f26b5643f" (UID: "401797c4-b8ea-463e-acca-b35f26b5643f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:08:01.965533 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:01.965449    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/401797c4-b8ea-463e-acca-b35f26b5643f-workload-certs" pod "401797c4-b8ea-463e-acca-b35f26b5643f" (UID: "401797c4-b8ea-463e-acca-b35f26b5643f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:08:01.965533 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:01.965473    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/401797c4-b8ea-463e-acca-b35f26b5643f-workload-socket" pod "401797c4-b8ea-463e-acca-b35f26b5643f" (UID: "401797c4-b8ea-463e-acca-b35f26b5643f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:08:01.965533 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:01.965525    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/401797c4-b8ea-463e-acca-b35f26b5643f-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:01.965861 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:01.965547    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/401797c4-b8ea-463e-acca-b35f26b5643f-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:01.965861 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:01.965562    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/401797c4-b8ea-463e-acca-b35f26b5643f-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:01.967143 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:01.967122    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/401797c4-b8ea-463e-acca-b35f26b5643f-istio-podinfo" pod "401797c4-b8ea-463e-acca-b35f26b5643f" (UID: "401797c4-b8ea-463e-acca-b35f26b5643f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 20:08:01.967238 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:01.967196    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/401797c4-b8ea-463e-acca-b35f26b5643f-kube-api-access-5lwfp" pod "401797c4-b8ea-463e-acca-b35f26b5643f" (UID: "401797c4-b8ea-463e-acca-b35f26b5643f"). InnerVolumeSpecName "kube-api-access-5lwfp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:08:01.967451 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:01.967435    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/401797c4-b8ea-463e-acca-b35f26b5643f-istio-envoy" pod "401797c4-b8ea-463e-acca-b35f26b5643f" (UID: "401797c4-b8ea-463e-acca-b35f26b5643f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:08:01.967538 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:01.967518    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/401797c4-b8ea-463e-acca-b35f26b5643f-istio-token" pod "401797c4-b8ea-463e-acca-b35f26b5643f" (UID: "401797c4-b8ea-463e-acca-b35f26b5643f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:08:02.066432 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:02.066402    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lwfp\" (UniqueName: \"kubernetes.io/projected/401797c4-b8ea-463e-acca-b35f26b5643f-kube-api-access-5lwfp\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:02.066432 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:02.066429    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/401797c4-b8ea-463e-acca-b35f26b5643f-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:02.066578 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:02.066446    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/401797c4-b8ea-463e-acca-b35f26b5643f-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:02.066578 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:02.066458    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/401797c4-b8ea-463e-acca-b35f26b5643f-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:02.066578 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:02.066470    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/401797c4-b8ea-463e-acca-b35f26b5643f-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:02.066578 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:02.066485    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/401797c4-b8ea-463e-acca-b35f26b5643f-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:02.079213 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:02.079185    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l"]
Oct 31 20:08:02.239544 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:02.239515    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:08:02.775982 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:02.775944    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5" event={"ID":"401797c4-b8ea-463e-acca-b35f26b5643f","Type":"ContainerDied","Data":"1ab04ea69e2a70bc311473e1ab5a54bcdd1f0766c38573fd52e34cb9ec0e8983"}
Oct 31 20:08:02.776617 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:02.776593    4645 scope.go:122] "RemoveContainer" containerID="271ce49aa8f7f10b0663d40744450f0f5c499fd51ac960453264139275f114a8"
Oct 31 20:08:02.777112 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:02.777091    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5"
Oct 31 20:08:02.806018 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:02.805981    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5"]
Oct 31 20:08:02.809488 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:02.809459    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-435728-openshift-default-5f778d7cf7-vbzt5"]
Oct 31 20:08:02.829044 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:02.829018    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:08:03.546897 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:03.546867    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:08:03.616576 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:03.616545    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="401797c4-b8ea-463e-acca-b35f26b5643f" path="/var/lib/kubelet/pods/401797c4-b8ea-463e-acca-b35f26b5643f/volumes"
Oct 31 20:08:04.379757 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:04.379727    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:08:05.178440 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:05.178404    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:08:05.957100 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:05.957058    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:08:06.246620 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:08:06.246585    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 20:08:06.470910 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:06.470884    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4"
Oct 31 20:08:06.607982 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:06.607905    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c010664-b8c0-4a2c-91d7-3f66e006b4ec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c010664-b8c0-4a2c-91d7-3f66e006b4ec-credential-socket\") pod \"5c010664-b8c0-4a2c-91d7-3f66e006b4ec\" (UID: \"5c010664-b8c0-4a2c-91d7-3f66e006b4ec\") "
Oct 31 20:08:06.608156 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:06.607987    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c010664-b8c0-4a2c-91d7-3f66e006b4ec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c010664-b8c0-4a2c-91d7-3f66e006b4ec-workload-socket\") pod \"5c010664-b8c0-4a2c-91d7-3f66e006b4ec\" (UID: \"5c010664-b8c0-4a2c-91d7-3f66e006b4ec\") "
Oct 31 20:08:06.608156 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:06.608017    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c010664-b8c0-4a2c-91d7-3f66e006b4ec-kube-api-access-n5f66\" (UniqueName: \"kubernetes.io/projected/5c010664-b8c0-4a2c-91d7-3f66e006b4ec-kube-api-access-n5f66\") pod \"5c010664-b8c0-4a2c-91d7-3f66e006b4ec\" (UID: \"5c010664-b8c0-4a2c-91d7-3f66e006b4ec\") "
Oct 31 20:08:06.608156 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:06.608042    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c010664-b8c0-4a2c-91d7-3f66e006b4ec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c010664-b8c0-4a2c-91d7-3f66e006b4ec-istio-envoy\") pod \"5c010664-b8c0-4a2c-91d7-3f66e006b4ec\" (UID: \"5c010664-b8c0-4a2c-91d7-3f66e006b4ec\") "
Oct 31 20:08:06.608156 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:06.608061    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c010664-b8c0-4a2c-91d7-3f66e006b4ec-istio-token\" (UniqueName: \"kubernetes.io/projected/5c010664-b8c0-4a2c-91d7-3f66e006b4ec-istio-token\") pod \"5c010664-b8c0-4a2c-91d7-3f66e006b4ec\" (UID: \"5c010664-b8c0-4a2c-91d7-3f66e006b4ec\") "
Oct 31 20:08:06.608156 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:06.608078    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c010664-b8c0-4a2c-91d7-3f66e006b4ec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c010664-b8c0-4a2c-91d7-3f66e006b4ec-istio-data\") pod \"5c010664-b8c0-4a2c-91d7-3f66e006b4ec\" (UID: \"5c010664-b8c0-4a2c-91d7-3f66e006b4ec\") "
Oct 31 20:08:06.608156 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:06.608118    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c010664-b8c0-4a2c-91d7-3f66e006b4ec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c010664-b8c0-4a2c-91d7-3f66e006b4ec-istiod-ca-cert\") pod \"5c010664-b8c0-4a2c-91d7-3f66e006b4ec\" (UID: \"5c010664-b8c0-4a2c-91d7-3f66e006b4ec\") "
Oct 31 20:08:06.608555 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:06.608179    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c010664-b8c0-4a2c-91d7-3f66e006b4ec-credential-socket" pod "5c010664-b8c0-4a2c-91d7-3f66e006b4ec" (UID: "5c010664-b8c0-4a2c-91d7-3f66e006b4ec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:08:06.608555 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:06.608318    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5c010664-b8c0-4a2c-91d7-3f66e006b4ec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c010664-b8c0-4a2c-91d7-3f66e006b4ec-istio-podinfo\") pod \"5c010664-b8c0-4a2c-91d7-3f66e006b4ec\" (UID: \"5c010664-b8c0-4a2c-91d7-3f66e006b4ec\") "
Oct 31 20:08:06.608555 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:06.608323    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c010664-b8c0-4a2c-91d7-3f66e006b4ec-workload-socket" pod "5c010664-b8c0-4a2c-91d7-3f66e006b4ec" (UID: "5c010664-b8c0-4a2c-91d7-3f66e006b4ec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:08:06.608555 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:06.608384    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c010664-b8c0-4a2c-91d7-3f66e006b4ec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c010664-b8c0-4a2c-91d7-3f66e006b4ec-workload-certs\") pod \"5c010664-b8c0-4a2c-91d7-3f66e006b4ec\" (UID: \"5c010664-b8c0-4a2c-91d7-3f66e006b4ec\") "
Oct 31 20:08:06.608555 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:06.608465    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c010664-b8c0-4a2c-91d7-3f66e006b4ec-istio-data" pod "5c010664-b8c0-4a2c-91d7-3f66e006b4ec" (UID: "5c010664-b8c0-4a2c-91d7-3f66e006b4ec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:08:06.608832 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:06.608562    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c010664-b8c0-4a2c-91d7-3f66e006b4ec-istiod-ca-cert" pod "5c010664-b8c0-4a2c-91d7-3f66e006b4ec" (UID: "5c010664-b8c0-4a2c-91d7-3f66e006b4ec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:08:06.608832 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:06.608758    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c010664-b8c0-4a2c-91d7-3f66e006b4ec-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:06.608832 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:06.608780    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c010664-b8c0-4a2c-91d7-3f66e006b4ec-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:06.608832 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:06.608784    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c010664-b8c0-4a2c-91d7-3f66e006b4ec-workload-certs" pod "5c010664-b8c0-4a2c-91d7-3f66e006b4ec" (UID: "5c010664-b8c0-4a2c-91d7-3f66e006b4ec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:08:06.608832 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:06.608794    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c010664-b8c0-4a2c-91d7-3f66e006b4ec-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:06.608832 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:06.608808    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c010664-b8c0-4a2c-91d7-3f66e006b4ec-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:06.610378 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:06.610301    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c010664-b8c0-4a2c-91d7-3f66e006b4ec-istio-envoy" pod "5c010664-b8c0-4a2c-91d7-3f66e006b4ec" (UID: "5c010664-b8c0-4a2c-91d7-3f66e006b4ec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:08:06.610378 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:06.610305    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c010664-b8c0-4a2c-91d7-3f66e006b4ec-kube-api-access-n5f66" pod "5c010664-b8c0-4a2c-91d7-3f66e006b4ec" (UID: "5c010664-b8c0-4a2c-91d7-3f66e006b4ec"). InnerVolumeSpecName "kube-api-access-n5f66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:08:06.610731 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:06.610709    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c010664-b8c0-4a2c-91d7-3f66e006b4ec-istio-token" pod "5c010664-b8c0-4a2c-91d7-3f66e006b4ec" (UID: "5c010664-b8c0-4a2c-91d7-3f66e006b4ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:08:06.610787 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:06.610742    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c010664-b8c0-4a2c-91d7-3f66e006b4ec-istio-podinfo" pod "5c010664-b8c0-4a2c-91d7-3f66e006b4ec" (UID: "5c010664-b8c0-4a2c-91d7-3f66e006b4ec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 20:08:06.710203 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:06.710175    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5f66\" (UniqueName: \"kubernetes.io/projected/5c010664-b8c0-4a2c-91d7-3f66e006b4ec-kube-api-access-n5f66\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:06.710203 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:06.710203    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c010664-b8c0-4a2c-91d7-3f66e006b4ec-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:06.710408 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:06.710213    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c010664-b8c0-4a2c-91d7-3f66e006b4ec-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:06.710408 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:06.710222    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c010664-b8c0-4a2c-91d7-3f66e006b4ec-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:06.710408 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:06.710231    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c010664-b8c0-4a2c-91d7-3f66e006b4ec-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:06.802344 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:06.802291    4645 generic.go:356] "Generic (PLEG): container finished" podID="5c010664-b8c0-4a2c-91d7-3f66e006b4ec" containerID="262b687feec098c344273a668a9b3207ee0dafc35a0ef0dc7a36fc538dd8a3bd" exitCode=0
Oct 31 20:08:06.802470 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:06.802362    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4" event={"ID":"5c010664-b8c0-4a2c-91d7-3f66e006b4ec","Type":"ContainerDied","Data":"262b687feec098c344273a668a9b3207ee0dafc35a0ef0dc7a36fc538dd8a3bd"}
Oct 31 20:08:06.802470 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:06.802416    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4" event={"ID":"5c010664-b8c0-4a2c-91d7-3f66e006b4ec","Type":"ContainerDied","Data":"72217eb365e7188264d60bd0b756c7ab89a3b7b88d21144e727a1255d35b6292"}
Oct 31 20:08:06.802470 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:06.802444    4645 scope.go:122] "RemoveContainer" containerID="262b687feec098c344273a668a9b3207ee0dafc35a0ef0dc7a36fc538dd8a3bd"
Oct 31 20:08:06.802470 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:06.802368    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4"
Oct 31 20:08:06.802804 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:06.802777    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:08:06.813168 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:06.813152    4645 scope.go:122] "RemoveContainer" containerID="1e4c85c9543fa3181e0d806d168f058638109986d6fdb39ba16812d43c75a0a6"
Oct 31 20:08:06.825372 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:06.825354    4645 scope.go:122] "RemoveContainer" containerID="262b687feec098c344273a668a9b3207ee0dafc35a0ef0dc7a36fc538dd8a3bd"
Oct 31 20:08:06.825664 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:08:06.825638    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"262b687feec098c344273a668a9b3207ee0dafc35a0ef0dc7a36fc538dd8a3bd\": container with ID starting with 262b687feec098c344273a668a9b3207ee0dafc35a0ef0dc7a36fc538dd8a3bd not found: ID does not exist" containerID="262b687feec098c344273a668a9b3207ee0dafc35a0ef0dc7a36fc538dd8a3bd"
Oct 31 20:08:06.825759 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:06.825671    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"262b687feec098c344273a668a9b3207ee0dafc35a0ef0dc7a36fc538dd8a3bd"} err="failed to get container status \"262b687feec098c344273a668a9b3207ee0dafc35a0ef0dc7a36fc538dd8a3bd\": rpc error: code = NotFound desc = could not find container \"262b687feec098c344273a668a9b3207ee0dafc35a0ef0dc7a36fc538dd8a3bd\": container with ID starting with 262b687feec098c344273a668a9b3207ee0dafc35a0ef0dc7a36fc538dd8a3bd not found: ID does not exist"
Oct 31 20:08:06.825759 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:06.825687    4645 scope.go:122] "RemoveContainer" containerID="1e4c85c9543fa3181e0d806d168f058638109986d6fdb39ba16812d43c75a0a6"
Oct 31 20:08:06.825910 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:08:06.825895    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e4c85c9543fa3181e0d806d168f058638109986d6fdb39ba16812d43c75a0a6\": container with ID starting with 1e4c85c9543fa3181e0d806d168f058638109986d6fdb39ba16812d43c75a0a6 not found: ID does not exist" containerID="1e4c85c9543fa3181e0d806d168f058638109986d6fdb39ba16812d43c75a0a6"
Oct 31 20:08:06.825960 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:06.825914    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e4c85c9543fa3181e0d806d168f058638109986d6fdb39ba16812d43c75a0a6"} err="failed to get container status \"1e4c85c9543fa3181e0d806d168f058638109986d6fdb39ba16812d43c75a0a6\": rpc error: code = NotFound desc = could not find container \"1e4c85c9543fa3181e0d806d168f058638109986d6fdb39ba16812d43c75a0a6\": container with ID starting with 1e4c85c9543fa3181e0d806d168f058638109986d6fdb39ba16812d43c75a0a6 not found: ID does not exist"
Oct 31 20:08:06.848214 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:06.848189    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4"]
Oct 31 20:08:06.862456 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:06.862400    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-ae4ce2-openshift-default-658c56f79f-sf9v4"]
Oct 31 20:08:06.959191 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:06.959154    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:08:07.581117 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:07.581084    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:08:07.616761 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:07.616728    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c010664-b8c0-4a2c-91d7-3f66e006b4ec" path="/var/lib/kubelet/pods/5c010664-b8c0-4a2c-91d7-3f66e006b4ec/volumes"
Oct 31 20:08:07.664173 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:07.664141    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f96b154d-q49db"]
Oct 31 20:08:07.664575 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:07.664557    4645 cpu_manager.go:414] "containerMap: removing container" podUID="401797c4-b8ea-463e-acca-b35f26b5643f" containerName="istio-proxy"
Oct 31 20:08:07.664669 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:07.664577    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="401797c4-b8ea-463e-acca-b35f26b5643f" containerName="istio-proxy"
Oct 31 20:08:07.664669 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:07.664597    4645 cpu_manager.go:414] "containerMap: removing container" podUID="f2be79e8-0166-4066-8e48-fad5a7e85684" containerName="istio-proxy"
Oct 31 20:08:07.664669 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:07.664606    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2be79e8-0166-4066-8e48-fad5a7e85684" containerName="istio-proxy"
Oct 31 20:08:07.664669 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:07.664620    4645 cpu_manager.go:414] "containerMap: removing container" podUID="401797c4-b8ea-463e-acca-b35f26b5643f" containerName="istio-proxy"
Oct 31 20:08:07.664669 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:07.664627    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="401797c4-b8ea-463e-acca-b35f26b5643f" containerName="istio-proxy"
Oct 31 20:08:07.664669 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:07.664640    4645 cpu_manager.go:414] "containerMap: removing container" podUID="f2be79e8-0166-4066-8e48-fad5a7e85684" containerName="istio-proxy"
Oct 31 20:08:07.664669 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:07.664648    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2be79e8-0166-4066-8e48-fad5a7e85684" containerName="istio-proxy"
Oct 31 20:08:07.665044 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:07.664678    4645 cpu_manager.go:414] "containerMap: removing container" podUID="5c010664-b8c0-4a2c-91d7-3f66e006b4ec" containerName="istio-proxy"
Oct 31 20:08:07.665044 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:07.664686    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c010664-b8c0-4a2c-91d7-3f66e006b4ec" containerName="istio-proxy"
Oct 31 20:08:07.665044 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:07.664699    4645 cpu_manager.go:414] "containerMap: removing container" podUID="5c010664-b8c0-4a2c-91d7-3f66e006b4ec" containerName="istio-proxy"
Oct 31 20:08:07.665044 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:07.664706    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c010664-b8c0-4a2c-91d7-3f66e006b4ec" containerName="istio-proxy"
Oct 31 20:08:07.665044 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:07.664720    4645 cpu_manager.go:414] "containerMap: removing container" podUID="e95b1cd3-9e02-41af-a69a-13270b64ec87" containerName="revoke-keys"
Oct 31 20:08:07.665044 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:07.664727    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e95b1cd3-9e02-41af-a69a-13270b64ec87" containerName="revoke-keys"
Oct 31 20:08:07.665044 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:07.664835    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c010664-b8c0-4a2c-91d7-3f66e006b4ec" containerName="istio-proxy"
Oct 31 20:08:07.665044 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:07.664848    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="e95b1cd3-9e02-41af-a69a-13270b64ec87" containerName="revoke-keys"
Oct 31 20:08:07.665044 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:07.664863    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="401797c4-b8ea-463e-acca-b35f26b5643f" containerName="istio-proxy"
Oct 31 20:08:07.665044 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:07.664874    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2be79e8-0166-4066-8e48-fad5a7e85684" containerName="istio-proxy"
Oct 31 20:08:07.665044 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:07.664883    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="401797c4-b8ea-463e-acca-b35f26b5643f" containerName="istio-proxy"
Oct 31 20:08:07.665044 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:07.664896    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c010664-b8c0-4a2c-91d7-3f66e006b4ec" containerName="istio-proxy"
Oct 31 20:08:07.669866 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:07.669846    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f96b154d-q49db"
Oct 31 20:08:07.682520 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:07.682494    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f96b154d-q49db"]
Oct 31 20:08:07.737367 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:07.736341    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:08:07.756147 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:07.756118    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:08:07.756373 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:07.756355    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 20:08:07.820101 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:07.820063    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e9a668b-bc28-4037-aab6-85f3a803497b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f96b154d-q49db\" (UID: \"7e9a668b-bc28-4037-aab6-85f3a803497b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f96b154d-q49db"
Oct 31 20:08:07.820312 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:07.820108    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e9a668b-bc28-4037-aab6-85f3a803497b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f96b154d-q49db\" (UID: \"7e9a668b-bc28-4037-aab6-85f3a803497b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f96b154d-q49db"
Oct 31 20:08:07.820312 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:07.820133    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwrwr\" (UniqueName: \"kubernetes.io/projected/7e9a668b-bc28-4037-aab6-85f3a803497b-kube-api-access-dwrwr\") pod \"maas-api-revoke-sub-f96b154d-q49db\" (UID: \"7e9a668b-bc28-4037-aab6-85f3a803497b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f96b154d-q49db"
Oct 31 20:08:07.921194 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:07.921156    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e9a668b-bc28-4037-aab6-85f3a803497b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f96b154d-q49db\" (UID: \"7e9a668b-bc28-4037-aab6-85f3a803497b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f96b154d-q49db"
Oct 31 20:08:07.921406 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:07.921199    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e9a668b-bc28-4037-aab6-85f3a803497b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f96b154d-q49db\" (UID: \"7e9a668b-bc28-4037-aab6-85f3a803497b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f96b154d-q49db"
Oct 31 20:08:07.921406 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:07.921221    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwrwr\" (UniqueName: \"kubernetes.io/projected/7e9a668b-bc28-4037-aab6-85f3a803497b-kube-api-access-dwrwr\") pod \"maas-api-revoke-sub-f96b154d-q49db\" (UID: \"7e9a668b-bc28-4037-aab6-85f3a803497b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f96b154d-q49db"
Oct 31 20:08:07.921969 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:07.921943    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e9a668b-bc28-4037-aab6-85f3a803497b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f96b154d-q49db\" (UID: \"7e9a668b-bc28-4037-aab6-85f3a803497b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f96b154d-q49db"
Oct 31 20:08:07.923769 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:07.923741    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e9a668b-bc28-4037-aab6-85f3a803497b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f96b154d-q49db\" (UID: \"7e9a668b-bc28-4037-aab6-85f3a803497b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f96b154d-q49db"
Oct 31 20:08:07.933506 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:07.933483    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwrwr\" (UniqueName: \"kubernetes.io/projected/7e9a668b-bc28-4037-aab6-85f3a803497b-kube-api-access-dwrwr\") pod \"maas-api-revoke-sub-f96b154d-q49db\" (UID: \"7e9a668b-bc28-4037-aab6-85f3a803497b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f96b154d-q49db"
Oct 31 20:08:07.961654 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:07.961622    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:08:07.989127 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:07.989095    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f96b154d-q49db"
Oct 31 20:08:08.305718 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:08.305695    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f96b154d-q49db"]
Oct 31 20:08:08.308169 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:08:08.308127    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e9a668b_bc28_4037_aab6_85f3a803497b.slice/crio-d0516febb4826dba43a1447e2685189978b0c998e2cefc27eef2d6eddc216e6e WatchSource:0}: Error finding container d0516febb4826dba43a1447e2685189978b0c998e2cefc27eef2d6eddc216e6e: Status 404 returned error can't find the container with id d0516febb4826dba43a1447e2685189978b0c998e2cefc27eef2d6eddc216e6e
Oct 31 20:08:08.821954 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:08.821867    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f96b154d-q49db" event={"ID":"7e9a668b-bc28-4037-aab6-85f3a803497b","Type":"ContainerStarted","Data":"d05d007f9f87ba18ec4d293e47facfeca77ad7a53c5e274a6a2a8888b06cfbbf"}
Oct 31 20:08:08.821954 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:08.821923    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f96b154d-q49db" event={"ID":"7e9a668b-bc28-4037-aab6-85f3a803497b","Type":"ContainerStarted","Data":"d0516febb4826dba43a1447e2685189978b0c998e2cefc27eef2d6eddc216e6e"}
Oct 31 20:08:08.840056 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:08.839994    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f96b154d-q49db" podStartSLOduration=1.839978897 podStartE2EDuration="1.839978897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:08:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:08:08.83836254 +0000 UTC m=+2919.411596659" watchObservedRunningTime="2026-10-31 20:08:08.839978897 +0000 UTC m=+2919.413213028"
Oct 31 20:08:08.978304 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:08.978276    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:08:09.828932 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:09.828900    4645 generic.go:356] "Generic (PLEG): container finished" podID="7e9a668b-bc28-4037-aab6-85f3a803497b" containerID="d05d007f9f87ba18ec4d293e47facfeca77ad7a53c5e274a6a2a8888b06cfbbf" exitCode=0
Oct 31 20:08:09.829111 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:09.828984    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f96b154d-q49db" event={"ID":"7e9a668b-bc28-4037-aab6-85f3a803497b","Type":"ContainerDied","Data":"d05d007f9f87ba18ec4d293e47facfeca77ad7a53c5e274a6a2a8888b06cfbbf"}
Oct 31 20:08:10.029774 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:10.029736    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:08:10.811132 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:10.811091    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:08:10.968746 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:10.968722    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f96b154d-q49db"
Oct 31 20:08:11.151905 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:11.151842    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e9a668b-bc28-4037-aab6-85f3a803497b-kube-api-access-dwrwr\" (UniqueName: \"kubernetes.io/projected/7e9a668b-bc28-4037-aab6-85f3a803497b-kube-api-access-dwrwr\") pod \"7e9a668b-bc28-4037-aab6-85f3a803497b\" (UID: \"7e9a668b-bc28-4037-aab6-85f3a803497b\") "
Oct 31 20:08:11.151905 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:11.151899    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e9a668b-bc28-4037-aab6-85f3a803497b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e9a668b-bc28-4037-aab6-85f3a803497b-maas-api-openshift-ca\") pod \"7e9a668b-bc28-4037-aab6-85f3a803497b\" (UID: \"7e9a668b-bc28-4037-aab6-85f3a803497b\") "
Oct 31 20:08:11.152266 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:11.151928    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e9a668b-bc28-4037-aab6-85f3a803497b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e9a668b-bc28-4037-aab6-85f3a803497b-maas-api-xks-ca\") pod \"7e9a668b-bc28-4037-aab6-85f3a803497b\" (UID: \"7e9a668b-bc28-4037-aab6-85f3a803497b\") "
Oct 31 20:08:11.152266 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:11.152241    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e9a668b-bc28-4037-aab6-85f3a803497b-maas-api-openshift-ca" pod "7e9a668b-bc28-4037-aab6-85f3a803497b" (UID: "7e9a668b-bc28-4037-aab6-85f3a803497b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:08:11.153928 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:11.153903    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e9a668b-bc28-4037-aab6-85f3a803497b-kube-api-access-dwrwr" pod "7e9a668b-bc28-4037-aab6-85f3a803497b" (UID: "7e9a668b-bc28-4037-aab6-85f3a803497b"). InnerVolumeSpecName "kube-api-access-dwrwr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:08:11.154087 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:11.154065    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e9a668b-bc28-4037-aab6-85f3a803497b-maas-api-xks-ca" pod "7e9a668b-bc28-4037-aab6-85f3a803497b" (UID: "7e9a668b-bc28-4037-aab6-85f3a803497b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:08:11.252857 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:11.252834    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwrwr\" (UniqueName: \"kubernetes.io/projected/7e9a668b-bc28-4037-aab6-85f3a803497b-kube-api-access-dwrwr\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:11.252969 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:11.252860    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e9a668b-bc28-4037-aab6-85f3a803497b-maas-api-openshift-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:11.252969 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:11.252874    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e9a668b-bc28-4037-aab6-85f3a803497b-maas-api-xks-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:11.608610 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:11.608556    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:08:11.842754 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:11.842717    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f96b154d-q49db" event={"ID":"7e9a668b-bc28-4037-aab6-85f3a803497b","Type":"ContainerDied","Data":"d0516febb4826dba43a1447e2685189978b0c998e2cefc27eef2d6eddc216e6e"}
Oct 31 20:08:11.842754 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:11.842736    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f96b154d-q49db"
Oct 31 20:08:11.843009 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:11.842765    4645 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0516febb4826dba43a1447e2685189978b0c998e2cefc27eef2d6eddc216e6e"
Oct 31 20:08:12.329380 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:12.329343    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:08:12.491595 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:12.491556    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:08:12.589501 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:12.589414    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9017f71-mvp5f"]
Oct 31 20:08:12.589842 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:12.589824    4645 cpu_manager.go:414] "containerMap: removing container" podUID="7e9a668b-bc28-4037-aab6-85f3a803497b" containerName="revoke-keys"
Oct 31 20:08:12.589842 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:12.589841    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e9a668b-bc28-4037-aab6-85f3a803497b" containerName="revoke-keys"
Oct 31 20:08:12.590008 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:12.589911    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2be79e8-0166-4066-8e48-fad5a7e85684" containerName="istio-proxy"
Oct 31 20:08:12.590008 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:12.589927    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e9a668b-bc28-4037-aab6-85f3a803497b" containerName="revoke-keys"
Oct 31 20:08:12.593336 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:12.593299    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9017f71-mvp5f"
Oct 31 20:08:12.614601 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:12.614573    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9017f71-mvp5f"]
Oct 31 20:08:12.648568 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:12.648528    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:08:12.648751 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:12.648686    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 20:08:12.771442 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:12.771403    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f82be1ed-85af-4e4e-8381-97c6f58648c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9017f71-mvp5f\" (UID: \"f82be1ed-85af-4e4e-8381-97c6f58648c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9017f71-mvp5f"
Oct 31 20:08:12.771649 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:12.771463    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjp6h\" (UniqueName: \"kubernetes.io/projected/f82be1ed-85af-4e4e-8381-97c6f58648c3-kube-api-access-tjp6h\") pod \"maas-api-revoke-sub-e9017f71-mvp5f\" (UID: \"f82be1ed-85af-4e4e-8381-97c6f58648c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9017f71-mvp5f"
Oct 31 20:08:12.771649 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:12.771575    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f82be1ed-85af-4e4e-8381-97c6f58648c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9017f71-mvp5f\" (UID: \"f82be1ed-85af-4e4e-8381-97c6f58648c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9017f71-mvp5f"
Oct 31 20:08:12.872785 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:12.872704    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f82be1ed-85af-4e4e-8381-97c6f58648c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9017f71-mvp5f\" (UID: \"f82be1ed-85af-4e4e-8381-97c6f58648c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9017f71-mvp5f"
Oct 31 20:08:12.872785 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:12.872759    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjp6h\" (UniqueName: \"kubernetes.io/projected/f82be1ed-85af-4e4e-8381-97c6f58648c3-kube-api-access-tjp6h\") pod \"maas-api-revoke-sub-e9017f71-mvp5f\" (UID: \"f82be1ed-85af-4e4e-8381-97c6f58648c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9017f71-mvp5f"
Oct 31 20:08:12.872978 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:12.872811    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f82be1ed-85af-4e4e-8381-97c6f58648c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9017f71-mvp5f\" (UID: \"f82be1ed-85af-4e4e-8381-97c6f58648c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9017f71-mvp5f"
Oct 31 20:08:12.873506 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:12.873482    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f82be1ed-85af-4e4e-8381-97c6f58648c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9017f71-mvp5f\" (UID: \"f82be1ed-85af-4e4e-8381-97c6f58648c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9017f71-mvp5f"
Oct 31 20:08:12.875214 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:12.875194    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f82be1ed-85af-4e4e-8381-97c6f58648c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9017f71-mvp5f\" (UID: \"f82be1ed-85af-4e4e-8381-97c6f58648c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9017f71-mvp5f"
Oct 31 20:08:12.894005 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:12.893982    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjp6h\" (UniqueName: \"kubernetes.io/projected/f82be1ed-85af-4e4e-8381-97c6f58648c3-kube-api-access-tjp6h\") pod \"maas-api-revoke-sub-e9017f71-mvp5f\" (UID: \"f82be1ed-85af-4e4e-8381-97c6f58648c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9017f71-mvp5f"
Oct 31 20:08:12.909324 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:12.909298    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9017f71-mvp5f"
Oct 31 20:08:13.192287 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:13.192260    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9017f71-mvp5f"]
Oct 31 20:08:13.194170 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:08:13.194137    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf82be1ed_85af_4e4e_8381_97c6f58648c3.slice/crio-70593b42ad358535705f7697837897296fe6509f758bd4d3d8f9750d52a41437 WatchSource:0}: Error finding container 70593b42ad358535705f7697837897296fe6509f758bd4d3d8f9750d52a41437: Status 404 returned error can't find the container with id 70593b42ad358535705f7697837897296fe6509f758bd4d3d8f9750d52a41437
Oct 31 20:08:13.858020 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:13.857928    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9017f71-mvp5f" event={"ID":"f82be1ed-85af-4e4e-8381-97c6f58648c3","Type":"ContainerStarted","Data":"5768066d7883465c069594a44564fff940022a86957cc0e7df52979f4557ac56"}
Oct 31 20:08:13.858020 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:13.857997    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9017f71-mvp5f" event={"ID":"f82be1ed-85af-4e4e-8381-97c6f58648c3","Type":"ContainerStarted","Data":"70593b42ad358535705f7697837897296fe6509f758bd4d3d8f9750d52a41437"}
Oct 31 20:08:13.883407 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:13.883359    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9017f71-mvp5f" podStartSLOduration=1.883315965 podStartE2EDuration="1.883315965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:08:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:08:13.880680997 +0000 UTC m=+2924.453915127" watchObservedRunningTime="2026-10-31 20:08:13.883315965 +0000 UTC m=+2924.456550094"
Oct 31 20:08:14.866775 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:14.866742    4645 generic.go:356] "Generic (PLEG): container finished" podID="f82be1ed-85af-4e4e-8381-97c6f58648c3" containerID="5768066d7883465c069594a44564fff940022a86957cc0e7df52979f4557ac56" exitCode=0
Oct 31 20:08:14.867151 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:14.866818    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9017f71-mvp5f" event={"ID":"f82be1ed-85af-4e4e-8381-97c6f58648c3","Type":"ContainerDied","Data":"5768066d7883465c069594a44564fff940022a86957cc0e7df52979f4557ac56"}
Oct 31 20:08:16.015717 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:16.015693    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9017f71-mvp5f"
Oct 31 20:08:16.101533 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:16.101508    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f82be1ed-85af-4e4e-8381-97c6f58648c3-kube-api-access-tjp6h\" (UniqueName: \"kubernetes.io/projected/f82be1ed-85af-4e4e-8381-97c6f58648c3-kube-api-access-tjp6h\") pod \"f82be1ed-85af-4e4e-8381-97c6f58648c3\" (UID: \"f82be1ed-85af-4e4e-8381-97c6f58648c3\") "
Oct 31 20:08:16.101676 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:16.101600    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f82be1ed-85af-4e4e-8381-97c6f58648c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f82be1ed-85af-4e4e-8381-97c6f58648c3-maas-api-xks-ca\") pod \"f82be1ed-85af-4e4e-8381-97c6f58648c3\" (UID: \"f82be1ed-85af-4e4e-8381-97c6f58648c3\") "
Oct 31 20:08:16.101676 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:16.101645    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f82be1ed-85af-4e4e-8381-97c6f58648c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f82be1ed-85af-4e4e-8381-97c6f58648c3-maas-api-openshift-ca\") pod \"f82be1ed-85af-4e4e-8381-97c6f58648c3\" (UID: \"f82be1ed-85af-4e4e-8381-97c6f58648c3\") "
Oct 31 20:08:16.102042 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:16.102012    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f82be1ed-85af-4e4e-8381-97c6f58648c3-maas-api-openshift-ca" pod "f82be1ed-85af-4e4e-8381-97c6f58648c3" (UID: "f82be1ed-85af-4e4e-8381-97c6f58648c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:08:16.103696 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:16.103663    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f82be1ed-85af-4e4e-8381-97c6f58648c3-maas-api-xks-ca" pod "f82be1ed-85af-4e4e-8381-97c6f58648c3" (UID: "f82be1ed-85af-4e4e-8381-97c6f58648c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:08:16.103789 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:16.103774    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f82be1ed-85af-4e4e-8381-97c6f58648c3-kube-api-access-tjp6h" pod "f82be1ed-85af-4e4e-8381-97c6f58648c3" (UID: "f82be1ed-85af-4e4e-8381-97c6f58648c3"). InnerVolumeSpecName "kube-api-access-tjp6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:08:16.202879 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:16.202855    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f82be1ed-85af-4e4e-8381-97c6f58648c3-maas-api-openshift-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:16.202879 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:16.202877    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjp6h\" (UniqueName: \"kubernetes.io/projected/f82be1ed-85af-4e4e-8381-97c6f58648c3-kube-api-access-tjp6h\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:16.203010 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:16.202888    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f82be1ed-85af-4e4e-8381-97c6f58648c3-maas-api-xks-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:16.886245 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:16.886211    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9017f71-mvp5f"
Oct 31 20:08:16.886245 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:16.886236    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9017f71-mvp5f" event={"ID":"f82be1ed-85af-4e4e-8381-97c6f58648c3","Type":"ContainerDied","Data":"70593b42ad358535705f7697837897296fe6509f758bd4d3d8f9750d52a41437"}
Oct 31 20:08:16.886483 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:16.886287    4645 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70593b42ad358535705f7697837897296fe6509f758bd4d3d8f9750d52a41437"
Oct 31 20:08:17.098690 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:17.098651    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l" podUID="12bea86c-61e5-465e-82fb-b4673a4d1be9" containerName="main" containerID="cri-o://1669a4f702146b54e4ce8bbaf55868405d56a79c124e01e0282af8325d0047bc" gracePeriod=45
Oct 31 20:08:17.432813 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:17.432512    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l"
Oct 31 20:08:17.689293 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:17.689254    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12bea86c-61e5-465e-82fb-b4673a4d1be9-kube-api-access-n6hqt\" (UniqueName: \"kubernetes.io/projected/12bea86c-61e5-465e-82fb-b4673a4d1be9-kube-api-access-n6hqt\") pod \"12bea86c-61e5-465e-82fb-b4673a4d1be9\" (UID: \"12bea86c-61e5-465e-82fb-b4673a4d1be9\") "
Oct 31 20:08:17.689293 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:17.689297    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12bea86c-61e5-465e-82fb-b4673a4d1be9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12bea86c-61e5-465e-82fb-b4673a4d1be9-model-cache\") pod \"12bea86c-61e5-465e-82fb-b4673a4d1be9\" (UID: \"12bea86c-61e5-465e-82fb-b4673a4d1be9\") "
Oct 31 20:08:17.689557 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:17.689381    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12bea86c-61e5-465e-82fb-b4673a4d1be9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/12bea86c-61e5-465e-82fb-b4673a4d1be9-dshm\") pod \"12bea86c-61e5-465e-82fb-b4673a4d1be9\" (UID: \"12bea86c-61e5-465e-82fb-b4673a4d1be9\") "
Oct 31 20:08:17.689557 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:17.689420    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12bea86c-61e5-465e-82fb-b4673a4d1be9-home\" (UniqueName: \"kubernetes.io/empty-dir/12bea86c-61e5-465e-82fb-b4673a4d1be9-home\") pod \"12bea86c-61e5-465e-82fb-b4673a4d1be9\" (UID: \"12bea86c-61e5-465e-82fb-b4673a4d1be9\") "
Oct 31 20:08:17.689557 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:17.689470    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12bea86c-61e5-465e-82fb-b4673a4d1be9-tls-certs\" (UniqueName: \"kubernetes.io/secret/12bea86c-61e5-465e-82fb-b4673a4d1be9-tls-certs\") pod \"12bea86c-61e5-465e-82fb-b4673a4d1be9\" (UID: \"12bea86c-61e5-465e-82fb-b4673a4d1be9\") "
Oct 31 20:08:17.689557 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:17.689534    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12bea86c-61e5-465e-82fb-b4673a4d1be9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12bea86c-61e5-465e-82fb-b4673a4d1be9-tmp-dir\") pod \"12bea86c-61e5-465e-82fb-b4673a4d1be9\" (UID: \"12bea86c-61e5-465e-82fb-b4673a4d1be9\") "
Oct 31 20:08:17.690117 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:17.690088    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12bea86c-61e5-465e-82fb-b4673a4d1be9-home" pod "12bea86c-61e5-465e-82fb-b4673a4d1be9" (UID: "12bea86c-61e5-465e-82fb-b4673a4d1be9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:08:17.690312 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:17.690288    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12bea86c-61e5-465e-82fb-b4673a4d1be9-model-cache" pod "12bea86c-61e5-465e-82fb-b4673a4d1be9" (UID: "12bea86c-61e5-465e-82fb-b4673a4d1be9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:08:17.690402 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:17.690361    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12bea86c-61e5-465e-82fb-b4673a4d1be9-tmp-dir" pod "12bea86c-61e5-465e-82fb-b4673a4d1be9" (UID: "12bea86c-61e5-465e-82fb-b4673a4d1be9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:08:17.690800 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:17.690780    4645 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12bea86c-61e5-465e-82fb-b4673a4d1be9-tmp-dir\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:17.692420 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:17.692400    4645 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12bea86c-61e5-465e-82fb-b4673a4d1be9-model-cache\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:17.692578 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:17.692564    4645 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12bea86c-61e5-465e-82fb-b4673a4d1be9-home\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:17.692669 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:17.692600    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12bea86c-61e5-465e-82fb-b4673a4d1be9-dshm" pod "12bea86c-61e5-465e-82fb-b4673a4d1be9" (UID: "12bea86c-61e5-465e-82fb-b4673a4d1be9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:08:17.692756 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:17.692657    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12bea86c-61e5-465e-82fb-b4673a4d1be9-tls-certs" pod "12bea86c-61e5-465e-82fb-b4673a4d1be9" (UID: "12bea86c-61e5-465e-82fb-b4673a4d1be9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:08:17.692756 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:17.692732    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12bea86c-61e5-465e-82fb-b4673a4d1be9-kube-api-access-n6hqt" pod "12bea86c-61e5-465e-82fb-b4673a4d1be9" (UID: "12bea86c-61e5-465e-82fb-b4673a4d1be9"). InnerVolumeSpecName "kube-api-access-n6hqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:08:17.793532 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:17.793491    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6hqt\" (UniqueName: \"kubernetes.io/projected/12bea86c-61e5-465e-82fb-b4673a4d1be9-kube-api-access-n6hqt\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:17.793532 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:17.793521    4645 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12bea86c-61e5-465e-82fb-b4673a4d1be9-dshm\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:17.793532 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:17.793531    4645 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12bea86c-61e5-465e-82fb-b4673a4d1be9-tls-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:17.895410 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:17.895371    4645 generic.go:356] "Generic (PLEG): container finished" podID="12bea86c-61e5-465e-82fb-b4673a4d1be9" containerID="1669a4f702146b54e4ce8bbaf55868405d56a79c124e01e0282af8325d0047bc" exitCode=0
Oct 31 20:08:17.895588 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:17.895453    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l"
Oct 31 20:08:17.895588 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:17.895466    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l" event={"ID":"12bea86c-61e5-465e-82fb-b4673a4d1be9","Type":"ContainerDied","Data":"1669a4f702146b54e4ce8bbaf55868405d56a79c124e01e0282af8325d0047bc"}
Oct 31 20:08:17.895588 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:17.895518    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l" event={"ID":"12bea86c-61e5-465e-82fb-b4673a4d1be9","Type":"ContainerDied","Data":"e8a0f20b7f3a66c704a6cdd4913fa29f426990ea6c33f3bc5507d18daf639cb9"}
Oct 31 20:08:17.895588 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:17.895538    4645 scope.go:122] "RemoveContainer" containerID="1669a4f702146b54e4ce8bbaf55868405d56a79c124e01e0282af8325d0047bc"
Oct 31 20:08:17.905451 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:17.905432    4645 scope.go:122] "RemoveContainer" containerID="1669a4f702146b54e4ce8bbaf55868405d56a79c124e01e0282af8325d0047bc"
Oct 31 20:08:17.905756 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:08:17.905736    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1669a4f702146b54e4ce8bbaf55868405d56a79c124e01e0282af8325d0047bc\": container with ID starting with 1669a4f702146b54e4ce8bbaf55868405d56a79c124e01e0282af8325d0047bc not found: ID does not exist" containerID="1669a4f702146b54e4ce8bbaf55868405d56a79c124e01e0282af8325d0047bc"
Oct 31 20:08:17.905838 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:17.905767    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1669a4f702146b54e4ce8bbaf55868405d56a79c124e01e0282af8325d0047bc"} err="failed to get container status \"1669a4f702146b54e4ce8bbaf55868405d56a79c124e01e0282af8325d0047bc\": rpc error: code = NotFound desc = could not find container \"1669a4f702146b54e4ce8bbaf55868405d56a79c124e01e0282af8325d0047bc\": container with ID starting with 1669a4f702146b54e4ce8bbaf55868405d56a79c124e01e0282af8325d0047bc not found: ID does not exist"
Oct 31 20:08:17.942952 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:17.942883    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l"]
Oct 31 20:08:17.948702 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:17.948670    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-8b083a/distinct-w1-8b083a-kserve-856d568d74-wvv9l"]
Oct 31 20:08:18.465756 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:18.465715    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:08:18.593155 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:18.593120    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:08:18.756912 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:18.756819    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:08:19.324255 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:19.324208    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:08:19.616154 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:19.616078    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12bea86c-61e5-465e-82fb-b4673a4d1be9" path="/var/lib/kubelet/pods/12bea86c-61e5-465e-82fb-b4673a4d1be9/volumes"
Oct 31 20:08:19.867887 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:19.867821    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:08:20.553776 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:20.553744    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:08:25.531021 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:25.530979    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:08:25.793436 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:25.793351    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:08:26.046142 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:26.046056    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:08:26.416746 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:26.416710    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:08:27.489386 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:27.489349    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:08:27.709042 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:27.709008    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:08:27.925600 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:27.925565    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd"]
Oct 31 20:08:27.925817 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:27.925773    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd" podUID="a28d9602-371e-4d21-b4f4-0ba0966dd9f0" containerName="istio-proxy" containerID="cri-o://7027ae75e445f2fddca99bbdbd51052db08abe34861186cff0c60f0742982ab4" gracePeriod=30
Oct 31 20:08:28.014006 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:28.013975    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:08:32.347157 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:32.347117    4645 scope.go:122] "RemoveContainer" containerID="beaf92083c6876ba81650f71806ab23dd0f58cddfc2050c7f2db0cde32cf5005"
Oct 31 20:08:32.358550 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:32.358505    4645 scope.go:122] "RemoveContainer" containerID="b218c166a957be7627b3663410a013d8e811514929e9c142dc8ec5073a3226b9"
Oct 31 20:08:32.671938 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:32.671900    4645 patch_prober.go:29] interesting pod/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 20:08:32.672052 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:32.671976    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd" podUID="a28d9602-371e-4d21-b4f4-0ba0966dd9f0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 20:08:33.177277 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:33.177236    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd"
Oct 31 20:08:33.222165 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:33.222106    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a28d9602-371e-4d21-b4f4-0ba0966dd9f0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a28d9602-371e-4d21-b4f4-0ba0966dd9f0-workload-certs\") pod \"a28d9602-371e-4d21-b4f4-0ba0966dd9f0\" (UID: \"a28d9602-371e-4d21-b4f4-0ba0966dd9f0\") "
Oct 31 20:08:33.222165 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:33.222155    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a28d9602-371e-4d21-b4f4-0ba0966dd9f0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a28d9602-371e-4d21-b4f4-0ba0966dd9f0-istio-podinfo\") pod \"a28d9602-371e-4d21-b4f4-0ba0966dd9f0\" (UID: \"a28d9602-371e-4d21-b4f4-0ba0966dd9f0\") "
Oct 31 20:08:33.222367 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:33.222205    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a28d9602-371e-4d21-b4f4-0ba0966dd9f0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a28d9602-371e-4d21-b4f4-0ba0966dd9f0-istiod-ca-cert\") pod \"a28d9602-371e-4d21-b4f4-0ba0966dd9f0\" (UID: \"a28d9602-371e-4d21-b4f4-0ba0966dd9f0\") "
Oct 31 20:08:33.222367 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:33.222239    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a28d9602-371e-4d21-b4f4-0ba0966dd9f0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a28d9602-371e-4d21-b4f4-0ba0966dd9f0-istio-data\") pod \"a28d9602-371e-4d21-b4f4-0ba0966dd9f0\" (UID: \"a28d9602-371e-4d21-b4f4-0ba0966dd9f0\") "
Oct 31 20:08:33.222367 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:33.222296    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a28d9602-371e-4d21-b4f4-0ba0966dd9f0-istio-token\" (UniqueName: \"kubernetes.io/projected/a28d9602-371e-4d21-b4f4-0ba0966dd9f0-istio-token\") pod \"a28d9602-371e-4d21-b4f4-0ba0966dd9f0\" (UID: \"a28d9602-371e-4d21-b4f4-0ba0966dd9f0\") "
Oct 31 20:08:33.222543 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:33.222369    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a28d9602-371e-4d21-b4f4-0ba0966dd9f0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a28d9602-371e-4d21-b4f4-0ba0966dd9f0-credential-socket\") pod \"a28d9602-371e-4d21-b4f4-0ba0966dd9f0\" (UID: \"a28d9602-371e-4d21-b4f4-0ba0966dd9f0\") "
Oct 31 20:08:33.222543 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:33.222442    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a28d9602-371e-4d21-b4f4-0ba0966dd9f0-workload-certs" pod "a28d9602-371e-4d21-b4f4-0ba0966dd9f0" (UID: "a28d9602-371e-4d21-b4f4-0ba0966dd9f0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:08:33.222543 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:33.222463    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a28d9602-371e-4d21-b4f4-0ba0966dd9f0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a28d9602-371e-4d21-b4f4-0ba0966dd9f0-istio-envoy\") pod \"a28d9602-371e-4d21-b4f4-0ba0966dd9f0\" (UID: \"a28d9602-371e-4d21-b4f4-0ba0966dd9f0\") "
Oct 31 20:08:33.222543 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:33.222493    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a28d9602-371e-4d21-b4f4-0ba0966dd9f0-kube-api-access-7sf6v\" (UniqueName: \"kubernetes.io/projected/a28d9602-371e-4d21-b4f4-0ba0966dd9f0-kube-api-access-7sf6v\") pod \"a28d9602-371e-4d21-b4f4-0ba0966dd9f0\" (UID: \"a28d9602-371e-4d21-b4f4-0ba0966dd9f0\") "
Oct 31 20:08:33.222543 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:33.222539    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a28d9602-371e-4d21-b4f4-0ba0966dd9f0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a28d9602-371e-4d21-b4f4-0ba0966dd9f0-workload-socket\") pod \"a28d9602-371e-4d21-b4f4-0ba0966dd9f0\" (UID: \"a28d9602-371e-4d21-b4f4-0ba0966dd9f0\") "
Oct 31 20:08:33.222834 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:33.222644    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a28d9602-371e-4d21-b4f4-0ba0966dd9f0-istio-data" pod "a28d9602-371e-4d21-b4f4-0ba0966dd9f0" (UID: "a28d9602-371e-4d21-b4f4-0ba0966dd9f0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:08:33.222834 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:33.222663    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a28d9602-371e-4d21-b4f4-0ba0966dd9f0-credential-socket" pod "a28d9602-371e-4d21-b4f4-0ba0966dd9f0" (UID: "a28d9602-371e-4d21-b4f4-0ba0966dd9f0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:08:33.222952 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:33.222845    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a28d9602-371e-4d21-b4f4-0ba0966dd9f0-workload-socket" pod "a28d9602-371e-4d21-b4f4-0ba0966dd9f0" (UID: "a28d9602-371e-4d21-b4f4-0ba0966dd9f0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:08:33.222952 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:33.222929    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a28d9602-371e-4d21-b4f4-0ba0966dd9f0-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:33.222952 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:33.222949    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a28d9602-371e-4d21-b4f4-0ba0966dd9f0-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:33.223140 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:33.222964    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a28d9602-371e-4d21-b4f4-0ba0966dd9f0-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:33.223140 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:33.222978    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a28d9602-371e-4d21-b4f4-0ba0966dd9f0-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:33.223283 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:33.223250    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a28d9602-371e-4d21-b4f4-0ba0966dd9f0-istiod-ca-cert" pod "a28d9602-371e-4d21-b4f4-0ba0966dd9f0" (UID: "a28d9602-371e-4d21-b4f4-0ba0966dd9f0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:08:33.224693 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:33.224666    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a28d9602-371e-4d21-b4f4-0ba0966dd9f0-istio-podinfo" pod "a28d9602-371e-4d21-b4f4-0ba0966dd9f0" (UID: "a28d9602-371e-4d21-b4f4-0ba0966dd9f0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 20:08:33.225032 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:33.225010    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a28d9602-371e-4d21-b4f4-0ba0966dd9f0-istio-envoy" pod "a28d9602-371e-4d21-b4f4-0ba0966dd9f0" (UID: "a28d9602-371e-4d21-b4f4-0ba0966dd9f0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:08:33.225236 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:33.225210    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a28d9602-371e-4d21-b4f4-0ba0966dd9f0-kube-api-access-7sf6v" pod "a28d9602-371e-4d21-b4f4-0ba0966dd9f0" (UID: "a28d9602-371e-4d21-b4f4-0ba0966dd9f0"). InnerVolumeSpecName "kube-api-access-7sf6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:08:33.225380 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:33.225242    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a28d9602-371e-4d21-b4f4-0ba0966dd9f0-istio-token" pod "a28d9602-371e-4d21-b4f4-0ba0966dd9f0" (UID: "a28d9602-371e-4d21-b4f4-0ba0966dd9f0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:08:33.324182 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:33.324158    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a28d9602-371e-4d21-b4f4-0ba0966dd9f0-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:33.324182 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:33.324182    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sf6v\" (UniqueName: \"kubernetes.io/projected/a28d9602-371e-4d21-b4f4-0ba0966dd9f0-kube-api-access-7sf6v\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:33.324353 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:33.324193    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a28d9602-371e-4d21-b4f4-0ba0966dd9f0-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:33.324353 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:33.324203    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a28d9602-371e-4d21-b4f4-0ba0966dd9f0-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:33.324353 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:33.324211    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a28d9602-371e-4d21-b4f4-0ba0966dd9f0-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:33.699304 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:33.699272    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:08:33.907045 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:33.906965    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:08:33.993803 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:33.993759    4645 generic.go:356] "Generic (PLEG): container finished" podID="a28d9602-371e-4d21-b4f4-0ba0966dd9f0" containerID="7027ae75e445f2fddca99bbdbd51052db08abe34861186cff0c60f0742982ab4" exitCode=0
Oct 31 20:08:33.993998 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:33.993815    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd"
Oct 31 20:08:33.993998 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:33.993837    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd" event={"ID":"a28d9602-371e-4d21-b4f4-0ba0966dd9f0","Type":"ContainerDied","Data":"7027ae75e445f2fddca99bbdbd51052db08abe34861186cff0c60f0742982ab4"}
Oct 31 20:08:33.993998 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:33.993904    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd" event={"ID":"a28d9602-371e-4d21-b4f4-0ba0966dd9f0","Type":"ContainerDied","Data":"69aeca40a4d178f613216a310e9de5d818f79541a8205d8e6a6bcc38ea7f31af"}
Oct 31 20:08:33.993998 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:33.993939    4645 scope.go:122] "RemoveContainer" containerID="7027ae75e445f2fddca99bbdbd51052db08abe34861186cff0c60f0742982ab4"
Oct 31 20:08:34.004482 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:34.004470    4645 scope.go:122] "RemoveContainer" containerID="7027ae75e445f2fddca99bbdbd51052db08abe34861186cff0c60f0742982ab4"
Oct 31 20:08:34.004773 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:08:34.004757    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7027ae75e445f2fddca99bbdbd51052db08abe34861186cff0c60f0742982ab4\": container with ID starting with 7027ae75e445f2fddca99bbdbd51052db08abe34861186cff0c60f0742982ab4 not found: ID does not exist" containerID="7027ae75e445f2fddca99bbdbd51052db08abe34861186cff0c60f0742982ab4"
Oct 31 20:08:34.004824 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:34.004782    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7027ae75e445f2fddca99bbdbd51052db08abe34861186cff0c60f0742982ab4"} err="failed to get container status \"7027ae75e445f2fddca99bbdbd51052db08abe34861186cff0c60f0742982ab4\": rpc error: code = NotFound desc = could not find container \"7027ae75e445f2fddca99bbdbd51052db08abe34861186cff0c60f0742982ab4\": container with ID starting with 7027ae75e445f2fddca99bbdbd51052db08abe34861186cff0c60f0742982ab4 not found: ID does not exist"
Oct 31 20:08:34.008462 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:34.008442    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt"]
Oct 31 20:08:34.008631 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:34.008613    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt" podUID="3fa97d00-d3af-4c53-bfce-83746cdd6ce3" containerName="istio-proxy" containerID="cri-o://8ed58bd86fce039d155cb3a7b6b9821ac00b1b0853fb3df59e5eee872377eae1" gracePeriod=30
Oct 31 20:08:34.040042 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:34.040016    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd"]
Oct 31 20:08:34.063167 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:34.063143    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b083a-openshift-default-f965c5bc5-zmszd"]
Oct 31 20:08:34.103963 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:34.103932    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:08:34.622099 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:34.622069    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:08:34.775526 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:34.775494    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:08:35.616417 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:35.616379    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a28d9602-371e-4d21-b4f4-0ba0966dd9f0" path="/var/lib/kubelet/pods/a28d9602-371e-4d21-b4f4-0ba0966dd9f0/volumes"
Oct 31 20:08:36.029389 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:36.029353    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09c55086-fx4jv"]
Oct 31 20:08:36.034682 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:36.034656    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09c55086-fx4jv"]
Oct 31 20:08:36.245881 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:08:36.245847    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 20:08:37.616579 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:37.616537    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48a098c1-470e-4994-845a-3d34a8618ba9" path="/var/lib/kubelet/pods/48a098c1-470e-4994-845a-3d34a8618ba9/volumes"
Oct 31 20:08:39.265254 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:39.265224    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt"
Oct 31 20:08:39.373872 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:39.373792    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fa97d00-d3af-4c53-bfce-83746cdd6ce3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fa97d00-d3af-4c53-bfce-83746cdd6ce3-istio-data\") pod \"3fa97d00-d3af-4c53-bfce-83746cdd6ce3\" (UID: \"3fa97d00-d3af-4c53-bfce-83746cdd6ce3\") "
Oct 31 20:08:39.374030 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:39.373892    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fa97d00-d3af-4c53-bfce-83746cdd6ce3-kube-api-access-xx678\" (UniqueName: \"kubernetes.io/projected/3fa97d00-d3af-4c53-bfce-83746cdd6ce3-kube-api-access-xx678\") pod \"3fa97d00-d3af-4c53-bfce-83746cdd6ce3\" (UID: \"3fa97d00-d3af-4c53-bfce-83746cdd6ce3\") "
Oct 31 20:08:39.374030 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:39.373947    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3fa97d00-d3af-4c53-bfce-83746cdd6ce3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fa97d00-d3af-4c53-bfce-83746cdd6ce3-istio-podinfo\") pod \"3fa97d00-d3af-4c53-bfce-83746cdd6ce3\" (UID: \"3fa97d00-d3af-4c53-bfce-83746cdd6ce3\") "
Oct 31 20:08:39.374169 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:39.374055    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fa97d00-d3af-4c53-bfce-83746cdd6ce3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fa97d00-d3af-4c53-bfce-83746cdd6ce3-istiod-ca-cert\") pod \"3fa97d00-d3af-4c53-bfce-83746cdd6ce3\" (UID: \"3fa97d00-d3af-4c53-bfce-83746cdd6ce3\") "
Oct 31 20:08:39.374169 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:39.374066    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fa97d00-d3af-4c53-bfce-83746cdd6ce3-istio-data" pod "3fa97d00-d3af-4c53-bfce-83746cdd6ce3" (UID: "3fa97d00-d3af-4c53-bfce-83746cdd6ce3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:08:39.374169 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:39.374107    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fa97d00-d3af-4c53-bfce-83746cdd6ce3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa97d00-d3af-4c53-bfce-83746cdd6ce3-credential-socket\") pod \"3fa97d00-d3af-4c53-bfce-83746cdd6ce3\" (UID: \"3fa97d00-d3af-4c53-bfce-83746cdd6ce3\") "
Oct 31 20:08:39.374169 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:39.374159    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fa97d00-d3af-4c53-bfce-83746cdd6ce3-istio-token\" (UniqueName: \"kubernetes.io/projected/3fa97d00-d3af-4c53-bfce-83746cdd6ce3-istio-token\") pod \"3fa97d00-d3af-4c53-bfce-83746cdd6ce3\" (UID: \"3fa97d00-d3af-4c53-bfce-83746cdd6ce3\") "
Oct 31 20:08:39.374451 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:39.374206    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fa97d00-d3af-4c53-bfce-83746cdd6ce3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fa97d00-d3af-4c53-bfce-83746cdd6ce3-workload-certs\") pod \"3fa97d00-d3af-4c53-bfce-83746cdd6ce3\" (UID: \"3fa97d00-d3af-4c53-bfce-83746cdd6ce3\") "
Oct 31 20:08:39.374451 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:39.374246    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fa97d00-d3af-4c53-bfce-83746cdd6ce3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa97d00-d3af-4c53-bfce-83746cdd6ce3-workload-socket\") pod \"3fa97d00-d3af-4c53-bfce-83746cdd6ce3\" (UID: \"3fa97d00-d3af-4c53-bfce-83746cdd6ce3\") "
Oct 31 20:08:39.374451 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:39.374294    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fa97d00-d3af-4c53-bfce-83746cdd6ce3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fa97d00-d3af-4c53-bfce-83746cdd6ce3-istio-envoy\") pod \"3fa97d00-d3af-4c53-bfce-83746cdd6ce3\" (UID: \"3fa97d00-d3af-4c53-bfce-83746cdd6ce3\") "
Oct 31 20:08:39.374629 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:39.374529    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fa97d00-d3af-4c53-bfce-83746cdd6ce3-workload-certs" pod "3fa97d00-d3af-4c53-bfce-83746cdd6ce3" (UID: "3fa97d00-d3af-4c53-bfce-83746cdd6ce3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:08:39.374691 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:39.374654    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fa97d00-d3af-4c53-bfce-83746cdd6ce3-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:39.374691 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:39.374675    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fa97d00-d3af-4c53-bfce-83746cdd6ce3-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:39.374691 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:39.374673    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fa97d00-d3af-4c53-bfce-83746cdd6ce3-credential-socket" pod "3fa97d00-d3af-4c53-bfce-83746cdd6ce3" (UID: "3fa97d00-d3af-4c53-bfce-83746cdd6ce3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:08:39.374935 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:39.374887    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fa97d00-d3af-4c53-bfce-83746cdd6ce3-workload-socket" pod "3fa97d00-d3af-4c53-bfce-83746cdd6ce3" (UID: "3fa97d00-d3af-4c53-bfce-83746cdd6ce3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:08:39.375113 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:39.375085    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fa97d00-d3af-4c53-bfce-83746cdd6ce3-istiod-ca-cert" pod "3fa97d00-d3af-4c53-bfce-83746cdd6ce3" (UID: "3fa97d00-d3af-4c53-bfce-83746cdd6ce3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:08:39.376751 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:39.376723    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3fa97d00-d3af-4c53-bfce-83746cdd6ce3-istio-podinfo" pod "3fa97d00-d3af-4c53-bfce-83746cdd6ce3" (UID: "3fa97d00-d3af-4c53-bfce-83746cdd6ce3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 20:08:39.377076 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:39.377052    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fa97d00-d3af-4c53-bfce-83746cdd6ce3-istio-envoy" pod "3fa97d00-d3af-4c53-bfce-83746cdd6ce3" (UID: "3fa97d00-d3af-4c53-bfce-83746cdd6ce3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:08:39.377206 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:39.377183    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fa97d00-d3af-4c53-bfce-83746cdd6ce3-istio-token" pod "3fa97d00-d3af-4c53-bfce-83746cdd6ce3" (UID: "3fa97d00-d3af-4c53-bfce-83746cdd6ce3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:08:39.377279 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:39.377211    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fa97d00-d3af-4c53-bfce-83746cdd6ce3-kube-api-access-xx678" pod "3fa97d00-d3af-4c53-bfce-83746cdd6ce3" (UID: "3fa97d00-d3af-4c53-bfce-83746cdd6ce3"). InnerVolumeSpecName "kube-api-access-xx678". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:08:39.476015 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:39.475983    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa97d00-d3af-4c53-bfce-83746cdd6ce3-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:39.476141 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:39.476017    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fa97d00-d3af-4c53-bfce-83746cdd6ce3-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:39.476141 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:39.476033    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xx678\" (UniqueName: \"kubernetes.io/projected/3fa97d00-d3af-4c53-bfce-83746cdd6ce3-kube-api-access-xx678\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:39.476141 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:39.476052    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fa97d00-d3af-4c53-bfce-83746cdd6ce3-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:39.476141 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:39.476066    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fa97d00-d3af-4c53-bfce-83746cdd6ce3-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:39.476141 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:39.476081    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa97d00-d3af-4c53-bfce-83746cdd6ce3-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:39.476141 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:39.476095    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fa97d00-d3af-4c53-bfce-83746cdd6ce3-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:08:40.034868 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:40.034830    4645 generic.go:356] "Generic (PLEG): container finished" podID="3fa97d00-d3af-4c53-bfce-83746cdd6ce3" containerID="8ed58bd86fce039d155cb3a7b6b9821ac00b1b0853fb3df59e5eee872377eae1" exitCode=0
Oct 31 20:08:40.035038 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:40.034886    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt"
Oct 31 20:08:40.035038 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:40.034916    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt" event={"ID":"3fa97d00-d3af-4c53-bfce-83746cdd6ce3","Type":"ContainerDied","Data":"8ed58bd86fce039d155cb3a7b6b9821ac00b1b0853fb3df59e5eee872377eae1"}
Oct 31 20:08:40.035038 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:40.034980    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt" event={"ID":"3fa97d00-d3af-4c53-bfce-83746cdd6ce3","Type":"ContainerDied","Data":"5ab1e8360c6b574dd6a2def2a174ab5b28b5c3bd044801e8d35d2866e6fae286"}
Oct 31 20:08:40.035038 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:40.035003    4645 scope.go:122] "RemoveContainer" containerID="8ed58bd86fce039d155cb3a7b6b9821ac00b1b0853fb3df59e5eee872377eae1"
Oct 31 20:08:40.048664 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:40.048638    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4c5b9b6-wzn7j"]
Oct 31 20:08:40.054863 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:40.054840    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4c5b9b6-wzn7j"]
Oct 31 20:08:40.067972 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:40.067953    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9d3273d-b8v6k"]
Oct 31 20:08:40.071306 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:40.071284    4645 scope.go:122] "RemoveContainer" containerID="fee0e0f90b8dd9575f381dfe8d2748ea2ac697946824aa35184c529981f7d64b"
Oct 31 20:08:40.073087 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:40.073064    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt"]
Oct 31 20:08:40.083954 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:40.083933    4645 scope.go:122] "RemoveContainer" containerID="8ed58bd86fce039d155cb3a7b6b9821ac00b1b0853fb3df59e5eee872377eae1"
Oct 31 20:08:40.084316 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:08:40.084296    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ed58bd86fce039d155cb3a7b6b9821ac00b1b0853fb3df59e5eee872377eae1\": container with ID starting with 8ed58bd86fce039d155cb3a7b6b9821ac00b1b0853fb3df59e5eee872377eae1 not found: ID does not exist" containerID="8ed58bd86fce039d155cb3a7b6b9821ac00b1b0853fb3df59e5eee872377eae1"
Oct 31 20:08:40.084431 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:40.084321    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ed58bd86fce039d155cb3a7b6b9821ac00b1b0853fb3df59e5eee872377eae1"} err="failed to get container status \"8ed58bd86fce039d155cb3a7b6b9821ac00b1b0853fb3df59e5eee872377eae1\": rpc error: code = NotFound desc = could not find container \"8ed58bd86fce039d155cb3a7b6b9821ac00b1b0853fb3df59e5eee872377eae1\": container with ID starting with 8ed58bd86fce039d155cb3a7b6b9821ac00b1b0853fb3df59e5eee872377eae1 not found: ID does not exist"
Oct 31 20:08:40.084431 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:40.084356    4645 scope.go:122] "RemoveContainer" containerID="fee0e0f90b8dd9575f381dfe8d2748ea2ac697946824aa35184c529981f7d64b"
Oct 31 20:08:40.084674 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:08:40.084656    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fee0e0f90b8dd9575f381dfe8d2748ea2ac697946824aa35184c529981f7d64b\": container with ID starting with fee0e0f90b8dd9575f381dfe8d2748ea2ac697946824aa35184c529981f7d64b not found: ID does not exist" containerID="fee0e0f90b8dd9575f381dfe8d2748ea2ac697946824aa35184c529981f7d64b"
Oct 31 20:08:40.084754 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:40.084679    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fee0e0f90b8dd9575f381dfe8d2748ea2ac697946824aa35184c529981f7d64b"} err="failed to get container status \"fee0e0f90b8dd9575f381dfe8d2748ea2ac697946824aa35184c529981f7d64b\": rpc error: code = NotFound desc = could not find container \"fee0e0f90b8dd9575f381dfe8d2748ea2ac697946824aa35184c529981f7d64b\": container with ID starting with fee0e0f90b8dd9575f381dfe8d2748ea2ac697946824aa35184c529981f7d64b not found: ID does not exist"
Oct 31 20:08:40.097736 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:40.095470    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9d3273d-b8v6k"]
Oct 31 20:08:40.097736 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:40.097402    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-23814d-openshift-default-775b6897bc-bgvwt"]
Oct 31 20:08:40.218264 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:40.218223    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:08:40.540856 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:40.540793    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:08:40.721500 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:40.721467    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:08:40.836647 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:40.836564    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:08:41.199645 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:41.199612    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:08:41.616091 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:41.616017    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fa97d00-d3af-4c53-bfce-83746cdd6ce3" path="/var/lib/kubelet/pods/3fa97d00-d3af-4c53-bfce-83746cdd6ce3/volumes"
Oct 31 20:08:41.616512 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:41.616496    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9841bd1b-461e-43fa-b433-793a0fd9fd97" path="/var/lib/kubelet/pods/9841bd1b-461e-43fa-b433-793a0fd9fd97/volumes"
Oct 31 20:08:41.616920 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:41.616905    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcceb7d6-67ba-4407-9609-ffdc6758004e" path="/var/lib/kubelet/pods/fcceb7d6-67ba-4407-9609-ffdc6758004e/volumes"
Oct 31 20:08:43.637096 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:43.637060    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:08:43.765609 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:43.765573    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:08:46.033480 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:46.033431    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25607571-wl7gs"]
Oct 31 20:08:46.036489 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:46.036455    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25607571-wl7gs"]
Oct 31 20:08:46.314776 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:46.314698    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:08:47.617229 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:47.617188    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65db169f-9733-4f67-b773-a012c8c308aa" path="/var/lib/kubelet/pods/65db169f-9733-4f67-b773-a012c8c308aa/volumes"
Oct 31 20:08:47.635757 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:47.635724    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:08:52.320445 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:52.320405    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:08:57.034424 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:57.034388    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8487c18-s65fh"]
Oct 31 20:08:57.040598 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:57.040571    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8487c18-s65fh"]
Oct 31 20:08:57.615771 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:08:57.615737    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e6f5b6b-6a09-4b98-82bd-cd6ad2899c0a" path="/var/lib/kubelet/pods/9e6f5b6b-6a09-4b98-82bd-cd6ad2899c0a/volumes"
Oct 31 20:09:06.243687 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:09:06.243651    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 20:09:08.541942 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:08.541905    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:09:09.304774 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.304739    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vkszn"]
Oct 31 20:09:09.305173 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.305156    4645 cpu_manager.go:414] "containerMap: removing container" podUID="3fa97d00-d3af-4c53-bfce-83746cdd6ce3" containerName="istio-proxy"
Oct 31 20:09:09.305246 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.305174    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fa97d00-d3af-4c53-bfce-83746cdd6ce3" containerName="istio-proxy"
Oct 31 20:09:09.305246 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.305189    4645 cpu_manager.go:414] "containerMap: removing container" podUID="3fa97d00-d3af-4c53-bfce-83746cdd6ce3" containerName="istio-proxy"
Oct 31 20:09:09.305246 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.305194    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fa97d00-d3af-4c53-bfce-83746cdd6ce3" containerName="istio-proxy"
Oct 31 20:09:09.305246 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.305202    4645 cpu_manager.go:414] "containerMap: removing container" podUID="a28d9602-371e-4d21-b4f4-0ba0966dd9f0" containerName="istio-proxy"
Oct 31 20:09:09.305246 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.305208    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a28d9602-371e-4d21-b4f4-0ba0966dd9f0" containerName="istio-proxy"
Oct 31 20:09:09.305246 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.305215    4645 cpu_manager.go:414] "containerMap: removing container" podUID="12bea86c-61e5-465e-82fb-b4673a4d1be9" containerName="main"
Oct 31 20:09:09.305246 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.305219    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12bea86c-61e5-465e-82fb-b4673a4d1be9" containerName="main"
Oct 31 20:09:09.305246 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.305238    4645 cpu_manager.go:414] "containerMap: removing container" podUID="f82be1ed-85af-4e4e-8381-97c6f58648c3" containerName="revoke-keys"
Oct 31 20:09:09.305246 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.305243    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f82be1ed-85af-4e4e-8381-97c6f58648c3" containerName="revoke-keys"
Oct 31 20:09:09.305583 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.305294    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fa97d00-d3af-4c53-bfce-83746cdd6ce3" containerName="istio-proxy"
Oct 31 20:09:09.305583 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.305304    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="a28d9602-371e-4d21-b4f4-0ba0966dd9f0" containerName="istio-proxy"
Oct 31 20:09:09.305583 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.305309    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="12bea86c-61e5-465e-82fb-b4673a4d1be9" containerName="main"
Oct 31 20:09:09.305583 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.305318    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="f82be1ed-85af-4e4e-8381-97c6f58648c3" containerName="revoke-keys"
Oct 31 20:09:09.308414 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.308387    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vkszn"
Oct 31 20:09:09.319205 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.319179    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-55znv"]
Oct 31 20:09:09.319557 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.319544    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fa97d00-d3af-4c53-bfce-83746cdd6ce3" containerName="istio-proxy"
Oct 31 20:09:09.322902 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.322876    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vkszn"]
Oct 31 20:09:09.323019 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.322981    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-55znv"
Oct 31 20:09:09.344424 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.344401    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-55znv"]
Oct 31 20:09:09.423967 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.423936    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0150a41c-8a80-46fb-b668-fcc8a4a48dcd-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-vkszn\" (UID: \"0150a41c-8a80-46fb-b668-fcc8a4a48dcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vkszn"
Oct 31 20:09:09.423967 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.423970    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86c63841-b547-43a5-9d82-e1cde5c8f2ed-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-55znv\" (UID: \"86c63841-b547-43a5-9d82-e1cde5c8f2ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-55znv"
Oct 31 20:09:09.424215 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.423993    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9bc2\" (UniqueName: \"kubernetes.io/projected/86c63841-b547-43a5-9d82-e1cde5c8f2ed-kube-api-access-r9bc2\") pod \"maas-default-gateway-openshift-default-75d7c785f-55znv\" (UID: \"86c63841-b547-43a5-9d82-e1cde5c8f2ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-55znv"
Oct 31 20:09:09.424215 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.424062    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0150a41c-8a80-46fb-b668-fcc8a4a48dcd-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-vkszn\" (UID: \"0150a41c-8a80-46fb-b668-fcc8a4a48dcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vkszn"
Oct 31 20:09:09.424215 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.424126    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0150a41c-8a80-46fb-b668-fcc8a4a48dcd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-vkszn\" (UID: \"0150a41c-8a80-46fb-b668-fcc8a4a48dcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vkszn"
Oct 31 20:09:09.424215 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.424174    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86c63841-b547-43a5-9d82-e1cde5c8f2ed-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-55znv\" (UID: \"86c63841-b547-43a5-9d82-e1cde5c8f2ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-55znv"
Oct 31 20:09:09.424449 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.424263    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0150a41c-8a80-46fb-b668-fcc8a4a48dcd-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-vkszn\" (UID: \"0150a41c-8a80-46fb-b668-fcc8a4a48dcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vkszn"
Oct 31 20:09:09.424449 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.424306    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86c63841-b547-43a5-9d82-e1cde5c8f2ed-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-55znv\" (UID: \"86c63841-b547-43a5-9d82-e1cde5c8f2ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-55znv"
Oct 31 20:09:09.424449 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.424410    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0150a41c-8a80-46fb-b668-fcc8a4a48dcd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-vkszn\" (UID: \"0150a41c-8a80-46fb-b668-fcc8a4a48dcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vkszn"
Oct 31 20:09:09.424628 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.424476    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0150a41c-8a80-46fb-b668-fcc8a4a48dcd-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-vkszn\" (UID: \"0150a41c-8a80-46fb-b668-fcc8a4a48dcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vkszn"
Oct 31 20:09:09.424628 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.424511    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvlrz\" (UniqueName: \"kubernetes.io/projected/0150a41c-8a80-46fb-b668-fcc8a4a48dcd-kube-api-access-qvlrz\") pod \"maas-default-gateway-openshift-default-75d7c785f-vkszn\" (UID: \"0150a41c-8a80-46fb-b668-fcc8a4a48dcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vkszn"
Oct 31 20:09:09.424628 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.424537    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86c63841-b547-43a5-9d82-e1cde5c8f2ed-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-55znv\" (UID: \"86c63841-b547-43a5-9d82-e1cde5c8f2ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-55znv"
Oct 31 20:09:09.424838 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.424658    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86c63841-b547-43a5-9d82-e1cde5c8f2ed-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-55znv\" (UID: \"86c63841-b547-43a5-9d82-e1cde5c8f2ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-55znv"
Oct 31 20:09:09.424838 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.424692    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86c63841-b547-43a5-9d82-e1cde5c8f2ed-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-55znv\" (UID: \"86c63841-b547-43a5-9d82-e1cde5c8f2ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-55znv"
Oct 31 20:09:09.424838 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.424724    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0150a41c-8a80-46fb-b668-fcc8a4a48dcd-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-vkszn\" (UID: \"0150a41c-8a80-46fb-b668-fcc8a4a48dcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vkszn"
Oct 31 20:09:09.424838 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.424788    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0150a41c-8a80-46fb-b668-fcc8a4a48dcd-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-vkszn\" (UID: \"0150a41c-8a80-46fb-b668-fcc8a4a48dcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vkszn"
Oct 31 20:09:09.424838 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.424831    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86c63841-b547-43a5-9d82-e1cde5c8f2ed-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-55znv\" (UID: \"86c63841-b547-43a5-9d82-e1cde5c8f2ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-55znv"
Oct 31 20:09:09.425032 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.424850    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86c63841-b547-43a5-9d82-e1cde5c8f2ed-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-55znv\" (UID: \"86c63841-b547-43a5-9d82-e1cde5c8f2ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-55znv"
Oct 31 20:09:09.525477 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.525447    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86c63841-b547-43a5-9d82-e1cde5c8f2ed-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-55znv\" (UID: \"86c63841-b547-43a5-9d82-e1cde5c8f2ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-55znv"
Oct 31 20:09:09.525612 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.525489    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86c63841-b547-43a5-9d82-e1cde5c8f2ed-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-55znv\" (UID: \"86c63841-b547-43a5-9d82-e1cde5c8f2ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-55znv"
Oct 31 20:09:09.525612 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.525515    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0150a41c-8a80-46fb-b668-fcc8a4a48dcd-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-vkszn\" (UID: \"0150a41c-8a80-46fb-b668-fcc8a4a48dcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vkszn"
Oct 31 20:09:09.525612 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.525549    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0150a41c-8a80-46fb-b668-fcc8a4a48dcd-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-vkszn\" (UID: \"0150a41c-8a80-46fb-b668-fcc8a4a48dcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vkszn"
Oct 31 20:09:09.525612 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.525591    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86c63841-b547-43a5-9d82-e1cde5c8f2ed-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-55znv\" (UID: \"86c63841-b547-43a5-9d82-e1cde5c8f2ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-55znv"
Oct 31 20:09:09.525854 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.525614    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86c63841-b547-43a5-9d82-e1cde5c8f2ed-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-55znv\" (UID: \"86c63841-b547-43a5-9d82-e1cde5c8f2ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-55znv"
Oct 31 20:09:09.525854 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.525661    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0150a41c-8a80-46fb-b668-fcc8a4a48dcd-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-vkszn\" (UID: \"0150a41c-8a80-46fb-b668-fcc8a4a48dcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vkszn"
Oct 31 20:09:09.525970 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.525888    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86c63841-b547-43a5-9d82-e1cde5c8f2ed-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-55znv\" (UID: \"86c63841-b547-43a5-9d82-e1cde5c8f2ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-55znv"
Oct 31 20:09:09.525970 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.525927    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9bc2\" (UniqueName: \"kubernetes.io/projected/86c63841-b547-43a5-9d82-e1cde5c8f2ed-kube-api-access-r9bc2\") pod \"maas-default-gateway-openshift-default-75d7c785f-55znv\" (UID: \"86c63841-b547-43a5-9d82-e1cde5c8f2ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-55znv"
Oct 31 20:09:09.525970 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.525952    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0150a41c-8a80-46fb-b668-fcc8a4a48dcd-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-vkszn\" (UID: \"0150a41c-8a80-46fb-b668-fcc8a4a48dcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vkszn"
Oct 31 20:09:09.526199 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.526165    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86c63841-b547-43a5-9d82-e1cde5c8f2ed-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-55znv\" (UID: \"86c63841-b547-43a5-9d82-e1cde5c8f2ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-55znv"
Oct 31 20:09:09.526266 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.525958    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0150a41c-8a80-46fb-b668-fcc8a4a48dcd-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-vkszn\" (UID: \"0150a41c-8a80-46fb-b668-fcc8a4a48dcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vkszn"
Oct 31 20:09:09.526266 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.526218    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0150a41c-8a80-46fb-b668-fcc8a4a48dcd-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-vkszn\" (UID: \"0150a41c-8a80-46fb-b668-fcc8a4a48dcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vkszn"
Oct 31 20:09:09.526266 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.526220    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86c63841-b547-43a5-9d82-e1cde5c8f2ed-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-55znv\" (UID: \"86c63841-b547-43a5-9d82-e1cde5c8f2ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-55znv"
Oct 31 20:09:09.526266 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.526229    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0150a41c-8a80-46fb-b668-fcc8a4a48dcd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-vkszn\" (UID: \"0150a41c-8a80-46fb-b668-fcc8a4a48dcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vkszn"
Oct 31 20:09:09.526486 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.526282    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86c63841-b547-43a5-9d82-e1cde5c8f2ed-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-55znv\" (UID: \"86c63841-b547-43a5-9d82-e1cde5c8f2ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-55znv"
Oct 31 20:09:09.526486 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.526320    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0150a41c-8a80-46fb-b668-fcc8a4a48dcd-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-vkszn\" (UID: \"0150a41c-8a80-46fb-b668-fcc8a4a48dcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vkszn"
Oct 31 20:09:09.526486 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.526323    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86c63841-b547-43a5-9d82-e1cde5c8f2ed-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-55znv\" (UID: \"86c63841-b547-43a5-9d82-e1cde5c8f2ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-55znv"
Oct 31 20:09:09.526486 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.526398    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86c63841-b547-43a5-9d82-e1cde5c8f2ed-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-55znv\" (UID: \"86c63841-b547-43a5-9d82-e1cde5c8f2ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-55znv"
Oct 31 20:09:09.526486 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.526438    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0150a41c-8a80-46fb-b668-fcc8a4a48dcd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-vkszn\" (UID: \"0150a41c-8a80-46fb-b668-fcc8a4a48dcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vkszn"
Oct 31 20:09:09.526486 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.526472    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0150a41c-8a80-46fb-b668-fcc8a4a48dcd-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-vkszn\" (UID: \"0150a41c-8a80-46fb-b668-fcc8a4a48dcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vkszn"
Oct 31 20:09:09.526818 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.526596    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvlrz\" (UniqueName: \"kubernetes.io/projected/0150a41c-8a80-46fb-b668-fcc8a4a48dcd-kube-api-access-qvlrz\") pod \"maas-default-gateway-openshift-default-75d7c785f-vkszn\" (UID: \"0150a41c-8a80-46fb-b668-fcc8a4a48dcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vkszn"
Oct 31 20:09:09.526818 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.526613    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0150a41c-8a80-46fb-b668-fcc8a4a48dcd-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-vkszn\" (UID: \"0150a41c-8a80-46fb-b668-fcc8a4a48dcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vkszn"
Oct 31 20:09:09.526818 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.526633    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86c63841-b547-43a5-9d82-e1cde5c8f2ed-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-55znv\" (UID: \"86c63841-b547-43a5-9d82-e1cde5c8f2ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-55znv"
Oct 31 20:09:09.527000 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.526930    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86c63841-b547-43a5-9d82-e1cde5c8f2ed-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-55znv\" (UID: \"86c63841-b547-43a5-9d82-e1cde5c8f2ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-55znv"
Oct 31 20:09:09.527065 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.526995    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0150a41c-8a80-46fb-b668-fcc8a4a48dcd-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-vkszn\" (UID: \"0150a41c-8a80-46fb-b668-fcc8a4a48dcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vkszn"
Oct 31 20:09:09.527163 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.527121    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0150a41c-8a80-46fb-b668-fcc8a4a48dcd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-vkszn\" (UID: \"0150a41c-8a80-46fb-b668-fcc8a4a48dcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vkszn"
Oct 31 20:09:09.527246 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.527221    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86c63841-b547-43a5-9d82-e1cde5c8f2ed-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-55znv\" (UID: \"86c63841-b547-43a5-9d82-e1cde5c8f2ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-55znv"
Oct 31 20:09:09.528558 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.528534    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86c63841-b547-43a5-9d82-e1cde5c8f2ed-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-55znv\" (UID: \"86c63841-b547-43a5-9d82-e1cde5c8f2ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-55znv"
Oct 31 20:09:09.528751 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.528734    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86c63841-b547-43a5-9d82-e1cde5c8f2ed-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-55znv\" (UID: \"86c63841-b547-43a5-9d82-e1cde5c8f2ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-55znv"
Oct 31 20:09:09.528977 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.528958    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0150a41c-8a80-46fb-b668-fcc8a4a48dcd-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-vkszn\" (UID: \"0150a41c-8a80-46fb-b668-fcc8a4a48dcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vkszn"
Oct 31 20:09:09.529187 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.529168    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0150a41c-8a80-46fb-b668-fcc8a4a48dcd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-vkszn\" (UID: \"0150a41c-8a80-46fb-b668-fcc8a4a48dcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vkszn"
Oct 31 20:09:09.542642 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.542620    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86c63841-b547-43a5-9d82-e1cde5c8f2ed-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-55znv\" (UID: \"86c63841-b547-43a5-9d82-e1cde5c8f2ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-55znv"
Oct 31 20:09:09.542948 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.542677    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0150a41c-8a80-46fb-b668-fcc8a4a48dcd-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-vkszn\" (UID: \"0150a41c-8a80-46fb-b668-fcc8a4a48dcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vkszn"
Oct 31 20:09:09.542948 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.542876    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvlrz\" (UniqueName: \"kubernetes.io/projected/0150a41c-8a80-46fb-b668-fcc8a4a48dcd-kube-api-access-qvlrz\") pod \"maas-default-gateway-openshift-default-75d7c785f-vkszn\" (UID: \"0150a41c-8a80-46fb-b668-fcc8a4a48dcd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vkszn"
Oct 31 20:09:09.543231 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.543214    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9bc2\" (UniqueName: \"kubernetes.io/projected/86c63841-b547-43a5-9d82-e1cde5c8f2ed-kube-api-access-r9bc2\") pod \"maas-default-gateway-openshift-default-75d7c785f-55znv\" (UID: \"86c63841-b547-43a5-9d82-e1cde5c8f2ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-55znv"
Oct 31 20:09:09.621196 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.621146    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vkszn"
Oct 31 20:09:09.637749 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.637729    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-55znv"
Oct 31 20:09:09.900392 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.900362    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vkszn"]
Oct 31 20:09:09.907689 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.907658    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:09:09.907784 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.907723    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:09:09.907784 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.907752    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:09:09.931045 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.931022    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-55znv"]
Oct 31 20:09:09.936453 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:09:09.936420    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86c63841_b547_43a5_9d82_e1cde5c8f2ed.slice/crio-2929889aca117ae1069c65e128833e9205ff43d055dd60ec479d43021869657b WatchSource:0}: Error finding container 2929889aca117ae1069c65e128833e9205ff43d055dd60ec479d43021869657b: Status 404 returned error can't find the container with id 2929889aca117ae1069c65e128833e9205ff43d055dd60ec479d43021869657b
Oct 31 20:09:09.938670 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.938638    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:09:09.938777 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.938699    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:09:09.938777 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:09.938730    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:09:10.217008 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:10.216966    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-55znv" event={"ID":"86c63841-b547-43a5-9d82-e1cde5c8f2ed","Type":"ContainerStarted","Data":"3a3359753776e5adcea55c368b8f64d1c2d4385862dec56548b86a44737b5446"}
Oct 31 20:09:10.217216 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:10.217022    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-55znv" event={"ID":"86c63841-b547-43a5-9d82-e1cde5c8f2ed","Type":"ContainerStarted","Data":"2929889aca117ae1069c65e128833e9205ff43d055dd60ec479d43021869657b"}
Oct 31 20:09:10.219073 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:10.219045    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vkszn" event={"ID":"0150a41c-8a80-46fb-b668-fcc8a4a48dcd","Type":"ContainerStarted","Data":"8de325a50771f5b9428b038cbdc86d5540d4a51df7f04714151f48dafaca92a8"}
Oct 31 20:09:10.219201 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:10.219092    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vkszn" event={"ID":"0150a41c-8a80-46fb-b668-fcc8a4a48dcd","Type":"ContainerStarted","Data":"2639340d1a92ab82431974157ac2f957a8661c0dd7c4a9fcc17443412d7b4b32"}
Oct 31 20:09:10.242766 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:10.242446    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-55znv" podStartSLOduration=1.242428857 podStartE2EDuration="1.242428857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:09:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:09:10.2384514 +0000 UTC m=+2980.811685543" watchObservedRunningTime="2026-10-31 20:09:10.242428857 +0000 UTC m=+2980.815662980"
Oct 31 20:09:10.261155 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:10.261101    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vkszn" podStartSLOduration=1.261083373 podStartE2EDuration="1.261083373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:09:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:09:10.260424358 +0000 UTC m=+2980.833658490" watchObservedRunningTime="2026-10-31 20:09:10.261083373 +0000 UTC m=+2980.834317505"
Oct 31 20:09:10.621847 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:10.621757    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vkszn"
Oct 31 20:09:10.623405 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:10.623375    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-vkszn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.64:15021/healthz/ready\": dial tcp 10.133.1.64:15021: connect: connection refused" start-of-body=
Oct 31 20:09:10.623566 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:10.623422    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vkszn" podUID="0150a41c-8a80-46fb-b668-fcc8a4a48dcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.64:15021/healthz/ready\": dial tcp 10.133.1.64:15021: connect: connection refused"
Oct 31 20:09:10.637949 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:10.637929    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-55znv"
Oct 31 20:09:10.639000 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:10.638977    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-55znv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused" start-of-body=
Oct 31 20:09:10.639105 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:10.639026    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-55znv" podUID="86c63841-b547-43a5-9d82-e1cde5c8f2ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused"
Oct 31 20:09:10.737025 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:10.736991    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:09:11.621972 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:11.621931    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-vkszn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.64:15021/healthz/ready\": dial tcp 10.133.1.64:15021: connect: connection refused" start-of-body=
Oct 31 20:09:11.622515 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:11.621991    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vkszn" podUID="0150a41c-8a80-46fb-b668-fcc8a4a48dcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.64:15021/healthz/ready\": dial tcp 10.133.1.64:15021: connect: connection refused"
Oct 31 20:09:11.638646 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:11.638622    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-55znv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused" start-of-body=
Oct 31 20:09:11.638712 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:11.638660    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-55znv" podUID="86c63841-b547-43a5-9d82-e1cde5c8f2ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused"
Oct 31 20:09:12.622043 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:12.622002    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-vkszn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.64:15021/healthz/ready\": dial tcp 10.133.1.64:15021: connect: connection refused" start-of-body=
Oct 31 20:09:12.622470 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:12.622074    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vkszn" podUID="0150a41c-8a80-46fb-b668-fcc8a4a48dcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.64:15021/healthz/ready\": dial tcp 10.133.1.64:15021: connect: connection refused"
Oct 31 20:09:12.638996 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:12.638971    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-55znv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused" start-of-body=
Oct 31 20:09:12.639055 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:12.639026    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-55znv" podUID="86c63841-b547-43a5-9d82-e1cde5c8f2ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused"
Oct 31 20:09:13.622284 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:13.622244    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-vkszn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.64:15021/healthz/ready\": dial tcp 10.133.1.64:15021: connect: connection refused" start-of-body=
Oct 31 20:09:13.622726 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:13.622310    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vkszn" podUID="0150a41c-8a80-46fb-b668-fcc8a4a48dcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.64:15021/healthz/ready\": dial tcp 10.133.1.64:15021: connect: connection refused"
Oct 31 20:09:13.638464 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:13.638437    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-55znv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused" start-of-body=
Oct 31 20:09:13.638538 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:13.638479    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-55znv" podUID="86c63841-b547-43a5-9d82-e1cde5c8f2ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused"
Oct 31 20:09:14.621632 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:14.621592    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-vkszn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.64:15021/healthz/ready\": dial tcp 10.133.1.64:15021: connect: connection refused" start-of-body=
Oct 31 20:09:14.621819 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:14.621647    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vkszn" podUID="0150a41c-8a80-46fb-b668-fcc8a4a48dcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.64:15021/healthz/ready\": dial tcp 10.133.1.64:15021: connect: connection refused"
Oct 31 20:09:14.638579 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:14.638545    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-55znv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused" start-of-body=
Oct 31 20:09:14.638977 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:14.638588    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-55znv" podUID="86c63841-b547-43a5-9d82-e1cde5c8f2ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused"
Oct 31 20:09:15.622408 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:15.622371    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-vkszn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.64:15021/healthz/ready\": dial tcp 10.133.1.64:15021: connect: connection refused" start-of-body=
Oct 31 20:09:15.622607 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:15.622423    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vkszn" podUID="0150a41c-8a80-46fb-b668-fcc8a4a48dcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.64:15021/healthz/ready\": dial tcp 10.133.1.64:15021: connect: connection refused"
Oct 31 20:09:15.638788 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:15.638760    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-55znv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused" start-of-body=
Oct 31 20:09:15.639116 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:15.638797    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-55znv" podUID="86c63841-b547-43a5-9d82-e1cde5c8f2ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused"
Oct 31 20:09:16.622014 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:16.621970    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-vkszn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.64:15021/healthz/ready\": dial tcp 10.133.1.64:15021: connect: connection refused" start-of-body=
Oct 31 20:09:16.622204 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:16.622039    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vkszn" podUID="0150a41c-8a80-46fb-b668-fcc8a4a48dcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.64:15021/healthz/ready\": dial tcp 10.133.1.64:15021: connect: connection refused"
Oct 31 20:09:16.639023 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:16.638994    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-55znv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused" start-of-body=
Oct 31 20:09:16.639392 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:16.639038    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-55znv" podUID="86c63841-b547-43a5-9d82-e1cde5c8f2ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.65:15021/healthz/ready\": dial tcp 10.133.1.65:15021: connect: connection refused"
Oct 31 20:09:18.135490 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:18.135454    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vkszn"
Oct 31 20:09:18.136315 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:18.136281    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vkszn"
Oct 31 20:09:18.137586 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:18.137567    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vkszn"
Oct 31 20:09:18.317026 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:18.316996    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-55znv"
Oct 31 20:09:18.317383 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:18.317337    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-55znv"
Oct 31 20:09:18.318170 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:18.318148    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-55znv"
Oct 31 20:09:18.494831 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:18.494796    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:09:19.559014 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:19.558976    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:09:20.594834 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:20.594801    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:09:24.341621 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:24.341585    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vj5qn"]
Oct 31 20:09:24.348138 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:24.348090    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vj5qn"
Oct 31 20:09:24.360610 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:24.360580    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vj5qn"]
Oct 31 20:09:24.394835 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:24.394806    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bdf127f-3310-4bc8-8f53-6a1e7dda4191-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-vj5qn\" (UID: \"8bdf127f-3310-4bc8-8f53-6a1e7dda4191\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vj5qn"
Oct 31 20:09:24.394982 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:24.394840    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bdf127f-3310-4bc8-8f53-6a1e7dda4191-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-vj5qn\" (UID: \"8bdf127f-3310-4bc8-8f53-6a1e7dda4191\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vj5qn"
Oct 31 20:09:24.394982 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:24.394863    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bdf127f-3310-4bc8-8f53-6a1e7dda4191-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-vj5qn\" (UID: \"8bdf127f-3310-4bc8-8f53-6a1e7dda4191\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vj5qn"
Oct 31 20:09:24.395113 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:24.395036    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bdf127f-3310-4bc8-8f53-6a1e7dda4191-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-vj5qn\" (UID: \"8bdf127f-3310-4bc8-8f53-6a1e7dda4191\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vj5qn"
Oct 31 20:09:24.395113 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:24.395078    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bdf127f-3310-4bc8-8f53-6a1e7dda4191-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-vj5qn\" (UID: \"8bdf127f-3310-4bc8-8f53-6a1e7dda4191\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vj5qn"
Oct 31 20:09:24.395191 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:24.395145    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktkz8\" (UniqueName: \"kubernetes.io/projected/8bdf127f-3310-4bc8-8f53-6a1e7dda4191-kube-api-access-ktkz8\") pod \"maas-default-gateway-openshift-default-75d7c785f-vj5qn\" (UID: \"8bdf127f-3310-4bc8-8f53-6a1e7dda4191\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vj5qn"
Oct 31 20:09:24.395191 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:24.395176    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bdf127f-3310-4bc8-8f53-6a1e7dda4191-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-vj5qn\" (UID: \"8bdf127f-3310-4bc8-8f53-6a1e7dda4191\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vj5qn"
Oct 31 20:09:24.395265 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:24.395195    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bdf127f-3310-4bc8-8f53-6a1e7dda4191-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-vj5qn\" (UID: \"8bdf127f-3310-4bc8-8f53-6a1e7dda4191\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vj5qn"
Oct 31 20:09:24.395265 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:24.395216    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bdf127f-3310-4bc8-8f53-6a1e7dda4191-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-vj5qn\" (UID: \"8bdf127f-3310-4bc8-8f53-6a1e7dda4191\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vj5qn"
Oct 31 20:09:24.496272 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:24.496248    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bdf127f-3310-4bc8-8f53-6a1e7dda4191-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-vj5qn\" (UID: \"8bdf127f-3310-4bc8-8f53-6a1e7dda4191\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vj5qn"
Oct 31 20:09:24.496408 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:24.496278    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bdf127f-3310-4bc8-8f53-6a1e7dda4191-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-vj5qn\" (UID: \"8bdf127f-3310-4bc8-8f53-6a1e7dda4191\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vj5qn"
Oct 31 20:09:24.496408 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:24.496317    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bdf127f-3310-4bc8-8f53-6a1e7dda4191-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-vj5qn\" (UID: \"8bdf127f-3310-4bc8-8f53-6a1e7dda4191\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vj5qn"
Oct 31 20:09:24.496408 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:24.496370    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bdf127f-3310-4bc8-8f53-6a1e7dda4191-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-vj5qn\" (UID: \"8bdf127f-3310-4bc8-8f53-6a1e7dda4191\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vj5qn"
Oct 31 20:09:24.496408 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:24.496405    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bdf127f-3310-4bc8-8f53-6a1e7dda4191-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-vj5qn\" (UID: \"8bdf127f-3310-4bc8-8f53-6a1e7dda4191\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vj5qn"
Oct 31 20:09:24.496606 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:24.496470    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bdf127f-3310-4bc8-8f53-6a1e7dda4191-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-vj5qn\" (UID: \"8bdf127f-3310-4bc8-8f53-6a1e7dda4191\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vj5qn"
Oct 31 20:09:24.496606 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:24.496495    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bdf127f-3310-4bc8-8f53-6a1e7dda4191-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-vj5qn\" (UID: \"8bdf127f-3310-4bc8-8f53-6a1e7dda4191\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vj5qn"
Oct 31 20:09:24.496719 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:24.496630    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktkz8\" (UniqueName: \"kubernetes.io/projected/8bdf127f-3310-4bc8-8f53-6a1e7dda4191-kube-api-access-ktkz8\") pod \"maas-default-gateway-openshift-default-75d7c785f-vj5qn\" (UID: \"8bdf127f-3310-4bc8-8f53-6a1e7dda4191\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vj5qn"
Oct 31 20:09:24.496719 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:24.496685    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bdf127f-3310-4bc8-8f53-6a1e7dda4191-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-vj5qn\" (UID: \"8bdf127f-3310-4bc8-8f53-6a1e7dda4191\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vj5qn"
Oct 31 20:09:24.496719 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:24.496692    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bdf127f-3310-4bc8-8f53-6a1e7dda4191-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-vj5qn\" (UID: \"8bdf127f-3310-4bc8-8f53-6a1e7dda4191\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vj5qn"
Oct 31 20:09:24.496949 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:24.496734    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bdf127f-3310-4bc8-8f53-6a1e7dda4191-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-vj5qn\" (UID: \"8bdf127f-3310-4bc8-8f53-6a1e7dda4191\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vj5qn"
Oct 31 20:09:24.496949 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:24.496792    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bdf127f-3310-4bc8-8f53-6a1e7dda4191-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-vj5qn\" (UID: \"8bdf127f-3310-4bc8-8f53-6a1e7dda4191\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vj5qn"
Oct 31 20:09:24.497154 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:24.497133    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bdf127f-3310-4bc8-8f53-6a1e7dda4191-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-vj5qn\" (UID: \"8bdf127f-3310-4bc8-8f53-6a1e7dda4191\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vj5qn"
Oct 31 20:09:24.497257 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:24.497235    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bdf127f-3310-4bc8-8f53-6a1e7dda4191-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-vj5qn\" (UID: \"8bdf127f-3310-4bc8-8f53-6a1e7dda4191\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vj5qn"
Oct 31 20:09:24.498644 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:24.498625    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bdf127f-3310-4bc8-8f53-6a1e7dda4191-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-vj5qn\" (UID: \"8bdf127f-3310-4bc8-8f53-6a1e7dda4191\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vj5qn"
Oct 31 20:09:24.499107 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:24.499085    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bdf127f-3310-4bc8-8f53-6a1e7dda4191-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-vj5qn\" (UID: \"8bdf127f-3310-4bc8-8f53-6a1e7dda4191\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vj5qn"
Oct 31 20:09:24.509154 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:24.509126    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bdf127f-3310-4bc8-8f53-6a1e7dda4191-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-vj5qn\" (UID: \"8bdf127f-3310-4bc8-8f53-6a1e7dda4191\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vj5qn"
Oct 31 20:09:24.509367 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:24.509349    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktkz8\" (UniqueName: \"kubernetes.io/projected/8bdf127f-3310-4bc8-8f53-6a1e7dda4191-kube-api-access-ktkz8\") pod \"maas-default-gateway-openshift-default-75d7c785f-vj5qn\" (UID: \"8bdf127f-3310-4bc8-8f53-6a1e7dda4191\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vj5qn"
Oct 31 20:09:24.662770 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:24.662748    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vj5qn"
Oct 31 20:09:24.947457 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:24.947426    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vj5qn"]
Oct 31 20:09:24.949001 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:09:24.948974    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bdf127f_3310_4bc8_8f53_6a1e7dda4191.slice/crio-bb58c80cb460b1a24b2c7d1ee776b379efe387068e2b3ac3318ea2918fd70bde WatchSource:0}: Error finding container bb58c80cb460b1a24b2c7d1ee776b379efe387068e2b3ac3318ea2918fd70bde: Status 404 returned error can't find the container with id bb58c80cb460b1a24b2c7d1ee776b379efe387068e2b3ac3318ea2918fd70bde
Oct 31 20:09:24.951195 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:24.951160    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:09:24.951304 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:24.951241    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:09:24.951304 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:24.951290    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:09:25.310353 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:25.310238    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vj5qn" event={"ID":"8bdf127f-3310-4bc8-8f53-6a1e7dda4191","Type":"ContainerStarted","Data":"8c3d4ba6e726f8f9462282daa0b9b123c97dc17a7db9a07361f56f2a5cd4f099"}
Oct 31 20:09:25.310353 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:25.310299    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vj5qn" event={"ID":"8bdf127f-3310-4bc8-8f53-6a1e7dda4191","Type":"ContainerStarted","Data":"bb58c80cb460b1a24b2c7d1ee776b379efe387068e2b3ac3318ea2918fd70bde"}
Oct 31 20:09:25.663808 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:25.663778    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vj5qn"
Oct 31 20:09:26.664793 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:26.664756    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-vj5qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused" start-of-body=
Oct 31 20:09:26.665400 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:26.665364    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vj5qn" podUID="8bdf127f-3310-4bc8-8f53-6a1e7dda4191" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused"
Oct 31 20:09:27.680354 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:27.679486    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-vj5qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused" start-of-body=
Oct 31 20:09:27.680354 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:27.679547    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vj5qn" podUID="8bdf127f-3310-4bc8-8f53-6a1e7dda4191" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused"
Oct 31 20:09:28.680353 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:28.679707    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-vj5qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused" start-of-body=
Oct 31 20:09:28.680353 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:28.679769    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vj5qn" podUID="8bdf127f-3310-4bc8-8f53-6a1e7dda4191" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused"
Oct 31 20:09:29.674829 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:29.674412    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-vj5qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused" start-of-body=
Oct 31 20:09:29.674829 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:29.674468    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vj5qn" podUID="8bdf127f-3310-4bc8-8f53-6a1e7dda4191" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused"
Oct 31 20:09:30.663957 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:30.663914    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-vj5qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused" start-of-body=
Oct 31 20:09:30.664407 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:30.663983    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vj5qn" podUID="8bdf127f-3310-4bc8-8f53-6a1e7dda4191" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused"
Oct 31 20:09:31.663007 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:31.662974    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-vj5qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused" start-of-body=
Oct 31 20:09:31.663191 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:31.663021    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vj5qn" podUID="8bdf127f-3310-4bc8-8f53-6a1e7dda4191" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused"
Oct 31 20:09:32.406516 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:32.406483    4645 scope.go:122] "RemoveContainer" containerID="e17cc4c3aeaab50d184bf6447361b1b299e4a7328d3205a677481d75c66ae785"
Oct 31 20:09:32.415825 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:32.415798    4645 scope.go:122] "RemoveContainer" containerID="3e10873a1057d46760e2c37f44d072523f0fcb8915b081c8d097f1d6a39376c2"
Oct 31 20:09:32.425382 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:32.425358    4645 scope.go:122] "RemoveContainer" containerID="e65baaa828d073b91a40188cbb32af573144a0318d023169cc6c6b5abc932c22"
Oct 31 20:09:32.433188 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:32.433167    4645 scope.go:122] "RemoveContainer" containerID="1440d2ae4202389fc3b0b6b17f75b39c86eaf3a2d847875e096b3ce8d02ff764"
Oct 31 20:09:32.891927 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:32.891895    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vj5qn"
Oct 31 20:09:32.892342 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:32.892300    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vj5qn"
Oct 31 20:09:32.893080 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:32.893055    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vj5qn"
Oct 31 20:09:32.923215 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:09:32.923173    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vj5qn" podStartSLOduration=8.923161002 podStartE2EDuration="8.923161002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:09:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:09:25.338668708 +0000 UTC m=+2995.911902857" watchObservedRunningTime="2026-10-31 20:09:32.923161002 +0000 UTC m=+3003.496395130"
Oct 31 20:09:36.244797 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:09:36.244751    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 20:10:03.944550 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:10:03.944457    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-vj5qn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 20:10:03.945433 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:10:03.945394    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vj5qn" podUID="8bdf127f-3310-4bc8-8f53-6a1e7dda4191" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 20:10:04.188479 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:10:04.188436    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-vkszn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 20:10:04.188808 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:10:04.188768    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vkszn" podUID="0150a41c-8a80-46fb-b668-fcc8a4a48dcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 20:10:06.202202 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:10:06.202152    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:10:06.321783 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:10:06.321730    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 20:10:17.774677 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:10:17.774643    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:10:17.878241 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:10:17.878213    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:10:18.565340 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:10:18.565297    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:10:19.631304 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:10:19.631264    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:10:20.637445 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:10:20.637406    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:10:32.482112 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:10:32.482078    4645 scope.go:122] "RemoveContainer" containerID="22256a7bcfc34444d4ebc822f63eed8c817aa87fd15bcb69ba53a472f960e054"
Oct 31 20:10:32.491241 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:10:32.491219    4645 scope.go:122] "RemoveContainer" containerID="b74947ef0513babd3d0cf7ff1ebfa0aeb5600e4869db5ed09e00075b04e5caf8"
Oct 31 20:10:32.499237 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:10:32.499218    4645 scope.go:122] "RemoveContainer" containerID="491df582afe74d08057005503e8b6ab0dda154ee0246dc6e759498c70e6c0fe4"
Oct 31 20:10:32.506848 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:10:32.506832    4645 scope.go:122] "RemoveContainer" containerID="0d14e2988c30c9a2204c0ae9b8625d5021f8c135f10fe67cd23beae095e5c238"
Oct 31 20:10:32.514416 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:10:32.514400    4645 scope.go:122] "RemoveContainer" containerID="7f78b2a9b929d5bceca795372443e33b46a80a12870f7bf5ea75967eb8e965f5"
Oct 31 20:10:32.521799 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:10:32.521779    4645 scope.go:122] "RemoveContainer" containerID="3d9c6e4480379edf4e051b7140bc29078012459e683a0c6dcac0af259d5a489c"
Oct 31 20:10:36.244423 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:10:36.244387    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 20:10:47.032882 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:10:47.032845    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9af3dc16-vxggv"]
Oct 31 20:10:47.035908 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:10:47.035880    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9af3dc16-vxggv"]
Oct 31 20:10:47.616573 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:10:47.616541    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fda00124-784f-4524-836b-ad9b3b9e23de" path="/var/lib/kubelet/pods/fda00124-784f-4524-836b-ad9b3b9e23de/volumes"
Oct 31 20:10:56.049669 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:10:56.049632    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dc93a5b-l8q72"]
Oct 31 20:10:56.054128 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:10:56.054100    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dc93a5b-l8q72"]
Oct 31 20:10:57.616285 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:10:57.616248    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48365630-1b94-419c-a248-39fd40ffb067" path="/var/lib/kubelet/pods/48365630-1b94-419c-a248-39fd40ffb067/volumes"
Oct 31 20:11:03.896116 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:11:03.896073    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-vj5qn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 20:11:03.896604 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:11:03.896161    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vj5qn" podUID="8bdf127f-3310-4bc8-8f53-6a1e7dda4191" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 20:11:05.043300 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:11:05.043258    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44ea86bf-bx7k4"]
Oct 31 20:11:05.050226 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:11:05.050199    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44ea86bf-bx7k4"]
Oct 31 20:11:05.616550 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:11:05.616517    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="994c4584-538a-4750-9fc8-242816cfadf8" path="/var/lib/kubelet/pods/994c4584-538a-4750-9fc8-242816cfadf8/volumes"
Oct 31 20:11:06.245277 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:11:06.245238    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 20:11:18.590840 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:11:18.590744    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:11:19.689713 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:11:19.689672    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:11:20.682399 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:11:20.682356    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:11:32.558712 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:11:32.558675    4645 scope.go:122] "RemoveContainer" containerID="44ff49ee1dabc8669a574a379e718960648e053ef65a77745bdb00971fbb3adf"
Oct 31 20:11:32.568122 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:11:32.568101    4645 scope.go:122] "RemoveContainer" containerID="7c73a5ff8bfc6ad062ba730349acd817c0efd4be88c24702906b5e71c7b32da7"
Oct 31 20:11:32.576442 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:11:32.576423    4645 scope.go:122] "RemoveContainer" containerID="f31a16feab5f034af33f270b2ad2eff05abb3b6b0f076d8d629ec3a8a84e63c9"
Oct 31 20:11:32.587700 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:11:32.587682    4645 scope.go:122] "RemoveContainer" containerID="83592c7b557b846803e645ebeb8f0000ef2ee39cc5c79d07175b3cf4f00d8bda"
Oct 31 20:11:34.612069 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:11:34.612028    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2q74f"]
Oct 31 20:11:34.612634 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:11:34.612256    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2q74f" podUID="ab41024e-f7d2-4167-be83-b7dae45cb624" containerName="discovery" containerID="cri-o://ae7178d8a40ffd5a4aa2fa56d51ac6443c1cbea8e924cef567f4632d835d44d8" gracePeriod=30
Oct 31 20:11:34.921837 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:11:34.921801    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2q74f"
Oct 31 20:11:35.036353 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:11:35.034601    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab41024e-f7d2-4167-be83-b7dae45cb624-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab41024e-f7d2-4167-be83-b7dae45cb624-local-certs\") pod \"ab41024e-f7d2-4167-be83-b7dae45cb624\" (UID: \"ab41024e-f7d2-4167-be83-b7dae45cb624\") "
Oct 31 20:11:35.036353 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:11:35.034741    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab41024e-f7d2-4167-be83-b7dae45cb624-kube-api-access-qrw7d\" (UniqueName: \"kubernetes.io/projected/ab41024e-f7d2-4167-be83-b7dae45cb624-kube-api-access-qrw7d\") pod \"ab41024e-f7d2-4167-be83-b7dae45cb624\" (UID: \"ab41024e-f7d2-4167-be83-b7dae45cb624\") "
Oct 31 20:11:35.036353 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:11:35.034808    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab41024e-f7d2-4167-be83-b7dae45cb624-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ab41024e-f7d2-4167-be83-b7dae45cb624-istio-csr-dns-cert\") pod \"ab41024e-f7d2-4167-be83-b7dae45cb624\" (UID: \"ab41024e-f7d2-4167-be83-b7dae45cb624\") "
Oct 31 20:11:35.036353 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:11:35.034843    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab41024e-f7d2-4167-be83-b7dae45cb624-cacerts\" (UniqueName: \"kubernetes.io/secret/ab41024e-f7d2-4167-be83-b7dae45cb624-cacerts\") pod \"ab41024e-f7d2-4167-be83-b7dae45cb624\" (UID: \"ab41024e-f7d2-4167-be83-b7dae45cb624\") "
Oct 31 20:11:35.036353 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:11:35.034874    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab41024e-f7d2-4167-be83-b7dae45cb624-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ab41024e-f7d2-4167-be83-b7dae45cb624-istio-csr-ca-configmap\") pod \"ab41024e-f7d2-4167-be83-b7dae45cb624\" (UID: \"ab41024e-f7d2-4167-be83-b7dae45cb624\") "
Oct 31 20:11:35.036353 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:11:35.034917    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab41024e-f7d2-4167-be83-b7dae45cb624-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ab41024e-f7d2-4167-be83-b7dae45cb624-istio-kubeconfig\") pod \"ab41024e-f7d2-4167-be83-b7dae45cb624\" (UID: \"ab41024e-f7d2-4167-be83-b7dae45cb624\") "
Oct 31 20:11:35.036353 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:11:35.034966    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab41024e-f7d2-4167-be83-b7dae45cb624-istio-token\" (UniqueName: \"kubernetes.io/projected/ab41024e-f7d2-4167-be83-b7dae45cb624-istio-token\") pod \"ab41024e-f7d2-4167-be83-b7dae45cb624\" (UID: \"ab41024e-f7d2-4167-be83-b7dae45cb624\") "
Oct 31 20:11:35.054764 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:11:35.052624    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab41024e-f7d2-4167-be83-b7dae45cb624-local-certs" pod "ab41024e-f7d2-4167-be83-b7dae45cb624" (UID: "ab41024e-f7d2-4167-be83-b7dae45cb624"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:11:35.054764 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:11:35.052962    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab41024e-f7d2-4167-be83-b7dae45cb624-istio-csr-ca-configmap" pod "ab41024e-f7d2-4167-be83-b7dae45cb624" (UID: "ab41024e-f7d2-4167-be83-b7dae45cb624"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:11:35.059638 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:11:35.055233    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab41024e-f7d2-4167-be83-b7dae45cb624-istio-kubeconfig" pod "ab41024e-f7d2-4167-be83-b7dae45cb624" (UID: "ab41024e-f7d2-4167-be83-b7dae45cb624"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:11:35.059638 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:11:35.055374    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab41024e-f7d2-4167-be83-b7dae45cb624-istio-token" pod "ab41024e-f7d2-4167-be83-b7dae45cb624" (UID: "ab41024e-f7d2-4167-be83-b7dae45cb624"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:11:35.071761 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:11:35.071712    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab41024e-f7d2-4167-be83-b7dae45cb624-istio-csr-dns-cert" pod "ab41024e-f7d2-4167-be83-b7dae45cb624" (UID: "ab41024e-f7d2-4167-be83-b7dae45cb624"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:11:35.077044 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:11:35.077014    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab41024e-f7d2-4167-be83-b7dae45cb624-cacerts" pod "ab41024e-f7d2-4167-be83-b7dae45cb624" (UID: "ab41024e-f7d2-4167-be83-b7dae45cb624"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:11:35.080389 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:11:35.077316    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab41024e-f7d2-4167-be83-b7dae45cb624-kube-api-access-qrw7d" pod "ab41024e-f7d2-4167-be83-b7dae45cb624" (UID: "ab41024e-f7d2-4167-be83-b7dae45cb624"). InnerVolumeSpecName "kube-api-access-qrw7d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:11:35.139172 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:11:35.138969    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrw7d\" (UniqueName: \"kubernetes.io/projected/ab41024e-f7d2-4167-be83-b7dae45cb624-kube-api-access-qrw7d\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:11:35.139172 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:11:35.139014    4645 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ab41024e-f7d2-4167-be83-b7dae45cb624-istio-csr-dns-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:11:35.139172 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:11:35.139033    4645 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ab41024e-f7d2-4167-be83-b7dae45cb624-cacerts\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:11:35.139172 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:11:35.139048    4645 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ab41024e-f7d2-4167-be83-b7dae45cb624-istio-csr-ca-configmap\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:11:35.139172 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:11:35.139063    4645 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ab41024e-f7d2-4167-be83-b7dae45cb624-istio-kubeconfig\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:11:35.139172 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:11:35.139077    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab41024e-f7d2-4167-be83-b7dae45cb624-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:11:35.139172 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:11:35.139092    4645 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab41024e-f7d2-4167-be83-b7dae45cb624-local-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:11:35.332357 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:11:35.329932    4645 generic.go:356] "Generic (PLEG): container finished" podID="ab41024e-f7d2-4167-be83-b7dae45cb624" containerID="ae7178d8a40ffd5a4aa2fa56d51ac6443c1cbea8e924cef567f4632d835d44d8" exitCode=0
Oct 31 20:11:35.332357 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:11:35.330107    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2q74f" event={"ID":"ab41024e-f7d2-4167-be83-b7dae45cb624","Type":"ContainerDied","Data":"ae7178d8a40ffd5a4aa2fa56d51ac6443c1cbea8e924cef567f4632d835d44d8"}
Oct 31 20:11:35.332357 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:11:35.330159    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2q74f" event={"ID":"ab41024e-f7d2-4167-be83-b7dae45cb624","Type":"ContainerDied","Data":"5b45961fef5c106db67580e5ee5cfd6f4580224cbeaf09a1b6c1ad732387cd21"}
Oct 31 20:11:35.332357 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:11:35.330189    4645 scope.go:122] "RemoveContainer" containerID="ae7178d8a40ffd5a4aa2fa56d51ac6443c1cbea8e924cef567f4632d835d44d8"
Oct 31 20:11:35.332357 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:11:35.330379    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2q74f"
Oct 31 20:11:35.467610 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:11:35.467485    4645 scope.go:122] "RemoveContainer" containerID="ae7178d8a40ffd5a4aa2fa56d51ac6443c1cbea8e924cef567f4632d835d44d8"
Oct 31 20:11:35.468218 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:11:35.468137    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae7178d8a40ffd5a4aa2fa56d51ac6443c1cbea8e924cef567f4632d835d44d8\": container with ID starting with ae7178d8a40ffd5a4aa2fa56d51ac6443c1cbea8e924cef567f4632d835d44d8 not found: ID does not exist" containerID="ae7178d8a40ffd5a4aa2fa56d51ac6443c1cbea8e924cef567f4632d835d44d8"
Oct 31 20:11:35.468218 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:11:35.468178    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae7178d8a40ffd5a4aa2fa56d51ac6443c1cbea8e924cef567f4632d835d44d8"} err="failed to get container status \"ae7178d8a40ffd5a4aa2fa56d51ac6443c1cbea8e924cef567f4632d835d44d8\": rpc error: code = NotFound desc = could not find container \"ae7178d8a40ffd5a4aa2fa56d51ac6443c1cbea8e924cef567f4632d835d44d8\": container with ID starting with ae7178d8a40ffd5a4aa2fa56d51ac6443c1cbea8e924cef567f4632d835d44d8 not found: ID does not exist"
Oct 31 20:11:35.472380 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:11:35.463939    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2q74f"]
Oct 31 20:11:35.490457 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:11:35.490401    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2q74f"]
Oct 31 20:11:35.655609 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:11:35.655135    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab41024e-f7d2-4167-be83-b7dae45cb624" path="/var/lib/kubelet/pods/ab41024e-f7d2-4167-be83-b7dae45cb624/volumes"
Oct 31 20:11:36.432769 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:11:36.432675    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 20:11:37.757526 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:11:37.757494    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:12:00.603807 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:12:00.603764    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:12:00.732979 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:12:00.732934    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:12:06.246107 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:12:06.246063    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 20:12:18.637213 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:12:18.637173    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:12:19.717639 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:12:19.717597    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:12:20.738842 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:12:20.738800    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:12:32.619439 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:12:32.619401    4645 scope.go:122] "RemoveContainer" containerID="7db09618b7ad28e7963ffd93be055a9b7dac17ee390545e461fb36616f853b71"
Oct 31 20:12:32.628572 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:12:32.628549    4645 scope.go:122] "RemoveContainer" containerID="495811b6c6299ae635e112d62653cec9dc48321145f081ae586bbb2753759844"
Oct 31 20:12:32.637145 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:12:32.637117    4645 scope.go:122] "RemoveContainer" containerID="f9f5f837be77eaf90db1bab308d1e9f4e4ee098280845b2a7a76bbacc97972f2"
Oct 31 20:12:32.644893 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:12:32.644873    4645 scope.go:122] "RemoveContainer" containerID="a67caed1dc85c660a3e02882942cc443b16f2c20b156cc17b4a742dded2bccc0"
Oct 31 20:12:32.652522 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:12:32.652502    4645 scope.go:122] "RemoveContainer" containerID="299ffbbdb82d1ca774d0af57189f340b71c58ff362090599ba92d558e76e6e38"
Oct 31 20:12:36.248410 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:12:36.248361    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 20:13:06.245991 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:13:06.245906    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 20:13:11.027848 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:13:11.027813    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f96b154d-q49db"]
Oct 31 20:13:11.034671 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:13:11.034642    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f96b154d-q49db"]
Oct 31 20:13:11.616861 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:13:11.616826    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e9a668b-bc28-4037-aab6-85f3a803497b" path="/var/lib/kubelet/pods/7e9a668b-bc28-4037-aab6-85f3a803497b/volumes"
Oct 31 20:13:16.026826 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:13:16.026759    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9017f71-mvp5f"]
Oct 31 20:13:16.028694 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:13:16.028667    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9017f71-mvp5f"]
Oct 31 20:13:17.616402 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:13:17.616356    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f82be1ed-85af-4e4e-8381-97c6f58648c3" path="/var/lib/kubelet/pods/f82be1ed-85af-4e4e-8381-97c6f58648c3/volumes"
Oct 31 20:13:20.040751 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:13:20.040699    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:13:20.151418 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:13:20.151374    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:13:20.807729 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:13:20.807695    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:13:32.693438 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:13:32.693403    4645 scope.go:122] "RemoveContainer" containerID="48c237d2655d1c08acf6f1f7ce2d5b20f8319f4fe37800f7d1a250d107a20a1b"
Oct 31 20:13:32.702136 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:13:32.702103    4645 scope.go:122] "RemoveContainer" containerID="17663dd659909937b7fed7b2d9abd901cf512b5dd21e352024b6e27abef64b5b"
Oct 31 20:13:32.710571 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:13:32.710542    4645 scope.go:122] "RemoveContainer" containerID="f016a15c329eaf18f51b90201d0343d124a87418f3448d853ef51b4b11fc1822"
Oct 31 20:13:32.718064 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:13:32.718045    4645 scope.go:122] "RemoveContainer" containerID="427309ca680a4c09425cac5edded8608be0b096985faf4485934154c7384cbb9"
Oct 31 20:13:36.247459 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:13:36.247413    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 20:14:06.248209 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:14:06.248165    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 20:14:18.732244 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:14:18.732150    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:14:19.807986 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:14:19.807952    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:14:20.931313 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:14:20.931271    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:14:32.747240 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:14:32.747208    4645 scope.go:122] "RemoveContainer" containerID="d05d007f9f87ba18ec4d293e47facfeca77ad7a53c5e274a6a2a8888b06cfbbf"
Oct 31 20:14:32.755249 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:14:32.755226    4645 scope.go:122] "RemoveContainer" containerID="5768066d7883465c069594a44564fff940022a86957cc0e7df52979f4557ac56"
Oct 31 20:14:32.762659 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:14:32.762639    4645 scope.go:122] "RemoveContainer" containerID="6525006b443ea29fb4b87648c87bd574857e233bbb4317f5108db94d110d1d75"
Oct 31 20:14:32.769444 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:14:32.769424    4645 scope.go:122] "RemoveContainer" containerID="98ee8c6535ecb67493125bb363ae7a74df109356eec6f975f8a83d1a80b38373"
Oct 31 20:14:36.244783 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:14:36.244744    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 20:15:06.247911 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:15:06.247863    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 20:15:13.821526 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:15:13.821486    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:15:13.926175 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:15:13.926140    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:15:18.771417 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:15:18.771373    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:15:19.863530 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:15:19.863492    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:15:20.897427 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:15:20.897387    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:15:36.248597 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:15:36.248549    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 20:16:06.249288 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:16:06.249197    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 20:16:18.844191 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:16:18.844149    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:16:19.907750 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:16:19.907715    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:16:20.920659 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:16:20.920617    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:16:36.247773 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:16:36.247730    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 20:17:06.245503 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:17:06.245463    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 20:17:18.899704 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:17:18.899622    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:17:19.945922 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:17:19.945887    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:17:21.042412 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:17:21.042367    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:17:36.249341 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:17:36.249286    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 20:18:06.249034 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:18:06.248979    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 20:18:18.924089 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:18:18.924055    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:18:20.058956 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:18:20.058914    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:18:21.026125 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:18:21.026079    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:18:36.245366 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:18:36.245308    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 20:18:55.087729 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:18:55.087684    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vkszn"]
Oct 31 20:18:55.088457 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:18:55.088421    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vj5qn"]
Oct 31 20:18:55.088653 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:18:55.088486    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-55znv"]
Oct 31 20:18:55.088889 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:18:55.088783    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vkszn" podUID="0150a41c-8a80-46fb-b668-fcc8a4a48dcd" containerName="istio-proxy" containerID="cri-o://8de325a50771f5b9428b038cbdc86d5540d4a51df7f04714151f48dafaca92a8" gracePeriod=30
Oct 31 20:18:55.089561 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:18:55.089416    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-55znv" podUID="86c63841-b547-43a5-9d82-e1cde5c8f2ed" containerName="istio-proxy" containerID="cri-o://3a3359753776e5adcea55c368b8f64d1c2d4385862dec56548b86a44737b5446" gracePeriod=30
Oct 31 20:18:55.090610 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:18:55.089997    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vj5qn" podUID="8bdf127f-3310-4bc8-8f53-6a1e7dda4191" containerName="istio-proxy" containerID="cri-o://8c3d4ba6e726f8f9462282daa0b9b123c97dc17a7db9a07361f56f2a5cd4f099" gracePeriod=30
Oct 31 20:18:55.091426 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:18:55.091401    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6k6cd"]
Oct 31 20:18:55.091637 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:18:55.091578    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6k6cd" podUID="a8fca631-527d-46c0-9197-5edd7762adb9" containerName="istio-proxy" containerID="cri-o://c32d0cfd6f6abb7c3db39cdc98f6040183989685f651797cfccbc4225e3e4835" gracePeriod=30
Oct 31 20:18:58.188822 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:18:58.188788    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:18:58.602507 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:18:58.602426    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w"]
Oct 31 20:18:58.602851 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:18:58.602835    4645 cpu_manager.go:414] "containerMap: removing container" podUID="ab41024e-f7d2-4167-be83-b7dae45cb624" containerName="discovery"
Oct 31 20:18:58.602915 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:18:58.602852    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab41024e-f7d2-4167-be83-b7dae45cb624" containerName="discovery"
Oct 31 20:18:58.602973 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:18:58.602923    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab41024e-f7d2-4167-be83-b7dae45cb624" containerName="discovery"
Oct 31 20:18:58.606064 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:18:58.606044    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w"
Oct 31 20:18:58.609850 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:18:58.609824    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-844db7-openshift-default-dockercfg-4llmm\""
Oct 31 20:18:58.622013 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:18:58.621989    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w"]
Oct 31 20:18:58.679849 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:18:58.679802    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/672d276a-4c71-4742-82f6-f5c3dc95e072-workload-socket\") pod \"e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w\" (UID: \"672d276a-4c71-4742-82f6-f5c3dc95e072\") " pod="openshift-ingress/e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w"
Oct 31 20:18:58.680968 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:18:58.679928    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/672d276a-4c71-4742-82f6-f5c3dc95e072-istio-envoy\") pod \"e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w\" (UID: \"672d276a-4c71-4742-82f6-f5c3dc95e072\") " pod="openshift-ingress/e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w"
Oct 31 20:18:58.680968 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:18:58.679988    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w5bm\" (UniqueName: \"kubernetes.io/projected/672d276a-4c71-4742-82f6-f5c3dc95e072-kube-api-access-7w5bm\") pod \"e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w\" (UID: \"672d276a-4c71-4742-82f6-f5c3dc95e072\") " pod="openshift-ingress/e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w"
Oct 31 20:18:58.680968 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:18:58.680019    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/672d276a-4c71-4742-82f6-f5c3dc95e072-istio-podinfo\") pod \"e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w\" (UID: \"672d276a-4c71-4742-82f6-f5c3dc95e072\") " pod="openshift-ingress/e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w"
Oct 31 20:18:58.680968 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:18:58.680049    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/672d276a-4c71-4742-82f6-f5c3dc95e072-credential-socket\") pod \"e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w\" (UID: \"672d276a-4c71-4742-82f6-f5c3dc95e072\") " pod="openshift-ingress/e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w"
Oct 31 20:18:58.680968 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:18:58.680091    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/672d276a-4c71-4742-82f6-f5c3dc95e072-istio-data\") pod \"e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w\" (UID: \"672d276a-4c71-4742-82f6-f5c3dc95e072\") " pod="openshift-ingress/e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w"
Oct 31 20:18:58.680968 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:18:58.680190    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/672d276a-4c71-4742-82f6-f5c3dc95e072-workload-certs\") pod \"e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w\" (UID: \"672d276a-4c71-4742-82f6-f5c3dc95e072\") " pod="openshift-ingress/e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w"
Oct 31 20:18:58.680968 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:18:58.680306    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/672d276a-4c71-4742-82f6-f5c3dc95e072-istiod-ca-cert\") pod \"e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w\" (UID: \"672d276a-4c71-4742-82f6-f5c3dc95e072\") " pod="openshift-ingress/e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w"
Oct 31 20:18:58.680968 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:18:58.680537    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/672d276a-4c71-4742-82f6-f5c3dc95e072-istio-token\") pod \"e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w\" (UID: \"672d276a-4c71-4742-82f6-f5c3dc95e072\") " pod="openshift-ingress/e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w"
Oct 31 20:18:58.692797 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:18:58.692764    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:18:58.781638 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:18:58.781593    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/672d276a-4c71-4742-82f6-f5c3dc95e072-istio-envoy\") pod \"e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w\" (UID: \"672d276a-4c71-4742-82f6-f5c3dc95e072\") " pod="openshift-ingress/e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w"
Oct 31 20:18:58.781841 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:18:58.781761    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w5bm\" (UniqueName: \"kubernetes.io/projected/672d276a-4c71-4742-82f6-f5c3dc95e072-kube-api-access-7w5bm\") pod \"e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w\" (UID: \"672d276a-4c71-4742-82f6-f5c3dc95e072\") " pod="openshift-ingress/e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w"
Oct 31 20:18:58.781841 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:18:58.781800    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/672d276a-4c71-4742-82f6-f5c3dc95e072-istio-podinfo\") pod \"e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w\" (UID: \"672d276a-4c71-4742-82f6-f5c3dc95e072\") " pod="openshift-ingress/e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w"
Oct 31 20:18:58.781841 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:18:58.781825    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/672d276a-4c71-4742-82f6-f5c3dc95e072-credential-socket\") pod \"e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w\" (UID: \"672d276a-4c71-4742-82f6-f5c3dc95e072\") " pod="openshift-ingress/e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w"
Oct 31 20:18:58.782050 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:18:58.781873    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/672d276a-4c71-4742-82f6-f5c3dc95e072-istio-data\") pod \"e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w\" (UID: \"672d276a-4c71-4742-82f6-f5c3dc95e072\") " pod="openshift-ingress/e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w"
Oct 31 20:18:58.782050 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:18:58.781955    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/672d276a-4c71-4742-82f6-f5c3dc95e072-workload-certs\") pod \"e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w\" (UID: \"672d276a-4c71-4742-82f6-f5c3dc95e072\") " pod="openshift-ingress/e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w"
Oct 31 20:18:58.782165 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:18:58.782125    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/672d276a-4c71-4742-82f6-f5c3dc95e072-istiod-ca-cert\") pod \"e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w\" (UID: \"672d276a-4c71-4742-82f6-f5c3dc95e072\") " pod="openshift-ingress/e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w"
Oct 31 20:18:58.782228 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:18:58.782178    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/672d276a-4c71-4742-82f6-f5c3dc95e072-istio-token\") pod \"e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w\" (UID: \"672d276a-4c71-4742-82f6-f5c3dc95e072\") " pod="openshift-ingress/e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w"
Oct 31 20:18:58.782228 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:18:58.782213    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/672d276a-4c71-4742-82f6-f5c3dc95e072-istio-data\") pod \"e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w\" (UID: \"672d276a-4c71-4742-82f6-f5c3dc95e072\") " pod="openshift-ingress/e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w"
Oct 31 20:18:58.782387 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:18:58.782237    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/672d276a-4c71-4742-82f6-f5c3dc95e072-workload-socket\") pod \"e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w\" (UID: \"672d276a-4c71-4742-82f6-f5c3dc95e072\") " pod="openshift-ingress/e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w"
Oct 31 20:18:58.782387 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:18:58.782293    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/672d276a-4c71-4742-82f6-f5c3dc95e072-workload-certs\") pod \"e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w\" (UID: \"672d276a-4c71-4742-82f6-f5c3dc95e072\") " pod="openshift-ingress/e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w"
Oct 31 20:18:58.782516 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:18:58.782380    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/672d276a-4c71-4742-82f6-f5c3dc95e072-credential-socket\") pod \"e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w\" (UID: \"672d276a-4c71-4742-82f6-f5c3dc95e072\") " pod="openshift-ingress/e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w"
Oct 31 20:18:58.782658 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:18:58.782618    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/672d276a-4c71-4742-82f6-f5c3dc95e072-workload-socket\") pod \"e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w\" (UID: \"672d276a-4c71-4742-82f6-f5c3dc95e072\") " pod="openshift-ingress/e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w"
Oct 31 20:18:58.782902 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:18:58.782876    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/672d276a-4c71-4742-82f6-f5c3dc95e072-istiod-ca-cert\") pod \"e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w\" (UID: \"672d276a-4c71-4742-82f6-f5c3dc95e072\") " pod="openshift-ingress/e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w"
Oct 31 20:18:58.784147 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:18:58.784124    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/672d276a-4c71-4742-82f6-f5c3dc95e072-istio-envoy\") pod \"e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w\" (UID: \"672d276a-4c71-4742-82f6-f5c3dc95e072\") " pod="openshift-ingress/e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w"
Oct 31 20:18:58.784503 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:18:58.784484    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/672d276a-4c71-4742-82f6-f5c3dc95e072-istio-podinfo\") pod \"e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w\" (UID: \"672d276a-4c71-4742-82f6-f5c3dc95e072\") " pod="openshift-ingress/e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w"
Oct 31 20:18:58.794598 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:18:58.794576    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/672d276a-4c71-4742-82f6-f5c3dc95e072-istio-token\") pod \"e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w\" (UID: \"672d276a-4c71-4742-82f6-f5c3dc95e072\") " pod="openshift-ingress/e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w"
Oct 31 20:18:58.795807 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:18:58.795780    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w5bm\" (UniqueName: \"kubernetes.io/projected/672d276a-4c71-4742-82f6-f5c3dc95e072-kube-api-access-7w5bm\") pod \"e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w\" (UID: \"672d276a-4c71-4742-82f6-f5c3dc95e072\") " pod="openshift-ingress/e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w"
Oct 31 20:18:58.857948 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:18:58.857858    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:18:58.919869 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:18:58.919834    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w"
Oct 31 20:18:59.198125 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:18:59.198088    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w"]
Oct 31 20:18:59.202040 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:18:59.202012    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod672d276a_4c71_4742_82f6_f5c3dc95e072.slice/crio-bb13ac498e25219c40643a49ed80fa8f629a280e874479ed3fa0c942d44c209d WatchSource:0}: Error finding container bb13ac498e25219c40643a49ed80fa8f629a280e874479ed3fa0c942d44c209d: Status 404 returned error can't find the container with id bb13ac498e25219c40643a49ed80fa8f629a280e874479ed3fa0c942d44c209d
Oct 31 20:18:59.204079 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:18:59.204046    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:18:59.204169 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:18:59.204112    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:18:59.204169 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:18:59.204139    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:18:59.633517 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:18:59.633437    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w" event={"ID":"672d276a-4c71-4742-82f6-f5c3dc95e072","Type":"ContainerStarted","Data":"20b1df8596b5b7fb6369a4036e590f2f4b5f358b74c5934466db94e801e83c41"}
Oct 31 20:18:59.633517 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:18:59.633492    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w" event={"ID":"672d276a-4c71-4742-82f6-f5c3dc95e072","Type":"ContainerStarted","Data":"bb13ac498e25219c40643a49ed80fa8f629a280e874479ed3fa0c942d44c209d"}
Oct 31 20:18:59.657951 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:18:59.657892    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w" podStartSLOduration=1.6578783860000001 podStartE2EDuration="1.657878386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:18:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:18:59.654870075 +0000 UTC m=+3570.228104204" watchObservedRunningTime="2026-10-31 20:18:59.657878386 +0000 UTC m=+3570.231112514"
Oct 31 20:18:59.920120 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:18:59.920089    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w"
Oct 31 20:19:00.350858 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.350832    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vj5qn"
Oct 31 20:19:00.377193 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.377172    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6k6cd"
Oct 31 20:19:00.383121 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.383099    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vkszn"
Oct 31 20:19:00.497969 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.497885    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0150a41c-8a80-46fb-b668-fcc8a4a48dcd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0150a41c-8a80-46fb-b668-fcc8a4a48dcd-istio-podinfo\") pod \"0150a41c-8a80-46fb-b668-fcc8a4a48dcd\" (UID: \"0150a41c-8a80-46fb-b668-fcc8a4a48dcd\") "
Oct 31 20:19:00.497969 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.497931    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a8fca631-527d-46c0-9197-5edd7762adb9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8fca631-527d-46c0-9197-5edd7762adb9-istio-podinfo\") pod \"a8fca631-527d-46c0-9197-5edd7762adb9\" (UID: \"a8fca631-527d-46c0-9197-5edd7762adb9\") "
Oct 31 20:19:00.497969 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.497961    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bdf127f-3310-4bc8-8f53-6a1e7dda4191-kube-api-access-ktkz8\" (UniqueName: \"kubernetes.io/projected/8bdf127f-3310-4bc8-8f53-6a1e7dda4191-kube-api-access-ktkz8\") pod \"8bdf127f-3310-4bc8-8f53-6a1e7dda4191\" (UID: \"8bdf127f-3310-4bc8-8f53-6a1e7dda4191\") "
Oct 31 20:19:00.498267 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.497992    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0150a41c-8a80-46fb-b668-fcc8a4a48dcd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0150a41c-8a80-46fb-b668-fcc8a4a48dcd-istio-envoy\") pod \"0150a41c-8a80-46fb-b668-fcc8a4a48dcd\" (UID: \"0150a41c-8a80-46fb-b668-fcc8a4a48dcd\") "
Oct 31 20:19:00.498267 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.498063    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8fca631-527d-46c0-9197-5edd7762adb9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8fca631-527d-46c0-9197-5edd7762adb9-credential-socket\") pod \"a8fca631-527d-46c0-9197-5edd7762adb9\" (UID: \"a8fca631-527d-46c0-9197-5edd7762adb9\") "
Oct 31 20:19:00.498267 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.498122    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bdf127f-3310-4bc8-8f53-6a1e7dda4191-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bdf127f-3310-4bc8-8f53-6a1e7dda4191-istio-data\") pod \"8bdf127f-3310-4bc8-8f53-6a1e7dda4191\" (UID: \"8bdf127f-3310-4bc8-8f53-6a1e7dda4191\") "
Oct 31 20:19:00.498267 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.498155    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0150a41c-8a80-46fb-b668-fcc8a4a48dcd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0150a41c-8a80-46fb-b668-fcc8a4a48dcd-workload-socket\") pod \"0150a41c-8a80-46fb-b668-fcc8a4a48dcd\" (UID: \"0150a41c-8a80-46fb-b668-fcc8a4a48dcd\") "
Oct 31 20:19:00.498267 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.498189    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8fca631-527d-46c0-9197-5edd7762adb9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8fca631-527d-46c0-9197-5edd7762adb9-istiod-ca-cert\") pod \"a8fca631-527d-46c0-9197-5edd7762adb9\" (UID: \"a8fca631-527d-46c0-9197-5edd7762adb9\") "
Oct 31 20:19:00.498267 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.498221    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0150a41c-8a80-46fb-b668-fcc8a4a48dcd-kube-api-access-qvlrz\" (UniqueName: \"kubernetes.io/projected/0150a41c-8a80-46fb-b668-fcc8a4a48dcd-kube-api-access-qvlrz\") pod \"0150a41c-8a80-46fb-b668-fcc8a4a48dcd\" (UID: \"0150a41c-8a80-46fb-b668-fcc8a4a48dcd\") "
Oct 31 20:19:00.498267 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.498266    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8bdf127f-3310-4bc8-8f53-6a1e7dda4191-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bdf127f-3310-4bc8-8f53-6a1e7dda4191-istio-podinfo\") pod \"8bdf127f-3310-4bc8-8f53-6a1e7dda4191\" (UID: \"8bdf127f-3310-4bc8-8f53-6a1e7dda4191\") "
Oct 31 20:19:00.498689 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.498303    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8fca631-527d-46c0-9197-5edd7762adb9-istio-token\" (UniqueName: \"kubernetes.io/projected/a8fca631-527d-46c0-9197-5edd7762adb9-istio-token\") pod \"a8fca631-527d-46c0-9197-5edd7762adb9\" (UID: \"a8fca631-527d-46c0-9197-5edd7762adb9\") "
Oct 31 20:19:00.498689 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.498373    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8fca631-527d-46c0-9197-5edd7762adb9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8fca631-527d-46c0-9197-5edd7762adb9-workload-certs\") pod \"a8fca631-527d-46c0-9197-5edd7762adb9\" (UID: \"a8fca631-527d-46c0-9197-5edd7762adb9\") "
Oct 31 20:19:00.498689 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.498404    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bdf127f-3310-4bc8-8f53-6a1e7dda4191-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bdf127f-3310-4bc8-8f53-6a1e7dda4191-credential-socket\") pod \"8bdf127f-3310-4bc8-8f53-6a1e7dda4191\" (UID: \"8bdf127f-3310-4bc8-8f53-6a1e7dda4191\") "
Oct 31 20:19:00.498689 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.498430    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8fca631-527d-46c0-9197-5edd7762adb9-credential-socket" pod "a8fca631-527d-46c0-9197-5edd7762adb9" (UID: "a8fca631-527d-46c0-9197-5edd7762adb9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:19:00.498689 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.498445    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8fca631-527d-46c0-9197-5edd7762adb9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8fca631-527d-46c0-9197-5edd7762adb9-istio-data\") pod \"a8fca631-527d-46c0-9197-5edd7762adb9\" (UID: \"a8fca631-527d-46c0-9197-5edd7762adb9\") "
Oct 31 20:19:00.498689 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.498446    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bdf127f-3310-4bc8-8f53-6a1e7dda4191-istio-data" pod "8bdf127f-3310-4bc8-8f53-6a1e7dda4191" (UID: "8bdf127f-3310-4bc8-8f53-6a1e7dda4191"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:19:00.498689 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.498489    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bdf127f-3310-4bc8-8f53-6a1e7dda4191-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bdf127f-3310-4bc8-8f53-6a1e7dda4191-istio-envoy\") pod \"8bdf127f-3310-4bc8-8f53-6a1e7dda4191\" (UID: \"8bdf127f-3310-4bc8-8f53-6a1e7dda4191\") "
Oct 31 20:19:00.498689 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.498540    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0150a41c-8a80-46fb-b668-fcc8a4a48dcd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0150a41c-8a80-46fb-b668-fcc8a4a48dcd-workload-certs\") pod \"0150a41c-8a80-46fb-b668-fcc8a4a48dcd\" (UID: \"0150a41c-8a80-46fb-b668-fcc8a4a48dcd\") "
Oct 31 20:19:00.498689 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.498567    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bdf127f-3310-4bc8-8f53-6a1e7dda4191-istio-token\" (UniqueName: \"kubernetes.io/projected/8bdf127f-3310-4bc8-8f53-6a1e7dda4191-istio-token\") pod \"8bdf127f-3310-4bc8-8f53-6a1e7dda4191\" (UID: \"8bdf127f-3310-4bc8-8f53-6a1e7dda4191\") "
Oct 31 20:19:00.498689 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.498618    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0150a41c-8a80-46fb-b668-fcc8a4a48dcd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0150a41c-8a80-46fb-b668-fcc8a4a48dcd-istio-data\") pod \"0150a41c-8a80-46fb-b668-fcc8a4a48dcd\" (UID: \"0150a41c-8a80-46fb-b668-fcc8a4a48dcd\") "
Oct 31 20:19:00.498689 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.498645    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0150a41c-8a80-46fb-b668-fcc8a4a48dcd-istio-token\" (UniqueName: \"kubernetes.io/projected/0150a41c-8a80-46fb-b668-fcc8a4a48dcd-istio-token\") pod \"0150a41c-8a80-46fb-b668-fcc8a4a48dcd\" (UID: \"0150a41c-8a80-46fb-b668-fcc8a4a48dcd\") "
Oct 31 20:19:00.498689 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.498675    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bdf127f-3310-4bc8-8f53-6a1e7dda4191-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bdf127f-3310-4bc8-8f53-6a1e7dda4191-istiod-ca-cert\") pod \"8bdf127f-3310-4bc8-8f53-6a1e7dda4191\" (UID: \"8bdf127f-3310-4bc8-8f53-6a1e7dda4191\") "
Oct 31 20:19:00.499389 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.498710    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8fca631-527d-46c0-9197-5edd7762adb9-kube-api-access-4m458\" (UniqueName: \"kubernetes.io/projected/a8fca631-527d-46c0-9197-5edd7762adb9-kube-api-access-4m458\") pod \"a8fca631-527d-46c0-9197-5edd7762adb9\" (UID: \"a8fca631-527d-46c0-9197-5edd7762adb9\") "
Oct 31 20:19:00.499389 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.498768    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8fca631-527d-46c0-9197-5edd7762adb9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8fca631-527d-46c0-9197-5edd7762adb9-workload-socket\") pod \"a8fca631-527d-46c0-9197-5edd7762adb9\" (UID: \"a8fca631-527d-46c0-9197-5edd7762adb9\") "
Oct 31 20:19:00.499389 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.498798    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bdf127f-3310-4bc8-8f53-6a1e7dda4191-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bdf127f-3310-4bc8-8f53-6a1e7dda4191-workload-certs\") pod \"8bdf127f-3310-4bc8-8f53-6a1e7dda4191\" (UID: \"8bdf127f-3310-4bc8-8f53-6a1e7dda4191\") "
Oct 31 20:19:00.499389 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.498853    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8fca631-527d-46c0-9197-5edd7762adb9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8fca631-527d-46c0-9197-5edd7762adb9-istio-envoy\") pod \"a8fca631-527d-46c0-9197-5edd7762adb9\" (UID: \"a8fca631-527d-46c0-9197-5edd7762adb9\") "
Oct 31 20:19:00.499389 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.498863    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8fca631-527d-46c0-9197-5edd7762adb9-istiod-ca-cert" pod "a8fca631-527d-46c0-9197-5edd7762adb9" (UID: "a8fca631-527d-46c0-9197-5edd7762adb9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:19:00.499389 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.498897    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bdf127f-3310-4bc8-8f53-6a1e7dda4191-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bdf127f-3310-4bc8-8f53-6a1e7dda4191-workload-socket\") pod \"8bdf127f-3310-4bc8-8f53-6a1e7dda4191\" (UID: \"8bdf127f-3310-4bc8-8f53-6a1e7dda4191\") "
Oct 31 20:19:00.499389 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.498946    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0150a41c-8a80-46fb-b668-fcc8a4a48dcd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0150a41c-8a80-46fb-b668-fcc8a4a48dcd-istiod-ca-cert\") pod \"0150a41c-8a80-46fb-b668-fcc8a4a48dcd\" (UID: \"0150a41c-8a80-46fb-b668-fcc8a4a48dcd\") "
Oct 31 20:19:00.499389 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.498988    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0150a41c-8a80-46fb-b668-fcc8a4a48dcd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0150a41c-8a80-46fb-b668-fcc8a4a48dcd-credential-socket\") pod \"0150a41c-8a80-46fb-b668-fcc8a4a48dcd\" (UID: \"0150a41c-8a80-46fb-b668-fcc8a4a48dcd\") "
Oct 31 20:19:00.499389 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.499067    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0150a41c-8a80-46fb-b668-fcc8a4a48dcd-workload-socket" pod "0150a41c-8a80-46fb-b668-fcc8a4a48dcd" (UID: "0150a41c-8a80-46fb-b668-fcc8a4a48dcd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:19:00.499389 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.499263    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0150a41c-8a80-46fb-b668-fcc8a4a48dcd-istio-data" pod "0150a41c-8a80-46fb-b668-fcc8a4a48dcd" (UID: "0150a41c-8a80-46fb-b668-fcc8a4a48dcd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:19:00.499939 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.499439    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0150a41c-8a80-46fb-b668-fcc8a4a48dcd-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:19:00.499939 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.499463    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8fca631-527d-46c0-9197-5edd7762adb9-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:19:00.499939 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.499480    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bdf127f-3310-4bc8-8f53-6a1e7dda4191-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:19:00.499939 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.499496    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0150a41c-8a80-46fb-b668-fcc8a4a48dcd-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:19:00.499939 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.499507    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0150a41c-8a80-46fb-b668-fcc8a4a48dcd-credential-socket" pod "0150a41c-8a80-46fb-b668-fcc8a4a48dcd" (UID: "0150a41c-8a80-46fb-b668-fcc8a4a48dcd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:19:00.499939 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.499512    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8fca631-527d-46c0-9197-5edd7762adb9-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:19:00.500321 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.500162    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8fca631-527d-46c0-9197-5edd7762adb9-workload-certs" pod "a8fca631-527d-46c0-9197-5edd7762adb9" (UID: "a8fca631-527d-46c0-9197-5edd7762adb9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:19:00.500421 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.500345    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8fca631-527d-46c0-9197-5edd7762adb9-istio-data" pod "a8fca631-527d-46c0-9197-5edd7762adb9" (UID: "a8fca631-527d-46c0-9197-5edd7762adb9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:19:00.500421 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.500387    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bdf127f-3310-4bc8-8f53-6a1e7dda4191-workload-certs" pod "8bdf127f-3310-4bc8-8f53-6a1e7dda4191" (UID: "8bdf127f-3310-4bc8-8f53-6a1e7dda4191"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:19:00.500630 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.500590    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bdf127f-3310-4bc8-8f53-6a1e7dda4191-credential-socket" pod "8bdf127f-3310-4bc8-8f53-6a1e7dda4191" (UID: "8bdf127f-3310-4bc8-8f53-6a1e7dda4191"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:19:00.500782 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.500760    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bdf127f-3310-4bc8-8f53-6a1e7dda4191-istiod-ca-cert" pod "8bdf127f-3310-4bc8-8f53-6a1e7dda4191" (UID: "8bdf127f-3310-4bc8-8f53-6a1e7dda4191"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:19:00.500853 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.500775    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bdf127f-3310-4bc8-8f53-6a1e7dda4191-workload-socket" pod "8bdf127f-3310-4bc8-8f53-6a1e7dda4191" (UID: "8bdf127f-3310-4bc8-8f53-6a1e7dda4191"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:19:00.500853 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.500841    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8bdf127f-3310-4bc8-8f53-6a1e7dda4191-istio-podinfo" pod "8bdf127f-3310-4bc8-8f53-6a1e7dda4191" (UID: "8bdf127f-3310-4bc8-8f53-6a1e7dda4191"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 20:19:00.500968 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.500889    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0150a41c-8a80-46fb-b668-fcc8a4a48dcd-istio-envoy" pod "0150a41c-8a80-46fb-b668-fcc8a4a48dcd" (UID: "0150a41c-8a80-46fb-b668-fcc8a4a48dcd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:19:00.501131 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.501107    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8fca631-527d-46c0-9197-5edd7762adb9-workload-socket" pod "a8fca631-527d-46c0-9197-5edd7762adb9" (UID: "a8fca631-527d-46c0-9197-5edd7762adb9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:19:00.501380 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.501356    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0150a41c-8a80-46fb-b668-fcc8a4a48dcd-workload-certs" pod "0150a41c-8a80-46fb-b668-fcc8a4a48dcd" (UID: "0150a41c-8a80-46fb-b668-fcc8a4a48dcd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:19:00.501475 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.501445    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0150a41c-8a80-46fb-b668-fcc8a4a48dcd-istiod-ca-cert" pod "0150a41c-8a80-46fb-b668-fcc8a4a48dcd" (UID: "0150a41c-8a80-46fb-b668-fcc8a4a48dcd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:19:00.503669 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.503571    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0150a41c-8a80-46fb-b668-fcc8a4a48dcd-istio-token" pod "0150a41c-8a80-46fb-b668-fcc8a4a48dcd" (UID: "0150a41c-8a80-46fb-b668-fcc8a4a48dcd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:19:00.503669 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.503644    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a8fca631-527d-46c0-9197-5edd7762adb9-istio-podinfo" pod "a8fca631-527d-46c0-9197-5edd7762adb9" (UID: "a8fca631-527d-46c0-9197-5edd7762adb9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 20:19:00.503669 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.503657    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8fca631-527d-46c0-9197-5edd7762adb9-istio-token" pod "a8fca631-527d-46c0-9197-5edd7762adb9" (UID: "a8fca631-527d-46c0-9197-5edd7762adb9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:19:00.504090 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.504054    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0150a41c-8a80-46fb-b668-fcc8a4a48dcd-istio-podinfo" pod "0150a41c-8a80-46fb-b668-fcc8a4a48dcd" (UID: "0150a41c-8a80-46fb-b668-fcc8a4a48dcd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 20:19:00.504090 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.504073    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bdf127f-3310-4bc8-8f53-6a1e7dda4191-kube-api-access-ktkz8" pod "8bdf127f-3310-4bc8-8f53-6a1e7dda4191" (UID: "8bdf127f-3310-4bc8-8f53-6a1e7dda4191"). InnerVolumeSpecName "kube-api-access-ktkz8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:19:00.504239 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.504133    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8fca631-527d-46c0-9197-5edd7762adb9-kube-api-access-4m458" pod "a8fca631-527d-46c0-9197-5edd7762adb9" (UID: "a8fca631-527d-46c0-9197-5edd7762adb9"). InnerVolumeSpecName "kube-api-access-4m458". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:19:00.504239 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.504194    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bdf127f-3310-4bc8-8f53-6a1e7dda4191-istio-envoy" pod "8bdf127f-3310-4bc8-8f53-6a1e7dda4191" (UID: "8bdf127f-3310-4bc8-8f53-6a1e7dda4191"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:19:00.505650 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.505613    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8fca631-527d-46c0-9197-5edd7762adb9-istio-envoy" pod "a8fca631-527d-46c0-9197-5edd7762adb9" (UID: "a8fca631-527d-46c0-9197-5edd7762adb9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:19:00.506765 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.506695    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0150a41c-8a80-46fb-b668-fcc8a4a48dcd-kube-api-access-qvlrz" pod "0150a41c-8a80-46fb-b668-fcc8a4a48dcd" (UID: "0150a41c-8a80-46fb-b668-fcc8a4a48dcd"). InnerVolumeSpecName "kube-api-access-qvlrz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:19:00.507016 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.506984    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bdf127f-3310-4bc8-8f53-6a1e7dda4191-istio-token" pod "8bdf127f-3310-4bc8-8f53-6a1e7dda4191" (UID: "8bdf127f-3310-4bc8-8f53-6a1e7dda4191"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:19:00.600095 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.600069    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvlrz\" (UniqueName: \"kubernetes.io/projected/0150a41c-8a80-46fb-b668-fcc8a4a48dcd-kube-api-access-qvlrz\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:19:00.600095 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.600096    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bdf127f-3310-4bc8-8f53-6a1e7dda4191-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:19:00.600255 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.600107    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8fca631-527d-46c0-9197-5edd7762adb9-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:19:00.600255 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.600116    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8fca631-527d-46c0-9197-5edd7762adb9-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:19:00.600255 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.600125    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bdf127f-3310-4bc8-8f53-6a1e7dda4191-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:19:00.600255 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.600133    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8fca631-527d-46c0-9197-5edd7762adb9-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:19:00.600255 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.600141    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bdf127f-3310-4bc8-8f53-6a1e7dda4191-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:19:00.600255 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.600149    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0150a41c-8a80-46fb-b668-fcc8a4a48dcd-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:19:00.600255 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.600159    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bdf127f-3310-4bc8-8f53-6a1e7dda4191-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:19:00.600255 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.600167    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0150a41c-8a80-46fb-b668-fcc8a4a48dcd-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:19:00.600255 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.600176    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bdf127f-3310-4bc8-8f53-6a1e7dda4191-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:19:00.600255 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.600185    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4m458\" (UniqueName: \"kubernetes.io/projected/a8fca631-527d-46c0-9197-5edd7762adb9-kube-api-access-4m458\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:19:00.600255 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.600193    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8fca631-527d-46c0-9197-5edd7762adb9-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:19:00.600255 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.600201    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bdf127f-3310-4bc8-8f53-6a1e7dda4191-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:19:00.600255 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.600209    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8fca631-527d-46c0-9197-5edd7762adb9-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:19:00.600255 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.600217    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bdf127f-3310-4bc8-8f53-6a1e7dda4191-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:19:00.600255 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.600225    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0150a41c-8a80-46fb-b668-fcc8a4a48dcd-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:19:00.600255 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.600233    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0150a41c-8a80-46fb-b668-fcc8a4a48dcd-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:19:00.600255 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.600242    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0150a41c-8a80-46fb-b668-fcc8a4a48dcd-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:19:00.600255 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.600260    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8fca631-527d-46c0-9197-5edd7762adb9-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:19:00.600872 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.600269    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktkz8\" (UniqueName: \"kubernetes.io/projected/8bdf127f-3310-4bc8-8f53-6a1e7dda4191-kube-api-access-ktkz8\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:19:00.600872 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.600277    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0150a41c-8a80-46fb-b668-fcc8a4a48dcd-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:19:00.639108 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.639080    4645 generic.go:356] "Generic (PLEG): container finished" podID="a8fca631-527d-46c0-9197-5edd7762adb9" containerID="c32d0cfd6f6abb7c3db39cdc98f6040183989685f651797cfccbc4225e3e4835" exitCode=0
Oct 31 20:19:00.639236 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.639128    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6k6cd"
Oct 31 20:19:00.639236 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.639157    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6k6cd" event={"ID":"a8fca631-527d-46c0-9197-5edd7762adb9","Type":"ContainerDied","Data":"c32d0cfd6f6abb7c3db39cdc98f6040183989685f651797cfccbc4225e3e4835"}
Oct 31 20:19:00.639236 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.639208    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6k6cd" event={"ID":"a8fca631-527d-46c0-9197-5edd7762adb9","Type":"ContainerDied","Data":"dbc30710aab6cec5232bf550442aa7007e18d311211c26f203deea5efa6c6a60"}
Oct 31 20:19:00.639236 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.639234    4645 scope.go:122] "RemoveContainer" containerID="c32d0cfd6f6abb7c3db39cdc98f6040183989685f651797cfccbc4225e3e4835"
Oct 31 20:19:00.640878 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.640854    4645 generic.go:356] "Generic (PLEG): container finished" podID="8bdf127f-3310-4bc8-8f53-6a1e7dda4191" containerID="8c3d4ba6e726f8f9462282daa0b9b123c97dc17a7db9a07361f56f2a5cd4f099" exitCode=0
Oct 31 20:19:00.641004 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.640900    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vj5qn"
Oct 31 20:19:00.641004 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.640933    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vj5qn" event={"ID":"8bdf127f-3310-4bc8-8f53-6a1e7dda4191","Type":"ContainerDied","Data":"8c3d4ba6e726f8f9462282daa0b9b123c97dc17a7db9a07361f56f2a5cd4f099"}
Oct 31 20:19:00.641004 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.640984    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vj5qn" event={"ID":"8bdf127f-3310-4bc8-8f53-6a1e7dda4191","Type":"ContainerDied","Data":"bb58c80cb460b1a24b2c7d1ee776b379efe387068e2b3ac3318ea2918fd70bde"}
Oct 31 20:19:00.642456 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.642434    4645 generic.go:356] "Generic (PLEG): container finished" podID="0150a41c-8a80-46fb-b668-fcc8a4a48dcd" containerID="8de325a50771f5b9428b038cbdc86d5540d4a51df7f04714151f48dafaca92a8" exitCode=0
Oct 31 20:19:00.642573 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.642484    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vkszn"
Oct 31 20:19:00.642573 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.642524    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vkszn" event={"ID":"0150a41c-8a80-46fb-b668-fcc8a4a48dcd","Type":"ContainerDied","Data":"8de325a50771f5b9428b038cbdc86d5540d4a51df7f04714151f48dafaca92a8"}
Oct 31 20:19:00.642710 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.642583    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vkszn" event={"ID":"0150a41c-8a80-46fb-b668-fcc8a4a48dcd","Type":"ContainerDied","Data":"2639340d1a92ab82431974157ac2f957a8661c0dd7c4a9fcc17443412d7b4b32"}
Oct 31 20:19:00.644643 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.644616    4645 generic.go:356] "Generic (PLEG): container finished" podID="86c63841-b547-43a5-9d82-e1cde5c8f2ed" containerID="3a3359753776e5adcea55c368b8f64d1c2d4385862dec56548b86a44737b5446" exitCode=0
Oct 31 20:19:00.645242 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.645220    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-55znv" event={"ID":"86c63841-b547-43a5-9d82-e1cde5c8f2ed","Type":"ContainerDied","Data":"3a3359753776e5adcea55c368b8f64d1c2d4385862dec56548b86a44737b5446"}
Oct 31 20:19:00.651424 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.651408    4645 scope.go:122] "RemoveContainer" containerID="c32d0cfd6f6abb7c3db39cdc98f6040183989685f651797cfccbc4225e3e4835"
Oct 31 20:19:00.651889 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:19:00.651741    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c32d0cfd6f6abb7c3db39cdc98f6040183989685f651797cfccbc4225e3e4835\": container with ID starting with c32d0cfd6f6abb7c3db39cdc98f6040183989685f651797cfccbc4225e3e4835 not found: ID does not exist" containerID="c32d0cfd6f6abb7c3db39cdc98f6040183989685f651797cfccbc4225e3e4835"
Oct 31 20:19:00.651889 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.651779    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c32d0cfd6f6abb7c3db39cdc98f6040183989685f651797cfccbc4225e3e4835"} err="failed to get container status \"c32d0cfd6f6abb7c3db39cdc98f6040183989685f651797cfccbc4225e3e4835\": rpc error: code = NotFound desc = could not find container \"c32d0cfd6f6abb7c3db39cdc98f6040183989685f651797cfccbc4225e3e4835\": container with ID starting with c32d0cfd6f6abb7c3db39cdc98f6040183989685f651797cfccbc4225e3e4835 not found: ID does not exist"
Oct 31 20:19:00.651889 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.651801    4645 scope.go:122] "RemoveContainer" containerID="8c3d4ba6e726f8f9462282daa0b9b123c97dc17a7db9a07361f56f2a5cd4f099"
Oct 31 20:19:00.664424 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.664403    4645 scope.go:122] "RemoveContainer" containerID="8c3d4ba6e726f8f9462282daa0b9b123c97dc17a7db9a07361f56f2a5cd4f099"
Oct 31 20:19:00.665161 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:19:00.665137    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c3d4ba6e726f8f9462282daa0b9b123c97dc17a7db9a07361f56f2a5cd4f099\": container with ID starting with 8c3d4ba6e726f8f9462282daa0b9b123c97dc17a7db9a07361f56f2a5cd4f099 not found: ID does not exist" containerID="8c3d4ba6e726f8f9462282daa0b9b123c97dc17a7db9a07361f56f2a5cd4f099"
Oct 31 20:19:00.665263 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.665166    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c3d4ba6e726f8f9462282daa0b9b123c97dc17a7db9a07361f56f2a5cd4f099"} err="failed to get container status \"8c3d4ba6e726f8f9462282daa0b9b123c97dc17a7db9a07361f56f2a5cd4f099\": rpc error: code = NotFound desc = could not find container \"8c3d4ba6e726f8f9462282daa0b9b123c97dc17a7db9a07361f56f2a5cd4f099\": container with ID starting with 8c3d4ba6e726f8f9462282daa0b9b123c97dc17a7db9a07361f56f2a5cd4f099 not found: ID does not exist"
Oct 31 20:19:00.665263 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.665182    4645 scope.go:122] "RemoveContainer" containerID="8de325a50771f5b9428b038cbdc86d5540d4a51df7f04714151f48dafaca92a8"
Oct 31 20:19:00.667989 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.667965    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6k6cd"]
Oct 31 20:19:00.672563 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.672541    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6k6cd"]
Oct 31 20:19:00.677750 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.677730    4645 scope.go:122] "RemoveContainer" containerID="8de325a50771f5b9428b038cbdc86d5540d4a51df7f04714151f48dafaca92a8"
Oct 31 20:19:00.683461 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.683439    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vj5qn"]
Oct 31 20:19:00.689100 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.688690    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vj5qn"]
Oct 31 20:19:00.700706 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:19:00.700680    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8de325a50771f5b9428b038cbdc86d5540d4a51df7f04714151f48dafaca92a8\": container with ID starting with 8de325a50771f5b9428b038cbdc86d5540d4a51df7f04714151f48dafaca92a8 not found: ID does not exist" containerID="8de325a50771f5b9428b038cbdc86d5540d4a51df7f04714151f48dafaca92a8"
Oct 31 20:19:00.700938 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.700906    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8de325a50771f5b9428b038cbdc86d5540d4a51df7f04714151f48dafaca92a8"} err="failed to get container status \"8de325a50771f5b9428b038cbdc86d5540d4a51df7f04714151f48dafaca92a8\": rpc error: code = NotFound desc = could not find container \"8de325a50771f5b9428b038cbdc86d5540d4a51df7f04714151f48dafaca92a8\": container with ID starting with 8de325a50771f5b9428b038cbdc86d5540d4a51df7f04714151f48dafaca92a8 not found: ID does not exist"
Oct 31 20:19:00.702018 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.701991    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vkszn"]
Oct 31 20:19:00.709175 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.709094    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-vkszn"]
Oct 31 20:19:00.747656 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.747634    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-55znv"
Oct 31 20:19:00.902911 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.902872    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86c63841-b547-43a5-9d82-e1cde5c8f2ed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86c63841-b547-43a5-9d82-e1cde5c8f2ed-workload-certs\") pod \"86c63841-b547-43a5-9d82-e1cde5c8f2ed\" (UID: \"86c63841-b547-43a5-9d82-e1cde5c8f2ed\") "
Oct 31 20:19:00.903047 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.902922    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86c63841-b547-43a5-9d82-e1cde5c8f2ed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86c63841-b547-43a5-9d82-e1cde5c8f2ed-credential-socket\") pod \"86c63841-b547-43a5-9d82-e1cde5c8f2ed\" (UID: \"86c63841-b547-43a5-9d82-e1cde5c8f2ed\") "
Oct 31 20:19:00.903047 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.902958    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86c63841-b547-43a5-9d82-e1cde5c8f2ed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86c63841-b547-43a5-9d82-e1cde5c8f2ed-istio-envoy\") pod \"86c63841-b547-43a5-9d82-e1cde5c8f2ed\" (UID: \"86c63841-b547-43a5-9d82-e1cde5c8f2ed\") "
Oct 31 20:19:00.903047 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.902993    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/86c63841-b547-43a5-9d82-e1cde5c8f2ed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86c63841-b547-43a5-9d82-e1cde5c8f2ed-istio-podinfo\") pod \"86c63841-b547-43a5-9d82-e1cde5c8f2ed\" (UID: \"86c63841-b547-43a5-9d82-e1cde5c8f2ed\") "
Oct 31 20:19:00.903047 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.903030    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86c63841-b547-43a5-9d82-e1cde5c8f2ed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86c63841-b547-43a5-9d82-e1cde5c8f2ed-workload-socket\") pod \"86c63841-b547-43a5-9d82-e1cde5c8f2ed\" (UID: \"86c63841-b547-43a5-9d82-e1cde5c8f2ed\") "
Oct 31 20:19:00.903047 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.903048    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86c63841-b547-43a5-9d82-e1cde5c8f2ed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86c63841-b547-43a5-9d82-e1cde5c8f2ed-istiod-ca-cert\") pod \"86c63841-b547-43a5-9d82-e1cde5c8f2ed\" (UID: \"86c63841-b547-43a5-9d82-e1cde5c8f2ed\") "
Oct 31 20:19:00.903382 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.903097    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86c63841-b547-43a5-9d82-e1cde5c8f2ed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86c63841-b547-43a5-9d82-e1cde5c8f2ed-istio-data\") pod \"86c63841-b547-43a5-9d82-e1cde5c8f2ed\" (UID: \"86c63841-b547-43a5-9d82-e1cde5c8f2ed\") "
Oct 31 20:19:00.903382 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.903135    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86c63841-b547-43a5-9d82-e1cde5c8f2ed-kube-api-access-r9bc2\" (UniqueName: \"kubernetes.io/projected/86c63841-b547-43a5-9d82-e1cde5c8f2ed-kube-api-access-r9bc2\") pod \"86c63841-b547-43a5-9d82-e1cde5c8f2ed\" (UID: \"86c63841-b547-43a5-9d82-e1cde5c8f2ed\") "
Oct 31 20:19:00.903382 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.903170    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86c63841-b547-43a5-9d82-e1cde5c8f2ed-istio-token\" (UniqueName: \"kubernetes.io/projected/86c63841-b547-43a5-9d82-e1cde5c8f2ed-istio-token\") pod \"86c63841-b547-43a5-9d82-e1cde5c8f2ed\" (UID: \"86c63841-b547-43a5-9d82-e1cde5c8f2ed\") "
Oct 31 20:19:00.903382 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.903174    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86c63841-b547-43a5-9d82-e1cde5c8f2ed-workload-certs" pod "86c63841-b547-43a5-9d82-e1cde5c8f2ed" (UID: "86c63841-b547-43a5-9d82-e1cde5c8f2ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:19:00.903382 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.903281    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86c63841-b547-43a5-9d82-e1cde5c8f2ed-credential-socket" pod "86c63841-b547-43a5-9d82-e1cde5c8f2ed" (UID: "86c63841-b547-43a5-9d82-e1cde5c8f2ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:19:00.903382 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.903316    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86c63841-b547-43a5-9d82-e1cde5c8f2ed-workload-socket" pod "86c63841-b547-43a5-9d82-e1cde5c8f2ed" (UID: "86c63841-b547-43a5-9d82-e1cde5c8f2ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:19:00.903736 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.903452    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86c63841-b547-43a5-9d82-e1cde5c8f2ed-istio-data" pod "86c63841-b547-43a5-9d82-e1cde5c8f2ed" (UID: "86c63841-b547-43a5-9d82-e1cde5c8f2ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:19:00.903736 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.903537    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86c63841-b547-43a5-9d82-e1cde5c8f2ed-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:19:00.903736 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.903560    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86c63841-b547-43a5-9d82-e1cde5c8f2ed-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:19:00.903736 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.903574    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86c63841-b547-43a5-9d82-e1cde5c8f2ed-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:19:00.903736 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.903588    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86c63841-b547-43a5-9d82-e1cde5c8f2ed-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:19:00.903736 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.903653    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86c63841-b547-43a5-9d82-e1cde5c8f2ed-istiod-ca-cert" pod "86c63841-b547-43a5-9d82-e1cde5c8f2ed" (UID: "86c63841-b547-43a5-9d82-e1cde5c8f2ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:19:00.905207 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.905183    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86c63841-b547-43a5-9d82-e1cde5c8f2ed-istio-envoy" pod "86c63841-b547-43a5-9d82-e1cde5c8f2ed" (UID: "86c63841-b547-43a5-9d82-e1cde5c8f2ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:19:00.905312 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.905245    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86c63841-b547-43a5-9d82-e1cde5c8f2ed-istio-token" pod "86c63841-b547-43a5-9d82-e1cde5c8f2ed" (UID: "86c63841-b547-43a5-9d82-e1cde5c8f2ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:19:00.905312 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.905267    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/86c63841-b547-43a5-9d82-e1cde5c8f2ed-istio-podinfo" pod "86c63841-b547-43a5-9d82-e1cde5c8f2ed" (UID: "86c63841-b547-43a5-9d82-e1cde5c8f2ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 20:19:00.905437 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.905320    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86c63841-b547-43a5-9d82-e1cde5c8f2ed-kube-api-access-r9bc2" pod "86c63841-b547-43a5-9d82-e1cde5c8f2ed" (UID: "86c63841-b547-43a5-9d82-e1cde5c8f2ed"). InnerVolumeSpecName "kube-api-access-r9bc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:19:00.925166 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:00.925138    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w"
Oct 31 20:19:01.004939 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:01.004911    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86c63841-b547-43a5-9d82-e1cde5c8f2ed-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:19:01.004939 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:01.004936    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86c63841-b547-43a5-9d82-e1cde5c8f2ed-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:19:01.005084 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:01.004951    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86c63841-b547-43a5-9d82-e1cde5c8f2ed-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:19:01.005084 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:01.004965    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9bc2\" (UniqueName: \"kubernetes.io/projected/86c63841-b547-43a5-9d82-e1cde5c8f2ed-kube-api-access-r9bc2\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:19:01.005084 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:01.004979    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86c63841-b547-43a5-9d82-e1cde5c8f2ed-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:19:01.616109 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:01.616075    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0150a41c-8a80-46fb-b668-fcc8a4a48dcd" path="/var/lib/kubelet/pods/0150a41c-8a80-46fb-b668-fcc8a4a48dcd/volumes"
Oct 31 20:19:01.616543 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:01.616527    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bdf127f-3310-4bc8-8f53-6a1e7dda4191" path="/var/lib/kubelet/pods/8bdf127f-3310-4bc8-8f53-6a1e7dda4191/volumes"
Oct 31 20:19:01.616928 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:01.616916    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8fca631-527d-46c0-9197-5edd7762adb9" path="/var/lib/kubelet/pods/a8fca631-527d-46c0-9197-5edd7762adb9/volumes"
Oct 31 20:19:01.652049 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:01.652025    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-55znv"
Oct 31 20:19:01.652170 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:01.652066    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-55znv" event={"ID":"86c63841-b547-43a5-9d82-e1cde5c8f2ed","Type":"ContainerDied","Data":"2929889aca117ae1069c65e128833e9205ff43d055dd60ec479d43021869657b"}
Oct 31 20:19:01.652170 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:01.652118    4645 scope.go:122] "RemoveContainer" containerID="3a3359753776e5adcea55c368b8f64d1c2d4385862dec56548b86a44737b5446"
Oct 31 20:19:01.652397 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:01.652376    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w"
Oct 31 20:19:01.653585 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:01.653361    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w"
Oct 31 20:19:01.695600 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:01.695558    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-55znv"]
Oct 31 20:19:01.708131 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:01.708097    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-55znv"]
Oct 31 20:19:02.074084 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:02.074051    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:02.250239 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:02.250210    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:03.212938 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:03.212899    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-844db7-5ff86b466f-wzmdl"]
Oct 31 20:19:03.213626 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:03.213546    4645 cpu_manager.go:414] "containerMap: removing container" podUID="8bdf127f-3310-4bc8-8f53-6a1e7dda4191" containerName="istio-proxy"
Oct 31 20:19:03.213626 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:03.213576    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bdf127f-3310-4bc8-8f53-6a1e7dda4191" containerName="istio-proxy"
Oct 31 20:19:03.213626 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:03.213595    4645 cpu_manager.go:414] "containerMap: removing container" podUID="a8fca631-527d-46c0-9197-5edd7762adb9" containerName="istio-proxy"
Oct 31 20:19:03.213626 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:03.213603    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8fca631-527d-46c0-9197-5edd7762adb9" containerName="istio-proxy"
Oct 31 20:19:03.213903 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:03.213636    4645 cpu_manager.go:414] "containerMap: removing container" podUID="86c63841-b547-43a5-9d82-e1cde5c8f2ed" containerName="istio-proxy"
Oct 31 20:19:03.213903 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:03.213646    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86c63841-b547-43a5-9d82-e1cde5c8f2ed" containerName="istio-proxy"
Oct 31 20:19:03.213903 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:03.213664    4645 cpu_manager.go:414] "containerMap: removing container" podUID="0150a41c-8a80-46fb-b668-fcc8a4a48dcd" containerName="istio-proxy"
Oct 31 20:19:03.213903 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:03.213672    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0150a41c-8a80-46fb-b668-fcc8a4a48dcd" containerName="istio-proxy"
Oct 31 20:19:03.213903 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:03.213794    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="86c63841-b547-43a5-9d82-e1cde5c8f2ed" containerName="istio-proxy"
Oct 31 20:19:03.213903 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:03.213813    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="0150a41c-8a80-46fb-b668-fcc8a4a48dcd" containerName="istio-proxy"
Oct 31 20:19:03.213903 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:03.213836    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bdf127f-3310-4bc8-8f53-6a1e7dda4191" containerName="istio-proxy"
Oct 31 20:19:03.213903 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:03.213846    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8fca631-527d-46c0-9197-5edd7762adb9" containerName="istio-proxy"
Oct 31 20:19:03.219071 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:03.219046    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-844db7-5ff86b466f-wzmdl"
Oct 31 20:19:03.228555 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:03.228528    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-844db7-5ff86b466f-wzmdl"]
Oct 31 20:19:03.270507 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:03.270479    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:03.293136 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:03.293103    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-844db7-serving-cert\""
Oct 31 20:19:03.328561 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:03.328524    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f67b87d1-b808-4393-9224-1c45bf38a373-metrics-certs\") pod \"maas-api-e2e-worker-single-main-844db7-5ff86b466f-wzmdl\" (UID: \"f67b87d1-b808-4393-9224-1c45bf38a373\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-844db7-5ff86b466f-wzmdl"
Oct 31 20:19:03.328712 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:03.328640    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f67b87d1-b808-4393-9224-1c45bf38a373-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-844db7-5ff86b466f-wzmdl\" (UID: \"f67b87d1-b808-4393-9224-1c45bf38a373\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-844db7-5ff86b466f-wzmdl"
Oct 31 20:19:03.328712 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:03.328675    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f67b87d1-b808-4393-9224-1c45bf38a373-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-844db7-5ff86b466f-wzmdl\" (UID: \"f67b87d1-b808-4393-9224-1c45bf38a373\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-844db7-5ff86b466f-wzmdl"
Oct 31 20:19:03.328848 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:03.328798    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mnqh\" (UniqueName: \"kubernetes.io/projected/f67b87d1-b808-4393-9224-1c45bf38a373-kube-api-access-5mnqh\") pod \"maas-api-e2e-worker-single-main-844db7-5ff86b466f-wzmdl\" (UID: \"f67b87d1-b808-4393-9224-1c45bf38a373\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-844db7-5ff86b466f-wzmdl"
Oct 31 20:19:03.389097 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:03.389055    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:03.395758 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:03.395718    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-844db7-d587bc6mhg"]
Oct 31 20:19:03.399154 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:03.399134    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-844db7-d587bc6mhg"
Oct 31 20:19:03.403503 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:03.403481    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-844db7\""
Oct 31 20:19:03.408732 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:03.408712    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-9b7e7362-dockercfg-hbhhs\""
Oct 31 20:19:03.414039 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:03.414013    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-844db7-d587bc6mhg"]
Oct 31 20:19:03.430077 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:03.430051    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f67b87d1-b808-4393-9224-1c45bf38a373-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-844db7-5ff86b466f-wzmdl\" (UID: \"f67b87d1-b808-4393-9224-1c45bf38a373\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-844db7-5ff86b466f-wzmdl"
Oct 31 20:19:03.430180 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:03.430083    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f67b87d1-b808-4393-9224-1c45bf38a373-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-844db7-5ff86b466f-wzmdl\" (UID: \"f67b87d1-b808-4393-9224-1c45bf38a373\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-844db7-5ff86b466f-wzmdl"
Oct 31 20:19:03.430313 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:03.430291    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mnqh\" (UniqueName: \"kubernetes.io/projected/f67b87d1-b808-4393-9224-1c45bf38a373-kube-api-access-5mnqh\") pod \"maas-api-e2e-worker-single-main-844db7-5ff86b466f-wzmdl\" (UID: \"f67b87d1-b808-4393-9224-1c45bf38a373\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-844db7-5ff86b466f-wzmdl"
Oct 31 20:19:03.430405 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:03.430377    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f67b87d1-b808-4393-9224-1c45bf38a373-metrics-certs\") pod \"maas-api-e2e-worker-single-main-844db7-5ff86b466f-wzmdl\" (UID: \"f67b87d1-b808-4393-9224-1c45bf38a373\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-844db7-5ff86b466f-wzmdl"
Oct 31 20:19:03.430793 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:03.430772    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f67b87d1-b808-4393-9224-1c45bf38a373-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-844db7-5ff86b466f-wzmdl\" (UID: \"f67b87d1-b808-4393-9224-1c45bf38a373\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-844db7-5ff86b466f-wzmdl"
Oct 31 20:19:03.432925 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:03.432897    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f67b87d1-b808-4393-9224-1c45bf38a373-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-844db7-5ff86b466f-wzmdl\" (UID: \"f67b87d1-b808-4393-9224-1c45bf38a373\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-844db7-5ff86b466f-wzmdl"
Oct 31 20:19:03.433067 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:03.433049    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f67b87d1-b808-4393-9224-1c45bf38a373-metrics-certs\") pod \"maas-api-e2e-worker-single-main-844db7-5ff86b466f-wzmdl\" (UID: \"f67b87d1-b808-4393-9224-1c45bf38a373\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-844db7-5ff86b466f-wzmdl"
Oct 31 20:19:03.437152 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:03.437128    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-844db7-6f7649b585pscw"]
Oct 31 20:19:03.441433 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:03.441415    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-844db7-6f7649b585pscw"
Oct 31 20:19:03.443174 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:03.443152    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mnqh\" (UniqueName: \"kubernetes.io/projected/f67b87d1-b808-4393-9224-1c45bf38a373-kube-api-access-5mnqh\") pod \"maas-api-e2e-worker-single-main-844db7-5ff86b466f-wzmdl\" (UID: \"f67b87d1-b808-4393-9224-1c45bf38a373\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-844db7-5ff86b466f-wzmdl"
Oct 31 20:19:03.450903 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:03.450879    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-844db7-6f7649b585pscw"]
Oct 31 20:19:03.503077 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:03.503020    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:03.531869 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:03.531841    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb7bf\" (UniqueName: \"kubernetes.io/projected/d8e68310-9416-42f7-9ec5-f2f3348b8029-kube-api-access-nb7bf\") pod \"payload-pre-processing-e2e-worker-single-main-844db7-d587bc6mhg\" (UID: \"d8e68310-9416-42f7-9ec5-f2f3348b8029\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-844db7-d587bc6mhg"
Oct 31 20:19:03.531995 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:03.531882    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8e68310-9416-42f7-9ec5-f2f3348b8029-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-844db7-d587bc6mhg\" (UID: \"d8e68310-9416-42f7-9ec5-f2f3348b8029\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-844db7-d587bc6mhg"
Oct 31 20:19:03.532073 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:03.532041    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50110354-1374-44e2-aced-b0b068cd1bba-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-844db7-6f7649b585pscw\" (UID: \"50110354-1374-44e2-aced-b0b068cd1bba\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-844db7-6f7649b585pscw"
Oct 31 20:19:03.532136 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:03.532090    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq8t5\" (UniqueName: \"kubernetes.io/projected/50110354-1374-44e2-aced-b0b068cd1bba-kube-api-access-wq8t5\") pod \"payload-processing-e2e-worker-single-main-844db7-6f7649b585pscw\" (UID: \"50110354-1374-44e2-aced-b0b068cd1bba\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-844db7-6f7649b585pscw"
Oct 31 20:19:03.533704 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:03.533680    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-844db7-5ff86b466f-wzmdl"
Oct 31 20:19:03.617012 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:03.616982    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86c63841-b547-43a5-9d82-e1cde5c8f2ed" path="/var/lib/kubelet/pods/86c63841-b547-43a5-9d82-e1cde5c8f2ed/volumes"
Oct 31 20:19:03.633251 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:03.633228    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb7bf\" (UniqueName: \"kubernetes.io/projected/d8e68310-9416-42f7-9ec5-f2f3348b8029-kube-api-access-nb7bf\") pod \"payload-pre-processing-e2e-worker-single-main-844db7-d587bc6mhg\" (UID: \"d8e68310-9416-42f7-9ec5-f2f3348b8029\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-844db7-d587bc6mhg"
Oct 31 20:19:03.633474 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:03.633262    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8e68310-9416-42f7-9ec5-f2f3348b8029-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-844db7-d587bc6mhg\" (UID: \"d8e68310-9416-42f7-9ec5-f2f3348b8029\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-844db7-d587bc6mhg"
Oct 31 20:19:03.633474 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:03.633313    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50110354-1374-44e2-aced-b0b068cd1bba-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-844db7-6f7649b585pscw\" (UID: \"50110354-1374-44e2-aced-b0b068cd1bba\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-844db7-6f7649b585pscw"
Oct 31 20:19:03.633474 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:03.633356    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq8t5\" (UniqueName: \"kubernetes.io/projected/50110354-1374-44e2-aced-b0b068cd1bba-kube-api-access-wq8t5\") pod \"payload-processing-e2e-worker-single-main-844db7-6f7649b585pscw\" (UID: \"50110354-1374-44e2-aced-b0b068cd1bba\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-844db7-6f7649b585pscw"
Oct 31 20:19:03.634091 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:03.634071    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8e68310-9416-42f7-9ec5-f2f3348b8029-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-844db7-d587bc6mhg\" (UID: \"d8e68310-9416-42f7-9ec5-f2f3348b8029\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-844db7-d587bc6mhg"
Oct 31 20:19:03.634135 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:03.634071    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50110354-1374-44e2-aced-b0b068cd1bba-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-844db7-6f7649b585pscw\" (UID: \"50110354-1374-44e2-aced-b0b068cd1bba\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-844db7-6f7649b585pscw"
Oct 31 20:19:03.646964 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:03.646941    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb7bf\" (UniqueName: \"kubernetes.io/projected/d8e68310-9416-42f7-9ec5-f2f3348b8029-kube-api-access-nb7bf\") pod \"payload-pre-processing-e2e-worker-single-main-844db7-d587bc6mhg\" (UID: \"d8e68310-9416-42f7-9ec5-f2f3348b8029\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-844db7-d587bc6mhg"
Oct 31 20:19:03.647123 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:03.647097    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq8t5\" (UniqueName: \"kubernetes.io/projected/50110354-1374-44e2-aced-b0b068cd1bba-kube-api-access-wq8t5\") pod \"payload-processing-e2e-worker-single-main-844db7-6f7649b585pscw\" (UID: \"50110354-1374-44e2-aced-b0b068cd1bba\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-844db7-6f7649b585pscw"
Oct 31 20:19:03.712208 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:03.712180    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-844db7-d587bc6mhg"
Oct 31 20:19:03.766371 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:03.765664    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-844db7-6f7649b585pscw"
Oct 31 20:19:03.840550 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:03.838582    4645 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 20:19:03.840550 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:03.839460    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-844db7-5ff86b466f-wzmdl"]
Oct 31 20:19:03.988858 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:03.988832    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-844db7-d587bc6mhg"]
Oct 31 20:19:03.990918 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:19:03.990893    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8e68310_9416_42f7_9ec5_f2f3348b8029.slice/crio-f6dea9fe6c37d41af72c0de865280953cd1d53a5d4df4598a80ce5917f69cb39 WatchSource:0}: Error finding container f6dea9fe6c37d41af72c0de865280953cd1d53a5d4df4598a80ce5917f69cb39: Status 404 returned error can't find the container with id f6dea9fe6c37d41af72c0de865280953cd1d53a5d4df4598a80ce5917f69cb39
Oct 31 20:19:04.053876 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:04.053851    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-844db7-6f7649b585pscw"]
Oct 31 20:19:04.056399 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:19:04.056319    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50110354_1374_44e2_aced_b0b068cd1bba.slice/crio-0c2a4ff8961fa296f66bee61022ec132fe3bb95994453709a395ca29089437d1 WatchSource:0}: Error finding container 0c2a4ff8961fa296f66bee61022ec132fe3bb95994453709a395ca29089437d1: Status 404 returned error can't find the container with id 0c2a4ff8961fa296f66bee61022ec132fe3bb95994453709a395ca29089437d1
Oct 31 20:19:04.301086 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:04.301012    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:04.667942 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:04.667906    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-844db7-6f7649b585pscw" event={"ID":"50110354-1374-44e2-aced-b0b068cd1bba","Type":"ContainerStarted","Data":"ace5fb30bec34aa7b38a8320dddaf0732c4ae27bed4bd611370c390bff1cb138"}
Oct 31 20:19:04.668132 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:04.667970    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-844db7-6f7649b585pscw"
Oct 31 20:19:04.668132 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:04.667994    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-844db7-6f7649b585pscw" event={"ID":"50110354-1374-44e2-aced-b0b068cd1bba","Type":"ContainerStarted","Data":"0c2a4ff8961fa296f66bee61022ec132fe3bb95994453709a395ca29089437d1"}
Oct 31 20:19:04.669404 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:04.669375    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-844db7-d587bc6mhg" event={"ID":"d8e68310-9416-42f7-9ec5-f2f3348b8029","Type":"ContainerStarted","Data":"d3cc5fd9d2e8c721f37d963b26864178acca8e7c80332ff2594309f36fa9aaa8"}
Oct 31 20:19:04.669544 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:04.669438    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-844db7-d587bc6mhg" event={"ID":"d8e68310-9416-42f7-9ec5-f2f3348b8029","Type":"ContainerStarted","Data":"f6dea9fe6c37d41af72c0de865280953cd1d53a5d4df4598a80ce5917f69cb39"}
Oct 31 20:19:04.669544 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:04.669470    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-844db7-d587bc6mhg"
Oct 31 20:19:04.670847 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:04.670824    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-844db7-5ff86b466f-wzmdl" event={"ID":"f67b87d1-b808-4393-9224-1c45bf38a373","Type":"ContainerStarted","Data":"cfb89fbb5424676bc488418f6948552e0fc21a8496294e419a715dd3eb290fd1"}
Oct 31 20:19:04.670957 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:04.670868    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-844db7-5ff86b466f-wzmdl" event={"ID":"f67b87d1-b808-4393-9224-1c45bf38a373","Type":"ContainerStarted","Data":"ee0fa4ff41ec434a8c05466a7de8f33e6c5a4f3a9b90bd0d52b157c41c1422f6"}
Oct 31 20:19:04.671007 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:04.670955    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-844db7-5ff86b466f-wzmdl"
Oct 31 20:19:04.687566 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:04.687517    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-844db7-6f7649b585pscw" podStartSLOduration=1.687500609 podStartE2EDuration="1.687500609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:19:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:19:04.684732086 +0000 UTC m=+3575.257966230" watchObservedRunningTime="2026-10-31 20:19:04.687500609 +0000 UTC m=+3575.260734738"
Oct 31 20:19:04.703736 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:04.703692    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-844db7-d587bc6mhg" podStartSLOduration=1.703673954 podStartE2EDuration="1.703673954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:19:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:19:04.702421075 +0000 UTC m=+3575.275655204" watchObservedRunningTime="2026-10-31 20:19:04.703673954 +0000 UTC m=+3575.276908085"
Oct 31 20:19:04.731951 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:04.731907    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-844db7-5ff86b466f-wzmdl" podStartSLOduration=1.350004866 podStartE2EDuration="1.73189378s" totalImagesPullingTime="381.888914ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:19:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:19:04.729631401 +0000 UTC m=+3575.302865529" watchObservedRunningTime="2026-10-31 20:19:04.73189378 +0000 UTC m=+3575.305128276"
Oct 31 20:19:05.135007 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:05.134920    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:05.943347 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:05.943294    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:06.244850 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:19:06.244754    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 20:19:06.669349 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:06.666914    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:07.480154 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:07.480122    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:08.273352 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:08.267140    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:08.991901 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:08.991870    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:09.753096 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:09.753067    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:10.544765 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:10.544727    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:10.685482 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:10.685451    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-844db7-5ff86b466f-wzmdl"
Oct 31 20:19:11.253542 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:11.253501    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:12.204879 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:12.204844    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:12.938997 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:12.938959    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:13.668156 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:13.668123    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:14.462877 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:14.462838    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:15.248866 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:15.248827    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:15.680307 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:15.680281    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-844db7-d587bc6mhg"
Oct 31 20:19:15.680529 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:15.680368    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-844db7-6f7649b585pscw"
Oct 31 20:19:15.731539 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:15.731507    4645 generic.go:356] "Generic (PLEG): container finished" podID="672d276a-4c71-4742-82f6-f5c3dc95e072" containerID="20b1df8596b5b7fb6369a4036e590f2f4b5f358b74c5934466db94e801e83c41" exitCode=0
Oct 31 20:19:15.731730 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:15.731589    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w" event={"ID":"672d276a-4c71-4742-82f6-f5c3dc95e072","Type":"ContainerDied","Data":"20b1df8596b5b7fb6369a4036e590f2f4b5f358b74c5934466db94e801e83c41"}
Oct 31 20:19:15.732013 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:15.731997    4645 scope.go:122] "RemoveContainer" containerID="20b1df8596b5b7fb6369a4036e590f2f4b5f358b74c5934466db94e801e83c41"
Oct 31 20:19:15.733319 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:15.733281    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:19:15.733418 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:15.733397    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:19:15.733482 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:15.733460    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:19:16.152103 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:16.152004    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:16.652894 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:16.652859    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w"
Oct 31 20:19:16.738726 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:16.738689    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w" event={"ID":"672d276a-4c71-4742-82f6-f5c3dc95e072","Type":"ContainerStarted","Data":"1670d35c1aefd30aa5445eeffdddfb1cfd992eb47dc592a912e6974972ecb7a7"}
Oct 31 20:19:16.920557 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:16.920525    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w"
Oct 31 20:19:16.921790 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:16.921760    4645 patch_prober.go:29] interesting pod/e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.67:15021/healthz/ready\": dial tcp 10.133.1.67:15021: connect: connection refused" start-of-body=
Oct 31 20:19:16.921933 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:16.921809    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w" podUID="672d276a-4c71-4742-82f6-f5c3dc95e072" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.67:15021/healthz/ready\": dial tcp 10.133.1.67:15021: connect: connection refused"
Oct 31 20:19:16.950120 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:16.950092    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:17.078353 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:17.077837    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:17.696787 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:17.696744    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:17.880968 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:17.880919    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:17.921183 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:17.921149    4645 patch_prober.go:29] interesting pod/e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.67:15021/healthz/ready\": dial tcp 10.133.1.67:15021: connect: connection refused" start-of-body=
Oct 31 20:19:17.921386 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:17.921228    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w" podUID="672d276a-4c71-4742-82f6-f5c3dc95e072" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.67:15021/healthz/ready\": dial tcp 10.133.1.67:15021: connect: connection refused"
Oct 31 20:19:18.007900 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:18.007821    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:18.802474 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:18.802439    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:18.920672 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:18.920631    4645 patch_prober.go:29] interesting pod/e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.67:15021/healthz/ready\": dial tcp 10.133.1.67:15021: connect: connection refused" start-of-body=
Oct 31 20:19:18.920884 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:18.920696    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w" podUID="672d276a-4c71-4742-82f6-f5c3dc95e072" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.67:15021/healthz/ready\": dial tcp 10.133.1.67:15021: connect: connection refused"
Oct 31 20:19:18.981215 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:18.981174    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:19.100195 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:19.100115    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:19.881228 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:19.881190    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:19.920985 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:19.920951    4645 patch_prober.go:29] interesting pod/e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.67:15021/healthz/ready\": dial tcp 10.133.1.67:15021: connect: connection refused" start-of-body=
Oct 31 20:19:19.921128 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:19.920999    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w" podUID="672d276a-4c71-4742-82f6-f5c3dc95e072" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.67:15021/healthz/ready\": dial tcp 10.133.1.67:15021: connect: connection refused"
Oct 31 20:19:20.079352 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:20.078709    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:20.921189 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:20.921123    4645 patch_prober.go:29] interesting pod/e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.67:15021/healthz/ready\": dial tcp 10.133.1.67:15021: connect: connection refused" start-of-body=
Oct 31 20:19:20.921189 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:20.921188    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w" podUID="672d276a-4c71-4742-82f6-f5c3dc95e072" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.67:15021/healthz/ready\": dial tcp 10.133.1.67:15021: connect: connection refused"
Oct 31 20:19:21.083458 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:21.083419    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:21.986846 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:21.986804    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:22.167324 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:22.167291    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w"
Oct 31 20:19:22.167768 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:22.167730    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w"
Oct 31 20:19:22.168746 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:22.168715    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w"
Oct 31 20:19:22.649793 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:22.649756    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:23.452902 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:23.452858    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:24.157422 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:24.157351    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:24.903755 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:24.903707    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:25.248827 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:25.248788    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6rwmb"]
Oct 31 20:19:25.252675 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:25.252653    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6rwmb"
Oct 31 20:19:25.263868 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:25.263813    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-jkgv9\""
Oct 31 20:19:25.264743 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:25.264716    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-68qbd"]
Oct 31 20:19:25.268424 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:25.268401    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-68qbd"
Oct 31 20:19:25.270938 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:25.270909    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6rwmb"]
Oct 31 20:19:25.282452 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:25.282423    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-68qbd"]
Oct 31 20:19:25.356568 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:25.356543    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a217b30-54c0-431f-90bc-0681901272f4-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-68qbd\" (UID: \"3a217b30-54c0-431f-90bc-0681901272f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-68qbd"
Oct 31 20:19:25.356707 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:25.356575    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsrk7\" (UniqueName: \"kubernetes.io/projected/b92c4b09-c2c0-4602-af78-2f410c843118-kube-api-access-qsrk7\") pod \"maas-default-gateway-openshift-default-75d7c785f-6rwmb\" (UID: \"b92c4b09-c2c0-4602-af78-2f410c843118\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6rwmb"
Oct 31 20:19:25.356707 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:25.356598    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a217b30-54c0-431f-90bc-0681901272f4-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-68qbd\" (UID: \"3a217b30-54c0-431f-90bc-0681901272f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-68qbd"
Oct 31 20:19:25.356832 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:25.356709    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a217b30-54c0-431f-90bc-0681901272f4-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-68qbd\" (UID: \"3a217b30-54c0-431f-90bc-0681901272f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-68qbd"
Oct 31 20:19:25.356832 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:25.356754    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b92c4b09-c2c0-4602-af78-2f410c843118-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-6rwmb\" (UID: \"b92c4b09-c2c0-4602-af78-2f410c843118\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6rwmb"
Oct 31 20:19:25.356832 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:25.356822    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b92c4b09-c2c0-4602-af78-2f410c843118-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-6rwmb\" (UID: \"b92c4b09-c2c0-4602-af78-2f410c843118\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6rwmb"
Oct 31 20:19:25.356954 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:25.356851    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b92c4b09-c2c0-4602-af78-2f410c843118-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-6rwmb\" (UID: \"b92c4b09-c2c0-4602-af78-2f410c843118\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6rwmb"
Oct 31 20:19:25.356954 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:25.356874    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b92c4b09-c2c0-4602-af78-2f410c843118-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-6rwmb\" (UID: \"b92c4b09-c2c0-4602-af78-2f410c843118\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6rwmb"
Oct 31 20:19:25.356954 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:25.356890    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b92c4b09-c2c0-4602-af78-2f410c843118-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-6rwmb\" (UID: \"b92c4b09-c2c0-4602-af78-2f410c843118\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6rwmb"
Oct 31 20:19:25.356954 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:25.356909    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a217b30-54c0-431f-90bc-0681901272f4-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-68qbd\" (UID: \"3a217b30-54c0-431f-90bc-0681901272f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-68qbd"
Oct 31 20:19:25.357228 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:25.356953    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b92c4b09-c2c0-4602-af78-2f410c843118-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-6rwmb\" (UID: \"b92c4b09-c2c0-4602-af78-2f410c843118\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6rwmb"
Oct 31 20:19:25.357228 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:25.357007    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a217b30-54c0-431f-90bc-0681901272f4-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-68qbd\" (UID: \"3a217b30-54c0-431f-90bc-0681901272f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-68qbd"
Oct 31 20:19:25.357228 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:25.357081    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a217b30-54c0-431f-90bc-0681901272f4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-68qbd\" (UID: \"3a217b30-54c0-431f-90bc-0681901272f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-68qbd"
Oct 31 20:19:25.357228 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:25.357120    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdvqh\" (UniqueName: \"kubernetes.io/projected/3a217b30-54c0-431f-90bc-0681901272f4-kube-api-access-jdvqh\") pod \"maas-default-gateway-openshift-default-75d7c785f-68qbd\" (UID: \"3a217b30-54c0-431f-90bc-0681901272f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-68qbd"
Oct 31 20:19:25.357228 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:25.357159    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b92c4b09-c2c0-4602-af78-2f410c843118-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-6rwmb\" (UID: \"b92c4b09-c2c0-4602-af78-2f410c843118\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6rwmb"
Oct 31 20:19:25.357228 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:25.357193    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a217b30-54c0-431f-90bc-0681901272f4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-68qbd\" (UID: \"3a217b30-54c0-431f-90bc-0681901272f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-68qbd"
Oct 31 20:19:25.357495 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:25.357229    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b92c4b09-c2c0-4602-af78-2f410c843118-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-6rwmb\" (UID: \"b92c4b09-c2c0-4602-af78-2f410c843118\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6rwmb"
Oct 31 20:19:25.357495 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:25.357279    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a217b30-54c0-431f-90bc-0681901272f4-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-68qbd\" (UID: \"3a217b30-54c0-431f-90bc-0681901272f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-68qbd"
Oct 31 20:19:25.458569 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:25.458540    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b92c4b09-c2c0-4602-af78-2f410c843118-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-6rwmb\" (UID: \"b92c4b09-c2c0-4602-af78-2f410c843118\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6rwmb"
Oct 31 20:19:25.458682 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:25.458580    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a217b30-54c0-431f-90bc-0681901272f4-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-68qbd\" (UID: \"3a217b30-54c0-431f-90bc-0681901272f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-68qbd"
Oct 31 20:19:25.458682 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:25.458604    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b92c4b09-c2c0-4602-af78-2f410c843118-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-6rwmb\" (UID: \"b92c4b09-c2c0-4602-af78-2f410c843118\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6rwmb"
Oct 31 20:19:25.458682 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:25.458633    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a217b30-54c0-431f-90bc-0681901272f4-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-68qbd\" (UID: \"3a217b30-54c0-431f-90bc-0681901272f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-68qbd"
Oct 31 20:19:25.458682 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:25.458677    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a217b30-54c0-431f-90bc-0681901272f4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-68qbd\" (UID: \"3a217b30-54c0-431f-90bc-0681901272f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-68qbd"
Oct 31 20:19:25.459461 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:25.458708    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdvqh\" (UniqueName: \"kubernetes.io/projected/3a217b30-54c0-431f-90bc-0681901272f4-kube-api-access-jdvqh\") pod \"maas-default-gateway-openshift-default-75d7c785f-68qbd\" (UID: \"3a217b30-54c0-431f-90bc-0681901272f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-68qbd"
Oct 31 20:19:25.459461 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:25.458742    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b92c4b09-c2c0-4602-af78-2f410c843118-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-6rwmb\" (UID: \"b92c4b09-c2c0-4602-af78-2f410c843118\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6rwmb"
Oct 31 20:19:25.459461 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:25.458772    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a217b30-54c0-431f-90bc-0681901272f4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-68qbd\" (UID: \"3a217b30-54c0-431f-90bc-0681901272f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-68qbd"
Oct 31 20:19:25.459461 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:25.458796    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b92c4b09-c2c0-4602-af78-2f410c843118-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-6rwmb\" (UID: \"b92c4b09-c2c0-4602-af78-2f410c843118\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6rwmb"
Oct 31 20:19:25.459461 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:25.459000    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b92c4b09-c2c0-4602-af78-2f410c843118-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-6rwmb\" (UID: \"b92c4b09-c2c0-4602-af78-2f410c843118\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6rwmb"
Oct 31 20:19:25.459461 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:25.459280    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a217b30-54c0-431f-90bc-0681901272f4-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-68qbd\" (UID: \"3a217b30-54c0-431f-90bc-0681901272f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-68qbd"
Oct 31 20:19:25.459832 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:25.459540    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a217b30-54c0-431f-90bc-0681901272f4-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-68qbd\" (UID: \"3a217b30-54c0-431f-90bc-0681901272f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-68qbd"
Oct 31 20:19:25.459832 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:25.459600    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a217b30-54c0-431f-90bc-0681901272f4-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-68qbd\" (UID: \"3a217b30-54c0-431f-90bc-0681901272f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-68qbd"
Oct 31 20:19:25.459832 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:25.459633    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsrk7\" (UniqueName: \"kubernetes.io/projected/b92c4b09-c2c0-4602-af78-2f410c843118-kube-api-access-qsrk7\") pod \"maas-default-gateway-openshift-default-75d7c785f-6rwmb\" (UID: \"b92c4b09-c2c0-4602-af78-2f410c843118\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6rwmb"
Oct 31 20:19:25.459832 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:25.459652    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a217b30-54c0-431f-90bc-0681901272f4-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-68qbd\" (UID: \"3a217b30-54c0-431f-90bc-0681901272f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-68qbd"
Oct 31 20:19:25.459832 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:25.459675    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a217b30-54c0-431f-90bc-0681901272f4-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-68qbd\" (UID: \"3a217b30-54c0-431f-90bc-0681901272f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-68qbd"
Oct 31 20:19:25.459832 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:25.459730    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a217b30-54c0-431f-90bc-0681901272f4-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-68qbd\" (UID: \"3a217b30-54c0-431f-90bc-0681901272f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-68qbd"
Oct 31 20:19:25.459832 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:25.459763    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b92c4b09-c2c0-4602-af78-2f410c843118-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-6rwmb\" (UID: \"b92c4b09-c2c0-4602-af78-2f410c843118\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6rwmb"
Oct 31 20:19:25.459832 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:25.459794    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b92c4b09-c2c0-4602-af78-2f410c843118-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-6rwmb\" (UID: \"b92c4b09-c2c0-4602-af78-2f410c843118\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6rwmb"
Oct 31 20:19:25.459832 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:25.459825    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b92c4b09-c2c0-4602-af78-2f410c843118-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-6rwmb\" (UID: \"b92c4b09-c2c0-4602-af78-2f410c843118\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6rwmb"
Oct 31 20:19:25.460378 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:25.459854    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b92c4b09-c2c0-4602-af78-2f410c843118-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-6rwmb\" (UID: \"b92c4b09-c2c0-4602-af78-2f410c843118\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6rwmb"
Oct 31 20:19:25.460378 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:25.460078    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a217b30-54c0-431f-90bc-0681901272f4-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-68qbd\" (UID: \"3a217b30-54c0-431f-90bc-0681901272f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-68qbd"
Oct 31 20:19:25.460378 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:25.460165    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a217b30-54c0-431f-90bc-0681901272f4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-68qbd\" (UID: \"3a217b30-54c0-431f-90bc-0681901272f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-68qbd"
Oct 31 20:19:25.460378 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:25.460179    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b92c4b09-c2c0-4602-af78-2f410c843118-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-6rwmb\" (UID: \"b92c4b09-c2c0-4602-af78-2f410c843118\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6rwmb"
Oct 31 20:19:25.460378 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:25.460206    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b92c4b09-c2c0-4602-af78-2f410c843118-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-6rwmb\" (UID: \"b92c4b09-c2c0-4602-af78-2f410c843118\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6rwmb"
Oct 31 20:19:25.460698 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:25.460385    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a217b30-54c0-431f-90bc-0681901272f4-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-68qbd\" (UID: \"3a217b30-54c0-431f-90bc-0681901272f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-68qbd"
Oct 31 20:19:25.460698 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:25.460571    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b92c4b09-c2c0-4602-af78-2f410c843118-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-6rwmb\" (UID: \"b92c4b09-c2c0-4602-af78-2f410c843118\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6rwmb"
Oct 31 20:19:25.460826 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:25.460776    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b92c4b09-c2c0-4602-af78-2f410c843118-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-6rwmb\" (UID: \"b92c4b09-c2c0-4602-af78-2f410c843118\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6rwmb"
Oct 31 20:19:25.461249 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:25.461224    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b92c4b09-c2c0-4602-af78-2f410c843118-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-6rwmb\" (UID: \"b92c4b09-c2c0-4602-af78-2f410c843118\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6rwmb"
Oct 31 20:19:25.462060 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:25.462039    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a217b30-54c0-431f-90bc-0681901272f4-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-68qbd\" (UID: \"3a217b30-54c0-431f-90bc-0681901272f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-68qbd"
Oct 31 20:19:25.462386 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:25.462365    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b92c4b09-c2c0-4602-af78-2f410c843118-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-6rwmb\" (UID: \"b92c4b09-c2c0-4602-af78-2f410c843118\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6rwmb"
Oct 31 20:19:25.462757 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:25.462740    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a217b30-54c0-431f-90bc-0681901272f4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-68qbd\" (UID: \"3a217b30-54c0-431f-90bc-0681901272f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-68qbd"
Oct 31 20:19:25.470984 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:25.470955    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsrk7\" (UniqueName: \"kubernetes.io/projected/b92c4b09-c2c0-4602-af78-2f410c843118-kube-api-access-qsrk7\") pod \"maas-default-gateway-openshift-default-75d7c785f-6rwmb\" (UID: \"b92c4b09-c2c0-4602-af78-2f410c843118\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6rwmb"
Oct 31 20:19:25.471128 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:25.471003    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdvqh\" (UniqueName: \"kubernetes.io/projected/3a217b30-54c0-431f-90bc-0681901272f4-kube-api-access-jdvqh\") pod \"maas-default-gateway-openshift-default-75d7c785f-68qbd\" (UID: \"3a217b30-54c0-431f-90bc-0681901272f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-68qbd"
Oct 31 20:19:25.471702 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:25.471683    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a217b30-54c0-431f-90bc-0681901272f4-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-68qbd\" (UID: \"3a217b30-54c0-431f-90bc-0681901272f4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-68qbd"
Oct 31 20:19:25.473273 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:25.473247    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b92c4b09-c2c0-4602-af78-2f410c843118-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-6rwmb\" (UID: \"b92c4b09-c2c0-4602-af78-2f410c843118\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6rwmb"
Oct 31 20:19:25.567726 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:25.567672    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6rwmb"
Oct 31 20:19:25.583800 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:25.583779    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-68qbd"
Oct 31 20:19:25.778959 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:25.778928    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:25.849887 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:25.849859    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6rwmb"]
Oct 31 20:19:25.852069 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:19:25.852042    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb92c4b09_c2c0_4602_af78_2f410c843118.slice/crio-2446edfeda56dcb6009743366dc63ffa30e36e9cea742e73fdf8cf0c909d948b WatchSource:0}: Error finding container 2446edfeda56dcb6009743366dc63ffa30e36e9cea742e73fdf8cf0c909d948b: Status 404 returned error can't find the container with id 2446edfeda56dcb6009743366dc63ffa30e36e9cea742e73fdf8cf0c909d948b
Oct 31 20:19:25.854167 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:25.854137    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:19:25.854240 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:25.854201    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:19:25.854240 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:25.854235    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:19:25.877713 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:25.877692    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-68qbd"]
Oct 31 20:19:25.883275 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:19:25.883249    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a217b30_54c0_431f_90bc_0681901272f4.slice/crio-ce7887fe553682c5dceb7a542bb0ce5ba3bdc252627c449eca90dc9e9bb37077 WatchSource:0}: Error finding container ce7887fe553682c5dceb7a542bb0ce5ba3bdc252627c449eca90dc9e9bb37077: Status 404 returned error can't find the container with id ce7887fe553682c5dceb7a542bb0ce5ba3bdc252627c449eca90dc9e9bb37077
Oct 31 20:19:25.885188 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:25.885163    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:19:25.885248 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:25.885216    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:19:25.885289 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:25.885246    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:19:26.721079 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:26.720640    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:26.818622 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:26.818575    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6rwmb" event={"ID":"b92c4b09-c2c0-4602-af78-2f410c843118","Type":"ContainerStarted","Data":"1ffeb4281c7f76e75277a131babb4052cf14443f4fe57a15f5ea9d012bc0b2de"}
Oct 31 20:19:26.818847 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:26.818649    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6rwmb" event={"ID":"b92c4b09-c2c0-4602-af78-2f410c843118","Type":"ContainerStarted","Data":"2446edfeda56dcb6009743366dc63ffa30e36e9cea742e73fdf8cf0c909d948b"}
Oct 31 20:19:26.820293 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:26.820266    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-68qbd" event={"ID":"3a217b30-54c0-431f-90bc-0681901272f4","Type":"ContainerStarted","Data":"0f55a00f888b17cbce28c912e85ee05389ce8655f75a493b26072ae703ade7ed"}
Oct 31 20:19:26.820399 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:26.820309    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-68qbd" event={"ID":"3a217b30-54c0-431f-90bc-0681901272f4","Type":"ContainerStarted","Data":"ce7887fe553682c5dceb7a542bb0ce5ba3bdc252627c449eca90dc9e9bb37077"}
Oct 31 20:19:26.845689 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:26.845642    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6rwmb" podStartSLOduration=1.845625648 podStartE2EDuration="1.845625648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:19:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:19:26.844229518 +0000 UTC m=+3597.417463662" watchObservedRunningTime="2026-10-31 20:19:26.845625648 +0000 UTC m=+3597.418859778"
Oct 31 20:19:26.878353 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:26.877644    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-68qbd" podStartSLOduration=1.8776281099999999 podStartE2EDuration="1.87762811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:19:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:19:26.876792566 +0000 UTC m=+3597.450026695" watchObservedRunningTime="2026-10-31 20:19:26.87762811 +0000 UTC m=+3597.450862239"
Oct 31 20:19:27.465352 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:27.465300    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:27.571154 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:27.571126    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6rwmb"
Oct 31 20:19:27.571553 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:27.571528    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-6rwmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.71:15021/healthz/ready\": dial tcp 10.133.1.71:15021: connect: connection refused" start-of-body=
Oct 31 20:19:27.571755 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:27.571729    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6rwmb" podUID="b92c4b09-c2c0-4602-af78-2f410c843118" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.71:15021/healthz/ready\": dial tcp 10.133.1.71:15021: connect: connection refused"
Oct 31 20:19:27.587411 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:27.587386    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-68qbd"
Oct 31 20:19:27.592512 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:27.592408    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-68qbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.72:15021/healthz/ready\": dial tcp 10.133.1.72:15021: connect: connection refused" start-of-body=
Oct 31 20:19:27.592512 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:27.592452    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-68qbd" podUID="3a217b30-54c0-431f-90bc-0681901272f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.72:15021/healthz/ready\": dial tcp 10.133.1.72:15021: connect: connection refused"
Oct 31 20:19:28.341776 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:28.341738    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:28.572350 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:28.571687    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-6rwmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.71:15021/healthz/ready\": dial tcp 10.133.1.71:15021: connect: connection refused" start-of-body=
Oct 31 20:19:28.572350 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:28.571748    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6rwmb" podUID="b92c4b09-c2c0-4602-af78-2f410c843118" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.71:15021/healthz/ready\": dial tcp 10.133.1.71:15021: connect: connection refused"
Oct 31 20:19:28.584697 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:28.584664    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-68qbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.72:15021/healthz/ready\": dial tcp 10.133.1.72:15021: connect: connection refused" start-of-body=
Oct 31 20:19:28.584935 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:28.584911    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-68qbd" podUID="3a217b30-54c0-431f-90bc-0681901272f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.72:15021/healthz/ready\": dial tcp 10.133.1.72:15021: connect: connection refused"
Oct 31 20:19:29.047811 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:29.047745    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:29.581364 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:29.578687    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-6rwmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.71:15021/healthz/ready\": dial tcp 10.133.1.71:15021: connect: connection refused" start-of-body=
Oct 31 20:19:29.581364 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:29.578741    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6rwmb" podUID="b92c4b09-c2c0-4602-af78-2f410c843118" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.71:15021/healthz/ready\": dial tcp 10.133.1.71:15021: connect: connection refused"
Oct 31 20:19:29.600350 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:29.591151    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-68qbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.72:15021/healthz/ready\": dial tcp 10.133.1.72:15021: connect: connection refused" start-of-body=
Oct 31 20:19:29.600350 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:29.591199    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-68qbd" podUID="3a217b30-54c0-431f-90bc-0681901272f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.72:15021/healthz/ready\": dial tcp 10.133.1.72:15021: connect: connection refused"
Oct 31 20:19:29.740105 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:29.740061    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:30.509356 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:30.509070    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:30.584361 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:30.583485    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-6rwmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.71:15021/healthz/ready\": dial tcp 10.133.1.71:15021: connect: connection refused" start-of-body=
Oct 31 20:19:30.584361 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:30.583575    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6rwmb" podUID="b92c4b09-c2c0-4602-af78-2f410c843118" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.71:15021/healthz/ready\": dial tcp 10.133.1.71:15021: connect: connection refused"
Oct 31 20:19:30.599352 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:30.595439    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-68qbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.72:15021/healthz/ready\": dial tcp 10.133.1.72:15021: connect: connection refused" start-of-body=
Oct 31 20:19:30.599352 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:30.595495    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-68qbd" podUID="3a217b30-54c0-431f-90bc-0681901272f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.72:15021/healthz/ready\": dial tcp 10.133.1.72:15021: connect: connection refused"
Oct 31 20:19:31.243479 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:31.243443    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:31.575407 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:31.571694    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-6rwmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.71:15021/healthz/ready\": dial tcp 10.133.1.71:15021: connect: connection refused" start-of-body=
Oct 31 20:19:31.575407 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:31.571756    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6rwmb" podUID="b92c4b09-c2c0-4602-af78-2f410c843118" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.71:15021/healthz/ready\": dial tcp 10.133.1.71:15021: connect: connection refused"
Oct 31 20:19:31.593350 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:31.587630    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-68qbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.72:15021/healthz/ready\": dial tcp 10.133.1.72:15021: connect: connection refused" start-of-body=
Oct 31 20:19:31.593350 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:31.587686    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-68qbd" podUID="3a217b30-54c0-431f-90bc-0681901272f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.72:15021/healthz/ready\": dial tcp 10.133.1.72:15021: connect: connection refused"
Oct 31 20:19:32.046008 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:32.045971    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:32.568825 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:32.568785    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-6rwmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.71:15021/healthz/ready\": dial tcp 10.133.1.71:15021: connect: connection refused" start-of-body=
Oct 31 20:19:32.569014 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:32.568854    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6rwmb" podUID="b92c4b09-c2c0-4602-af78-2f410c843118" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.71:15021/healthz/ready\": dial tcp 10.133.1.71:15021: connect: connection refused"
Oct 31 20:19:32.584920 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:32.584883    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-68qbd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.72:15021/healthz/ready\": dial tcp 10.133.1.72:15021: connect: connection refused" start-of-body=
Oct 31 20:19:32.585065 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:32.584948    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-68qbd" podUID="3a217b30-54c0-431f-90bc-0681901272f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.72:15021/healthz/ready\": dial tcp 10.133.1.72:15021: connect: connection refused"
Oct 31 20:19:32.850563 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:32.850528    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:33.561971 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:33.561930    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:34.139475 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:34.139430    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6rwmb"
Oct 31 20:19:34.140104 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:34.139964    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6rwmb"
Oct 31 20:19:34.140731 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:34.140703    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6rwmb"
Oct 31 20:19:34.151816 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:34.151789    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-68qbd"
Oct 31 20:19:34.152186 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:34.152165    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-68qbd"
Oct 31 20:19:34.153896 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:34.153846    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-68qbd"
Oct 31 20:19:34.281393 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:34.281354    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:34.966969 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:34.966938    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:35.739384 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:35.739347    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:36.245350 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:19:36.245286    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 20:19:36.451760 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:36.451732    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:37.343706 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:37.343666    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:38.168234 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:38.168200    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:38.855558 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:38.855521    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:39.550402 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:39.550361    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:40.244504 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:40.244465    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lwvpn"]
Oct 31 20:19:40.248577 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:40.248550    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lwvpn"
Oct 31 20:19:40.263296 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:40.263262    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lwvpn"]
Oct 31 20:19:40.273899 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:40.273871    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t27sn"]
Oct 31 20:19:40.277517 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:40.277493    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t27sn"
Oct 31 20:19:40.294115 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:40.294078    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t27sn"]
Oct 31 20:19:40.353481 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:40.353454    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d401fef-1568-4d16-88fa-92aeaaea68b0-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-lwvpn\" (UID: \"3d401fef-1568-4d16-88fa-92aeaaea68b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lwvpn"
Oct 31 20:19:40.353592 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:40.353490    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d401fef-1568-4d16-88fa-92aeaaea68b0-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-lwvpn\" (UID: \"3d401fef-1568-4d16-88fa-92aeaaea68b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lwvpn"
Oct 31 20:19:40.353592 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:40.353511    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3994be2-19d9-4e2d-b3ce-03f63d2c9c32-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-t27sn\" (UID: \"f3994be2-19d9-4e2d-b3ce-03f63d2c9c32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t27sn"
Oct 31 20:19:40.353681 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:40.353575    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3994be2-19d9-4e2d-b3ce-03f63d2c9c32-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-t27sn\" (UID: \"f3994be2-19d9-4e2d-b3ce-03f63d2c9c32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t27sn"
Oct 31 20:19:40.353681 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:40.353670    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3994be2-19d9-4e2d-b3ce-03f63d2c9c32-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-t27sn\" (UID: \"f3994be2-19d9-4e2d-b3ce-03f63d2c9c32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t27sn"
Oct 31 20:19:40.353784 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:40.353707    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3994be2-19d9-4e2d-b3ce-03f63d2c9c32-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-t27sn\" (UID: \"f3994be2-19d9-4e2d-b3ce-03f63d2c9c32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t27sn"
Oct 31 20:19:40.353784 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:40.353735    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3994be2-19d9-4e2d-b3ce-03f63d2c9c32-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-t27sn\" (UID: \"f3994be2-19d9-4e2d-b3ce-03f63d2c9c32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t27sn"
Oct 31 20:19:40.353874 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:40.353815    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh6gp\" (UniqueName: \"kubernetes.io/projected/3d401fef-1568-4d16-88fa-92aeaaea68b0-kube-api-access-hh6gp\") pod \"maas-default-gateway-openshift-default-75d7c785f-lwvpn\" (UID: \"3d401fef-1568-4d16-88fa-92aeaaea68b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lwvpn"
Oct 31 20:19:40.353874 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:40.353839    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d401fef-1568-4d16-88fa-92aeaaea68b0-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-lwvpn\" (UID: \"3d401fef-1568-4d16-88fa-92aeaaea68b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lwvpn"
Oct 31 20:19:40.353874 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:40.353862    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d401fef-1568-4d16-88fa-92aeaaea68b0-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-lwvpn\" (UID: \"3d401fef-1568-4d16-88fa-92aeaaea68b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lwvpn"
Oct 31 20:19:40.354043 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:40.353880    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3994be2-19d9-4e2d-b3ce-03f63d2c9c32-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-t27sn\" (UID: \"f3994be2-19d9-4e2d-b3ce-03f63d2c9c32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t27sn"
Oct 31 20:19:40.354043 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:40.353922    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d401fef-1568-4d16-88fa-92aeaaea68b0-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-lwvpn\" (UID: \"3d401fef-1568-4d16-88fa-92aeaaea68b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lwvpn"
Oct 31 20:19:40.354161 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:40.354074    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d401fef-1568-4d16-88fa-92aeaaea68b0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-lwvpn\" (UID: \"3d401fef-1568-4d16-88fa-92aeaaea68b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lwvpn"
Oct 31 20:19:40.354161 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:40.354137    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d401fef-1568-4d16-88fa-92aeaaea68b0-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-lwvpn\" (UID: \"3d401fef-1568-4d16-88fa-92aeaaea68b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lwvpn"
Oct 31 20:19:40.354272 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:40.354193    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3994be2-19d9-4e2d-b3ce-03f63d2c9c32-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-t27sn\" (UID: \"f3994be2-19d9-4e2d-b3ce-03f63d2c9c32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t27sn"
Oct 31 20:19:40.354360 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:40.354322    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d401fef-1568-4d16-88fa-92aeaaea68b0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-lwvpn\" (UID: \"3d401fef-1568-4d16-88fa-92aeaaea68b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lwvpn"
Oct 31 20:19:40.354407 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:40.354376    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2nv6\" (UniqueName: \"kubernetes.io/projected/f3994be2-19d9-4e2d-b3ce-03f63d2c9c32-kube-api-access-n2nv6\") pod \"maas-default-gateway-openshift-default-75d7c785f-t27sn\" (UID: \"f3994be2-19d9-4e2d-b3ce-03f63d2c9c32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t27sn"
Oct 31 20:19:40.354407 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:40.354400    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3994be2-19d9-4e2d-b3ce-03f63d2c9c32-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-t27sn\" (UID: \"f3994be2-19d9-4e2d-b3ce-03f63d2c9c32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t27sn"
Oct 31 20:19:40.402049 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:40.402021    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:40.455708 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:40.455673    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d401fef-1568-4d16-88fa-92aeaaea68b0-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-lwvpn\" (UID: \"3d401fef-1568-4d16-88fa-92aeaaea68b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lwvpn"
Oct 31 20:19:40.455875 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:40.455716    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3994be2-19d9-4e2d-b3ce-03f63d2c9c32-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-t27sn\" (UID: \"f3994be2-19d9-4e2d-b3ce-03f63d2c9c32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t27sn"
Oct 31 20:19:40.455875 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:40.455743    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d401fef-1568-4d16-88fa-92aeaaea68b0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-lwvpn\" (UID: \"3d401fef-1568-4d16-88fa-92aeaaea68b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lwvpn"
Oct 31 20:19:40.455875 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:40.455770    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2nv6\" (UniqueName: \"kubernetes.io/projected/f3994be2-19d9-4e2d-b3ce-03f63d2c9c32-kube-api-access-n2nv6\") pod \"maas-default-gateway-openshift-default-75d7c785f-t27sn\" (UID: \"f3994be2-19d9-4e2d-b3ce-03f63d2c9c32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t27sn"
Oct 31 20:19:40.455875 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:40.455795    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3994be2-19d9-4e2d-b3ce-03f63d2c9c32-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-t27sn\" (UID: \"f3994be2-19d9-4e2d-b3ce-03f63d2c9c32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t27sn"
Oct 31 20:19:40.455875 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:40.455824    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d401fef-1568-4d16-88fa-92aeaaea68b0-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-lwvpn\" (UID: \"3d401fef-1568-4d16-88fa-92aeaaea68b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lwvpn"
Oct 31 20:19:40.456205 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:40.456178    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d401fef-1568-4d16-88fa-92aeaaea68b0-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-lwvpn\" (UID: \"3d401fef-1568-4d16-88fa-92aeaaea68b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lwvpn"
Oct 31 20:19:40.456299 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:40.456231    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3994be2-19d9-4e2d-b3ce-03f63d2c9c32-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-t27sn\" (UID: \"f3994be2-19d9-4e2d-b3ce-03f63d2c9c32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t27sn"
Oct 31 20:19:40.456299 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:40.456269    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3994be2-19d9-4e2d-b3ce-03f63d2c9c32-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-t27sn\" (UID: \"f3994be2-19d9-4e2d-b3ce-03f63d2c9c32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t27sn"
Oct 31 20:19:40.456299 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:40.456282    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3994be2-19d9-4e2d-b3ce-03f63d2c9c32-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-t27sn\" (UID: \"f3994be2-19d9-4e2d-b3ce-03f63d2c9c32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t27sn"
Oct 31 20:19:40.456506 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:40.456389    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3994be2-19d9-4e2d-b3ce-03f63d2c9c32-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-t27sn\" (UID: \"f3994be2-19d9-4e2d-b3ce-03f63d2c9c32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t27sn"
Oct 31 20:19:40.456506 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:40.456435    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3994be2-19d9-4e2d-b3ce-03f63d2c9c32-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-t27sn\" (UID: \"f3994be2-19d9-4e2d-b3ce-03f63d2c9c32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t27sn"
Oct 31 20:19:40.456506 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:40.456467    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3994be2-19d9-4e2d-b3ce-03f63d2c9c32-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-t27sn\" (UID: \"f3994be2-19d9-4e2d-b3ce-03f63d2c9c32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t27sn"
Oct 31 20:19:40.456506 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:40.456485    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d401fef-1568-4d16-88fa-92aeaaea68b0-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-lwvpn\" (UID: \"3d401fef-1568-4d16-88fa-92aeaaea68b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lwvpn"
Oct 31 20:19:40.456506 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:40.456501    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh6gp\" (UniqueName: \"kubernetes.io/projected/3d401fef-1568-4d16-88fa-92aeaaea68b0-kube-api-access-hh6gp\") pod \"maas-default-gateway-openshift-default-75d7c785f-lwvpn\" (UID: \"3d401fef-1568-4d16-88fa-92aeaaea68b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lwvpn"
Oct 31 20:19:40.456776 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:40.456545    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d401fef-1568-4d16-88fa-92aeaaea68b0-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-lwvpn\" (UID: \"3d401fef-1568-4d16-88fa-92aeaaea68b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lwvpn"
Oct 31 20:19:40.456776 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:40.456588    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d401fef-1568-4d16-88fa-92aeaaea68b0-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-lwvpn\" (UID: \"3d401fef-1568-4d16-88fa-92aeaaea68b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lwvpn"
Oct 31 20:19:40.456776 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:40.456618    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3994be2-19d9-4e2d-b3ce-03f63d2c9c32-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-t27sn\" (UID: \"f3994be2-19d9-4e2d-b3ce-03f63d2c9c32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t27sn"
Oct 31 20:19:40.456776 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:40.456653    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d401fef-1568-4d16-88fa-92aeaaea68b0-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-lwvpn\" (UID: \"3d401fef-1568-4d16-88fa-92aeaaea68b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lwvpn"
Oct 31 20:19:40.456776 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:40.456703    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d401fef-1568-4d16-88fa-92aeaaea68b0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-lwvpn\" (UID: \"3d401fef-1568-4d16-88fa-92aeaaea68b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lwvpn"
Oct 31 20:19:40.457073 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:40.456932    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d401fef-1568-4d16-88fa-92aeaaea68b0-credential-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-lwvpn\" (UID: \"3d401fef-1568-4d16-88fa-92aeaaea68b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lwvpn"
Oct 31 20:19:40.457073 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:40.457031    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3994be2-19d9-4e2d-b3ce-03f63d2c9c32-workload-socket\") pod \"maas-default-gateway-openshift-default-75d7c785f-t27sn\" (UID: \"f3994be2-19d9-4e2d-b3ce-03f63d2c9c32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t27sn"
Oct 31 20:19:40.457254 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:40.457229    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d401fef-1568-4d16-88fa-92aeaaea68b0-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-lwvpn\" (UID: \"3d401fef-1568-4d16-88fa-92aeaaea68b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lwvpn"
Oct 31 20:19:40.457346 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:40.457315    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3994be2-19d9-4e2d-b3ce-03f63d2c9c32-workload-certs\") pod \"maas-default-gateway-openshift-default-75d7c785f-t27sn\" (UID: \"f3994be2-19d9-4e2d-b3ce-03f63d2c9c32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t27sn"
Oct 31 20:19:40.457610 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:40.457540    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3994be2-19d9-4e2d-b3ce-03f63d2c9c32-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-t27sn\" (UID: \"f3994be2-19d9-4e2d-b3ce-03f63d2c9c32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t27sn"
Oct 31 20:19:40.457610 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:40.457580    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d401fef-1568-4d16-88fa-92aeaaea68b0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-lwvpn\" (UID: \"3d401fef-1568-4d16-88fa-92aeaaea68b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lwvpn"
Oct 31 20:19:40.457820 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:40.457664    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3994be2-19d9-4e2d-b3ce-03f63d2c9c32-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-75d7c785f-t27sn\" (UID: \"f3994be2-19d9-4e2d-b3ce-03f63d2c9c32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t27sn"
Oct 31 20:19:40.457820 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:40.456179    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d401fef-1568-4d16-88fa-92aeaaea68b0-istio-data\") pod \"maas-default-gateway-openshift-default-75d7c785f-lwvpn\" (UID: \"3d401fef-1568-4d16-88fa-92aeaaea68b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lwvpn"
Oct 31 20:19:40.458554 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:40.458526    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d401fef-1568-4d16-88fa-92aeaaea68b0-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-lwvpn\" (UID: \"3d401fef-1568-4d16-88fa-92aeaaea68b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lwvpn"
Oct 31 20:19:40.458800 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:40.458773    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d401fef-1568-4d16-88fa-92aeaaea68b0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-lwvpn\" (UID: \"3d401fef-1568-4d16-88fa-92aeaaea68b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lwvpn"
Oct 31 20:19:40.459091 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:40.459074    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3994be2-19d9-4e2d-b3ce-03f63d2c9c32-istio-envoy\") pod \"maas-default-gateway-openshift-default-75d7c785f-t27sn\" (UID: \"f3994be2-19d9-4e2d-b3ce-03f63d2c9c32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t27sn"
Oct 31 20:19:40.459554 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:40.459528    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3994be2-19d9-4e2d-b3ce-03f63d2c9c32-istio-podinfo\") pod \"maas-default-gateway-openshift-default-75d7c785f-t27sn\" (UID: \"f3994be2-19d9-4e2d-b3ce-03f63d2c9c32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t27sn"
Oct 31 20:19:40.467227 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:40.467197    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3994be2-19d9-4e2d-b3ce-03f63d2c9c32-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-t27sn\" (UID: \"f3994be2-19d9-4e2d-b3ce-03f63d2c9c32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t27sn"
Oct 31 20:19:40.467557 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:40.467531    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh6gp\" (UniqueName: \"kubernetes.io/projected/3d401fef-1568-4d16-88fa-92aeaaea68b0-kube-api-access-hh6gp\") pod \"maas-default-gateway-openshift-default-75d7c785f-lwvpn\" (UID: \"3d401fef-1568-4d16-88fa-92aeaaea68b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lwvpn"
Oct 31 20:19:40.467648 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:40.467624    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2nv6\" (UniqueName: \"kubernetes.io/projected/f3994be2-19d9-4e2d-b3ce-03f63d2c9c32-kube-api-access-n2nv6\") pod \"maas-default-gateway-openshift-default-75d7c785f-t27sn\" (UID: \"f3994be2-19d9-4e2d-b3ce-03f63d2c9c32\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t27sn"
Oct 31 20:19:40.468827 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:40.468791    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d401fef-1568-4d16-88fa-92aeaaea68b0-istio-token\") pod \"maas-default-gateway-openshift-default-75d7c785f-lwvpn\" (UID: \"3d401fef-1568-4d16-88fa-92aeaaea68b0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lwvpn"
Oct 31 20:19:40.565310 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:40.565259    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lwvpn"
Oct 31 20:19:40.591388 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:40.591367    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t27sn"
Oct 31 20:19:40.851556 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:40.851527    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lwvpn"]
Oct 31 20:19:40.854171 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:19:40.854146    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d401fef_1568_4d16_88fa_92aeaaea68b0.slice/crio-3e3b4946198f06990628ebc500b5a668e8514c65992287d8e2a26cc4a6a22392 WatchSource:0}: Error finding container 3e3b4946198f06990628ebc500b5a668e8514c65992287d8e2a26cc4a6a22392: Status 404 returned error can't find the container with id 3e3b4946198f06990628ebc500b5a668e8514c65992287d8e2a26cc4a6a22392
Oct 31 20:19:40.856516 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:40.856481    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:19:40.856601 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:40.856548    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:19:40.856601 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:40.856578    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:19:40.888827 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:40.888799    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lwvpn" event={"ID":"3d401fef-1568-4d16-88fa-92aeaaea68b0","Type":"ContainerStarted","Data":"3e3b4946198f06990628ebc500b5a668e8514c65992287d8e2a26cc4a6a22392"}
Oct 31 20:19:40.896020 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:40.895996    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t27sn"]
Oct 31 20:19:40.898121 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:19:40.898095    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3994be2_19d9_4e2d_b3ce_03f63d2c9c32.slice/crio-86ea53ca77c784696b5d793123045c4222a87946507d029b36a2648f42775445 WatchSource:0}: Error finding container 86ea53ca77c784696b5d793123045c4222a87946507d029b36a2648f42775445: Status 404 returned error can't find the container with id 86ea53ca77c784696b5d793123045c4222a87946507d029b36a2648f42775445
Oct 31 20:19:40.900357 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:40.900308    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:19:40.900429 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:40.900404    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:19:40.900470 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:40.900451    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:19:41.178358 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:41.178306    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:41.900932 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:41.900891    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t27sn" event={"ID":"f3994be2-19d9-4e2d-b3ce-03f63d2c9c32","Type":"ContainerStarted","Data":"5027002db1aa1f864c3da61a83032e923da3d0e161341c31b60303d4916d25bb"}
Oct 31 20:19:41.901602 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:41.901562    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t27sn" event={"ID":"f3994be2-19d9-4e2d-b3ce-03f63d2c9c32","Type":"ContainerStarted","Data":"86ea53ca77c784696b5d793123045c4222a87946507d029b36a2648f42775445"}
Oct 31 20:19:41.904879 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:41.904852    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lwvpn" event={"ID":"3d401fef-1568-4d16-88fa-92aeaaea68b0","Type":"ContainerStarted","Data":"712900a9a313180d3a21620d4a8438d2bfbd336595c50b3d4479796c3be973ca"}
Oct 31 20:19:41.928418 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:41.928319    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t27sn" podStartSLOduration=1.928303123 podStartE2EDuration="1.928303123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:19:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:19:41.926666189 +0000 UTC m=+3612.499900319" watchObservedRunningTime="2026-10-31 20:19:41.928303123 +0000 UTC m=+3612.501537253"
Oct 31 20:19:41.947649 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:41.947612    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lwvpn" podStartSLOduration=1.947601557 podStartE2EDuration="1.947601557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:19:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:19:41.947217531 +0000 UTC m=+3612.520451660" watchObservedRunningTime="2026-10-31 20:19:41.947601557 +0000 UTC m=+3612.520835682"
Oct 31 20:19:41.985353 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:41.984914    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:42.571354 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:42.568826    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lwvpn"
Oct 31 20:19:42.571354 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:42.568948    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-lwvpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.73:15021/healthz/ready\": dial tcp 10.133.1.73:15021: connect: connection refused" start-of-body=
Oct 31 20:19:42.571354 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:42.568991    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lwvpn" podUID="3d401fef-1568-4d16-88fa-92aeaaea68b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.73:15021/healthz/ready\": dial tcp 10.133.1.73:15021: connect: connection refused"
Oct 31 20:19:42.601351 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:42.600754    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t27sn"
Oct 31 20:19:42.601351 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:42.600860    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-t27sn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.74:15021/healthz/ready\": dial tcp 10.133.1.74:15021: connect: connection refused" start-of-body=
Oct 31 20:19:42.601351 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:42.600897    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t27sn" podUID="f3994be2-19d9-4e2d-b3ce-03f63d2c9c32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.74:15021/healthz/ready\": dial tcp 10.133.1.74:15021: connect: connection refused"
Oct 31 20:19:42.777355 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:42.775838    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:43.574707 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:43.574247    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-lwvpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.73:15021/healthz/ready\": dial tcp 10.133.1.73:15021: connect: connection refused" start-of-body=
Oct 31 20:19:43.574707 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:43.574305    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lwvpn" podUID="3d401fef-1568-4d16-88fa-92aeaaea68b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.73:15021/healthz/ready\": dial tcp 10.133.1.73:15021: connect: connection refused"
Oct 31 20:19:43.586352 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:43.576292    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:43.603351 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:43.594829    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-t27sn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.74:15021/healthz/ready\": dial tcp 10.133.1.74:15021: connect: connection refused" start-of-body=
Oct 31 20:19:43.603351 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:43.594895    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t27sn" podUID="f3994be2-19d9-4e2d-b3ce-03f63d2c9c32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.74:15021/healthz/ready\": dial tcp 10.133.1.74:15021: connect: connection refused"
Oct 31 20:19:44.380350 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:44.376809    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:44.579441 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:44.579403    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-lwvpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.73:15021/healthz/ready\": dial tcp 10.133.1.73:15021: connect: connection refused" start-of-body=
Oct 31 20:19:44.580033 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:44.580000    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lwvpn" podUID="3d401fef-1568-4d16-88fa-92aeaaea68b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.73:15021/healthz/ready\": dial tcp 10.133.1.73:15021: connect: connection refused"
Oct 31 20:19:44.595659 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:44.595634    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-t27sn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.74:15021/healthz/ready\": dial tcp 10.133.1.74:15021: connect: connection refused" start-of-body=
Oct 31 20:19:44.595938 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:44.595913    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t27sn" podUID="f3994be2-19d9-4e2d-b3ce-03f63d2c9c32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.74:15021/healthz/ready\": dial tcp 10.133.1.74:15021: connect: connection refused"
Oct 31 20:19:45.282374 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:45.281346    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:45.574410 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:45.571737    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-lwvpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.73:15021/healthz/ready\": dial tcp 10.133.1.73:15021: connect: connection refused" start-of-body=
Oct 31 20:19:45.574410 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:45.571800    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lwvpn" podUID="3d401fef-1568-4d16-88fa-92aeaaea68b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.73:15021/healthz/ready\": dial tcp 10.133.1.73:15021: connect: connection refused"
Oct 31 20:19:45.594347 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:45.593738    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-t27sn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.74:15021/healthz/ready\": dial tcp 10.133.1.74:15021: connect: connection refused" start-of-body=
Oct 31 20:19:45.594347 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:45.593790    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t27sn" podUID="f3994be2-19d9-4e2d-b3ce-03f63d2c9c32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.74:15021/healthz/ready\": dial tcp 10.133.1.74:15021: connect: connection refused"
Oct 31 20:19:46.079353 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:46.078619    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:46.568785 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:46.568746    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-lwvpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.73:15021/healthz/ready\": dial tcp 10.133.1.73:15021: connect: connection refused" start-of-body=
Oct 31 20:19:46.569109 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:46.569077    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lwvpn" podUID="3d401fef-1568-4d16-88fa-92aeaaea68b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.73:15021/healthz/ready\": dial tcp 10.133.1.73:15021: connect: connection refused"
Oct 31 20:19:46.592707 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:46.592674    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-t27sn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.74:15021/healthz/ready\": dial tcp 10.133.1.74:15021: connect: connection refused" start-of-body=
Oct 31 20:19:46.592924 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:46.592898    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t27sn" podUID="f3994be2-19d9-4e2d-b3ce-03f63d2c9c32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.74:15021/healthz/ready\": dial tcp 10.133.1.74:15021: connect: connection refused"
Oct 31 20:19:46.863241 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:46.863146    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:47.566121 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:47.566081    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-lwvpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.73:15021/healthz/ready\": dial tcp 10.133.1.73:15021: connect: connection refused" start-of-body=
Oct 31 20:19:47.566381 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:47.566153    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lwvpn" podUID="3d401fef-1568-4d16-88fa-92aeaaea68b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.73:15021/healthz/ready\": dial tcp 10.133.1.73:15021: connect: connection refused"
Oct 31 20:19:47.591801 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:47.591769    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-t27sn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.74:15021/healthz/ready\": dial tcp 10.133.1.74:15021: connect: connection refused" start-of-body=
Oct 31 20:19:47.591941 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:47.591827    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t27sn" podUID="f3994be2-19d9-4e2d-b3ce-03f63d2c9c32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.74:15021/healthz/ready\": dial tcp 10.133.1.74:15021: connect: connection refused"
Oct 31 20:19:47.774323 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:47.774287    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:48.541624 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:48.541588    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:49.322862 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:49.322829    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t27sn"
Oct 31 20:19:49.323390 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:49.323363    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t27sn"
Oct 31 20:19:49.324378 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:49.324355    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t27sn"
Oct 31 20:19:49.338376 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:49.338340    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lwvpn"
Oct 31 20:19:49.339671 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:49.339650    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lwvpn"
Oct 31 20:19:49.340560 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:49.340542    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-lwvpn"
Oct 31 20:19:49.395084 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:49.395041    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:50.247969 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:50.247888    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:51.056692 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:51.056654    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:51.775438 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:51.775402    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:52.532960 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:52.532923    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:53.353415 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:53.353377    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:54.057202 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:54.057167    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:54.833801 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:54.833764    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:55.649115 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:55.649065    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:56.361459 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:56.361421    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:57.156359 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:57.156306    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:57.643348 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:57.643295    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:57.890974 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:57.890932    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:57.997146 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:57.997100    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:58.629182 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:58.629148    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:19:59.358102 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:19:59.358065    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:20:00.079456 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:00.079422    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:20:00.316981 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:00.316946    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:20:01.048463 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:01.048427    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:20:01.853681 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:01.853642    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:20:02.775765 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:02.775723    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:20:03.457375 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:03.457314    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:20:04.335565 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:04.335533    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:20:05.152221 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:05.152180    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:20:05.354314 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:05.354278    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:20:06.033974 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:06.033929    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:20:06.245089 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:20:06.245053    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 20:20:06.748173 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:06.748139    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:20:07.452225 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:07.452182    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:20:08.149274 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:08.149244    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:20:08.844549 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:08.844516    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:20:09.560967 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:09.560934    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:20:10.251034 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:10.250984    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:20:10.596830 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:10.596721    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-844db7-6f7649b585pscw"]
Oct 31 20:20:10.609079 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:10.609037    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-844db7-d587bc6mhg"]
Oct 31 20:20:10.680481 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:10.680443    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:20:10.793151 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:10.793116    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:20:11.051896 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:11.051847    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:20:11.692189 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:11.692140    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-844db7-5ff86b466f-wzmdl"]
Oct 31 20:20:11.692660 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:11.692425    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-844db7-5ff86b466f-wzmdl" podUID="f67b87d1-b808-4393-9224-1c45bf38a373" containerName="maas-api" containerID="cri-o://cfb89fbb5424676bc488418f6948552e0fc21a8496294e419a715dd3eb290fd1" gracePeriod=30
Oct 31 20:20:11.745355 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:11.745300    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:20:11.914681 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:11.914387    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:20:11.968093 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:11.968069    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-844db7-5ff86b466f-wzmdl"
Oct 31 20:20:12.014657 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:12.014603    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:20:12.048890 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:12.048862    4645 generic.go:356] "Generic (PLEG): container finished" podID="f67b87d1-b808-4393-9224-1c45bf38a373" containerID="cfb89fbb5424676bc488418f6948552e0fc21a8496294e419a715dd3eb290fd1" exitCode=0
Oct 31 20:20:12.049000 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:12.048909    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-844db7-5ff86b466f-wzmdl"
Oct 31 20:20:12.049000 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:12.048948    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-844db7-5ff86b466f-wzmdl" event={"ID":"f67b87d1-b808-4393-9224-1c45bf38a373","Type":"ContainerDied","Data":"cfb89fbb5424676bc488418f6948552e0fc21a8496294e419a715dd3eb290fd1"}
Oct 31 20:20:12.049092 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:12.049000    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-844db7-5ff86b466f-wzmdl" event={"ID":"f67b87d1-b808-4393-9224-1c45bf38a373","Type":"ContainerDied","Data":"ee0fa4ff41ec434a8c05466a7de8f33e6c5a4f3a9b90bd0d52b157c41c1422f6"}
Oct 31 20:20:12.049092 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:12.049024    4645 scope.go:122] "RemoveContainer" containerID="cfb89fbb5424676bc488418f6948552e0fc21a8496294e419a715dd3eb290fd1"
Oct 31 20:20:12.057879 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:12.057862    4645 scope.go:122] "RemoveContainer" containerID="cfb89fbb5424676bc488418f6948552e0fc21a8496294e419a715dd3eb290fd1"
Oct 31 20:20:12.058150 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:20:12.058122    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfb89fbb5424676bc488418f6948552e0fc21a8496294e419a715dd3eb290fd1\": container with ID starting with cfb89fbb5424676bc488418f6948552e0fc21a8496294e419a715dd3eb290fd1 not found: ID does not exist" containerID="cfb89fbb5424676bc488418f6948552e0fc21a8496294e419a715dd3eb290fd1"
Oct 31 20:20:12.058277 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:12.058156    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfb89fbb5424676bc488418f6948552e0fc21a8496294e419a715dd3eb290fd1"} err="failed to get container status \"cfb89fbb5424676bc488418f6948552e0fc21a8496294e419a715dd3eb290fd1\": rpc error: code = NotFound desc = could not find container \"cfb89fbb5424676bc488418f6948552e0fc21a8496294e419a715dd3eb290fd1\": container with ID starting with cfb89fbb5424676bc488418f6948552e0fc21a8496294e419a715dd3eb290fd1 not found: ID does not exist"
Oct 31 20:20:12.128665 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:12.128632    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f67b87d1-b808-4393-9224-1c45bf38a373-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f67b87d1-b808-4393-9224-1c45bf38a373-openshift-service-ca\") pod \"f67b87d1-b808-4393-9224-1c45bf38a373\" (UID: \"f67b87d1-b808-4393-9224-1c45bf38a373\") "
Oct 31 20:20:12.128860 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:12.128740    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f67b87d1-b808-4393-9224-1c45bf38a373-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f67b87d1-b808-4393-9224-1c45bf38a373-maas-api-tls\") pod \"f67b87d1-b808-4393-9224-1c45bf38a373\" (UID: \"f67b87d1-b808-4393-9224-1c45bf38a373\") "
Oct 31 20:20:12.128860 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:12.128795    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f67b87d1-b808-4393-9224-1c45bf38a373-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f67b87d1-b808-4393-9224-1c45bf38a373-metrics-certs\") pod \"f67b87d1-b808-4393-9224-1c45bf38a373\" (UID: \"f67b87d1-b808-4393-9224-1c45bf38a373\") "
Oct 31 20:20:12.128860 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:12.128846    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f67b87d1-b808-4393-9224-1c45bf38a373-kube-api-access-5mnqh\" (UniqueName: \"kubernetes.io/projected/f67b87d1-b808-4393-9224-1c45bf38a373-kube-api-access-5mnqh\") pod \"f67b87d1-b808-4393-9224-1c45bf38a373\" (UID: \"f67b87d1-b808-4393-9224-1c45bf38a373\") "
Oct 31 20:20:12.129054 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:12.129021    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f67b87d1-b808-4393-9224-1c45bf38a373-openshift-service-ca" pod "f67b87d1-b808-4393-9224-1c45bf38a373" (UID: "f67b87d1-b808-4393-9224-1c45bf38a373"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:20:12.129212 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:12.129184    4645 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f67b87d1-b808-4393-9224-1c45bf38a373-openshift-service-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:20:12.131056 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:12.131029    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f67b87d1-b808-4393-9224-1c45bf38a373-metrics-certs" pod "f67b87d1-b808-4393-9224-1c45bf38a373" (UID: "f67b87d1-b808-4393-9224-1c45bf38a373"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:20:12.131269 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:12.131247    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f67b87d1-b808-4393-9224-1c45bf38a373-maas-api-tls" pod "f67b87d1-b808-4393-9224-1c45bf38a373" (UID: "f67b87d1-b808-4393-9224-1c45bf38a373"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:20:12.131383 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:12.131365    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f67b87d1-b808-4393-9224-1c45bf38a373-kube-api-access-5mnqh" pod "f67b87d1-b808-4393-9224-1c45bf38a373" (UID: "f67b87d1-b808-4393-9224-1c45bf38a373"). InnerVolumeSpecName "kube-api-access-5mnqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:20:12.229966 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:12.229891    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f67b87d1-b808-4393-9224-1c45bf38a373-maas-api-tls\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:20:12.229966 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:12.229922    4645 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f67b87d1-b808-4393-9224-1c45bf38a373-metrics-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:20:12.229966 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:12.229934    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mnqh\" (UniqueName: \"kubernetes.io/projected/f67b87d1-b808-4393-9224-1c45bf38a373-kube-api-access-5mnqh\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:20:12.371141 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:12.371111    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-844db7-5ff86b466f-wzmdl"]
Oct 31 20:20:12.376183 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:12.376159    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-844db7-5ff86b466f-wzmdl"]
Oct 31 20:20:12.542628 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:12.542569    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:20:13.615776 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:13.615745    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f67b87d1-b808-4393-9224-1c45bf38a373" path="/var/lib/kubelet/pods/f67b87d1-b808-4393-9224-1c45bf38a373/volumes"
Oct 31 20:20:15.598030 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:15.597963    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-844db7-6f7649b585pscw" podUID="50110354-1374-44e2-aced-b0b068cd1bba" containerName="payload-processing" containerID="cri-o://ace5fb30bec34aa7b38a8320dddaf0732c4ae27bed4bd611370c390bff1cb138" gracePeriod=25
Oct 31 20:20:15.610504 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:15.610451    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-844db7-d587bc6mhg" podUID="d8e68310-9416-42f7-9ec5-f2f3348b8029" containerName="payload-pre-processing" containerID="cri-o://d3cc5fd9d2e8c721f37d963b26864178acca8e7c80332ff2594309f36fa9aaa8" gracePeriod=25
Oct 31 20:20:15.678187 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:15.678156    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-844db7-6f7649b585pscw" podUID="50110354-1374-44e2-aced-b0b068cd1bba" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:20:15.678315 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:15.678187    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-844db7-d587bc6mhg" podUID="d8e68310-9416-42f7-9ec5-f2f3348b8029" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:20:18.945250 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:18.945208    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:20:19.077969 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:19.077929    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:20:19.285928 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:19.285851    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w"]
Oct 31 20:20:19.286092 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:19.286070    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w" podUID="672d276a-4c71-4742-82f6-f5c3dc95e072" containerName="istio-proxy" containerID="cri-o://1670d35c1aefd30aa5445eeffdddfb1cfd992eb47dc592a912e6974972ecb7a7" gracePeriod=30
Oct 31 20:20:19.320412 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:19.320373    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:20:20.095071 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:20.095030    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:20:21.127795 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:21.127748    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:20:22.170148 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:22.170099    4645 patch_prober.go:29] interesting pod/e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 20:20:22.170714 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:22.170166    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w" podUID="672d276a-4c71-4742-82f6-f5c3dc95e072" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 20:20:24.538368 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:24.538339    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w"
Oct 31 20:20:24.639259 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:24.639196    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/672d276a-4c71-4742-82f6-f5c3dc95e072-istio-token\" (UniqueName: \"kubernetes.io/projected/672d276a-4c71-4742-82f6-f5c3dc95e072-istio-token\") pod \"672d276a-4c71-4742-82f6-f5c3dc95e072\" (UID: \"672d276a-4c71-4742-82f6-f5c3dc95e072\") "
Oct 31 20:20:24.639259 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:24.639250    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/672d276a-4c71-4742-82f6-f5c3dc95e072-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/672d276a-4c71-4742-82f6-f5c3dc95e072-istio-podinfo\") pod \"672d276a-4c71-4742-82f6-f5c3dc95e072\" (UID: \"672d276a-4c71-4742-82f6-f5c3dc95e072\") "
Oct 31 20:20:24.639502 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:24.639318    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/672d276a-4c71-4742-82f6-f5c3dc95e072-kube-api-access-7w5bm\" (UniqueName: \"kubernetes.io/projected/672d276a-4c71-4742-82f6-f5c3dc95e072-kube-api-access-7w5bm\") pod \"672d276a-4c71-4742-82f6-f5c3dc95e072\" (UID: \"672d276a-4c71-4742-82f6-f5c3dc95e072\") "
Oct 31 20:20:24.639502 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:24.639372    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/672d276a-4c71-4742-82f6-f5c3dc95e072-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/672d276a-4c71-4742-82f6-f5c3dc95e072-workload-certs\") pod \"672d276a-4c71-4742-82f6-f5c3dc95e072\" (UID: \"672d276a-4c71-4742-82f6-f5c3dc95e072\") "
Oct 31 20:20:24.639502 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:24.639408    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/672d276a-4c71-4742-82f6-f5c3dc95e072-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/672d276a-4c71-4742-82f6-f5c3dc95e072-istio-envoy\") pod \"672d276a-4c71-4742-82f6-f5c3dc95e072\" (UID: \"672d276a-4c71-4742-82f6-f5c3dc95e072\") "
Oct 31 20:20:24.639502 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:24.639440    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/672d276a-4c71-4742-82f6-f5c3dc95e072-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/672d276a-4c71-4742-82f6-f5c3dc95e072-workload-socket\") pod \"672d276a-4c71-4742-82f6-f5c3dc95e072\" (UID: \"672d276a-4c71-4742-82f6-f5c3dc95e072\") "
Oct 31 20:20:24.639762 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:24.639543    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/672d276a-4c71-4742-82f6-f5c3dc95e072-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/672d276a-4c71-4742-82f6-f5c3dc95e072-credential-socket\") pod \"672d276a-4c71-4742-82f6-f5c3dc95e072\" (UID: \"672d276a-4c71-4742-82f6-f5c3dc95e072\") "
Oct 31 20:20:24.639762 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:24.639578    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/672d276a-4c71-4742-82f6-f5c3dc95e072-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/672d276a-4c71-4742-82f6-f5c3dc95e072-istiod-ca-cert\") pod \"672d276a-4c71-4742-82f6-f5c3dc95e072\" (UID: \"672d276a-4c71-4742-82f6-f5c3dc95e072\") "
Oct 31 20:20:24.639762 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:24.639630    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/672d276a-4c71-4742-82f6-f5c3dc95e072-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/672d276a-4c71-4742-82f6-f5c3dc95e072-istio-data\") pod \"672d276a-4c71-4742-82f6-f5c3dc95e072\" (UID: \"672d276a-4c71-4742-82f6-f5c3dc95e072\") "
Oct 31 20:20:24.639762 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:24.639681    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/672d276a-4c71-4742-82f6-f5c3dc95e072-workload-certs" pod "672d276a-4c71-4742-82f6-f5c3dc95e072" (UID: "672d276a-4c71-4742-82f6-f5c3dc95e072"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:20:24.640001 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:24.639791    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/672d276a-4c71-4742-82f6-f5c3dc95e072-workload-socket" pod "672d276a-4c71-4742-82f6-f5c3dc95e072" (UID: "672d276a-4c71-4742-82f6-f5c3dc95e072"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:20:24.640001 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:24.639855    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/672d276a-4c71-4742-82f6-f5c3dc95e072-credential-socket" pod "672d276a-4c71-4742-82f6-f5c3dc95e072" (UID: "672d276a-4c71-4742-82f6-f5c3dc95e072"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:20:24.640001 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:24.639958    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/672d276a-4c71-4742-82f6-f5c3dc95e072-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:20:24.640001 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:24.639980    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/672d276a-4c71-4742-82f6-f5c3dc95e072-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:20:24.640001 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:24.639994    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/672d276a-4c71-4742-82f6-f5c3dc95e072-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:20:24.640308 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:24.640038    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/672d276a-4c71-4742-82f6-f5c3dc95e072-istio-data" pod "672d276a-4c71-4742-82f6-f5c3dc95e072" (UID: "672d276a-4c71-4742-82f6-f5c3dc95e072"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:20:24.640308 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:24.640068    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/672d276a-4c71-4742-82f6-f5c3dc95e072-istiod-ca-cert" pod "672d276a-4c71-4742-82f6-f5c3dc95e072" (UID: "672d276a-4c71-4742-82f6-f5c3dc95e072"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:20:24.641666 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:24.641642    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/672d276a-4c71-4742-82f6-f5c3dc95e072-kube-api-access-7w5bm" pod "672d276a-4c71-4742-82f6-f5c3dc95e072" (UID: "672d276a-4c71-4742-82f6-f5c3dc95e072"). InnerVolumeSpecName "kube-api-access-7w5bm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:20:24.641666 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:24.641649    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/672d276a-4c71-4742-82f6-f5c3dc95e072-istio-podinfo" pod "672d276a-4c71-4742-82f6-f5c3dc95e072" (UID: "672d276a-4c71-4742-82f6-f5c3dc95e072"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 20:20:24.642031 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:24.642010    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/672d276a-4c71-4742-82f6-f5c3dc95e072-istio-envoy" pod "672d276a-4c71-4742-82f6-f5c3dc95e072" (UID: "672d276a-4c71-4742-82f6-f5c3dc95e072"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:20:24.642031 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:24.642021    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/672d276a-4c71-4742-82f6-f5c3dc95e072-istio-token" pod "672d276a-4c71-4742-82f6-f5c3dc95e072" (UID: "672d276a-4c71-4742-82f6-f5c3dc95e072"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:20:24.740974 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:24.740946    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7w5bm\" (UniqueName: \"kubernetes.io/projected/672d276a-4c71-4742-82f6-f5c3dc95e072-kube-api-access-7w5bm\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:20:24.740974 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:24.740970    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/672d276a-4c71-4742-82f6-f5c3dc95e072-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:20:24.741123 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:24.740980    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/672d276a-4c71-4742-82f6-f5c3dc95e072-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:20:24.741123 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:24.740990    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/672d276a-4c71-4742-82f6-f5c3dc95e072-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:20:24.741123 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:24.740999    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/672d276a-4c71-4742-82f6-f5c3dc95e072-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:20:24.741123 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:24.741008    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/672d276a-4c71-4742-82f6-f5c3dc95e072-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:20:25.122742 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:25.122710    4645 generic.go:356] "Generic (PLEG): container finished" podID="672d276a-4c71-4742-82f6-f5c3dc95e072" containerID="1670d35c1aefd30aa5445eeffdddfb1cfd992eb47dc592a912e6974972ecb7a7" exitCode=0
Oct 31 20:20:25.122896 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:25.122764    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w"
Oct 31 20:20:25.122896 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:25.122775    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w" event={"ID":"672d276a-4c71-4742-82f6-f5c3dc95e072","Type":"ContainerDied","Data":"1670d35c1aefd30aa5445eeffdddfb1cfd992eb47dc592a912e6974972ecb7a7"}
Oct 31 20:20:25.122896 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:25.122820    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w" event={"ID":"672d276a-4c71-4742-82f6-f5c3dc95e072","Type":"ContainerDied","Data":"bb13ac498e25219c40643a49ed80fa8f629a280e874479ed3fa0c942d44c209d"}
Oct 31 20:20:25.122896 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:25.122842    4645 scope.go:122] "RemoveContainer" containerID="1670d35c1aefd30aa5445eeffdddfb1cfd992eb47dc592a912e6974972ecb7a7"
Oct 31 20:20:25.132834 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:25.132803    4645 scope.go:122] "RemoveContainer" containerID="20b1df8596b5b7fb6369a4036e590f2f4b5f358b74c5934466db94e801e83c41"
Oct 31 20:20:25.141249 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:25.141227    4645 scope.go:122] "RemoveContainer" containerID="1670d35c1aefd30aa5445eeffdddfb1cfd992eb47dc592a912e6974972ecb7a7"
Oct 31 20:20:25.141573 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:20:25.141548    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1670d35c1aefd30aa5445eeffdddfb1cfd992eb47dc592a912e6974972ecb7a7\": container with ID starting with 1670d35c1aefd30aa5445eeffdddfb1cfd992eb47dc592a912e6974972ecb7a7 not found: ID does not exist" containerID="1670d35c1aefd30aa5445eeffdddfb1cfd992eb47dc592a912e6974972ecb7a7"
Oct 31 20:20:25.141650 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:25.141584    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1670d35c1aefd30aa5445eeffdddfb1cfd992eb47dc592a912e6974972ecb7a7"} err="failed to get container status \"1670d35c1aefd30aa5445eeffdddfb1cfd992eb47dc592a912e6974972ecb7a7\": rpc error: code = NotFound desc = could not find container \"1670d35c1aefd30aa5445eeffdddfb1cfd992eb47dc592a912e6974972ecb7a7\": container with ID starting with 1670d35c1aefd30aa5445eeffdddfb1cfd992eb47dc592a912e6974972ecb7a7 not found: ID does not exist"
Oct 31 20:20:25.141650 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:25.141603    4645 scope.go:122] "RemoveContainer" containerID="20b1df8596b5b7fb6369a4036e590f2f4b5f358b74c5934466db94e801e83c41"
Oct 31 20:20:25.141851 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:20:25.141832    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20b1df8596b5b7fb6369a4036e590f2f4b5f358b74c5934466db94e801e83c41\": container with ID starting with 20b1df8596b5b7fb6369a4036e590f2f4b5f358b74c5934466db94e801e83c41 not found: ID does not exist" containerID="20b1df8596b5b7fb6369a4036e590f2f4b5f358b74c5934466db94e801e83c41"
Oct 31 20:20:25.141918 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:25.141856    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20b1df8596b5b7fb6369a4036e590f2f4b5f358b74c5934466db94e801e83c41"} err="failed to get container status \"20b1df8596b5b7fb6369a4036e590f2f4b5f358b74c5934466db94e801e83c41\": rpc error: code = NotFound desc = could not find container \"20b1df8596b5b7fb6369a4036e590f2f4b5f358b74c5934466db94e801e83c41\": container with ID starting with 20b1df8596b5b7fb6369a4036e590f2f4b5f358b74c5934466db94e801e83c41 not found: ID does not exist"
Oct 31 20:20:25.147992 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:25.147960    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w"]
Oct 31 20:20:25.151573 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:25.151542    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-844db7-openshift-default-54d4f5f85-p827w"]
Oct 31 20:20:25.616428 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:25.616395    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="672d276a-4c71-4742-82f6-f5c3dc95e072" path="/var/lib/kubelet/pods/672d276a-4c71-4742-82f6-f5c3dc95e072/volumes"
Oct 31 20:20:25.678178 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:25.678154    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-844db7-6f7649b585pscw" podUID="50110354-1374-44e2-aced-b0b068cd1bba" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:20:25.678178 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:25.678164    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-844db7-d587bc6mhg" podUID="d8e68310-9416-42f7-9ec5-f2f3348b8029" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:20:35.190354 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:35.189407    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-6rwmb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 20:20:35.190354 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:35.189531    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6rwmb" podUID="b92c4b09-c2c0-4602-af78-2f410c843118" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 20:20:35.208359 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:35.207414    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-68qbd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 20:20:35.208359 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:35.207488    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-68qbd" podUID="3a217b30-54c0-431f-90bc-0681901272f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 20:20:35.684349 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:20:35.677903    4645 logging.go:55] [core] [Channel #6308 SubChannel #6309]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.1.69:50052", ServerName: "10.133.1.69:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.1.69:50052: connect: connection refused"
Oct 31 20:20:35.684349 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:20:35.678190    4645 logging.go:55] [core] [Channel #6310 SubChannel #6311]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.1.70:50052", ServerName: "10.133.1.70:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.1.70:50052: connect: connection refused"
Oct 31 20:20:35.924138 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:35.923761    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-844db7-6f7649b585pscw"
Oct 31 20:20:35.965493 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:35.965214    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-844db7-d587bc6mhg"
Oct 31 20:20:35.992225 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:35.992197    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50110354-1374-44e2-aced-b0b068cd1bba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50110354-1374-44e2-aced-b0b068cd1bba-plugins-config-volume\") pod \"50110354-1374-44e2-aced-b0b068cd1bba\" (UID: \"50110354-1374-44e2-aced-b0b068cd1bba\") "
Oct 31 20:20:35.992530 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:35.992509    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50110354-1374-44e2-aced-b0b068cd1bba-kube-api-access-wq8t5\" (UniqueName: \"kubernetes.io/projected/50110354-1374-44e2-aced-b0b068cd1bba-kube-api-access-wq8t5\") pod \"50110354-1374-44e2-aced-b0b068cd1bba\" (UID: \"50110354-1374-44e2-aced-b0b068cd1bba\") "
Oct 31 20:20:35.994695 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:35.994651    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50110354-1374-44e2-aced-b0b068cd1bba-plugins-config-volume" pod "50110354-1374-44e2-aced-b0b068cd1bba" (UID: "50110354-1374-44e2-aced-b0b068cd1bba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:20:35.995653 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:35.995626    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50110354-1374-44e2-aced-b0b068cd1bba-kube-api-access-wq8t5" pod "50110354-1374-44e2-aced-b0b068cd1bba" (UID: "50110354-1374-44e2-aced-b0b068cd1bba"). InnerVolumeSpecName "kube-api-access-wq8t5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:20:36.094084 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:36.094048    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8e68310-9416-42f7-9ec5-f2f3348b8029-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8e68310-9416-42f7-9ec5-f2f3348b8029-plugins-config-volume\") pod \"d8e68310-9416-42f7-9ec5-f2f3348b8029\" (UID: \"d8e68310-9416-42f7-9ec5-f2f3348b8029\") "
Oct 31 20:20:36.094230 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:36.094166    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8e68310-9416-42f7-9ec5-f2f3348b8029-kube-api-access-nb7bf\" (UniqueName: \"kubernetes.io/projected/d8e68310-9416-42f7-9ec5-f2f3348b8029-kube-api-access-nb7bf\") pod \"d8e68310-9416-42f7-9ec5-f2f3348b8029\" (UID: \"d8e68310-9416-42f7-9ec5-f2f3348b8029\") "
Oct 31 20:20:36.094478 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:36.094454    4645 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50110354-1374-44e2-aced-b0b068cd1bba-plugins-config-volume\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:20:36.094659 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:36.094634    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wq8t5\" (UniqueName: \"kubernetes.io/projected/50110354-1374-44e2-aced-b0b068cd1bba-kube-api-access-wq8t5\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:20:36.094659 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:36.094527    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8e68310-9416-42f7-9ec5-f2f3348b8029-plugins-config-volume" pod "d8e68310-9416-42f7-9ec5-f2f3348b8029" (UID: "d8e68310-9416-42f7-9ec5-f2f3348b8029"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:20:36.096997 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:36.096963    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8e68310-9416-42f7-9ec5-f2f3348b8029-kube-api-access-nb7bf" pod "d8e68310-9416-42f7-9ec5-f2f3348b8029" (UID: "d8e68310-9416-42f7-9ec5-f2f3348b8029"). InnerVolumeSpecName "kube-api-access-nb7bf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:20:36.195757 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:36.195718    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nb7bf\" (UniqueName: \"kubernetes.io/projected/d8e68310-9416-42f7-9ec5-f2f3348b8029-kube-api-access-nb7bf\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:20:36.196175 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:36.195763    4645 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8e68310-9416-42f7-9ec5-f2f3348b8029-plugins-config-volume\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:20:36.217891 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:36.217857    4645 generic.go:356] "Generic (PLEG): container finished" podID="50110354-1374-44e2-aced-b0b068cd1bba" containerID="ace5fb30bec34aa7b38a8320dddaf0732c4ae27bed4bd611370c390bff1cb138" exitCode=0
Oct 31 20:20:36.217995 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:36.217922    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-844db7-6f7649b585pscw"
Oct 31 20:20:36.217995 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:36.217953    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-844db7-6f7649b585pscw" event={"ID":"50110354-1374-44e2-aced-b0b068cd1bba","Type":"ContainerDied","Data":"ace5fb30bec34aa7b38a8320dddaf0732c4ae27bed4bd611370c390bff1cb138"}
Oct 31 20:20:36.218132 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:36.218016    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-844db7-6f7649b585pscw" event={"ID":"50110354-1374-44e2-aced-b0b068cd1bba","Type":"ContainerDied","Data":"0c2a4ff8961fa296f66bee61022ec132fe3bb95994453709a395ca29089437d1"}
Oct 31 20:20:36.218132 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:36.218052    4645 scope.go:122] "RemoveContainer" containerID="ace5fb30bec34aa7b38a8320dddaf0732c4ae27bed4bd611370c390bff1cb138"
Oct 31 20:20:36.222676 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:36.220177    4645 generic.go:356] "Generic (PLEG): container finished" podID="d8e68310-9416-42f7-9ec5-f2f3348b8029" containerID="d3cc5fd9d2e8c721f37d963b26864178acca8e7c80332ff2594309f36fa9aaa8" exitCode=0
Oct 31 20:20:36.222676 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:36.220250    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-844db7-d587bc6mhg" event={"ID":"d8e68310-9416-42f7-9ec5-f2f3348b8029","Type":"ContainerDied","Data":"d3cc5fd9d2e8c721f37d963b26864178acca8e7c80332ff2594309f36fa9aaa8"}
Oct 31 20:20:36.222676 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:36.220301    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-844db7-d587bc6mhg" event={"ID":"d8e68310-9416-42f7-9ec5-f2f3348b8029","Type":"ContainerDied","Data":"f6dea9fe6c37d41af72c0de865280953cd1d53a5d4df4598a80ce5917f69cb39"}
Oct 31 20:20:36.222676 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:36.220391    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-844db7-d587bc6mhg"
Oct 31 20:20:36.231710 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:36.231471    4645 scope.go:122] "RemoveContainer" containerID="ace5fb30bec34aa7b38a8320dddaf0732c4ae27bed4bd611370c390bff1cb138"
Oct 31 20:20:36.231809 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:20:36.231785    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ace5fb30bec34aa7b38a8320dddaf0732c4ae27bed4bd611370c390bff1cb138\": container with ID starting with ace5fb30bec34aa7b38a8320dddaf0732c4ae27bed4bd611370c390bff1cb138 not found: ID does not exist" containerID="ace5fb30bec34aa7b38a8320dddaf0732c4ae27bed4bd611370c390bff1cb138"
Oct 31 20:20:36.232175 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:36.231818    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ace5fb30bec34aa7b38a8320dddaf0732c4ae27bed4bd611370c390bff1cb138"} err="failed to get container status \"ace5fb30bec34aa7b38a8320dddaf0732c4ae27bed4bd611370c390bff1cb138\": rpc error: code = NotFound desc = could not find container \"ace5fb30bec34aa7b38a8320dddaf0732c4ae27bed4bd611370c390bff1cb138\": container with ID starting with ace5fb30bec34aa7b38a8320dddaf0732c4ae27bed4bd611370c390bff1cb138 not found: ID does not exist"
Oct 31 20:20:36.232175 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:36.231840    4645 scope.go:122] "RemoveContainer" containerID="d3cc5fd9d2e8c721f37d963b26864178acca8e7c80332ff2594309f36fa9aaa8"
Oct 31 20:20:36.253769 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:36.252163    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-844db7-6f7649b585pscw"]
Oct 31 20:20:36.254801 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:36.254767    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-844db7-6f7649b585pscw"]
Oct 31 20:20:36.258396 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:36.258275    4645 scope.go:122] "RemoveContainer" containerID="d3cc5fd9d2e8c721f37d963b26864178acca8e7c80332ff2594309f36fa9aaa8"
Oct 31 20:20:36.259780 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:20:36.259679    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3cc5fd9d2e8c721f37d963b26864178acca8e7c80332ff2594309f36fa9aaa8\": container with ID starting with d3cc5fd9d2e8c721f37d963b26864178acca8e7c80332ff2594309f36fa9aaa8 not found: ID does not exist" containerID="d3cc5fd9d2e8c721f37d963b26864178acca8e7c80332ff2594309f36fa9aaa8"
Oct 31 20:20:36.259780 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:36.259717    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3cc5fd9d2e8c721f37d963b26864178acca8e7c80332ff2594309f36fa9aaa8"} err="failed to get container status \"d3cc5fd9d2e8c721f37d963b26864178acca8e7c80332ff2594309f36fa9aaa8\": rpc error: code = NotFound desc = could not find container \"d3cc5fd9d2e8c721f37d963b26864178acca8e7c80332ff2594309f36fa9aaa8\": container with ID starting with d3cc5fd9d2e8c721f37d963b26864178acca8e7c80332ff2594309f36fa9aaa8 not found: ID does not exist"
Oct 31 20:20:36.264072 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:36.264041    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-844db7-d587bc6mhg"]
Oct 31 20:20:36.265742 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:20:36.265713    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 20:20:36.271669 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:36.271645    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-844db7-d587bc6mhg"]
Oct 31 20:20:36.682832 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:36.682274    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-844db7-6f7649b585pscw" podUID="50110354-1374-44e2-aced-b0b068cd1bba" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.1.70:50052\" within 1s: context deadline exceeded"
Oct 31 20:20:36.682832 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:36.682443    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-844db7-d587bc6mhg" podUID="d8e68310-9416-42f7-9ec5-f2f3348b8029" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.1.69:50052\" within 1s: context deadline exceeded"
Oct 31 20:20:36.682832 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:20:36.682577    4645 logging.go:55] [core] [Channel #6310 SubChannel #6311]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.1.70:50052", ServerName: "10.133.1.70:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.1.70:50052: operation was canceled"
Oct 31 20:20:37.634353 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:37.634162    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50110354-1374-44e2-aced-b0b068cd1bba" path="/var/lib/kubelet/pods/50110354-1374-44e2-aced-b0b068cd1bba/volumes"
Oct 31 20:20:37.638348 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:37.635879    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8e68310-9416-42f7-9ec5-f2f3348b8029" path="/var/lib/kubelet/pods/d8e68310-9416-42f7-9ec5-f2f3348b8029/volumes"
Oct 31 20:20:41.284574 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:41.284534    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:20:41.388686 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:41.388654    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:20:42.558181 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:42.558147    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:20:43.683421 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:43.683389    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:20:49.818921 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:20:49.818862    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:21:06.245385 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:21:06.245310    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 20:21:19.079937 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:21:19.079898    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:21:20.137355 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:21:20.137305    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:21:21.176770 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:21:21.176738    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:21:36.244471 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:21:36.244433    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 20:21:38.487411 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:21:38.487358    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:22:05.183580 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:05.183527    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-6rwmb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 20:22:05.184168 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:05.183631    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6rwmb" podUID="b92c4b09-c2c0-4602-af78-2f410c843118" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 20:22:05.196362 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:05.195542    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-68qbd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 20:22:05.196362 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:05.195630    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-68qbd" podUID="3a217b30-54c0-431f-90bc-0681901272f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 20:22:05.349356 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:05.348426    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-t27sn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 20:22:05.349356 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:05.348511    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-t27sn" podUID="f3994be2-19d9-4e2d-b3ce-03f63d2c9c32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 20:22:06.246115 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:22:06.246073    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 20:22:19.124447 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:19.124403    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:22:20.200555 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:20.200510    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:22:21.319359 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:21.310816    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:22:36.245911 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:22:36.245863    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 20:22:37.109159 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:37.109123    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:22:37.522999 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:37.522963    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj"]
Oct 31 20:22:37.523534 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:37.523394    4645 cpu_manager.go:414] "containerMap: removing container" podUID="d8e68310-9416-42f7-9ec5-f2f3348b8029" containerName="payload-pre-processing"
Oct 31 20:22:37.523534 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:37.523409    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8e68310-9416-42f7-9ec5-f2f3348b8029" containerName="payload-pre-processing"
Oct 31 20:22:37.523534 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:37.523419    4645 cpu_manager.go:414] "containerMap: removing container" podUID="672d276a-4c71-4742-82f6-f5c3dc95e072" containerName="istio-proxy"
Oct 31 20:22:37.523534 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:37.523424    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="672d276a-4c71-4742-82f6-f5c3dc95e072" containerName="istio-proxy"
Oct 31 20:22:37.523534 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:37.523446    4645 cpu_manager.go:414] "containerMap: removing container" podUID="f67b87d1-b808-4393-9224-1c45bf38a373" containerName="maas-api"
Oct 31 20:22:37.523534 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:37.523453    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f67b87d1-b808-4393-9224-1c45bf38a373" containerName="maas-api"
Oct 31 20:22:37.523534 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:37.523468    4645 cpu_manager.go:414] "containerMap: removing container" podUID="50110354-1374-44e2-aced-b0b068cd1bba" containerName="payload-processing"
Oct 31 20:22:37.523534 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:37.523473    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50110354-1374-44e2-aced-b0b068cd1bba" containerName="payload-processing"
Oct 31 20:22:37.523964 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:37.523548    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="f67b87d1-b808-4393-9224-1c45bf38a373" containerName="maas-api"
Oct 31 20:22:37.523964 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:37.523559    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="50110354-1374-44e2-aced-b0b068cd1bba" containerName="payload-processing"
Oct 31 20:22:37.523964 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:37.523567    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="672d276a-4c71-4742-82f6-f5c3dc95e072" containerName="istio-proxy"
Oct 31 20:22:37.523964 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:37.523575    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8e68310-9416-42f7-9ec5-f2f3348b8029" containerName="payload-pre-processing"
Oct 31 20:22:37.523964 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:37.523582    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="672d276a-4c71-4742-82f6-f5c3dc95e072" containerName="istio-proxy"
Oct 31 20:22:37.526662 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:37.526645    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj"
Oct 31 20:22:37.532876 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:37.532842    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-8407e9-openshift-default-dockercfg-9wqml\""
Oct 31 20:22:37.546218 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:37.546187    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj"]
Oct 31 20:22:37.598570 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:37.598528    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:22:37.604289 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:37.604236    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3d499c3-4252-4867-a0f7-917c0001d831-istio-envoy\") pod \"e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj\" (UID: \"a3d499c3-4252-4867-a0f7-917c0001d831\") " pod="openshift-ingress/e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj"
Oct 31 20:22:37.604670 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:37.604640    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3d499c3-4252-4867-a0f7-917c0001d831-istio-token\") pod \"e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj\" (UID: \"a3d499c3-4252-4867-a0f7-917c0001d831\") " pod="openshift-ingress/e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj"
Oct 31 20:22:37.604925 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:37.604899    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3d499c3-4252-4867-a0f7-917c0001d831-workload-certs\") pod \"e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj\" (UID: \"a3d499c3-4252-4867-a0f7-917c0001d831\") " pod="openshift-ingress/e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj"
Oct 31 20:22:37.605067 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:37.604935    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3d499c3-4252-4867-a0f7-917c0001d831-credential-socket\") pod \"e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj\" (UID: \"a3d499c3-4252-4867-a0f7-917c0001d831\") " pod="openshift-ingress/e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj"
Oct 31 20:22:37.605067 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:37.604990    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3d499c3-4252-4867-a0f7-917c0001d831-istio-data\") pod \"e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj\" (UID: \"a3d499c3-4252-4867-a0f7-917c0001d831\") " pod="openshift-ingress/e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj"
Oct 31 20:22:37.605067 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:37.605045    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3d499c3-4252-4867-a0f7-917c0001d831-workload-socket\") pod \"e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj\" (UID: \"a3d499c3-4252-4867-a0f7-917c0001d831\") " pod="openshift-ingress/e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj"
Oct 31 20:22:37.605264 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:37.605077    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xw4m7\" (UniqueName: \"kubernetes.io/projected/a3d499c3-4252-4867-a0f7-917c0001d831-kube-api-access-xw4m7\") pod \"e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj\" (UID: \"a3d499c3-4252-4867-a0f7-917c0001d831\") " pod="openshift-ingress/e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj"
Oct 31 20:22:37.605264 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:37.605122    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3d499c3-4252-4867-a0f7-917c0001d831-istiod-ca-cert\") pod \"e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj\" (UID: \"a3d499c3-4252-4867-a0f7-917c0001d831\") " pod="openshift-ingress/e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj"
Oct 31 20:22:37.605264 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:37.605151    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3d499c3-4252-4867-a0f7-917c0001d831-istio-podinfo\") pod \"e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj\" (UID: \"a3d499c3-4252-4867-a0f7-917c0001d831\") " pod="openshift-ingress/e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj"
Oct 31 20:22:37.706107 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:37.706064    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3d499c3-4252-4867-a0f7-917c0001d831-workload-socket\") pod \"e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj\" (UID: \"a3d499c3-4252-4867-a0f7-917c0001d831\") " pod="openshift-ingress/e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj"
Oct 31 20:22:37.706349 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:37.706125    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xw4m7\" (UniqueName: \"kubernetes.io/projected/a3d499c3-4252-4867-a0f7-917c0001d831-kube-api-access-xw4m7\") pod \"e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj\" (UID: \"a3d499c3-4252-4867-a0f7-917c0001d831\") " pod="openshift-ingress/e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj"
Oct 31 20:22:37.706349 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:37.706160    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3d499c3-4252-4867-a0f7-917c0001d831-istiod-ca-cert\") pod \"e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj\" (UID: \"a3d499c3-4252-4867-a0f7-917c0001d831\") " pod="openshift-ingress/e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj"
Oct 31 20:22:37.706349 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:37.706178    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3d499c3-4252-4867-a0f7-917c0001d831-istio-podinfo\") pod \"e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj\" (UID: \"a3d499c3-4252-4867-a0f7-917c0001d831\") " pod="openshift-ingress/e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj"
Oct 31 20:22:37.706349 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:37.706216    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3d499c3-4252-4867-a0f7-917c0001d831-istio-envoy\") pod \"e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj\" (UID: \"a3d499c3-4252-4867-a0f7-917c0001d831\") " pod="openshift-ingress/e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj"
Oct 31 20:22:37.706349 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:37.706291    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3d499c3-4252-4867-a0f7-917c0001d831-istio-token\") pod \"e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj\" (UID: \"a3d499c3-4252-4867-a0f7-917c0001d831\") " pod="openshift-ingress/e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj"
Oct 31 20:22:37.706727 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:37.706369    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3d499c3-4252-4867-a0f7-917c0001d831-workload-certs\") pod \"e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj\" (UID: \"a3d499c3-4252-4867-a0f7-917c0001d831\") " pod="openshift-ingress/e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj"
Oct 31 20:22:37.706727 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:37.706394    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3d499c3-4252-4867-a0f7-917c0001d831-credential-socket\") pod \"e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj\" (UID: \"a3d499c3-4252-4867-a0f7-917c0001d831\") " pod="openshift-ingress/e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj"
Oct 31 20:22:37.706727 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:37.706449    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3d499c3-4252-4867-a0f7-917c0001d831-istio-data\") pod \"e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj\" (UID: \"a3d499c3-4252-4867-a0f7-917c0001d831\") " pod="openshift-ingress/e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj"
Oct 31 20:22:37.706921 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:37.706734    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3d499c3-4252-4867-a0f7-917c0001d831-workload-socket\") pod \"e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj\" (UID: \"a3d499c3-4252-4867-a0f7-917c0001d831\") " pod="openshift-ingress/e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj"
Oct 31 20:22:37.706921 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:37.706840    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3d499c3-4252-4867-a0f7-917c0001d831-istio-data\") pod \"e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj\" (UID: \"a3d499c3-4252-4867-a0f7-917c0001d831\") " pod="openshift-ingress/e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj"
Oct 31 20:22:37.707050 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:37.707017    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3d499c3-4252-4867-a0f7-917c0001d831-workload-certs\") pod \"e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj\" (UID: \"a3d499c3-4252-4867-a0f7-917c0001d831\") " pod="openshift-ingress/e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj"
Oct 31 20:22:37.707050 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:37.707034    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3d499c3-4252-4867-a0f7-917c0001d831-credential-socket\") pod \"e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj\" (UID: \"a3d499c3-4252-4867-a0f7-917c0001d831\") " pod="openshift-ingress/e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj"
Oct 31 20:22:37.707175 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:37.707117    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3d499c3-4252-4867-a0f7-917c0001d831-istiod-ca-cert\") pod \"e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj\" (UID: \"a3d499c3-4252-4867-a0f7-917c0001d831\") " pod="openshift-ingress/e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj"
Oct 31 20:22:37.708854 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:37.708829    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3d499c3-4252-4867-a0f7-917c0001d831-istio-envoy\") pod \"e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj\" (UID: \"a3d499c3-4252-4867-a0f7-917c0001d831\") " pod="openshift-ingress/e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj"
Oct 31 20:22:37.709225 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:37.709205    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3d499c3-4252-4867-a0f7-917c0001d831-istio-podinfo\") pod \"e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj\" (UID: \"a3d499c3-4252-4867-a0f7-917c0001d831\") " pod="openshift-ingress/e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj"
Oct 31 20:22:37.720654 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:37.720627    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3d499c3-4252-4867-a0f7-917c0001d831-istio-token\") pod \"e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj\" (UID: \"a3d499c3-4252-4867-a0f7-917c0001d831\") " pod="openshift-ingress/e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj"
Oct 31 20:22:37.721864 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:37.721833    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xw4m7\" (UniqueName: \"kubernetes.io/projected/a3d499c3-4252-4867-a0f7-917c0001d831-kube-api-access-xw4m7\") pod \"e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj\" (UID: \"a3d499c3-4252-4867-a0f7-917c0001d831\") " pod="openshift-ingress/e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj"
Oct 31 20:22:37.808788 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:37.808693    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:22:37.840626 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:37.840587    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj"
Oct 31 20:22:38.124155 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:38.124075    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj"]
Oct 31 20:22:38.127598 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:22:38.127571    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3d499c3_4252_4867_a0f7_917c0001d831.slice/crio-0bdf85a23e6bc849fa89ab2a36661f06a644b26548748321877d3e1d8467a209 WatchSource:0}: Error finding container 0bdf85a23e6bc849fa89ab2a36661f06a644b26548748321877d3e1d8467a209: Status 404 returned error can't find the container with id 0bdf85a23e6bc849fa89ab2a36661f06a644b26548748321877d3e1d8467a209
Oct 31 20:22:38.130082 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:38.130048    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:22:38.130162 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:38.130117    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:22:38.130162 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:38.130151    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:22:38.920806 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:38.920767    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj" event={"ID":"a3d499c3-4252-4867-a0f7-917c0001d831","Type":"ContainerStarted","Data":"33a75414c2d8d8cd467e9b695ac022a291852a45c020eb2c04c0542b051478e0"}
Oct 31 20:22:38.921242 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:38.920822    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj" event={"ID":"a3d499c3-4252-4867-a0f7-917c0001d831","Type":"ContainerStarted","Data":"0bdf85a23e6bc849fa89ab2a36661f06a644b26548748321877d3e1d8467a209"}
Oct 31 20:22:38.942824 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:38.942759    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj" podStartSLOduration=1.942742588 podStartE2EDuration="1.942742588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:22:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:22:38.942125883 +0000 UTC m=+3789.515360011" watchObservedRunningTime="2026-10-31 20:22:38.942742588 +0000 UTC m=+3789.515976743"
Oct 31 20:22:39.841162 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:39.841124    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj"
Oct 31 20:22:39.846262 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:39.846232    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj"
Oct 31 20:22:39.925458 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:39.925426    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj"
Oct 31 20:22:39.926362 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:39.926342    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj"
Oct 31 20:22:41.310501 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:41.310465    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:22:41.641802 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:41.641718    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:22:42.560172 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:42.560109    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:22:42.643280 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:42.643242    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-8407e9-544db86kfw"]
Oct 31 20:22:42.643682 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:42.643669    4645 cpu_manager.go:414] "containerMap: removing container" podUID="672d276a-4c71-4742-82f6-f5c3dc95e072" containerName="istio-proxy"
Oct 31 20:22:42.643731 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:42.643684    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="672d276a-4c71-4742-82f6-f5c3dc95e072" containerName="istio-proxy"
Oct 31 20:22:42.646859 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:42.646841    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8407e9-544db86kfw"
Oct 31 20:22:42.658085 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:42.658061    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-8407e9\""
Oct 31 20:22:42.660202 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:42.660176    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-8407e9-544db86kfw"]
Oct 31 20:22:42.672106 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:42.672075    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-16df549e-dockercfg-kzg8f\""
Oct 31 20:22:42.690750 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:42.690704    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:22:42.761905 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:42.761869    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9eadbab-6f06-49a9-87ba-9853a3a7e8f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-8407e9-544db86kfw\" (UID: \"f9eadbab-6f06-49a9-87ba-9853a3a7e8f4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8407e9-544db86kfw"
Oct 31 20:22:42.762096 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:42.761930    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzrlc\" (UniqueName: \"kubernetes.io/projected/f9eadbab-6f06-49a9-87ba-9853a3a7e8f4-kube-api-access-tzrlc\") pod \"payload-pre-processing-e2e-worker-single-main-8407e9-544db86kfw\" (UID: \"f9eadbab-6f06-49a9-87ba-9853a3a7e8f4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8407e9-544db86kfw"
Oct 31 20:22:42.794745 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:42.794702    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:22:42.863044 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:42.862946    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzrlc\" (UniqueName: \"kubernetes.io/projected/f9eadbab-6f06-49a9-87ba-9853a3a7e8f4-kube-api-access-tzrlc\") pod \"payload-pre-processing-e2e-worker-single-main-8407e9-544db86kfw\" (UID: \"f9eadbab-6f06-49a9-87ba-9853a3a7e8f4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8407e9-544db86kfw"
Oct 31 20:22:42.863218 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:42.863075    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9eadbab-6f06-49a9-87ba-9853a3a7e8f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-8407e9-544db86kfw\" (UID: \"f9eadbab-6f06-49a9-87ba-9853a3a7e8f4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8407e9-544db86kfw"
Oct 31 20:22:42.863842 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:42.863821    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9eadbab-6f06-49a9-87ba-9853a3a7e8f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-8407e9-544db86kfw\" (UID: \"f9eadbab-6f06-49a9-87ba-9853a3a7e8f4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8407e9-544db86kfw"
Oct 31 20:22:42.874437 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:42.874411    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzrlc\" (UniqueName: \"kubernetes.io/projected/f9eadbab-6f06-49a9-87ba-9853a3a7e8f4-kube-api-access-tzrlc\") pod \"payload-pre-processing-e2e-worker-single-main-8407e9-544db86kfw\" (UID: \"f9eadbab-6f06-49a9-87ba-9853a3a7e8f4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8407e9-544db86kfw"
Oct 31 20:22:42.961223 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:42.961184    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8407e9-544db86kfw"
Oct 31 20:22:43.239525 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:43.239486    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-8407e9-544db86kfw"]
Oct 31 20:22:43.242894 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:22:43.242869    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9eadbab_6f06_49a9_87ba_9853a3a7e8f4.slice/crio-b0df389a78f0d895abe1d8fe564b752b3cbd340a21e3dc29e376a7357558ba5b WatchSource:0}: Error finding container b0df389a78f0d895abe1d8fe564b752b3cbd340a21e3dc29e376a7357558ba5b: Status 404 returned error can't find the container with id b0df389a78f0d895abe1d8fe564b752b3cbd340a21e3dc29e376a7357558ba5b
Oct 31 20:22:43.472442 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:43.472400    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:22:43.945683 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:43.945642    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8407e9-544db86kfw" event={"ID":"f9eadbab-6f06-49a9-87ba-9853a3a7e8f4","Type":"ContainerStarted","Data":"07b51a343c0d4eeb4990eac8e6b2e3f9b4cbfd8de91982ad2a7e3fc120392e4e"}
Oct 31 20:22:43.946108 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:43.945701    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8407e9-544db86kfw" event={"ID":"f9eadbab-6f06-49a9-87ba-9853a3a7e8f4","Type":"ContainerStarted","Data":"b0df389a78f0d895abe1d8fe564b752b3cbd340a21e3dc29e376a7357558ba5b"}
Oct 31 20:22:43.946108 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:43.945742    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8407e9-544db86kfw"
Oct 31 20:22:43.964138 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:43.964084    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8407e9-544db86kfw" podStartSLOduration=1.96407029 podStartE2EDuration="1.96407029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:22:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:22:43.963149846 +0000 UTC m=+3794.536383975" watchObservedRunningTime="2026-10-31 20:22:43.96407029 +0000 UTC m=+3794.537304418"
Oct 31 20:22:44.366873 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:44.366785    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:22:45.280298 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:45.280261    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:22:46.053351 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:46.053287    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:22:46.781863 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:46.781811    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:22:47.648719 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:47.648675    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:22:48.449553 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:48.449515    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:22:49.178777 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:49.178719    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:22:49.851935 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:49.851894    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:22:50.737699 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:50.737664    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:22:51.529183 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:51.529139    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:22:52.399674 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:52.399566    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:22:53.067458 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:53.067417    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:22:53.957219 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:53.957178    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:22:54.668540 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:54.668502    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:22:54.926914 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:54.926827    4645 patch_prober.go:29] interesting pod/e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.75:15021/healthz/ready\": dial tcp 10.133.1.75:15021: connect: connection refused" start-of-body=
Oct 31 20:22:54.926914 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:54.926884    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj" podUID="a3d499c3-4252-4867-a0f7-917c0001d831" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.75:15021/healthz/ready\": dial tcp 10.133.1.75:15021: connect: connection refused"
Oct 31 20:22:54.953387 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:54.953355    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8407e9-544db86kfw"
Oct 31 20:22:55.017776 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:55.017741    4645 generic.go:356] "Generic (PLEG): container finished" podID="a3d499c3-4252-4867-a0f7-917c0001d831" containerID="33a75414c2d8d8cd467e9b695ac022a291852a45c020eb2c04c0542b051478e0" exitCode=0
Oct 31 20:22:55.017965 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:55.017816    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj" event={"ID":"a3d499c3-4252-4867-a0f7-917c0001d831","Type":"ContainerDied","Data":"33a75414c2d8d8cd467e9b695ac022a291852a45c020eb2c04c0542b051478e0"}
Oct 31 20:22:55.018224 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:55.018208    4645 scope.go:122] "RemoveContainer" containerID="33a75414c2d8d8cd467e9b695ac022a291852a45c020eb2c04c0542b051478e0"
Oct 31 20:22:55.019596 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:55.019563    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:22:55.019711 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:55.019635    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:22:55.019711 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:55.019666    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:22:55.444632 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:55.444473    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:22:56.024238 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:56.024201    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj" event={"ID":"a3d499c3-4252-4867-a0f7-917c0001d831","Type":"ContainerStarted","Data":"e25c44efa7bdecc373a062d44502396db771aaff9a00f6937bad44cd3437446d"}
Oct 31 20:22:56.303517 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:56.303426    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:22:56.441423 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:56.441385    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:22:56.841212 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:56.841168    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj"
Oct 31 20:22:56.841456 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:56.841417    4645 patch_prober.go:29] interesting pod/e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.75:15021/healthz/ready\": dial tcp 10.133.1.75:15021: connect: connection refused" start-of-body=
Oct 31 20:22:56.841513 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:56.841465    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj" podUID="a3d499c3-4252-4867-a0f7-917c0001d831" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.75:15021/healthz/ready\": dial tcp 10.133.1.75:15021: connect: connection refused"
Oct 31 20:22:57.177572 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:57.177533    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:22:57.312870 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:57.312830    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:22:57.841806 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:57.841768    4645 patch_prober.go:29] interesting pod/e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.75:15021/healthz/ready\": dial tcp 10.133.1.75:15021: connect: connection refused" start-of-body=
Oct 31 20:22:57.842017 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:57.841821    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj" podUID="a3d499c3-4252-4867-a0f7-917c0001d831" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.75:15021/healthz/ready\": dial tcp 10.133.1.75:15021: connect: connection refused"
Oct 31 20:22:58.093531 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:58.093451    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:22:58.338426 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:58.338381    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:22:58.840997 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:58.840960    4645 patch_prober.go:29] interesting pod/e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.75:15021/healthz/ready\": dial tcp 10.133.1.75:15021: connect: connection refused" start-of-body=
Oct 31 20:22:58.841205 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:58.841013    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj" podUID="a3d499c3-4252-4867-a0f7-917c0001d831" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.75:15021/healthz/ready\": dial tcp 10.133.1.75:15021: connect: connection refused"
Oct 31 20:22:59.082104 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:59.082062    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:22:59.841163 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:59.841115    4645 patch_prober.go:29] interesting pod/e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.75:15021/healthz/ready\": dial tcp 10.133.1.75:15021: connect: connection refused" start-of-body=
Oct 31 20:22:59.841659 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:59.841195    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj" podUID="a3d499c3-4252-4867-a0f7-917c0001d831" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.75:15021/healthz/ready\": dial tcp 10.133.1.75:15021: connect: connection refused"
Oct 31 20:22:59.969354 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:22:59.965746    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:00.841826 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:00.841783    4645 patch_prober.go:29] interesting pod/e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.75:15021/healthz/ready\": dial tcp 10.133.1.75:15021: connect: connection refused" start-of-body=
Oct 31 20:23:00.842268 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:00.841842    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj" podUID="a3d499c3-4252-4867-a0f7-917c0001d831" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.75:15021/healthz/ready\": dial tcp 10.133.1.75:15021: connect: connection refused"
Oct 31 20:23:00.852020 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:00.851992    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:01.664519 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:01.664481    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:01.845689 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:01.845659    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj"
Oct 31 20:23:01.846195 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:01.846050    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj"
Oct 31 20:23:01.846770 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:01.846749    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj"
Oct 31 20:23:02.468107 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:02.468061    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:03.235569 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:03.235497    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:04.002831 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:04.002787    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:04.762978 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:04.762946    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:05.548987 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:05.548949    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:06.245914 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:23:06.245873    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 20:23:06.256963 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:06.256924    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:07.016852 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:07.016815    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:07.849693 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:07.849654    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:08.761548 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:08.761508    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:09.559721 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:09.559685    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:10.330779 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:10.330732    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:11.150832 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:11.150797    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:11.844667 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:11.844627    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:12.561221 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:12.561186    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:13.353311 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:13.353270    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:14.123439 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:14.123393    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:15.008493 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:15.008451    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:15.879847 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:15.879805    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:17.148593 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:17.148554    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:18.066988 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:18.066945    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:18.918653 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:18.918617    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:19.159308 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:19.159266    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:19.656842 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:19.656803    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:20.264549 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:20.264505    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:20.540480 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:20.540383    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:21.359402 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:21.359367    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:21.461586 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:21.461542    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:22.136342 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:22.136291    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:23.063528 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:23.063484    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:23.869551 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:23.869508    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:24.636441 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:24.636393    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:25.361765 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:25.361730    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:26.149089 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:26.149048    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:26.925978 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:26.925915    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:27.648001 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:27.647966    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:28.454847 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:28.454811    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:29.255057 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:29.255026    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:30.156878 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:30.156839    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:30.868570 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:30.868533    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:31.642025 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:31.641980    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:32.371114 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:32.371076    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:33.154025 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:33.153986    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:33.857431 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:33.857398    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:34.558279 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:34.558242    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:35.253412 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:35.253365    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:36.048485 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:36.048446    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:36.245685 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:23:36.245649    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 20:23:36.849670 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:36.849634    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:37.572853 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:37.572816    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:38.312003 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:38.311966    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:39.162382 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:39.162345    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:39.544092 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:39.544005    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:40.035945 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:40.035891    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:40.961807 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:40.961770    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:41.656317 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:41.656279    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:42.600020 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:42.599983    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:43.358504 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:43.358468    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:44.091559 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:44.091528    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:44.601145 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:44.601108    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:44.947451 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:44.947416    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:45.751125 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:45.751085    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:46.453687 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:46.453650    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:47.183411 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:47.183376    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:47.939848 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:47.939792    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:48.648760 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:48.648704    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:49.387118 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:49.387081    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:49.949211 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:49.949176    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-8407e9-544db86kfw"]
Oct 31 20:23:50.003919 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:50.003884    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:50.121661 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:50.121626    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:50.370386 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:50.370253    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:50.570086 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:50.570039    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:50.947776 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:50.947745    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:54.949703 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:54.949660    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8407e9-544db86kfw" podUID="f9eadbab-6f06-49a9-87ba-9853a3a7e8f4" containerName="payload-pre-processing" containerID="cri-o://07b51a343c0d4eeb4990eac8e6b2e3f9b4cbfd8de91982ad2a7e3fc120392e4e" gracePeriod=25
Oct 31 20:23:57.309314 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:57.309277    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:57.416691 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:57.416648    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:23:57.622626 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:57.622540    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj"]
Oct 31 20:23:57.622850 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:57.622814    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj" podUID="a3d499c3-4252-4867-a0f7-917c0001d831" containerName="istio-proxy" containerID="cri-o://e25c44efa7bdecc373a062d44502396db771aaff9a00f6937bad44cd3437446d" gracePeriod=30
Oct 31 20:23:57.674172 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:23:57.674146    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:24:01.848578 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:01.848544    4645 patch_prober.go:29] interesting pod/e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 20:24:01.848969 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:01.848604    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj" podUID="a3d499c3-4252-4867-a0f7-917c0001d831" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 20:24:02.871079 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:02.871049    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj"
Oct 31 20:24:02.942123 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:02.942097    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3d499c3-4252-4867-a0f7-917c0001d831-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3d499c3-4252-4867-a0f7-917c0001d831-istio-envoy\") pod \"a3d499c3-4252-4867-a0f7-917c0001d831\" (UID: \"a3d499c3-4252-4867-a0f7-917c0001d831\") "
Oct 31 20:24:02.942275 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:02.942149    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a3d499c3-4252-4867-a0f7-917c0001d831-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3d499c3-4252-4867-a0f7-917c0001d831-istio-podinfo\") pod \"a3d499c3-4252-4867-a0f7-917c0001d831\" (UID: \"a3d499c3-4252-4867-a0f7-917c0001d831\") "
Oct 31 20:24:02.942275 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:02.942208    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3d499c3-4252-4867-a0f7-917c0001d831-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3d499c3-4252-4867-a0f7-917c0001d831-credential-socket\") pod \"a3d499c3-4252-4867-a0f7-917c0001d831\" (UID: \"a3d499c3-4252-4867-a0f7-917c0001d831\") "
Oct 31 20:24:02.942275 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:02.942232    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3d499c3-4252-4867-a0f7-917c0001d831-istio-token\" (UniqueName: \"kubernetes.io/projected/a3d499c3-4252-4867-a0f7-917c0001d831-istio-token\") pod \"a3d499c3-4252-4867-a0f7-917c0001d831\" (UID: \"a3d499c3-4252-4867-a0f7-917c0001d831\") "
Oct 31 20:24:02.942275 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:02.942273    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3d499c3-4252-4867-a0f7-917c0001d831-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3d499c3-4252-4867-a0f7-917c0001d831-workload-socket\") pod \"a3d499c3-4252-4867-a0f7-917c0001d831\" (UID: \"a3d499c3-4252-4867-a0f7-917c0001d831\") "
Oct 31 20:24:02.942544 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:02.942297    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3d499c3-4252-4867-a0f7-917c0001d831-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3d499c3-4252-4867-a0f7-917c0001d831-istio-data\") pod \"a3d499c3-4252-4867-a0f7-917c0001d831\" (UID: \"a3d499c3-4252-4867-a0f7-917c0001d831\") "
Oct 31 20:24:02.942544 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:02.942360    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3d499c3-4252-4867-a0f7-917c0001d831-kube-api-access-xw4m7\" (UniqueName: \"kubernetes.io/projected/a3d499c3-4252-4867-a0f7-917c0001d831-kube-api-access-xw4m7\") pod \"a3d499c3-4252-4867-a0f7-917c0001d831\" (UID: \"a3d499c3-4252-4867-a0f7-917c0001d831\") "
Oct 31 20:24:02.942544 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:02.942401    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3d499c3-4252-4867-a0f7-917c0001d831-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3d499c3-4252-4867-a0f7-917c0001d831-workload-certs\") pod \"a3d499c3-4252-4867-a0f7-917c0001d831\" (UID: \"a3d499c3-4252-4867-a0f7-917c0001d831\") "
Oct 31 20:24:02.942544 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:02.942462    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3d499c3-4252-4867-a0f7-917c0001d831-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3d499c3-4252-4867-a0f7-917c0001d831-istiod-ca-cert\") pod \"a3d499c3-4252-4867-a0f7-917c0001d831\" (UID: \"a3d499c3-4252-4867-a0f7-917c0001d831\") "
Oct 31 20:24:02.942852 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:02.942540    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3d499c3-4252-4867-a0f7-917c0001d831-credential-socket" pod "a3d499c3-4252-4867-a0f7-917c0001d831" (UID: "a3d499c3-4252-4867-a0f7-917c0001d831"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:24:02.942852 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:02.942634    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3d499c3-4252-4867-a0f7-917c0001d831-workload-socket" pod "a3d499c3-4252-4867-a0f7-917c0001d831" (UID: "a3d499c3-4252-4867-a0f7-917c0001d831"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:24:02.942852 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:02.942784    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3d499c3-4252-4867-a0f7-917c0001d831-workload-certs" pod "a3d499c3-4252-4867-a0f7-917c0001d831" (UID: "a3d499c3-4252-4867-a0f7-917c0001d831"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:24:02.942852 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:02.942805    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3d499c3-4252-4867-a0f7-917c0001d831-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:24:02.943087 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:02.942865    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3d499c3-4252-4867-a0f7-917c0001d831-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:24:02.943087 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:02.942915    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3d499c3-4252-4867-a0f7-917c0001d831-istio-data" pod "a3d499c3-4252-4867-a0f7-917c0001d831" (UID: "a3d499c3-4252-4867-a0f7-917c0001d831"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:24:02.943087 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:02.942977    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3d499c3-4252-4867-a0f7-917c0001d831-istiod-ca-cert" pod "a3d499c3-4252-4867-a0f7-917c0001d831" (UID: "a3d499c3-4252-4867-a0f7-917c0001d831"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:24:02.944478 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:02.944455    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a3d499c3-4252-4867-a0f7-917c0001d831-istio-podinfo" pod "a3d499c3-4252-4867-a0f7-917c0001d831" (UID: "a3d499c3-4252-4867-a0f7-917c0001d831"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 20:24:02.944607 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:02.944585    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3d499c3-4252-4867-a0f7-917c0001d831-kube-api-access-xw4m7" pod "a3d499c3-4252-4867-a0f7-917c0001d831" (UID: "a3d499c3-4252-4867-a0f7-917c0001d831"). InnerVolumeSpecName "kube-api-access-xw4m7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:24:02.944709 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:02.944689    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3d499c3-4252-4867-a0f7-917c0001d831-istio-envoy" pod "a3d499c3-4252-4867-a0f7-917c0001d831" (UID: "a3d499c3-4252-4867-a0f7-917c0001d831"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:24:02.944759 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:02.944694    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3d499c3-4252-4867-a0f7-917c0001d831-istio-token" pod "a3d499c3-4252-4867-a0f7-917c0001d831" (UID: "a3d499c3-4252-4867-a0f7-917c0001d831"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:24:03.044184 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:03.044154    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3d499c3-4252-4867-a0f7-917c0001d831-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:24:03.044184 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:03.044181    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3d499c3-4252-4867-a0f7-917c0001d831-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:24:03.044377 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:03.044197    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3d499c3-4252-4867-a0f7-917c0001d831-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:24:03.044377 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:03.044210    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xw4m7\" (UniqueName: \"kubernetes.io/projected/a3d499c3-4252-4867-a0f7-917c0001d831-kube-api-access-xw4m7\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:24:03.044377 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:03.044225    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3d499c3-4252-4867-a0f7-917c0001d831-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:24:03.044377 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:03.044237    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3d499c3-4252-4867-a0f7-917c0001d831-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:24:03.044377 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:03.044258    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3d499c3-4252-4867-a0f7-917c0001d831-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:24:03.257372 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:03.257278    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:24:03.344907 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:03.344874    4645 generic.go:356] "Generic (PLEG): container finished" podID="a3d499c3-4252-4867-a0f7-917c0001d831" containerID="e25c44efa7bdecc373a062d44502396db771aaff9a00f6937bad44cd3437446d" exitCode=0
Oct 31 20:24:03.345081 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:03.344928    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj"
Oct 31 20:24:03.345081 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:03.344951    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj" event={"ID":"a3d499c3-4252-4867-a0f7-917c0001d831","Type":"ContainerDied","Data":"e25c44efa7bdecc373a062d44502396db771aaff9a00f6937bad44cd3437446d"}
Oct 31 20:24:03.345081 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:03.345019    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj" event={"ID":"a3d499c3-4252-4867-a0f7-917c0001d831","Type":"ContainerDied","Data":"0bdf85a23e6bc849fa89ab2a36661f06a644b26548748321877d3e1d8467a209"}
Oct 31 20:24:03.345081 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:03.345051    4645 scope.go:122] "RemoveContainer" containerID="e25c44efa7bdecc373a062d44502396db771aaff9a00f6937bad44cd3437446d"
Oct 31 20:24:03.354155 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:03.354138    4645 scope.go:122] "RemoveContainer" containerID="33a75414c2d8d8cd467e9b695ac022a291852a45c020eb2c04c0542b051478e0"
Oct 31 20:24:03.361920 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:03.361905    4645 scope.go:122] "RemoveContainer" containerID="e25c44efa7bdecc373a062d44502396db771aaff9a00f6937bad44cd3437446d"
Oct 31 20:24:03.362169 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:24:03.362153    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e25c44efa7bdecc373a062d44502396db771aaff9a00f6937bad44cd3437446d\": container with ID starting with e25c44efa7bdecc373a062d44502396db771aaff9a00f6937bad44cd3437446d not found: ID does not exist" containerID="e25c44efa7bdecc373a062d44502396db771aaff9a00f6937bad44cd3437446d"
Oct 31 20:24:03.362214 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:03.362181    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e25c44efa7bdecc373a062d44502396db771aaff9a00f6937bad44cd3437446d"} err="failed to get container status \"e25c44efa7bdecc373a062d44502396db771aaff9a00f6937bad44cd3437446d\": rpc error: code = NotFound desc = could not find container \"e25c44efa7bdecc373a062d44502396db771aaff9a00f6937bad44cd3437446d\": container with ID starting with e25c44efa7bdecc373a062d44502396db771aaff9a00f6937bad44cd3437446d not found: ID does not exist"
Oct 31 20:24:03.362214 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:03.362198    4645 scope.go:122] "RemoveContainer" containerID="33a75414c2d8d8cd467e9b695ac022a291852a45c020eb2c04c0542b051478e0"
Oct 31 20:24:03.362452 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:24:03.362433    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33a75414c2d8d8cd467e9b695ac022a291852a45c020eb2c04c0542b051478e0\": container with ID starting with 33a75414c2d8d8cd467e9b695ac022a291852a45c020eb2c04c0542b051478e0 not found: ID does not exist" containerID="33a75414c2d8d8cd467e9b695ac022a291852a45c020eb2c04c0542b051478e0"
Oct 31 20:24:03.362497 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:03.362457    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33a75414c2d8d8cd467e9b695ac022a291852a45c020eb2c04c0542b051478e0"} err="failed to get container status \"33a75414c2d8d8cd467e9b695ac022a291852a45c020eb2c04c0542b051478e0\": rpc error: code = NotFound desc = could not find container \"33a75414c2d8d8cd467e9b695ac022a291852a45c020eb2c04c0542b051478e0\": container with ID starting with 33a75414c2d8d8cd467e9b695ac022a291852a45c020eb2c04c0542b051478e0 not found: ID does not exist"
Oct 31 20:24:03.366733 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:03.366712    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj"]
Oct 31 20:24:03.371364 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:03.371342    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-8407e9-openshift-default-5cf8b85f7ckdtlj"]
Oct 31 20:24:03.616382 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:03.616301    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3d499c3-4252-4867-a0f7-917c0001d831" path="/var/lib/kubelet/pods/a3d499c3-4252-4867-a0f7-917c0001d831/volumes"
Oct 31 20:24:04.951728 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:04.951679    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8407e9-544db86kfw" podUID="f9eadbab-6f06-49a9-87ba-9853a3a7e8f4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:24:06.368351 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:24:06.367451    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 20:24:14.952399 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:14.952351    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8407e9-544db86kfw" podUID="f9eadbab-6f06-49a9-87ba-9853a3a7e8f4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 20:24:15.202927 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:15.202858    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8407e9-544db86kfw"
Oct 31 20:24:15.320729 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:15.320685    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9eadbab-6f06-49a9-87ba-9853a3a7e8f4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9eadbab-6f06-49a9-87ba-9853a3a7e8f4-plugins-config-volume\") pod \"f9eadbab-6f06-49a9-87ba-9853a3a7e8f4\" (UID: \"f9eadbab-6f06-49a9-87ba-9853a3a7e8f4\") "
Oct 31 20:24:15.320921 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:15.320745    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9eadbab-6f06-49a9-87ba-9853a3a7e8f4-kube-api-access-tzrlc\" (UniqueName: \"kubernetes.io/projected/f9eadbab-6f06-49a9-87ba-9853a3a7e8f4-kube-api-access-tzrlc\") pod \"f9eadbab-6f06-49a9-87ba-9853a3a7e8f4\" (UID: \"f9eadbab-6f06-49a9-87ba-9853a3a7e8f4\") "
Oct 31 20:24:15.321094 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:15.321068    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9eadbab-6f06-49a9-87ba-9853a3a7e8f4-plugins-config-volume" pod "f9eadbab-6f06-49a9-87ba-9853a3a7e8f4" (UID: "f9eadbab-6f06-49a9-87ba-9853a3a7e8f4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:24:15.322978 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:15.322950    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9eadbab-6f06-49a9-87ba-9853a3a7e8f4-kube-api-access-tzrlc" pod "f9eadbab-6f06-49a9-87ba-9853a3a7e8f4" (UID: "f9eadbab-6f06-49a9-87ba-9853a3a7e8f4"). InnerVolumeSpecName "kube-api-access-tzrlc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:24:15.422462 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:15.422426    4645 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9eadbab-6f06-49a9-87ba-9853a3a7e8f4-plugins-config-volume\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:24:15.422462 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:15.422455    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzrlc\" (UniqueName: \"kubernetes.io/projected/f9eadbab-6f06-49a9-87ba-9853a3a7e8f4-kube-api-access-tzrlc\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:24:15.444511 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:15.444469    4645 generic.go:356] "Generic (PLEG): container finished" podID="f9eadbab-6f06-49a9-87ba-9853a3a7e8f4" containerID="07b51a343c0d4eeb4990eac8e6b2e3f9b4cbfd8de91982ad2a7e3fc120392e4e" exitCode=0
Oct 31 20:24:15.444713 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:15.444523    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8407e9-544db86kfw"
Oct 31 20:24:15.444713 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:15.444562    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8407e9-544db86kfw" event={"ID":"f9eadbab-6f06-49a9-87ba-9853a3a7e8f4","Type":"ContainerDied","Data":"07b51a343c0d4eeb4990eac8e6b2e3f9b4cbfd8de91982ad2a7e3fc120392e4e"}
Oct 31 20:24:15.444713 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:15.444627    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-8407e9-544db86kfw" event={"ID":"f9eadbab-6f06-49a9-87ba-9853a3a7e8f4","Type":"ContainerDied","Data":"b0df389a78f0d895abe1d8fe564b752b3cbd340a21e3dc29e376a7357558ba5b"}
Oct 31 20:24:15.444713 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:15.444657    4645 scope.go:122] "RemoveContainer" containerID="07b51a343c0d4eeb4990eac8e6b2e3f9b4cbfd8de91982ad2a7e3fc120392e4e"
Oct 31 20:24:15.453799 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:15.453774    4645 scope.go:122] "RemoveContainer" containerID="07b51a343c0d4eeb4990eac8e6b2e3f9b4cbfd8de91982ad2a7e3fc120392e4e"
Oct 31 20:24:15.454118 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:24:15.454096    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07b51a343c0d4eeb4990eac8e6b2e3f9b4cbfd8de91982ad2a7e3fc120392e4e\": container with ID starting with 07b51a343c0d4eeb4990eac8e6b2e3f9b4cbfd8de91982ad2a7e3fc120392e4e not found: ID does not exist" containerID="07b51a343c0d4eeb4990eac8e6b2e3f9b4cbfd8de91982ad2a7e3fc120392e4e"
Oct 31 20:24:15.454204 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:15.454126    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07b51a343c0d4eeb4990eac8e6b2e3f9b4cbfd8de91982ad2a7e3fc120392e4e"} err="failed to get container status \"07b51a343c0d4eeb4990eac8e6b2e3f9b4cbfd8de91982ad2a7e3fc120392e4e\": rpc error: code = NotFound desc = could not find container \"07b51a343c0d4eeb4990eac8e6b2e3f9b4cbfd8de91982ad2a7e3fc120392e4e\": container with ID starting with 07b51a343c0d4eeb4990eac8e6b2e3f9b4cbfd8de91982ad2a7e3fc120392e4e not found: ID does not exist"
Oct 31 20:24:15.467713 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:15.467681    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-8407e9-544db86kfw"]
Oct 31 20:24:15.472365 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:15.472314    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-8407e9-544db86kfw"]
Oct 31 20:24:15.616481 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:15.616448    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9eadbab-6f06-49a9-87ba-9853a3a7e8f4" path="/var/lib/kubelet/pods/f9eadbab-6f06-49a9-87ba-9853a3a7e8f4/volumes"
Oct 31 20:24:18.303000 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:18.302963    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-098205f1-kserve-6d7744b6c7-jsbt7"]
Oct 31 20:24:18.303420 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:18.303317    4645 cpu_manager.go:414] "containerMap: removing container" podUID="a3d499c3-4252-4867-a0f7-917c0001d831" containerName="istio-proxy"
Oct 31 20:24:18.303420 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:18.303343    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3d499c3-4252-4867-a0f7-917c0001d831" containerName="istio-proxy"
Oct 31 20:24:18.303420 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:18.303375    4645 cpu_manager.go:414] "containerMap: removing container" podUID="f9eadbab-6f06-49a9-87ba-9853a3a7e8f4" containerName="payload-pre-processing"
Oct 31 20:24:18.303420 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:18.303381    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9eadbab-6f06-49a9-87ba-9853a3a7e8f4" containerName="payload-pre-processing"
Oct 31 20:24:18.303575 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:18.303450    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3d499c3-4252-4867-a0f7-917c0001d831" containerName="istio-proxy"
Oct 31 20:24:18.303575 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:18.303459    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9eadbab-6f06-49a9-87ba-9853a3a7e8f4" containerName="payload-pre-processing"
Oct 31 20:24:18.303575 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:18.303471    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3d499c3-4252-4867-a0f7-917c0001d831" containerName="istio-proxy"
Oct 31 20:24:18.307940 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:18.307908    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-098205f1-kserve-6d7744b6c7-jsbt7"
Oct 31 20:24:18.312452 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:18.312426    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-098205f1-kserve-self-signed-certs\""
Oct 31 20:24:18.322318 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:18.322289    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-098205f1-kserve-6d7744b6c7-jsbt7"]
Oct 31 20:24:18.348894 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:18.348856    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-776kf\" (UniqueName: \"kubernetes.io/projected/d6057165-0484-4b37-9cbd-9e0d68ab8893-kube-api-access-776kf\") pod \"e2e-same-id-b-098205f1-kserve-6d7744b6c7-jsbt7\" (UID: \"d6057165-0484-4b37-9cbd-9e0d68ab8893\") " pod="llm/e2e-same-id-b-098205f1-kserve-6d7744b6c7-jsbt7"
Oct 31 20:24:18.348894 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:18.348899    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6057165-0484-4b37-9cbd-9e0d68ab8893-model-cache\") pod \"e2e-same-id-b-098205f1-kserve-6d7744b6c7-jsbt7\" (UID: \"d6057165-0484-4b37-9cbd-9e0d68ab8893\") " pod="llm/e2e-same-id-b-098205f1-kserve-6d7744b6c7-jsbt7"
Oct 31 20:24:18.349145 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:18.348923    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6057165-0484-4b37-9cbd-9e0d68ab8893-tls-certs\") pod \"e2e-same-id-b-098205f1-kserve-6d7744b6c7-jsbt7\" (UID: \"d6057165-0484-4b37-9cbd-9e0d68ab8893\") " pod="llm/e2e-same-id-b-098205f1-kserve-6d7744b6c7-jsbt7"
Oct 31 20:24:18.349145 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:18.349035    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6057165-0484-4b37-9cbd-9e0d68ab8893-tmp-dir\") pod \"e2e-same-id-b-098205f1-kserve-6d7744b6c7-jsbt7\" (UID: \"d6057165-0484-4b37-9cbd-9e0d68ab8893\") " pod="llm/e2e-same-id-b-098205f1-kserve-6d7744b6c7-jsbt7"
Oct 31 20:24:18.349145 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:18.349099    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6057165-0484-4b37-9cbd-9e0d68ab8893-home\") pod \"e2e-same-id-b-098205f1-kserve-6d7744b6c7-jsbt7\" (UID: \"d6057165-0484-4b37-9cbd-9e0d68ab8893\") " pod="llm/e2e-same-id-b-098205f1-kserve-6d7744b6c7-jsbt7"
Oct 31 20:24:18.349262 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:18.349159    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6057165-0484-4b37-9cbd-9e0d68ab8893-dshm\") pod \"e2e-same-id-b-098205f1-kserve-6d7744b6c7-jsbt7\" (UID: \"d6057165-0484-4b37-9cbd-9e0d68ab8893\") " pod="llm/e2e-same-id-b-098205f1-kserve-6d7744b6c7-jsbt7"
Oct 31 20:24:18.378388 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:18.378352    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:24:18.449758 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:18.449722    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6057165-0484-4b37-9cbd-9e0d68ab8893-dshm\") pod \"e2e-same-id-b-098205f1-kserve-6d7744b6c7-jsbt7\" (UID: \"d6057165-0484-4b37-9cbd-9e0d68ab8893\") " pod="llm/e2e-same-id-b-098205f1-kserve-6d7744b6c7-jsbt7"
Oct 31 20:24:18.449963 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:18.449797    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-776kf\" (UniqueName: \"kubernetes.io/projected/d6057165-0484-4b37-9cbd-9e0d68ab8893-kube-api-access-776kf\") pod \"e2e-same-id-b-098205f1-kserve-6d7744b6c7-jsbt7\" (UID: \"d6057165-0484-4b37-9cbd-9e0d68ab8893\") " pod="llm/e2e-same-id-b-098205f1-kserve-6d7744b6c7-jsbt7"
Oct 31 20:24:18.449963 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:18.449830    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6057165-0484-4b37-9cbd-9e0d68ab8893-model-cache\") pod \"e2e-same-id-b-098205f1-kserve-6d7744b6c7-jsbt7\" (UID: \"d6057165-0484-4b37-9cbd-9e0d68ab8893\") " pod="llm/e2e-same-id-b-098205f1-kserve-6d7744b6c7-jsbt7"
Oct 31 20:24:18.449963 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:18.449864    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6057165-0484-4b37-9cbd-9e0d68ab8893-tls-certs\") pod \"e2e-same-id-b-098205f1-kserve-6d7744b6c7-jsbt7\" (UID: \"d6057165-0484-4b37-9cbd-9e0d68ab8893\") " pod="llm/e2e-same-id-b-098205f1-kserve-6d7744b6c7-jsbt7"
Oct 31 20:24:18.449963 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:18.449911    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6057165-0484-4b37-9cbd-9e0d68ab8893-tmp-dir\") pod \"e2e-same-id-b-098205f1-kserve-6d7744b6c7-jsbt7\" (UID: \"d6057165-0484-4b37-9cbd-9e0d68ab8893\") " pod="llm/e2e-same-id-b-098205f1-kserve-6d7744b6c7-jsbt7"
Oct 31 20:24:18.449963 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:18.449941    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6057165-0484-4b37-9cbd-9e0d68ab8893-home\") pod \"e2e-same-id-b-098205f1-kserve-6d7744b6c7-jsbt7\" (UID: \"d6057165-0484-4b37-9cbd-9e0d68ab8893\") " pod="llm/e2e-same-id-b-098205f1-kserve-6d7744b6c7-jsbt7"
Oct 31 20:24:18.450317 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:18.450288    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6057165-0484-4b37-9cbd-9e0d68ab8893-model-cache\") pod \"e2e-same-id-b-098205f1-kserve-6d7744b6c7-jsbt7\" (UID: \"d6057165-0484-4b37-9cbd-9e0d68ab8893\") " pod="llm/e2e-same-id-b-098205f1-kserve-6d7744b6c7-jsbt7"
Oct 31 20:24:18.450435 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:18.450300    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6057165-0484-4b37-9cbd-9e0d68ab8893-tmp-dir\") pod \"e2e-same-id-b-098205f1-kserve-6d7744b6c7-jsbt7\" (UID: \"d6057165-0484-4b37-9cbd-9e0d68ab8893\") " pod="llm/e2e-same-id-b-098205f1-kserve-6d7744b6c7-jsbt7"
Oct 31 20:24:18.450435 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:18.450400    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6057165-0484-4b37-9cbd-9e0d68ab8893-home\") pod \"e2e-same-id-b-098205f1-kserve-6d7744b6c7-jsbt7\" (UID: \"d6057165-0484-4b37-9cbd-9e0d68ab8893\") " pod="llm/e2e-same-id-b-098205f1-kserve-6d7744b6c7-jsbt7"
Oct 31 20:24:18.452215 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:18.452196    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6057165-0484-4b37-9cbd-9e0d68ab8893-dshm\") pod \"e2e-same-id-b-098205f1-kserve-6d7744b6c7-jsbt7\" (UID: \"d6057165-0484-4b37-9cbd-9e0d68ab8893\") " pod="llm/e2e-same-id-b-098205f1-kserve-6d7744b6c7-jsbt7"
Oct 31 20:24:18.452581 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:18.452562    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6057165-0484-4b37-9cbd-9e0d68ab8893-tls-certs\") pod \"e2e-same-id-b-098205f1-kserve-6d7744b6c7-jsbt7\" (UID: \"d6057165-0484-4b37-9cbd-9e0d68ab8893\") " pod="llm/e2e-same-id-b-098205f1-kserve-6d7744b6c7-jsbt7"
Oct 31 20:24:18.462903 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:18.462873    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-776kf\" (UniqueName: \"kubernetes.io/projected/d6057165-0484-4b37-9cbd-9e0d68ab8893-kube-api-access-776kf\") pod \"e2e-same-id-b-098205f1-kserve-6d7744b6c7-jsbt7\" (UID: \"d6057165-0484-4b37-9cbd-9e0d68ab8893\") " pod="llm/e2e-same-id-b-098205f1-kserve-6d7744b6c7-jsbt7"
Oct 31 20:24:18.620416 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:18.620302    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-098205f1-kserve-6d7744b6c7-jsbt7"
Oct 31 20:24:18.897068 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:18.897033    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-098205f1-kserve-6d7744b6c7-jsbt7"]
Oct 31 20:24:18.899476 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:24:18.899446    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd6057165_0484_4b37_9cbd_9e0d68ab8893.slice/crio-a9c42a8f376ec37e67d51c00036ff0c640f0a8d0f84bce017125c4460d00f97b WatchSource:0}: Error finding container a9c42a8f376ec37e67d51c00036ff0c640f0a8d0f84bce017125c4460d00f97b: Status 404 returned error can't find the container with id a9c42a8f376ec37e67d51c00036ff0c640f0a8d0f84bce017125c4460d00f97b
Oct 31 20:24:19.222708 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:19.222673    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:24:19.465097 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:19.465058    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-098205f1-kserve-6d7744b6c7-jsbt7" event={"ID":"d6057165-0484-4b37-9cbd-9e0d68ab8893","Type":"ContainerStarted","Data":"f8c2018335e4b797e2d11cc20af29c03efc3d9db2f18c45bd8ed487ddecbf3d9"}
Oct 31 20:24:19.465097 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:19.465110    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-098205f1-kserve-6d7744b6c7-jsbt7" event={"ID":"d6057165-0484-4b37-9cbd-9e0d68ab8893","Type":"ContainerStarted","Data":"a9c42a8f376ec37e67d51c00036ff0c640f0a8d0f84bce017125c4460d00f97b"}
Oct 31 20:24:19.549224 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:19.549105    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-098205f1-kserve-6d7744b6c7-jsbt7" podStartSLOduration=1.549086671 podStartE2EDuration="1.549086671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:24:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:24:19.547504078 +0000 UTC m=+3890.120738206" watchObservedRunningTime="2026-10-31 20:24:19.549086671 +0000 UTC m=+3890.122320799"
Oct 31 20:24:19.620736 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:19.620703    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-098205f1-kserve-6d7744b6c7-jsbt7"
Oct 31 20:24:20.300454 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:20.300408    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:24:21.521230 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:21.521185    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:24:21.632930 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:21.632229    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:24:23.040364 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:23.039929    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:24:28.620765 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:28.620721    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-098205f1-kserve-6d7744b6c7-jsbt7"
Oct 31 20:24:28.633361 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:28.633310    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-098205f1-kserve-6d7744b6c7-jsbt7"
Oct 31 20:24:29.549938 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:29.549895    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:24:29.559094 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:29.559064    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-098205f1-kserve-6d7744b6c7-jsbt7"
Oct 31 20:24:36.246024 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:24:36.245982    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 20:24:46.790625 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:46.790530    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:24:48.037949 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:48.037909    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-098205f1-kserve-6d7744b6c7-jsbt7"]
Oct 31 20:24:48.091406 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:24:48.091362    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:25:03.058477 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:03.058410    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-098205f1-kserve-6d7744b6c7-jsbt7" podUID="d6057165-0484-4b37-9cbd-9e0d68ab8893" containerName="main" containerID="cri-o://f8c2018335e4b797e2d11cc20af29c03efc3d9db2f18c45bd8ed487ddecbf3d9" gracePeriod=45
Oct 31 20:25:03.323249 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:03.323174    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-098205f1-kserve-6d7744b6c7-jsbt7"
Oct 31 20:25:03.478355 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:03.477669    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6057165-0484-4b37-9cbd-9e0d68ab8893-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6057165-0484-4b37-9cbd-9e0d68ab8893-tmp-dir\") pod \"d6057165-0484-4b37-9cbd-9e0d68ab8893\" (UID: \"d6057165-0484-4b37-9cbd-9e0d68ab8893\") "
Oct 31 20:25:03.478355 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:03.477759    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6057165-0484-4b37-9cbd-9e0d68ab8893-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6057165-0484-4b37-9cbd-9e0d68ab8893-model-cache\") pod \"d6057165-0484-4b37-9cbd-9e0d68ab8893\" (UID: \"d6057165-0484-4b37-9cbd-9e0d68ab8893\") "
Oct 31 20:25:03.478355 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:03.477824    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6057165-0484-4b37-9cbd-9e0d68ab8893-tls-certs\" (UniqueName: \"kubernetes.io/secret/d6057165-0484-4b37-9cbd-9e0d68ab8893-tls-certs\") pod \"d6057165-0484-4b37-9cbd-9e0d68ab8893\" (UID: \"d6057165-0484-4b37-9cbd-9e0d68ab8893\") "
Oct 31 20:25:03.478355 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:03.477890    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6057165-0484-4b37-9cbd-9e0d68ab8893-home\" (UniqueName: \"kubernetes.io/empty-dir/d6057165-0484-4b37-9cbd-9e0d68ab8893-home\") pod \"d6057165-0484-4b37-9cbd-9e0d68ab8893\" (UID: \"d6057165-0484-4b37-9cbd-9e0d68ab8893\") "
Oct 31 20:25:03.478355 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:03.477940    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6057165-0484-4b37-9cbd-9e0d68ab8893-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6057165-0484-4b37-9cbd-9e0d68ab8893-dshm\") pod \"d6057165-0484-4b37-9cbd-9e0d68ab8893\" (UID: \"d6057165-0484-4b37-9cbd-9e0d68ab8893\") "
Oct 31 20:25:03.478355 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:03.477985    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6057165-0484-4b37-9cbd-9e0d68ab8893-kube-api-access-776kf\" (UniqueName: \"kubernetes.io/projected/d6057165-0484-4b37-9cbd-9e0d68ab8893-kube-api-access-776kf\") pod \"d6057165-0484-4b37-9cbd-9e0d68ab8893\" (UID: \"d6057165-0484-4b37-9cbd-9e0d68ab8893\") "
Oct 31 20:25:03.478921 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:03.478628    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6057165-0484-4b37-9cbd-9e0d68ab8893-tmp-dir" pod "d6057165-0484-4b37-9cbd-9e0d68ab8893" (UID: "d6057165-0484-4b37-9cbd-9e0d68ab8893"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:25:03.478921 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:03.478826    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6057165-0484-4b37-9cbd-9e0d68ab8893-model-cache" pod "d6057165-0484-4b37-9cbd-9e0d68ab8893" (UID: "d6057165-0484-4b37-9cbd-9e0d68ab8893"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:25:03.538395 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:03.534444    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6057165-0484-4b37-9cbd-9e0d68ab8893-home" pod "d6057165-0484-4b37-9cbd-9e0d68ab8893" (UID: "d6057165-0484-4b37-9cbd-9e0d68ab8893"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:25:03.549633 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:03.549583    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6057165-0484-4b37-9cbd-9e0d68ab8893-tls-certs" pod "d6057165-0484-4b37-9cbd-9e0d68ab8893" (UID: "d6057165-0484-4b37-9cbd-9e0d68ab8893"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:25:03.552470 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:03.552420    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6057165-0484-4b37-9cbd-9e0d68ab8893-dshm" pod "d6057165-0484-4b37-9cbd-9e0d68ab8893" (UID: "d6057165-0484-4b37-9cbd-9e0d68ab8893"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:25:03.552780 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:03.552743    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6057165-0484-4b37-9cbd-9e0d68ab8893-kube-api-access-776kf" pod "d6057165-0484-4b37-9cbd-9e0d68ab8893" (UID: "d6057165-0484-4b37-9cbd-9e0d68ab8893"). InnerVolumeSpecName "kube-api-access-776kf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:25:03.582141 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:03.581945    4645 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6057165-0484-4b37-9cbd-9e0d68ab8893-tls-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:25:03.582141 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:03.581990    4645 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6057165-0484-4b37-9cbd-9e0d68ab8893-home\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:25:03.582141 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:03.582006    4645 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6057165-0484-4b37-9cbd-9e0d68ab8893-dshm\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:25:03.582141 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:03.582020    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-776kf\" (UniqueName: \"kubernetes.io/projected/d6057165-0484-4b37-9cbd-9e0d68ab8893-kube-api-access-776kf\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:25:03.582141 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:03.582037    4645 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6057165-0484-4b37-9cbd-9e0d68ab8893-tmp-dir\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:25:03.582141 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:03.582052    4645 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6057165-0484-4b37-9cbd-9e0d68ab8893-model-cache\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:25:03.946858 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:03.945459    4645 generic.go:356] "Generic (PLEG): container finished" podID="d6057165-0484-4b37-9cbd-9e0d68ab8893" containerID="f8c2018335e4b797e2d11cc20af29c03efc3d9db2f18c45bd8ed487ddecbf3d9" exitCode=0
Oct 31 20:25:03.946858 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:03.945589    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-098205f1-kserve-6d7744b6c7-jsbt7" event={"ID":"d6057165-0484-4b37-9cbd-9e0d68ab8893","Type":"ContainerDied","Data":"f8c2018335e4b797e2d11cc20af29c03efc3d9db2f18c45bd8ed487ddecbf3d9"}
Oct 31 20:25:03.946858 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:03.945618    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-098205f1-kserve-6d7744b6c7-jsbt7" event={"ID":"d6057165-0484-4b37-9cbd-9e0d68ab8893","Type":"ContainerDied","Data":"a9c42a8f376ec37e67d51c00036ff0c640f0a8d0f84bce017125c4460d00f97b"}
Oct 31 20:25:03.946858 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:03.945635    4645 scope.go:122] "RemoveContainer" containerID="f8c2018335e4b797e2d11cc20af29c03efc3d9db2f18c45bd8ed487ddecbf3d9"
Oct 31 20:25:03.946858 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:03.945770    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-098205f1-kserve-6d7744b6c7-jsbt7"
Oct 31 20:25:03.972514 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:03.971993    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-098205f1-kserve-6d7744b6c7-jsbt7"]
Oct 31 20:25:03.982466 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:03.982433    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-098205f1-kserve-6d7744b6c7-jsbt7"]
Oct 31 20:25:03.982977 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:03.982957    4645 scope.go:122] "RemoveContainer" containerID="f8c2018335e4b797e2d11cc20af29c03efc3d9db2f18c45bd8ed487ddecbf3d9"
Oct 31 20:25:03.983732 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:25:03.983694    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8c2018335e4b797e2d11cc20af29c03efc3d9db2f18c45bd8ed487ddecbf3d9\": container with ID starting with f8c2018335e4b797e2d11cc20af29c03efc3d9db2f18c45bd8ed487ddecbf3d9 not found: ID does not exist" containerID="f8c2018335e4b797e2d11cc20af29c03efc3d9db2f18c45bd8ed487ddecbf3d9"
Oct 31 20:25:03.983915 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:03.983885    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8c2018335e4b797e2d11cc20af29c03efc3d9db2f18c45bd8ed487ddecbf3d9"} err="failed to get container status \"f8c2018335e4b797e2d11cc20af29c03efc3d9db2f18c45bd8ed487ddecbf3d9\": rpc error: code = NotFound desc = could not find container \"f8c2018335e4b797e2d11cc20af29c03efc3d9db2f18c45bd8ed487ddecbf3d9\": container with ID starting with f8c2018335e4b797e2d11cc20af29c03efc3d9db2f18c45bd8ed487ddecbf3d9 not found: ID does not exist"
Oct 31 20:25:05.192047 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:05.191998    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-6rwmb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 20:25:05.192760 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:05.192721    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-6rwmb" podUID="b92c4b09-c2c0-4602-af78-2f410c843118" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 20:25:05.196458 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:05.196427    4645 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-75d7c785f-68qbd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 20:25:05.196669 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:05.196645    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-75d7c785f-68qbd" podUID="3a217b30-54c0-431f-90bc-0681901272f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 20:25:05.617075 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:05.616989    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6057165-0484-4b37-9cbd-9e0d68ab8893" path="/var/lib/kubelet/pods/d6057165-0484-4b37-9cbd-9e0d68ab8893/volumes"
Oct 31 20:25:06.262011 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:25:06.261962    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 20:25:19.307380 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:19.307314    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:25:20.346158 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:20.346111    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:25:21.479082 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:21.479035    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:25:31.758129 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:31.758089    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:25:32.187557 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:32.187522    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs"]
Oct 31 20:25:32.187913 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:32.187899    4645 cpu_manager.go:414] "containerMap: removing container" podUID="a3d499c3-4252-4867-a0f7-917c0001d831" containerName="istio-proxy"
Oct 31 20:25:32.187970 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:32.187914    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3d499c3-4252-4867-a0f7-917c0001d831" containerName="istio-proxy"
Oct 31 20:25:32.187970 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:32.187951    4645 cpu_manager.go:414] "containerMap: removing container" podUID="d6057165-0484-4b37-9cbd-9e0d68ab8893" containerName="main"
Oct 31 20:25:32.187970 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:32.187957    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6057165-0484-4b37-9cbd-9e0d68ab8893" containerName="main"
Oct 31 20:25:32.188081 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:32.188045    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6057165-0484-4b37-9cbd-9e0d68ab8893" containerName="main"
Oct 31 20:25:32.191261 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:32.191234    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs"
Oct 31 20:25:32.196652 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:32.196630    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-4554a17c-openshift-default-dockercfg-6m6xj\""
Oct 31 20:25:32.207962 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:32.207937    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs"]
Oct 31 20:25:32.235466 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:32.235440    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2cf6a6f-1268-4997-8728-e8ca29514c4f-istio-data\") pod \"e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs\" (UID: \"f2cf6a6f-1268-4997-8728-e8ca29514c4f\") " pod="openshift-ingress/e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs"
Oct 31 20:25:32.235595 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:32.235486    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2cf6a6f-1268-4997-8728-e8ca29514c4f-istiod-ca-cert\") pod \"e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs\" (UID: \"f2cf6a6f-1268-4997-8728-e8ca29514c4f\") " pod="openshift-ingress/e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs"
Oct 31 20:25:32.235595 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:32.235568    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcsff\" (UniqueName: \"kubernetes.io/projected/f2cf6a6f-1268-4997-8728-e8ca29514c4f-kube-api-access-xcsff\") pod \"e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs\" (UID: \"f2cf6a6f-1268-4997-8728-e8ca29514c4f\") " pod="openshift-ingress/e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs"
Oct 31 20:25:32.235706 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:32.235683    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2cf6a6f-1268-4997-8728-e8ca29514c4f-istio-token\") pod \"e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs\" (UID: \"f2cf6a6f-1268-4997-8728-e8ca29514c4f\") " pod="openshift-ingress/e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs"
Oct 31 20:25:32.235770 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:32.235751    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2cf6a6f-1268-4997-8728-e8ca29514c4f-credential-socket\") pod \"e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs\" (UID: \"f2cf6a6f-1268-4997-8728-e8ca29514c4f\") " pod="openshift-ingress/e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs"
Oct 31 20:25:32.235820 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:32.235798    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2cf6a6f-1268-4997-8728-e8ca29514c4f-workload-socket\") pod \"e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs\" (UID: \"f2cf6a6f-1268-4997-8728-e8ca29514c4f\") " pod="openshift-ingress/e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs"
Oct 31 20:25:32.235912 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:32.235895    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2cf6a6f-1268-4997-8728-e8ca29514c4f-istio-podinfo\") pod \"e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs\" (UID: \"f2cf6a6f-1268-4997-8728-e8ca29514c4f\") " pod="openshift-ingress/e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs"
Oct 31 20:25:32.235979 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:32.235964    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2cf6a6f-1268-4997-8728-e8ca29514c4f-workload-certs\") pod \"e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs\" (UID: \"f2cf6a6f-1268-4997-8728-e8ca29514c4f\") " pod="openshift-ingress/e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs"
Oct 31 20:25:32.236069 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:32.235990    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2cf6a6f-1268-4997-8728-e8ca29514c4f-istio-envoy\") pod \"e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs\" (UID: \"f2cf6a6f-1268-4997-8728-e8ca29514c4f\") " pod="openshift-ingress/e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs"
Oct 31 20:25:32.243552 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:32.243528    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:25:32.336654 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:32.336616    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2cf6a6f-1268-4997-8728-e8ca29514c4f-credential-socket\") pod \"e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs\" (UID: \"f2cf6a6f-1268-4997-8728-e8ca29514c4f\") " pod="openshift-ingress/e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs"
Oct 31 20:25:32.336848 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:32.336663    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2cf6a6f-1268-4997-8728-e8ca29514c4f-workload-socket\") pod \"e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs\" (UID: \"f2cf6a6f-1268-4997-8728-e8ca29514c4f\") " pod="openshift-ingress/e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs"
Oct 31 20:25:32.336848 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:32.336693    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2cf6a6f-1268-4997-8728-e8ca29514c4f-istio-podinfo\") pod \"e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs\" (UID: \"f2cf6a6f-1268-4997-8728-e8ca29514c4f\") " pod="openshift-ingress/e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs"
Oct 31 20:25:32.336848 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:32.336718    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2cf6a6f-1268-4997-8728-e8ca29514c4f-workload-certs\") pod \"e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs\" (UID: \"f2cf6a6f-1268-4997-8728-e8ca29514c4f\") " pod="openshift-ingress/e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs"
Oct 31 20:25:32.336848 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:32.336735    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2cf6a6f-1268-4997-8728-e8ca29514c4f-istio-envoy\") pod \"e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs\" (UID: \"f2cf6a6f-1268-4997-8728-e8ca29514c4f\") " pod="openshift-ingress/e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs"
Oct 31 20:25:32.336848 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:32.336766    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2cf6a6f-1268-4997-8728-e8ca29514c4f-istio-data\") pod \"e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs\" (UID: \"f2cf6a6f-1268-4997-8728-e8ca29514c4f\") " pod="openshift-ingress/e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs"
Oct 31 20:25:32.336848 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:32.336798    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2cf6a6f-1268-4997-8728-e8ca29514c4f-istiod-ca-cert\") pod \"e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs\" (UID: \"f2cf6a6f-1268-4997-8728-e8ca29514c4f\") " pod="openshift-ingress/e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs"
Oct 31 20:25:32.336848 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:32.336827    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcsff\" (UniqueName: \"kubernetes.io/projected/f2cf6a6f-1268-4997-8728-e8ca29514c4f-kube-api-access-xcsff\") pod \"e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs\" (UID: \"f2cf6a6f-1268-4997-8728-e8ca29514c4f\") " pod="openshift-ingress/e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs"
Oct 31 20:25:32.337233 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:32.336879    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2cf6a6f-1268-4997-8728-e8ca29514c4f-istio-token\") pod \"e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs\" (UID: \"f2cf6a6f-1268-4997-8728-e8ca29514c4f\") " pod="openshift-ingress/e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs"
Oct 31 20:25:32.337233 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:32.337191    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2cf6a6f-1268-4997-8728-e8ca29514c4f-workload-certs\") pod \"e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs\" (UID: \"f2cf6a6f-1268-4997-8728-e8ca29514c4f\") " pod="openshift-ingress/e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs"
Oct 31 20:25:32.337386 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:32.337271    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2cf6a6f-1268-4997-8728-e8ca29514c4f-credential-socket\") pod \"e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs\" (UID: \"f2cf6a6f-1268-4997-8728-e8ca29514c4f\") " pod="openshift-ingress/e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs"
Oct 31 20:25:32.337455 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:32.337413    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2cf6a6f-1268-4997-8728-e8ca29514c4f-workload-socket\") pod \"e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs\" (UID: \"f2cf6a6f-1268-4997-8728-e8ca29514c4f\") " pod="openshift-ingress/e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs"
Oct 31 20:25:32.337525 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:32.337447    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2cf6a6f-1268-4997-8728-e8ca29514c4f-istio-data\") pod \"e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs\" (UID: \"f2cf6a6f-1268-4997-8728-e8ca29514c4f\") " pod="openshift-ingress/e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs"
Oct 31 20:25:32.337750 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:32.337726    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2cf6a6f-1268-4997-8728-e8ca29514c4f-istiod-ca-cert\") pod \"e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs\" (UID: \"f2cf6a6f-1268-4997-8728-e8ca29514c4f\") " pod="openshift-ingress/e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs"
Oct 31 20:25:32.338977 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:32.338957    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2cf6a6f-1268-4997-8728-e8ca29514c4f-istio-envoy\") pod \"e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs\" (UID: \"f2cf6a6f-1268-4997-8728-e8ca29514c4f\") " pod="openshift-ingress/e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs"
Oct 31 20:25:32.339489 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:32.339471    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2cf6a6f-1268-4997-8728-e8ca29514c4f-istio-podinfo\") pod \"e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs\" (UID: \"f2cf6a6f-1268-4997-8728-e8ca29514c4f\") " pod="openshift-ingress/e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs"
Oct 31 20:25:32.348919 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:32.348896    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcsff\" (UniqueName: \"kubernetes.io/projected/f2cf6a6f-1268-4997-8728-e8ca29514c4f-kube-api-access-xcsff\") pod \"e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs\" (UID: \"f2cf6a6f-1268-4997-8728-e8ca29514c4f\") " pod="openshift-ingress/e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs"
Oct 31 20:25:32.349047 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:32.348969    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2cf6a6f-1268-4997-8728-e8ca29514c4f-istio-token\") pod \"e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs\" (UID: \"f2cf6a6f-1268-4997-8728-e8ca29514c4f\") " pod="openshift-ingress/e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs"
Oct 31 20:25:32.403590 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:32.403529    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:25:32.505093 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:32.505032    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs"
Oct 31 20:25:32.789374 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:32.789346    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs"]
Oct 31 20:25:32.791828 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:25:32.791789    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2cf6a6f_1268_4997_8728_e8ca29514c4f.slice/crio-8d82aba4cf43c1a0b4d58a4aad36644891699de2d10659b052dcf29309f403d5 WatchSource:0}: Error finding container 8d82aba4cf43c1a0b4d58a4aad36644891699de2d10659b052dcf29309f403d5: Status 404 returned error can't find the container with id 8d82aba4cf43c1a0b4d58a4aad36644891699de2d10659b052dcf29309f403d5
Oct 31 20:25:32.794255 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:32.794219    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:25:32.794388 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:32.794295    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:25:32.794388 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:32.794357    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:25:33.124236 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:33.124148    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs" event={"ID":"f2cf6a6f-1268-4997-8728-e8ca29514c4f","Type":"ContainerStarted","Data":"b5ddce04c86daa9592ec01b999d322e7dba8134b9e147d400781de9a0faaf7ae"}
Oct 31 20:25:33.124236 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:33.124214    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs" event={"ID":"f2cf6a6f-1268-4997-8728-e8ca29514c4f","Type":"ContainerStarted","Data":"8d82aba4cf43c1a0b4d58a4aad36644891699de2d10659b052dcf29309f403d5"}
Oct 31 20:25:33.150928 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:33.150874    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs" podStartSLOduration=1.150856808 podStartE2EDuration="1.150856808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:25:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:25:33.1486066 +0000 UTC m=+3963.721840729" watchObservedRunningTime="2026-10-31 20:25:33.150856808 +0000 UTC m=+3963.724090938"
Oct 31 20:25:33.505159 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:33.505125    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs"
Oct 31 20:25:33.510350 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:33.510304    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs"
Oct 31 20:25:34.129033 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:34.128997    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs"
Oct 31 20:25:34.129924 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:34.129904    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs"
Oct 31 20:25:34.987406 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:34.987370    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:25:35.181234 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:35.181198    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:25:36.239989 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:36.239943    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:25:36.248122 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:25:36.248080    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 20:25:36.296378 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:36.296340    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4554a17c-5776dfb9cf-x8n6g"]
Oct 31 20:25:36.301922 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:36.301898    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4554a17c-5776dfb9cf-x8n6g"
Oct 31 20:25:36.320931 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:36.320892    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4554a17c-bdcd45c4c-84k8c"]
Oct 31 20:25:36.326400 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:36.326369    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4554a17c-5776dfb9cf-x8n6g"]
Oct 31 20:25:36.326717 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:36.326693    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4554a17c-bdcd45c4c-84k8c"
Oct 31 20:25:36.343508 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:36.343449    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4554a17c-bdcd45c4c-84k8c"]
Oct 31 20:25:36.369785 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:36.369754    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d773219-9039-4e52-b5dd-f20cebacf1ab-metrics-certs\") pod \"maas-api-e2e-ait-4554a17c-bdcd45c4c-84k8c\" (UID: \"6d773219-9039-4e52-b5dd-f20cebacf1ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4554a17c-bdcd45c4c-84k8c"
Oct 31 20:25:36.369969 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:36.369812    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c336e91-9d54-4a61-9bcb-1fb9bc3832ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4554a17c-5776dfb9cf-x8n6g\" (UID: \"8c336e91-9d54-4a61-9bcb-1fb9bc3832ab\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4554a17c-5776dfb9cf-x8n6g"
Oct 31 20:25:36.369969 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:36.369912    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d773219-9039-4e52-b5dd-f20cebacf1ab-openshift-service-ca\") pod \"maas-api-e2e-ait-4554a17c-bdcd45c4c-84k8c\" (UID: \"6d773219-9039-4e52-b5dd-f20cebacf1ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4554a17c-bdcd45c4c-84k8c"
Oct 31 20:25:36.369969 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:36.369938    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h6c7\" (UniqueName: \"kubernetes.io/projected/6d773219-9039-4e52-b5dd-f20cebacf1ab-kube-api-access-4h6c7\") pod \"maas-api-e2e-ait-4554a17c-bdcd45c4c-84k8c\" (UID: \"6d773219-9039-4e52-b5dd-f20cebacf1ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4554a17c-bdcd45c4c-84k8c"
Oct 31 20:25:36.370137 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:36.369974    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhh74\" (UniqueName: \"kubernetes.io/projected/8c336e91-9d54-4a61-9bcb-1fb9bc3832ab-kube-api-access-qhh74\") pod \"payload-pre-processing-e2e-ait-4554a17c-5776dfb9cf-x8n6g\" (UID: \"8c336e91-9d54-4a61-9bcb-1fb9bc3832ab\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4554a17c-5776dfb9cf-x8n6g"
Oct 31 20:25:36.370137 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:36.370104    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d773219-9039-4e52-b5dd-f20cebacf1ab-maas-api-tls\") pod \"maas-api-e2e-ait-4554a17c-bdcd45c4c-84k8c\" (UID: \"6d773219-9039-4e52-b5dd-f20cebacf1ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4554a17c-bdcd45c4c-84k8c"
Oct 31 20:25:36.396202 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:36.396163    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:25:36.405928 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:36.405896    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-4554a17c-serving-cert\""
Oct 31 20:25:36.406065 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:36.405998    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-4554a17c-dockercfg-mp8hz\""
Oct 31 20:25:36.406065 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:36.406019    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-4554a17c\""
Oct 31 20:25:36.471423 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:36.471385    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c336e91-9d54-4a61-9bcb-1fb9bc3832ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4554a17c-5776dfb9cf-x8n6g\" (UID: \"8c336e91-9d54-4a61-9bcb-1fb9bc3832ab\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4554a17c-5776dfb9cf-x8n6g"
Oct 31 20:25:36.471619 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:36.471440    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d773219-9039-4e52-b5dd-f20cebacf1ab-openshift-service-ca\") pod \"maas-api-e2e-ait-4554a17c-bdcd45c4c-84k8c\" (UID: \"6d773219-9039-4e52-b5dd-f20cebacf1ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4554a17c-bdcd45c4c-84k8c"
Oct 31 20:25:36.471619 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:36.471466    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h6c7\" (UniqueName: \"kubernetes.io/projected/6d773219-9039-4e52-b5dd-f20cebacf1ab-kube-api-access-4h6c7\") pod \"maas-api-e2e-ait-4554a17c-bdcd45c4c-84k8c\" (UID: \"6d773219-9039-4e52-b5dd-f20cebacf1ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4554a17c-bdcd45c4c-84k8c"
Oct 31 20:25:36.471619 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:36.471503    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhh74\" (UniqueName: \"kubernetes.io/projected/8c336e91-9d54-4a61-9bcb-1fb9bc3832ab-kube-api-access-qhh74\") pod \"payload-pre-processing-e2e-ait-4554a17c-5776dfb9cf-x8n6g\" (UID: \"8c336e91-9d54-4a61-9bcb-1fb9bc3832ab\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4554a17c-5776dfb9cf-x8n6g"
Oct 31 20:25:36.471619 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:36.471529    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d773219-9039-4e52-b5dd-f20cebacf1ab-maas-api-tls\") pod \"maas-api-e2e-ait-4554a17c-bdcd45c4c-84k8c\" (UID: \"6d773219-9039-4e52-b5dd-f20cebacf1ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4554a17c-bdcd45c4c-84k8c"
Oct 31 20:25:36.471619 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:36.471583    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d773219-9039-4e52-b5dd-f20cebacf1ab-metrics-certs\") pod \"maas-api-e2e-ait-4554a17c-bdcd45c4c-84k8c\" (UID: \"6d773219-9039-4e52-b5dd-f20cebacf1ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4554a17c-bdcd45c4c-84k8c"
Oct 31 20:25:36.472295 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:36.472268    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d773219-9039-4e52-b5dd-f20cebacf1ab-openshift-service-ca\") pod \"maas-api-e2e-ait-4554a17c-bdcd45c4c-84k8c\" (UID: \"6d773219-9039-4e52-b5dd-f20cebacf1ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4554a17c-bdcd45c4c-84k8c"
Oct 31 20:25:36.472446 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:36.472391    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c336e91-9d54-4a61-9bcb-1fb9bc3832ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4554a17c-5776dfb9cf-x8n6g\" (UID: \"8c336e91-9d54-4a61-9bcb-1fb9bc3832ab\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4554a17c-5776dfb9cf-x8n6g"
Oct 31 20:25:36.474162 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:36.474137    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d773219-9039-4e52-b5dd-f20cebacf1ab-maas-api-tls\") pod \"maas-api-e2e-ait-4554a17c-bdcd45c4c-84k8c\" (UID: \"6d773219-9039-4e52-b5dd-f20cebacf1ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4554a17c-bdcd45c4c-84k8c"
Oct 31 20:25:36.474304 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:36.474282    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d773219-9039-4e52-b5dd-f20cebacf1ab-metrics-certs\") pod \"maas-api-e2e-ait-4554a17c-bdcd45c4c-84k8c\" (UID: \"6d773219-9039-4e52-b5dd-f20cebacf1ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4554a17c-bdcd45c4c-84k8c"
Oct 31 20:25:36.491703 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:36.491642    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h6c7\" (UniqueName: \"kubernetes.io/projected/6d773219-9039-4e52-b5dd-f20cebacf1ab-kube-api-access-4h6c7\") pod \"maas-api-e2e-ait-4554a17c-bdcd45c4c-84k8c\" (UID: \"6d773219-9039-4e52-b5dd-f20cebacf1ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4554a17c-bdcd45c4c-84k8c"
Oct 31 20:25:36.498695 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:36.498642    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhh74\" (UniqueName: \"kubernetes.io/projected/8c336e91-9d54-4a61-9bcb-1fb9bc3832ab-kube-api-access-qhh74\") pod \"payload-pre-processing-e2e-ait-4554a17c-5776dfb9cf-x8n6g\" (UID: \"8c336e91-9d54-4a61-9bcb-1fb9bc3832ab\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4554a17c-5776dfb9cf-x8n6g"
Oct 31 20:25:36.559515 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:36.559024    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:25:36.618670 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:36.618631    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4554a17c-5776dfb9cf-x8n6g"
Oct 31 20:25:36.644887 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:36.644855    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4554a17c-bdcd45c4c-84k8c"
Oct 31 20:25:36.717465 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:36.715984    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:25:36.923067 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:36.923034    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4554a17c-5776dfb9cf-x8n6g"]
Oct 31 20:25:36.923230 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:25:36.923202    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c336e91_9d54_4a61_9bcb_1fb9bc3832ab.slice/crio-0979dbc131352fd95c5127e79945068f656e09af1a00dab5b80b41810fbfa0fa WatchSource:0}: Error finding container 0979dbc131352fd95c5127e79945068f656e09af1a00dab5b80b41810fbfa0fa: Status 404 returned error can't find the container with id 0979dbc131352fd95c5127e79945068f656e09af1a00dab5b80b41810fbfa0fa
Oct 31 20:25:36.942238 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:36.942176    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4554a17c-bdcd45c4c-84k8c"]
Oct 31 20:25:36.951427 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:25:36.951403    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d773219_9039_4e52_b5dd_f20cebacf1ab.slice/crio-bd6c6a96eb9d99245f2b7ecb028ffbf14ac7f7747c3a9f0eb194c1bfe5872885 WatchSource:0}: Error finding container bd6c6a96eb9d99245f2b7ecb028ffbf14ac7f7747c3a9f0eb194c1bfe5872885: Status 404 returned error can't find the container with id bd6c6a96eb9d99245f2b7ecb028ffbf14ac7f7747c3a9f0eb194c1bfe5872885
Oct 31 20:25:36.952774 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:36.952752    4645 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 20:25:37.146199 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:37.146164    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4554a17c-bdcd45c4c-84k8c" event={"ID":"6d773219-9039-4e52-b5dd-f20cebacf1ab","Type":"ContainerStarted","Data":"bd6c6a96eb9d99245f2b7ecb028ffbf14ac7f7747c3a9f0eb194c1bfe5872885"}
Oct 31 20:25:37.147740 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:37.147711    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4554a17c-5776dfb9cf-x8n6g" event={"ID":"8c336e91-9d54-4a61-9bcb-1fb9bc3832ab","Type":"ContainerStarted","Data":"b4af2f3890932d692321abf503ad22f0cc599e4761d28c14c1320c88e05aaf8a"}
Oct 31 20:25:37.147864 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:37.147756    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4554a17c-5776dfb9cf-x8n6g" event={"ID":"8c336e91-9d54-4a61-9bcb-1fb9bc3832ab","Type":"ContainerStarted","Data":"0979dbc131352fd95c5127e79945068f656e09af1a00dab5b80b41810fbfa0fa"}
Oct 31 20:25:37.147864 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:37.147825    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-4554a17c-5776dfb9cf-x8n6g"
Oct 31 20:25:37.169414 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:37.169367    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-4554a17c-5776dfb9cf-x8n6g" podStartSLOduration=1.169354374 podStartE2EDuration="1.169354374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:25:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:25:37.167269448 +0000 UTC m=+3967.740503576" watchObservedRunningTime="2026-10-31 20:25:37.169354374 +0000 UTC m=+3967.742588497"
Oct 31 20:25:37.369166 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:37.369141    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:25:38.153797 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:38.153763    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4554a17c-bdcd45c4c-84k8c" event={"ID":"6d773219-9039-4e52-b5dd-f20cebacf1ab","Type":"ContainerStarted","Data":"157d7625b9ba28991a23f7a16369e08358cfa37b5e5629bb7a095cb5e4638605"}
Oct 31 20:25:38.153990 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:38.153856    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4554a17c-bdcd45c4c-84k8c"
Oct 31 20:25:38.180665 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:38.180621    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4554a17c-bdcd45c4c-84k8c" podStartSLOduration=1.807761205 podStartE2EDuration="2.180609044s" totalImagesPullingTime="372.847839ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:25:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:25:38.17872977 +0000 UTC m=+3968.751963900" watchObservedRunningTime="2026-10-31 20:25:38.180609044 +0000 UTC m=+3968.753843174"
Oct 31 20:25:38.246625 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:38.246598    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:25:39.028365 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:39.025857    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:25:39.860657 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:39.860624    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:25:40.543543 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:40.543500    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:25:40.998448 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:40.998414    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b784f56d7-phm8m"]
Oct 31 20:25:40.998649 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:40.998627    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-7b784f56d7-phm8m" podUID="835bc094-57b0-4d4e-adf9-df4f189ca317" containerName="maas-api" containerID="cri-o://6f677490caaaf6fd588a1cbaad6bfe170c2bf86eb22db68931ab850b874a5b13" gracePeriod=30
Oct 31 20:25:41.381401 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:41.381232    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:25:42.198732 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:42.198695    4645 generic.go:356] "Generic (PLEG): container finished" podID="835bc094-57b0-4d4e-adf9-df4f189ca317" containerID="6f677490caaaf6fd588a1cbaad6bfe170c2bf86eb22db68931ab850b874a5b13" exitCode=0
Oct 31 20:25:42.199281 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:42.198825    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7b784f56d7-phm8m" event={"ID":"835bc094-57b0-4d4e-adf9-df4f189ca317","Type":"ContainerDied","Data":"6f677490caaaf6fd588a1cbaad6bfe170c2bf86eb22db68931ab850b874a5b13"}
Oct 31 20:25:42.294368 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:42.277912    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:25:42.300669 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:42.300643    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7b784f56d7-phm8m"
Oct 31 20:25:42.334798 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:42.334735    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/835bc094-57b0-4d4e-adf9-df4f189ca317-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/835bc094-57b0-4d4e-adf9-df4f189ca317-openshift-service-ca\") pod \"835bc094-57b0-4d4e-adf9-df4f189ca317\" (UID: \"835bc094-57b0-4d4e-adf9-df4f189ca317\") "
Oct 31 20:25:42.335098 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:42.335074    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/835bc094-57b0-4d4e-adf9-df4f189ca317-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/835bc094-57b0-4d4e-adf9-df4f189ca317-maas-api-tls\") pod \"835bc094-57b0-4d4e-adf9-df4f189ca317\" (UID: \"835bc094-57b0-4d4e-adf9-df4f189ca317\") "
Oct 31 20:25:42.335279 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:42.335261    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/835bc094-57b0-4d4e-adf9-df4f189ca317-kube-api-access-ncrz8\" (UniqueName: \"kubernetes.io/projected/835bc094-57b0-4d4e-adf9-df4f189ca317-kube-api-access-ncrz8\") pod \"835bc094-57b0-4d4e-adf9-df4f189ca317\" (UID: \"835bc094-57b0-4d4e-adf9-df4f189ca317\") "
Oct 31 20:25:42.335463 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:42.335447    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/835bc094-57b0-4d4e-adf9-df4f189ca317-metrics-certs\" (UniqueName: \"kubernetes.io/secret/835bc094-57b0-4d4e-adf9-df4f189ca317-metrics-certs\") pod \"835bc094-57b0-4d4e-adf9-df4f189ca317\" (UID: \"835bc094-57b0-4d4e-adf9-df4f189ca317\") "
Oct 31 20:25:42.335653 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:42.335098    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/835bc094-57b0-4d4e-adf9-df4f189ca317-openshift-service-ca" pod "835bc094-57b0-4d4e-adf9-df4f189ca317" (UID: "835bc094-57b0-4d4e-adf9-df4f189ca317"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:25:42.335909 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:42.335884    4645 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/835bc094-57b0-4d4e-adf9-df4f189ca317-openshift-service-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:25:42.337259 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:42.337235    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/835bc094-57b0-4d4e-adf9-df4f189ca317-maas-api-tls" pod "835bc094-57b0-4d4e-adf9-df4f189ca317" (UID: "835bc094-57b0-4d4e-adf9-df4f189ca317"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:25:42.337565 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:42.337544    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/835bc094-57b0-4d4e-adf9-df4f189ca317-metrics-certs" pod "835bc094-57b0-4d4e-adf9-df4f189ca317" (UID: "835bc094-57b0-4d4e-adf9-df4f189ca317"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:25:42.337646 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:42.337549    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/835bc094-57b0-4d4e-adf9-df4f189ca317-kube-api-access-ncrz8" pod "835bc094-57b0-4d4e-adf9-df4f189ca317" (UID: "835bc094-57b0-4d4e-adf9-df4f189ca317"). InnerVolumeSpecName "kube-api-access-ncrz8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:25:42.437201 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:42.437162    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/835bc094-57b0-4d4e-adf9-df4f189ca317-maas-api-tls\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:25:42.437201 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:42.437195    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncrz8\" (UniqueName: \"kubernetes.io/projected/835bc094-57b0-4d4e-adf9-df4f189ca317-kube-api-access-ncrz8\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:25:42.437201 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:42.437205    4645 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/835bc094-57b0-4d4e-adf9-df4f189ca317-metrics-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:25:42.981155 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:42.980875    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:25:43.204925 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:43.204894    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7b784f56d7-phm8m"
Oct 31 20:25:43.205373 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:43.204937    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7b784f56d7-phm8m" event={"ID":"835bc094-57b0-4d4e-adf9-df4f189ca317","Type":"ContainerDied","Data":"427b9a575c78470484fc503dd60547533c2b7349a4db61702fcda5f72e2e1ca0"}
Oct 31 20:25:43.205373 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:43.204998    4645 scope.go:122] "RemoveContainer" containerID="6f677490caaaf6fd588a1cbaad6bfe170c2bf86eb22db68931ab850b874a5b13"
Oct 31 20:25:43.228092 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:43.228061    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b784f56d7-phm8m"]
Oct 31 20:25:43.234916 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:43.234825    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b784f56d7-phm8m"]
Oct 31 20:25:43.616378 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:43.616285    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="835bc094-57b0-4d4e-adf9-df4f189ca317" path="/var/lib/kubelet/pods/835bc094-57b0-4d4e-adf9-df4f189ca317/volumes"
Oct 31 20:25:43.682135 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:43.682102    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:25:44.182531 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:44.182499    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4554a17c-bdcd45c4c-84k8c"
Oct 31 20:25:44.422454 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:44.422419    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:25:45.217556 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:45.217523    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:25:46.144040 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:46.143988    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:25:46.839719 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:46.839674    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:25:47.642523 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:47.642489    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:25:48.157112 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:48.157077    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-4554a17c-5776dfb9cf-x8n6g"
Oct 31 20:25:48.372782 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:48.372740    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:25:49.130336 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:49.130303    4645 patch_prober.go:29] interesting pod/e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.78:15021/healthz/ready\": dial tcp 10.133.1.78:15021: connect: connection refused" start-of-body=
Oct 31 20:25:49.130843 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:49.130387    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs" podUID="f2cf6a6f-1268-4997-8728-e8ca29514c4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.78:15021/healthz/ready\": dial tcp 10.133.1.78:15021: connect: connection refused"
Oct 31 20:25:49.137597 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:49.137563    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:25:49.236395 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:49.236360    4645 generic.go:356] "Generic (PLEG): container finished" podID="f2cf6a6f-1268-4997-8728-e8ca29514c4f" containerID="b5ddce04c86daa9592ec01b999d322e7dba8134b9e147d400781de9a0faaf7ae" exitCode=0
Oct 31 20:25:49.236596 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:49.236438    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs" event={"ID":"f2cf6a6f-1268-4997-8728-e8ca29514c4f","Type":"ContainerDied","Data":"b5ddce04c86daa9592ec01b999d322e7dba8134b9e147d400781de9a0faaf7ae"}
Oct 31 20:25:49.236900 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:49.236882    4645 scope.go:122] "RemoveContainer" containerID="b5ddce04c86daa9592ec01b999d322e7dba8134b9e147d400781de9a0faaf7ae"
Oct 31 20:25:49.238121 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:49.238090    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:25:49.238209 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:49.238159    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:25:49.238209 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:49.238194    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:25:49.980353 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:49.977981    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:25:50.243055 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:50.242971    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs" event={"ID":"f2cf6a6f-1268-4997-8728-e8ca29514c4f","Type":"ContainerStarted","Data":"162682d2600ff8c5e65b0f70b5549e46eeb8165ab9e6b78e1ead8723c6edf1e1"}
Oct 31 20:25:50.505825 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:50.505737    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs"
Oct 31 20:25:50.505996 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:50.505967    4645 patch_prober.go:29] interesting pod/e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.78:15021/healthz/ready\": dial tcp 10.133.1.78:15021: connect: connection refused" start-of-body=
Oct 31 20:25:50.506051 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:50.506017    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs" podUID="f2cf6a6f-1268-4997-8728-e8ca29514c4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.78:15021/healthz/ready\": dial tcp 10.133.1.78:15021: connect: connection refused"
Oct 31 20:25:50.779460 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:50.766687    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4554a17c-sdwsx"]
Oct 31 20:25:50.779460 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:50.767270    4645 cpu_manager.go:414] "containerMap: removing container" podUID="835bc094-57b0-4d4e-adf9-df4f189ca317" containerName="maas-api"
Oct 31 20:25:50.779460 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:50.767290    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="835bc094-57b0-4d4e-adf9-df4f189ca317" containerName="maas-api"
Oct 31 20:25:50.779460 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:50.772771    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="835bc094-57b0-4d4e-adf9-df4f189ca317" containerName="maas-api"
Oct 31 20:25:50.779839 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:50.779675    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4554a17c-sdwsx"
Oct 31 20:25:50.783791 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:50.783766    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:25:50.783947 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:50.783858    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r8bvf\""
Oct 31 20:25:50.785092 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:50.785071    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4554a17c-sdwsx"]
Oct 31 20:25:50.846285 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:50.846260    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f6b5a22-b637-4b2b-87dc-f1d668b5cfea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-4554a17c-sdwsx\" (UID: \"7f6b5a22-b637-4b2b-87dc-f1d668b5cfea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4554a17c-sdwsx"
Oct 31 20:25:50.846427 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:50.846360    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r8qz\" (UniqueName: \"kubernetes.io/projected/7f6b5a22-b637-4b2b-87dc-f1d668b5cfea-kube-api-access-2r8qz\") pod \"maas-api-revoke-keys-e2e-ait-4554a17c-sdwsx\" (UID: \"7f6b5a22-b637-4b2b-87dc-f1d668b5cfea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4554a17c-sdwsx"
Oct 31 20:25:50.846427 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:50.846395    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f6b5a22-b637-4b2b-87dc-f1d668b5cfea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-4554a17c-sdwsx\" (UID: \"7f6b5a22-b637-4b2b-87dc-f1d668b5cfea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4554a17c-sdwsx"
Oct 31 20:25:50.896157 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:50.896128    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:25:50.947103 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:50.947075    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f6b5a22-b637-4b2b-87dc-f1d668b5cfea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-4554a17c-sdwsx\" (UID: \"7f6b5a22-b637-4b2b-87dc-f1d668b5cfea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4554a17c-sdwsx"
Oct 31 20:25:50.947217 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:50.947151    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r8qz\" (UniqueName: \"kubernetes.io/projected/7f6b5a22-b637-4b2b-87dc-f1d668b5cfea-kube-api-access-2r8qz\") pod \"maas-api-revoke-keys-e2e-ait-4554a17c-sdwsx\" (UID: \"7f6b5a22-b637-4b2b-87dc-f1d668b5cfea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4554a17c-sdwsx"
Oct 31 20:25:50.947217 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:50.947181    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f6b5a22-b637-4b2b-87dc-f1d668b5cfea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-4554a17c-sdwsx\" (UID: \"7f6b5a22-b637-4b2b-87dc-f1d668b5cfea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4554a17c-sdwsx"
Oct 31 20:25:50.947736 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:50.947718    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f6b5a22-b637-4b2b-87dc-f1d668b5cfea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-4554a17c-sdwsx\" (UID: \"7f6b5a22-b637-4b2b-87dc-f1d668b5cfea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4554a17c-sdwsx"
Oct 31 20:25:50.949595 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:50.949576    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f6b5a22-b637-4b2b-87dc-f1d668b5cfea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-4554a17c-sdwsx\" (UID: \"7f6b5a22-b637-4b2b-87dc-f1d668b5cfea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4554a17c-sdwsx"
Oct 31 20:25:50.982037 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:50.982008    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r8qz\" (UniqueName: \"kubernetes.io/projected/7f6b5a22-b637-4b2b-87dc-f1d668b5cfea-kube-api-access-2r8qz\") pod \"maas-api-revoke-keys-e2e-ait-4554a17c-sdwsx\" (UID: \"7f6b5a22-b637-4b2b-87dc-f1d668b5cfea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4554a17c-sdwsx"
Oct 31 20:25:51.096137 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:51.096048    4645 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4554a17c-sdwsx" secret="" err="secret \"maas-api-cleanup-dockercfg-r8bvf\" not found"
Oct 31 20:25:51.096137 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:51.096105    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4554a17c-sdwsx"
Oct 31 20:25:51.377831 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:51.377686    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4554a17c-sdwsx"]
Oct 31 20:25:51.385452 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:25:51.385423    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f6b5a22_b637_4b2b_87dc_f1d668b5cfea.slice/crio-80938341f65855a2b749aa276d6562fa5fa0ae8c470bc9e0a1d4dd591237ee16 WatchSource:0}: Error finding container 80938341f65855a2b749aa276d6562fa5fa0ae8c470bc9e0a1d4dd591237ee16: Status 404 returned error can't find the container with id 80938341f65855a2b749aa276d6562fa5fa0ae8c470bc9e0a1d4dd591237ee16
Oct 31 20:25:51.506192 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:51.506163    4645 patch_prober.go:29] interesting pod/e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.78:15021/healthz/ready\": dial tcp 10.133.1.78:15021: connect: connection refused" start-of-body=
Oct 31 20:25:51.506308 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:51.506212    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs" podUID="f2cf6a6f-1268-4997-8728-e8ca29514c4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.78:15021/healthz/ready\": dial tcp 10.133.1.78:15021: connect: connection refused"
Oct 31 20:25:51.878347 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:51.877068    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:25:52.253773 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:52.253735    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4554a17c-sdwsx" event={"ID":"7f6b5a22-b637-4b2b-87dc-f1d668b5cfea","Type":"ContainerStarted","Data":"550aac81e47546613c0c86c10522f6122dbcf66ce029de2cc9f9e187489e3af2"}
Oct 31 20:25:52.253965 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:52.253814    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4554a17c-sdwsx" event={"ID":"7f6b5a22-b637-4b2b-87dc-f1d668b5cfea","Type":"ContainerStarted","Data":"80938341f65855a2b749aa276d6562fa5fa0ae8c470bc9e0a1d4dd591237ee16"}
Oct 31 20:25:52.254042 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:52.254026    4645 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4554a17c-sdwsx" secret="" err="secret \"maas-api-cleanup-dockercfg-r8bvf\" not found"
Oct 31 20:25:52.296520 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:52.296468    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4554a17c-sdwsx" podStartSLOduration=2.296455634 podStartE2EDuration="2.296455634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:25:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:25:52.294757572 +0000 UTC m=+3982.867991700" watchObservedRunningTime="2026-10-31 20:25:52.296455634 +0000 UTC m=+3982.869689763"
Oct 31 20:25:52.505607 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:52.505538    4645 patch_prober.go:29] interesting pod/e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.78:15021/healthz/ready\": dial tcp 10.133.1.78:15021: connect: connection refused" start-of-body=
Oct 31 20:25:52.505607 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:52.505589    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs" podUID="f2cf6a6f-1268-4997-8728-e8ca29514c4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.78:15021/healthz/ready\": dial tcp 10.133.1.78:15021: connect: connection refused"
Oct 31 20:25:52.623368 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:52.623313    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:25:53.287736 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:53.287693    4645 generic.go:356] "Generic (PLEG): container finished" podID="7f6b5a22-b637-4b2b-87dc-f1d668b5cfea" containerID="550aac81e47546613c0c86c10522f6122dbcf66ce029de2cc9f9e187489e3af2" exitCode=0
Oct 31 20:25:53.287973 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:53.287753    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4554a17c-sdwsx" event={"ID":"7f6b5a22-b637-4b2b-87dc-f1d668b5cfea","Type":"ContainerDied","Data":"550aac81e47546613c0c86c10522f6122dbcf66ce029de2cc9f9e187489e3af2"}
Oct 31 20:25:53.355008 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:53.354966    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:25:53.505598 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:53.505554    4645 patch_prober.go:29] interesting pod/e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.78:15021/healthz/ready\": dial tcp 10.133.1.78:15021: connect: connection refused" start-of-body=
Oct 31 20:25:53.506024 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:53.505617    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs" podUID="f2cf6a6f-1268-4997-8728-e8ca29514c4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.78:15021/healthz/ready\": dial tcp 10.133.1.78:15021: connect: connection refused"
Oct 31 20:25:54.135207 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:54.135119    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:25:54.442310 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:54.442286    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4554a17c-sdwsx"
Oct 31 20:25:54.505950 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:54.505912    4645 patch_prober.go:29] interesting pod/e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.78:15021/healthz/ready\": dial tcp 10.133.1.78:15021: connect: connection refused" start-of-body=
Oct 31 20:25:54.506344 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:54.505963    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs" podUID="f2cf6a6f-1268-4997-8728-e8ca29514c4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.78:15021/healthz/ready\": dial tcp 10.133.1.78:15021: connect: connection refused"
Oct 31 20:25:54.608077 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:54.608042    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f6b5a22-b637-4b2b-87dc-f1d668b5cfea-kube-api-access-2r8qz\" (UniqueName: \"kubernetes.io/projected/7f6b5a22-b637-4b2b-87dc-f1d668b5cfea-kube-api-access-2r8qz\") pod \"7f6b5a22-b637-4b2b-87dc-f1d668b5cfea\" (UID: \"7f6b5a22-b637-4b2b-87dc-f1d668b5cfea\") "
Oct 31 20:25:54.608257 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:54.608172    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f6b5a22-b637-4b2b-87dc-f1d668b5cfea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f6b5a22-b637-4b2b-87dc-f1d668b5cfea-maas-api-openshift-ca\") pod \"7f6b5a22-b637-4b2b-87dc-f1d668b5cfea\" (UID: \"7f6b5a22-b637-4b2b-87dc-f1d668b5cfea\") "
Oct 31 20:25:54.608257 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:54.608229    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f6b5a22-b637-4b2b-87dc-f1d668b5cfea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f6b5a22-b637-4b2b-87dc-f1d668b5cfea-maas-api-xks-ca\") pod \"7f6b5a22-b637-4b2b-87dc-f1d668b5cfea\" (UID: \"7f6b5a22-b637-4b2b-87dc-f1d668b5cfea\") "
Oct 31 20:25:54.608538 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:54.608514    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f6b5a22-b637-4b2b-87dc-f1d668b5cfea-maas-api-openshift-ca" pod "7f6b5a22-b637-4b2b-87dc-f1d668b5cfea" (UID: "7f6b5a22-b637-4b2b-87dc-f1d668b5cfea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:25:54.610280 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:54.610250    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f6b5a22-b637-4b2b-87dc-f1d668b5cfea-maas-api-xks-ca" pod "7f6b5a22-b637-4b2b-87dc-f1d668b5cfea" (UID: "7f6b5a22-b637-4b2b-87dc-f1d668b5cfea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:25:54.610507 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:54.610410    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f6b5a22-b637-4b2b-87dc-f1d668b5cfea-kube-api-access-2r8qz" pod "7f6b5a22-b637-4b2b-87dc-f1d668b5cfea" (UID: "7f6b5a22-b637-4b2b-87dc-f1d668b5cfea"). InnerVolumeSpecName "kube-api-access-2r8qz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:25:54.709764 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:54.709686    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f6b5a22-b637-4b2b-87dc-f1d668b5cfea-maas-api-xks-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:25:54.709764 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:54.709718    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2r8qz\" (UniqueName: \"kubernetes.io/projected/7f6b5a22-b637-4b2b-87dc-f1d668b5cfea-kube-api-access-2r8qz\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:25:54.709764 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:54.709734    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f6b5a22-b637-4b2b-87dc-f1d668b5cfea-maas-api-openshift-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:25:54.866274 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:54.866157    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:25:55.298516 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:55.298483    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4554a17c-sdwsx"
Oct 31 20:25:55.298713 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:55.298485    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4554a17c-sdwsx" event={"ID":"7f6b5a22-b637-4b2b-87dc-f1d668b5cfea","Type":"ContainerDied","Data":"80938341f65855a2b749aa276d6562fa5fa0ae8c470bc9e0a1d4dd591237ee16"}
Oct 31 20:25:55.298713 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:55.298612    4645 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80938341f65855a2b749aa276d6562fa5fa0ae8c470bc9e0a1d4dd591237ee16"
Oct 31 20:25:55.509131 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:55.509105    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs"
Oct 31 20:25:55.509575 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:55.509561    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs"
Oct 31 20:25:55.510166 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:55.510145    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs"
Oct 31 20:25:55.653476 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:55.653391    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:25:56.567738 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:56.567700    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:25:57.351563 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:57.351525    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:25:58.065078 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:58.065044    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:25:58.831259 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:58.831219    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:25:59.640148 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:25:59.640108    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:00.587896 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:00.587851    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:00.937304 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:00.937256    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4554a17c-5776dfb9cf-x8n6g"]
Oct 31 20:26:01.059836 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:01.059797    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:01.202534 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:01.202464    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:01.411891 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:01.411854    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:02.019414 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:02.019379    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4554a17c-bdcd45c4c-84k8c"]
Oct 31 20:26:02.019879 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:02.019635    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4554a17c-bdcd45c4c-84k8c" podUID="6d773219-9039-4e52-b5dd-f20cebacf1ab" containerName="maas-api" containerID="cri-o://157d7625b9ba28991a23f7a16369e08358cfa37b5e5629bb7a095cb5e4638605" gracePeriod=30
Oct 31 20:26:02.128705 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:02.128673    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:02.265352 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:02.265313    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4554a17c-bdcd45c4c-84k8c"
Oct 31 20:26:02.278917 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:02.278861    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d773219-9039-4e52-b5dd-f20cebacf1ab-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d773219-9039-4e52-b5dd-f20cebacf1ab-openshift-service-ca\") pod \"6d773219-9039-4e52-b5dd-f20cebacf1ab\" (UID: \"6d773219-9039-4e52-b5dd-f20cebacf1ab\") "
Oct 31 20:26:02.279045 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:02.278927    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d773219-9039-4e52-b5dd-f20cebacf1ab-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d773219-9039-4e52-b5dd-f20cebacf1ab-metrics-certs\") pod \"6d773219-9039-4e52-b5dd-f20cebacf1ab\" (UID: \"6d773219-9039-4e52-b5dd-f20cebacf1ab\") "
Oct 31 20:26:02.279045 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:02.278986    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d773219-9039-4e52-b5dd-f20cebacf1ab-kube-api-access-4h6c7\" (UniqueName: \"kubernetes.io/projected/6d773219-9039-4e52-b5dd-f20cebacf1ab-kube-api-access-4h6c7\") pod \"6d773219-9039-4e52-b5dd-f20cebacf1ab\" (UID: \"6d773219-9039-4e52-b5dd-f20cebacf1ab\") "
Oct 31 20:26:02.279162 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:02.279069    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d773219-9039-4e52-b5dd-f20cebacf1ab-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d773219-9039-4e52-b5dd-f20cebacf1ab-maas-api-tls\") pod \"6d773219-9039-4e52-b5dd-f20cebacf1ab\" (UID: \"6d773219-9039-4e52-b5dd-f20cebacf1ab\") "
Oct 31 20:26:02.279285 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:02.279255    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d773219-9039-4e52-b5dd-f20cebacf1ab-openshift-service-ca" pod "6d773219-9039-4e52-b5dd-f20cebacf1ab" (UID: "6d773219-9039-4e52-b5dd-f20cebacf1ab"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:26:02.279473 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:02.279451    4645 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d773219-9039-4e52-b5dd-f20cebacf1ab-openshift-service-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:26:02.281094 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:02.281068    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d773219-9039-4e52-b5dd-f20cebacf1ab-kube-api-access-4h6c7" pod "6d773219-9039-4e52-b5dd-f20cebacf1ab" (UID: "6d773219-9039-4e52-b5dd-f20cebacf1ab"). InnerVolumeSpecName "kube-api-access-4h6c7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:26:02.281179 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:02.281136    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d773219-9039-4e52-b5dd-f20cebacf1ab-metrics-certs" pod "6d773219-9039-4e52-b5dd-f20cebacf1ab" (UID: "6d773219-9039-4e52-b5dd-f20cebacf1ab"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:26:02.281179 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:02.281157    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d773219-9039-4e52-b5dd-f20cebacf1ab-maas-api-tls" pod "6d773219-9039-4e52-b5dd-f20cebacf1ab" (UID: "6d773219-9039-4e52-b5dd-f20cebacf1ab"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:26:02.333553 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:02.333519    4645 generic.go:356] "Generic (PLEG): container finished" podID="6d773219-9039-4e52-b5dd-f20cebacf1ab" containerID="157d7625b9ba28991a23f7a16369e08358cfa37b5e5629bb7a095cb5e4638605" exitCode=0
Oct 31 20:26:02.333718 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:02.333559    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4554a17c-bdcd45c4c-84k8c"
Oct 31 20:26:02.333718 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:02.333598    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4554a17c-bdcd45c4c-84k8c" event={"ID":"6d773219-9039-4e52-b5dd-f20cebacf1ab","Type":"ContainerDied","Data":"157d7625b9ba28991a23f7a16369e08358cfa37b5e5629bb7a095cb5e4638605"}
Oct 31 20:26:02.333718 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:02.333675    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4554a17c-bdcd45c4c-84k8c" event={"ID":"6d773219-9039-4e52-b5dd-f20cebacf1ab","Type":"ContainerDied","Data":"bd6c6a96eb9d99245f2b7ecb028ffbf14ac7f7747c3a9f0eb194c1bfe5872885"}
Oct 31 20:26:02.333718 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:02.333697    4645 scope.go:122] "RemoveContainer" containerID="157d7625b9ba28991a23f7a16369e08358cfa37b5e5629bb7a095cb5e4638605"
Oct 31 20:26:02.343291 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:02.343275    4645 scope.go:122] "RemoveContainer" containerID="157d7625b9ba28991a23f7a16369e08358cfa37b5e5629bb7a095cb5e4638605"
Oct 31 20:26:02.343585 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:26:02.343566    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"157d7625b9ba28991a23f7a16369e08358cfa37b5e5629bb7a095cb5e4638605\": container with ID starting with 157d7625b9ba28991a23f7a16369e08358cfa37b5e5629bb7a095cb5e4638605 not found: ID does not exist" containerID="157d7625b9ba28991a23f7a16369e08358cfa37b5e5629bb7a095cb5e4638605"
Oct 31 20:26:02.343647 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:02.343594    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"157d7625b9ba28991a23f7a16369e08358cfa37b5e5629bb7a095cb5e4638605"} err="failed to get container status \"157d7625b9ba28991a23f7a16369e08358cfa37b5e5629bb7a095cb5e4638605\": rpc error: code = NotFound desc = could not find container \"157d7625b9ba28991a23f7a16369e08358cfa37b5e5629bb7a095cb5e4638605\": container with ID starting with 157d7625b9ba28991a23f7a16369e08358cfa37b5e5629bb7a095cb5e4638605 not found: ID does not exist"
Oct 31 20:26:02.365892 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:02.365860    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4554a17c-bdcd45c4c-84k8c"]
Oct 31 20:26:02.370069 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:02.370047    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4554a17c-bdcd45c4c-84k8c"]
Oct 31 20:26:02.380623 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:02.380603    4645 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d773219-9039-4e52-b5dd-f20cebacf1ab-metrics-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:26:02.380693 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:02.380629    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4h6c7\" (UniqueName: \"kubernetes.io/projected/6d773219-9039-4e52-b5dd-f20cebacf1ab-kube-api-access-4h6c7\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:26:02.380693 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:02.380644    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d773219-9039-4e52-b5dd-f20cebacf1ab-maas-api-tls\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:26:02.391647 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:02.391625    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:02.444108 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:02.444078    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4554a17c-sdwsx"]
Oct 31 20:26:02.446413 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:02.446391    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4554a17c-sdwsx"]
Oct 31 20:26:02.533423 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:02.533310    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:03.616832 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:03.616798    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d773219-9039-4e52-b5dd-f20cebacf1ab" path="/var/lib/kubelet/pods/6d773219-9039-4e52-b5dd-f20cebacf1ab/volumes"
Oct 31 20:26:03.617226 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:03.617212    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f6b5a22-b637-4b2b-87dc-f1d668b5cfea" path="/var/lib/kubelet/pods/7f6b5a22-b637-4b2b-87dc-f1d668b5cfea/volumes"
Oct 31 20:26:05.937744 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:05.937697    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-4554a17c-5776dfb9cf-x8n6g" podUID="8c336e91-9d54-4a61-9bcb-1fb9bc3832ab" containerName="payload-pre-processing" containerID="cri-o://b4af2f3890932d692321abf503ad22f0cc599e4761d28c14c1320c88e05aaf8a" gracePeriod=25
Oct 31 20:26:06.205764 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:06.205739    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4554a17c-5776dfb9cf-x8n6g"
Oct 31 20:26:06.245845 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:26:06.245805    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 20:26:06.323412 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:06.323372    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c336e91-9d54-4a61-9bcb-1fb9bc3832ab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c336e91-9d54-4a61-9bcb-1fb9bc3832ab-plugins-config-volume\") pod \"8c336e91-9d54-4a61-9bcb-1fb9bc3832ab\" (UID: \"8c336e91-9d54-4a61-9bcb-1fb9bc3832ab\") "
Oct 31 20:26:06.323628 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:06.323465    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c336e91-9d54-4a61-9bcb-1fb9bc3832ab-kube-api-access-qhh74\" (UniqueName: \"kubernetes.io/projected/8c336e91-9d54-4a61-9bcb-1fb9bc3832ab-kube-api-access-qhh74\") pod \"8c336e91-9d54-4a61-9bcb-1fb9bc3832ab\" (UID: \"8c336e91-9d54-4a61-9bcb-1fb9bc3832ab\") "
Oct 31 20:26:06.323774 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:06.323747    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c336e91-9d54-4a61-9bcb-1fb9bc3832ab-plugins-config-volume" pod "8c336e91-9d54-4a61-9bcb-1fb9bc3832ab" (UID: "8c336e91-9d54-4a61-9bcb-1fb9bc3832ab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:26:06.325559 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:06.325536    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c336e91-9d54-4a61-9bcb-1fb9bc3832ab-kube-api-access-qhh74" pod "8c336e91-9d54-4a61-9bcb-1fb9bc3832ab" (UID: "8c336e91-9d54-4a61-9bcb-1fb9bc3832ab"). InnerVolumeSpecName "kube-api-access-qhh74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:26:06.353539 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:06.353507    4645 generic.go:356] "Generic (PLEG): container finished" podID="8c336e91-9d54-4a61-9bcb-1fb9bc3832ab" containerID="b4af2f3890932d692321abf503ad22f0cc599e4761d28c14c1320c88e05aaf8a" exitCode=0
Oct 31 20:26:06.353696 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:06.353551    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4554a17c-5776dfb9cf-x8n6g"
Oct 31 20:26:06.353696 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:06.353589    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4554a17c-5776dfb9cf-x8n6g" event={"ID":"8c336e91-9d54-4a61-9bcb-1fb9bc3832ab","Type":"ContainerDied","Data":"b4af2f3890932d692321abf503ad22f0cc599e4761d28c14c1320c88e05aaf8a"}
Oct 31 20:26:06.353696 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:06.353643    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4554a17c-5776dfb9cf-x8n6g" event={"ID":"8c336e91-9d54-4a61-9bcb-1fb9bc3832ab","Type":"ContainerDied","Data":"0979dbc131352fd95c5127e79945068f656e09af1a00dab5b80b41810fbfa0fa"}
Oct 31 20:26:06.353696 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:06.353666    4645 scope.go:122] "RemoveContainer" containerID="b4af2f3890932d692321abf503ad22f0cc599e4761d28c14c1320c88e05aaf8a"
Oct 31 20:26:06.363202 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:06.363182    4645 scope.go:122] "RemoveContainer" containerID="b4af2f3890932d692321abf503ad22f0cc599e4761d28c14c1320c88e05aaf8a"
Oct 31 20:26:06.363519 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:26:06.363485    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4af2f3890932d692321abf503ad22f0cc599e4761d28c14c1320c88e05aaf8a\": container with ID starting with b4af2f3890932d692321abf503ad22f0cc599e4761d28c14c1320c88e05aaf8a not found: ID does not exist" containerID="b4af2f3890932d692321abf503ad22f0cc599e4761d28c14c1320c88e05aaf8a"
Oct 31 20:26:06.363602 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:06.363534    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4af2f3890932d692321abf503ad22f0cc599e4761d28c14c1320c88e05aaf8a"} err="failed to get container status \"b4af2f3890932d692321abf503ad22f0cc599e4761d28c14c1320c88e05aaf8a\": rpc error: code = NotFound desc = could not find container \"b4af2f3890932d692321abf503ad22f0cc599e4761d28c14c1320c88e05aaf8a\": container with ID starting with b4af2f3890932d692321abf503ad22f0cc599e4761d28c14c1320c88e05aaf8a not found: ID does not exist"
Oct 31 20:26:06.386349 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:06.386299    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4554a17c-5776dfb9cf-x8n6g"]
Oct 31 20:26:06.388062 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:06.388038    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4554a17c-5776dfb9cf-x8n6g"]
Oct 31 20:26:06.424390 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:06.424357    4645 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c336e91-9d54-4a61-9bcb-1fb9bc3832ab-plugins-config-volume\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:26:06.424390 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:06.424385    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhh74\" (UniqueName: \"kubernetes.io/projected/8c336e91-9d54-4a61-9bcb-1fb9bc3832ab-kube-api-access-qhh74\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:26:07.224442 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:07.224406    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs"]
Oct 31 20:26:07.224986 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:07.224666    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs" podUID="f2cf6a6f-1268-4997-8728-e8ca29514c4f" containerName="istio-proxy" containerID="cri-o://162682d2600ff8c5e65b0f70b5549e46eeb8165ab9e6b78e1ead8723c6edf1e1" gracePeriod=30
Oct 31 20:26:07.280135 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:07.280096    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:07.616526 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:07.616450    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c336e91-9d54-4a61-9bcb-1fb9bc3832ab" path="/var/lib/kubelet/pods/8c336e91-9d54-4a61-9bcb-1fb9bc3832ab/volumes"
Oct 31 20:26:08.604345 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:08.604291    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:09.098531 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:09.098498    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww"]
Oct 31 20:26:09.099025 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:09.099009    4645 cpu_manager.go:414] "containerMap: removing container" podUID="7f6b5a22-b637-4b2b-87dc-f1d668b5cfea" containerName="revoke-keys"
Oct 31 20:26:09.099079 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:09.099026    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f6b5a22-b637-4b2b-87dc-f1d668b5cfea" containerName="revoke-keys"
Oct 31 20:26:09.099079 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:09.099064    4645 cpu_manager.go:414] "containerMap: removing container" podUID="6d773219-9039-4e52-b5dd-f20cebacf1ab" containerName="maas-api"
Oct 31 20:26:09.099079 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:09.099072    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d773219-9039-4e52-b5dd-f20cebacf1ab" containerName="maas-api"
Oct 31 20:26:09.099194 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:09.099092    4645 cpu_manager.go:414] "containerMap: removing container" podUID="8c336e91-9d54-4a61-9bcb-1fb9bc3832ab" containerName="payload-pre-processing"
Oct 31 20:26:09.099194 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:09.099100    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c336e91-9d54-4a61-9bcb-1fb9bc3832ab" containerName="payload-pre-processing"
Oct 31 20:26:09.099194 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:09.099179    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d773219-9039-4e52-b5dd-f20cebacf1ab" containerName="maas-api"
Oct 31 20:26:09.099194 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:09.099191    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c336e91-9d54-4a61-9bcb-1fb9bc3832ab" containerName="payload-pre-processing"
Oct 31 20:26:09.099396 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:09.099203    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f6b5a22-b637-4b2b-87dc-f1d668b5cfea" containerName="revoke-keys"
Oct 31 20:26:09.102585 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:09.102566    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww"
Oct 31 20:26:09.109375 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:09.109044    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-authcfg-http-33c6d1-openshift-default-dockercfg-8pvhq\""
Oct 31 20:26:09.118298 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:09.118274    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww"]
Oct 31 20:26:09.151585 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:09.151558    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6bcb2da-7676-4255-a33a-084bb67f1bdf-istio-token\") pod \"e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww\" (UID: \"d6bcb2da-7676-4255-a33a-084bb67f1bdf\") " pod="openshift-ingress/e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww"
Oct 31 20:26:09.151706 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:09.151602    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6bcb2da-7676-4255-a33a-084bb67f1bdf-workload-socket\") pod \"e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww\" (UID: \"d6bcb2da-7676-4255-a33a-084bb67f1bdf\") " pod="openshift-ingress/e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww"
Oct 31 20:26:09.152002 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:09.151719    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6bcb2da-7676-4255-a33a-084bb67f1bdf-istiod-ca-cert\") pod \"e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww\" (UID: \"d6bcb2da-7676-4255-a33a-084bb67f1bdf\") " pod="openshift-ingress/e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww"
Oct 31 20:26:09.152002 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:09.151762    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6bcb2da-7676-4255-a33a-084bb67f1bdf-istio-envoy\") pod \"e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww\" (UID: \"d6bcb2da-7676-4255-a33a-084bb67f1bdf\") " pod="openshift-ingress/e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww"
Oct 31 20:26:09.152002 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:09.151793    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6bcb2da-7676-4255-a33a-084bb67f1bdf-workload-certs\") pod \"e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww\" (UID: \"d6bcb2da-7676-4255-a33a-084bb67f1bdf\") " pod="openshift-ingress/e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww"
Oct 31 20:26:09.152002 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:09.151860    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f468\" (UniqueName: \"kubernetes.io/projected/d6bcb2da-7676-4255-a33a-084bb67f1bdf-kube-api-access-2f468\") pod \"e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww\" (UID: \"d6bcb2da-7676-4255-a33a-084bb67f1bdf\") " pod="openshift-ingress/e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww"
Oct 31 20:26:09.152002 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:09.151915    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6bcb2da-7676-4255-a33a-084bb67f1bdf-istio-podinfo\") pod \"e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww\" (UID: \"d6bcb2da-7676-4255-a33a-084bb67f1bdf\") " pod="openshift-ingress/e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww"
Oct 31 20:26:09.152002 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:09.151949    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6bcb2da-7676-4255-a33a-084bb67f1bdf-istio-data\") pod \"e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww\" (UID: \"d6bcb2da-7676-4255-a33a-084bb67f1bdf\") " pod="openshift-ingress/e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww"
Oct 31 20:26:09.152397 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:09.152003    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6bcb2da-7676-4255-a33a-084bb67f1bdf-credential-socket\") pod \"e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww\" (UID: \"d6bcb2da-7676-4255-a33a-084bb67f1bdf\") " pod="openshift-ingress/e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww"
Oct 31 20:26:09.216595 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:09.216556    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:09.252725 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:09.252693    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6bcb2da-7676-4255-a33a-084bb67f1bdf-istio-envoy\") pod \"e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww\" (UID: \"d6bcb2da-7676-4255-a33a-084bb67f1bdf\") " pod="openshift-ingress/e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww"
Oct 31 20:26:09.252896 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:09.252731    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6bcb2da-7676-4255-a33a-084bb67f1bdf-workload-certs\") pod \"e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww\" (UID: \"d6bcb2da-7676-4255-a33a-084bb67f1bdf\") " pod="openshift-ingress/e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww"
Oct 31 20:26:09.252896 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:09.252752    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f468\" (UniqueName: \"kubernetes.io/projected/d6bcb2da-7676-4255-a33a-084bb67f1bdf-kube-api-access-2f468\") pod \"e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww\" (UID: \"d6bcb2da-7676-4255-a33a-084bb67f1bdf\") " pod="openshift-ingress/e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww"
Oct 31 20:26:09.252896 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:09.252876    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6bcb2da-7676-4255-a33a-084bb67f1bdf-istio-podinfo\") pod \"e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww\" (UID: \"d6bcb2da-7676-4255-a33a-084bb67f1bdf\") " pod="openshift-ingress/e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww"
Oct 31 20:26:09.253073 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:09.252924    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6bcb2da-7676-4255-a33a-084bb67f1bdf-istio-data\") pod \"e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww\" (UID: \"d6bcb2da-7676-4255-a33a-084bb67f1bdf\") " pod="openshift-ingress/e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww"
Oct 31 20:26:09.253073 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:09.252949    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6bcb2da-7676-4255-a33a-084bb67f1bdf-credential-socket\") pod \"e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww\" (UID: \"d6bcb2da-7676-4255-a33a-084bb67f1bdf\") " pod="openshift-ingress/e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww"
Oct 31 20:26:09.253073 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:09.253048    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6bcb2da-7676-4255-a33a-084bb67f1bdf-istio-token\") pod \"e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww\" (UID: \"d6bcb2da-7676-4255-a33a-084bb67f1bdf\") " pod="openshift-ingress/e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww"
Oct 31 20:26:09.253243 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:09.253094    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6bcb2da-7676-4255-a33a-084bb67f1bdf-workload-socket\") pod \"e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww\" (UID: \"d6bcb2da-7676-4255-a33a-084bb67f1bdf\") " pod="openshift-ingress/e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww"
Oct 31 20:26:09.253243 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:09.253137    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6bcb2da-7676-4255-a33a-084bb67f1bdf-workload-certs\") pod \"e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww\" (UID: \"d6bcb2da-7676-4255-a33a-084bb67f1bdf\") " pod="openshift-ingress/e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww"
Oct 31 20:26:09.253243 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:09.253161    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6bcb2da-7676-4255-a33a-084bb67f1bdf-istiod-ca-cert\") pod \"e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww\" (UID: \"d6bcb2da-7676-4255-a33a-084bb67f1bdf\") " pod="openshift-ingress/e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww"
Oct 31 20:26:09.253439 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:09.253358    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6bcb2da-7676-4255-a33a-084bb67f1bdf-credential-socket\") pod \"e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww\" (UID: \"d6bcb2da-7676-4255-a33a-084bb67f1bdf\") " pod="openshift-ingress/e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww"
Oct 31 20:26:09.253502 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:09.253453    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6bcb2da-7676-4255-a33a-084bb67f1bdf-workload-socket\") pod \"e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww\" (UID: \"d6bcb2da-7676-4255-a33a-084bb67f1bdf\") " pod="openshift-ingress/e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww"
Oct 31 20:26:09.253628 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:09.253607    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6bcb2da-7676-4255-a33a-084bb67f1bdf-istio-data\") pod \"e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww\" (UID: \"d6bcb2da-7676-4255-a33a-084bb67f1bdf\") " pod="openshift-ingress/e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww"
Oct 31 20:26:09.253821 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:09.253803    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6bcb2da-7676-4255-a33a-084bb67f1bdf-istiod-ca-cert\") pod \"e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww\" (UID: \"d6bcb2da-7676-4255-a33a-084bb67f1bdf\") " pod="openshift-ingress/e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww"
Oct 31 20:26:09.255132 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:09.255110    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6bcb2da-7676-4255-a33a-084bb67f1bdf-istio-envoy\") pod \"e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww\" (UID: \"d6bcb2da-7676-4255-a33a-084bb67f1bdf\") " pod="openshift-ingress/e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww"
Oct 31 20:26:09.255365 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:09.255322    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6bcb2da-7676-4255-a33a-084bb67f1bdf-istio-podinfo\") pod \"e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww\" (UID: \"d6bcb2da-7676-4255-a33a-084bb67f1bdf\") " pod="openshift-ingress/e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww"
Oct 31 20:26:09.272626 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:09.272601    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f468\" (UniqueName: \"kubernetes.io/projected/d6bcb2da-7676-4255-a33a-084bb67f1bdf-kube-api-access-2f468\") pod \"e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww\" (UID: \"d6bcb2da-7676-4255-a33a-084bb67f1bdf\") " pod="openshift-ingress/e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww"
Oct 31 20:26:09.277564 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:09.277541    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6bcb2da-7676-4255-a33a-084bb67f1bdf-istio-token\") pod \"e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww\" (UID: \"d6bcb2da-7676-4255-a33a-084bb67f1bdf\") " pod="openshift-ingress/e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww"
Oct 31 20:26:09.415235 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:09.415206    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww"
Oct 31 20:26:09.469365 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:09.469318    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:09.715653 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:09.715623    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww"]
Oct 31 20:26:09.718442 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:26:09.718416    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6bcb2da_7676_4255_a33a_084bb67f1bdf.slice/crio-048e8242dca79f48f2c02114193700328be1946f81471b0b7cf4d3ecbe429eeb WatchSource:0}: Error finding container 048e8242dca79f48f2c02114193700328be1946f81471b0b7cf4d3ecbe429eeb: Status 404 returned error can't find the container with id 048e8242dca79f48f2c02114193700328be1946f81471b0b7cf4d3ecbe429eeb
Oct 31 20:26:09.720572 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:09.720540    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:26:09.720675 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:09.720614    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:26:09.720675 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:09.720659    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:26:10.375016 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:10.374977    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww" event={"ID":"d6bcb2da-7676-4255-a33a-084bb67f1bdf","Type":"ContainerStarted","Data":"eb730b35f6ec8107fcb8c2b029c3b54924a37029598fcf041b82fc85da8b9f54"}
Oct 31 20:26:10.375240 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:10.375036    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww" event={"ID":"d6bcb2da-7676-4255-a33a-084bb67f1bdf","Type":"ContainerStarted","Data":"048e8242dca79f48f2c02114193700328be1946f81471b0b7cf4d3ecbe429eeb"}
Oct 31 20:26:10.398904 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:10.398856    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww" podStartSLOduration=1.398843004 podStartE2EDuration="1.398843004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:26:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:26:10.396649727 +0000 UTC m=+4000.969883858" watchObservedRunningTime="2026-10-31 20:26:10.398843004 +0000 UTC m=+4000.972077133"
Oct 31 20:26:10.416305 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:10.416277    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww"
Oct 31 20:26:10.420892 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:10.420871    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww"
Oct 31 20:26:10.511016 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:10.510987    4645 patch_prober.go:29] interesting pod/e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 20:26:10.511140 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:10.511038    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs" podUID="f2cf6a6f-1268-4997-8728-e8ca29514c4f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 20:26:11.380204 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:11.380163    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww"
Oct 31 20:26:11.381087 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:11.381063    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww"
Oct 31 20:26:12.387103 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:12.387074    4645 generic.go:356] "Generic (PLEG): container finished" podID="f2cf6a6f-1268-4997-8728-e8ca29514c4f" containerID="162682d2600ff8c5e65b0f70b5549e46eeb8165ab9e6b78e1ead8723c6edf1e1" exitCode=0
Oct 31 20:26:12.387594 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:12.387147    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs" event={"ID":"f2cf6a6f-1268-4997-8728-e8ca29514c4f","Type":"ContainerDied","Data":"162682d2600ff8c5e65b0f70b5549e46eeb8165ab9e6b78e1ead8723c6edf1e1"}
Oct 31 20:26:12.387594 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:12.387209    4645 scope.go:122] "RemoveContainer" containerID="b5ddce04c86daa9592ec01b999d322e7dba8134b9e147d400781de9a0faaf7ae"
Oct 31 20:26:12.481128 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:12.481101    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs"
Oct 31 20:26:12.586148 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:12.586090    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2cf6a6f-1268-4997-8728-e8ca29514c4f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2cf6a6f-1268-4997-8728-e8ca29514c4f-credential-socket\") pod \"f2cf6a6f-1268-4997-8728-e8ca29514c4f\" (UID: \"f2cf6a6f-1268-4997-8728-e8ca29514c4f\") "
Oct 31 20:26:12.586148 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:12.586130    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2cf6a6f-1268-4997-8728-e8ca29514c4f-kube-api-access-xcsff\" (UniqueName: \"kubernetes.io/projected/f2cf6a6f-1268-4997-8728-e8ca29514c4f-kube-api-access-xcsff\") pod \"f2cf6a6f-1268-4997-8728-e8ca29514c4f\" (UID: \"f2cf6a6f-1268-4997-8728-e8ca29514c4f\") "
Oct 31 20:26:12.586372 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:12.586149    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2cf6a6f-1268-4997-8728-e8ca29514c4f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2cf6a6f-1268-4997-8728-e8ca29514c4f-istio-envoy\") pod \"f2cf6a6f-1268-4997-8728-e8ca29514c4f\" (UID: \"f2cf6a6f-1268-4997-8728-e8ca29514c4f\") "
Oct 31 20:26:12.586372 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:12.586174    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2cf6a6f-1268-4997-8728-e8ca29514c4f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2cf6a6f-1268-4997-8728-e8ca29514c4f-istiod-ca-cert\") pod \"f2cf6a6f-1268-4997-8728-e8ca29514c4f\" (UID: \"f2cf6a6f-1268-4997-8728-e8ca29514c4f\") "
Oct 31 20:26:12.586372 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:12.586198    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2cf6a6f-1268-4997-8728-e8ca29514c4f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2cf6a6f-1268-4997-8728-e8ca29514c4f-workload-socket\") pod \"f2cf6a6f-1268-4997-8728-e8ca29514c4f\" (UID: \"f2cf6a6f-1268-4997-8728-e8ca29514c4f\") "
Oct 31 20:26:12.586372 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:12.586223    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2cf6a6f-1268-4997-8728-e8ca29514c4f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2cf6a6f-1268-4997-8728-e8ca29514c4f-istio-data\") pod \"f2cf6a6f-1268-4997-8728-e8ca29514c4f\" (UID: \"f2cf6a6f-1268-4997-8728-e8ca29514c4f\") "
Oct 31 20:26:12.586372 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:12.586251    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2cf6a6f-1268-4997-8728-e8ca29514c4f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2cf6a6f-1268-4997-8728-e8ca29514c4f-workload-certs\") pod \"f2cf6a6f-1268-4997-8728-e8ca29514c4f\" (UID: \"f2cf6a6f-1268-4997-8728-e8ca29514c4f\") "
Oct 31 20:26:12.586372 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:12.586291    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2cf6a6f-1268-4997-8728-e8ca29514c4f-istio-token\" (UniqueName: \"kubernetes.io/projected/f2cf6a6f-1268-4997-8728-e8ca29514c4f-istio-token\") pod \"f2cf6a6f-1268-4997-8728-e8ca29514c4f\" (UID: \"f2cf6a6f-1268-4997-8728-e8ca29514c4f\") "
Oct 31 20:26:12.586372 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:12.586353    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f2cf6a6f-1268-4997-8728-e8ca29514c4f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2cf6a6f-1268-4997-8728-e8ca29514c4f-istio-podinfo\") pod \"f2cf6a6f-1268-4997-8728-e8ca29514c4f\" (UID: \"f2cf6a6f-1268-4997-8728-e8ca29514c4f\") "
Oct 31 20:26:12.586786 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:12.586380    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2cf6a6f-1268-4997-8728-e8ca29514c4f-credential-socket" pod "f2cf6a6f-1268-4997-8728-e8ca29514c4f" (UID: "f2cf6a6f-1268-4997-8728-e8ca29514c4f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:26:12.586786 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:12.586576    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2cf6a6f-1268-4997-8728-e8ca29514c4f-workload-certs" pod "f2cf6a6f-1268-4997-8728-e8ca29514c4f" (UID: "f2cf6a6f-1268-4997-8728-e8ca29514c4f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:26:12.586786 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:12.586684    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2cf6a6f-1268-4997-8728-e8ca29514c4f-istiod-ca-cert" pod "f2cf6a6f-1268-4997-8728-e8ca29514c4f" (UID: "f2cf6a6f-1268-4997-8728-e8ca29514c4f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:26:12.586936 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:12.586794    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2cf6a6f-1268-4997-8728-e8ca29514c4f-workload-socket" pod "f2cf6a6f-1268-4997-8728-e8ca29514c4f" (UID: "f2cf6a6f-1268-4997-8728-e8ca29514c4f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:26:12.586936 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:12.586864    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2cf6a6f-1268-4997-8728-e8ca29514c4f-istio-data" pod "f2cf6a6f-1268-4997-8728-e8ca29514c4f" (UID: "f2cf6a6f-1268-4997-8728-e8ca29514c4f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:26:12.587060 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:12.587002    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2cf6a6f-1268-4997-8728-e8ca29514c4f-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:26:12.587060 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:12.587024    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2cf6a6f-1268-4997-8728-e8ca29514c4f-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:26:12.587060 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:12.587038    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2cf6a6f-1268-4997-8728-e8ca29514c4f-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:26:12.587060 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:12.587052    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2cf6a6f-1268-4997-8728-e8ca29514c4f-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:26:12.587282 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:12.587066    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2cf6a6f-1268-4997-8728-e8ca29514c4f-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:26:12.588620 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:12.588600    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2cf6a6f-1268-4997-8728-e8ca29514c4f-kube-api-access-xcsff" pod "f2cf6a6f-1268-4997-8728-e8ca29514c4f" (UID: "f2cf6a6f-1268-4997-8728-e8ca29514c4f"). InnerVolumeSpecName "kube-api-access-xcsff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:26:12.588724 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:12.588656    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2cf6a6f-1268-4997-8728-e8ca29514c4f-istio-envoy" pod "f2cf6a6f-1268-4997-8728-e8ca29514c4f" (UID: "f2cf6a6f-1268-4997-8728-e8ca29514c4f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:26:12.588910 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:12.588894    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2cf6a6f-1268-4997-8728-e8ca29514c4f-istio-token" pod "f2cf6a6f-1268-4997-8728-e8ca29514c4f" (UID: "f2cf6a6f-1268-4997-8728-e8ca29514c4f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:26:12.589098 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:12.589078    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f2cf6a6f-1268-4997-8728-e8ca29514c4f-istio-podinfo" pod "f2cf6a6f-1268-4997-8728-e8ca29514c4f" (UID: "f2cf6a6f-1268-4997-8728-e8ca29514c4f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 20:26:12.688411 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:12.688386    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcsff\" (UniqueName: \"kubernetes.io/projected/f2cf6a6f-1268-4997-8728-e8ca29514c4f-kube-api-access-xcsff\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:26:12.688411 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:12.688408    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2cf6a6f-1268-4997-8728-e8ca29514c4f-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:26:12.688544 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:12.688418    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2cf6a6f-1268-4997-8728-e8ca29514c4f-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:26:12.688544 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:12.688426    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2cf6a6f-1268-4997-8728-e8ca29514c4f-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:26:12.789179 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:12.789152    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:13.000216 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:13.000187    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:13.393146 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:13.393078    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs"
Oct 31 20:26:13.393146 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:13.393106    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs" event={"ID":"f2cf6a6f-1268-4997-8728-e8ca29514c4f","Type":"ContainerDied","Data":"8d82aba4cf43c1a0b4d58a4aad36644891699de2d10659b052dcf29309f403d5"}
Oct 31 20:26:13.393662 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:13.393194    4645 scope.go:122] "RemoveContainer" containerID="162682d2600ff8c5e65b0f70b5549e46eeb8165ab9e6b78e1ead8723c6edf1e1"
Oct 31 20:26:13.431390 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:13.431365    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs"]
Oct 31 20:26:13.444491 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:13.444467    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-4554a17c-openshift-default-757d594ccc-gbdvs"]
Oct 31 20:26:13.616015 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:13.615990    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2cf6a6f-1268-4997-8728-e8ca29514c4f" path="/var/lib/kubelet/pods/f2cf6a6f-1268-4997-8728-e8ca29514c4f/volumes"
Oct 31 20:26:13.897257 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:13.897223    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-33c6d1-f5d545cfb-9clf6"]
Oct 31 20:26:13.897762 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:13.897744    4645 cpu_manager.go:414] "containerMap: removing container" podUID="f2cf6a6f-1268-4997-8728-e8ca29514c4f" containerName="istio-proxy"
Oct 31 20:26:13.897822 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:13.897765    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2cf6a6f-1268-4997-8728-e8ca29514c4f" containerName="istio-proxy"
Oct 31 20:26:13.897822 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:13.897799    4645 cpu_manager.go:414] "containerMap: removing container" podUID="f2cf6a6f-1268-4997-8728-e8ca29514c4f" containerName="istio-proxy"
Oct 31 20:26:13.897822 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:13.897807    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2cf6a6f-1268-4997-8728-e8ca29514c4f" containerName="istio-proxy"
Oct 31 20:26:13.897933 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:13.897896    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2cf6a6f-1268-4997-8728-e8ca29514c4f" containerName="istio-proxy"
Oct 31 20:26:13.902664 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:13.902639    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-33c6d1-f5d545cfb-9clf6"
Oct 31 20:26:13.932436 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:13.932406    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-33c6d1-f5d545cfb-9clf6"]
Oct 31 20:26:13.935970 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:13.935945    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 20:26:13.935970 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:13.935952    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 20:26:13.986106 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:13.986069    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-authcfg-http-33c6d1-serving-cert\""
Oct 31 20:26:13.986248 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:13.986119    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9dsr5\""
Oct 31 20:26:13.986248 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:13.986127    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 31 20:26:13.986950 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:13.986923    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:14.000386 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:14.000354    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwrhv\" (UniqueName: \"kubernetes.io/projected/6468c090-c0fd-49b2-b2f1-8830d600232e-kube-api-access-hwrhv\") pod \"maas-api-e2e-authcfg-http-33c6d1-f5d545cfb-9clf6\" (UID: \"6468c090-c0fd-49b2-b2f1-8830d600232e\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-33c6d1-f5d545cfb-9clf6"
Oct 31 20:26:14.000583 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:14.000561    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6468c090-c0fd-49b2-b2f1-8830d600232e-metrics-certs\") pod \"maas-api-e2e-authcfg-http-33c6d1-f5d545cfb-9clf6\" (UID: \"6468c090-c0fd-49b2-b2f1-8830d600232e\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-33c6d1-f5d545cfb-9clf6"
Oct 31 20:26:14.000644 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:14.000599    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6468c090-c0fd-49b2-b2f1-8830d600232e-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-33c6d1-f5d545cfb-9clf6\" (UID: \"6468c090-c0fd-49b2-b2f1-8830d600232e\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-33c6d1-f5d545cfb-9clf6"
Oct 31 20:26:14.000644 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:14.000623    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6468c090-c0fd-49b2-b2f1-8830d600232e-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-33c6d1-f5d545cfb-9clf6\" (UID: \"6468c090-c0fd-49b2-b2f1-8830d600232e\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-33c6d1-f5d545cfb-9clf6"
Oct 31 20:26:14.101448 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:14.101417    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6468c090-c0fd-49b2-b2f1-8830d600232e-metrics-certs\") pod \"maas-api-e2e-authcfg-http-33c6d1-f5d545cfb-9clf6\" (UID: \"6468c090-c0fd-49b2-b2f1-8830d600232e\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-33c6d1-f5d545cfb-9clf6"
Oct 31 20:26:14.101448 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:14.101450    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6468c090-c0fd-49b2-b2f1-8830d600232e-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-33c6d1-f5d545cfb-9clf6\" (UID: \"6468c090-c0fd-49b2-b2f1-8830d600232e\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-33c6d1-f5d545cfb-9clf6"
Oct 31 20:26:14.101690 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:14.101470    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6468c090-c0fd-49b2-b2f1-8830d600232e-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-33c6d1-f5d545cfb-9clf6\" (UID: \"6468c090-c0fd-49b2-b2f1-8830d600232e\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-33c6d1-f5d545cfb-9clf6"
Oct 31 20:26:14.101690 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:14.101497    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwrhv\" (UniqueName: \"kubernetes.io/projected/6468c090-c0fd-49b2-b2f1-8830d600232e-kube-api-access-hwrhv\") pod \"maas-api-e2e-authcfg-http-33c6d1-f5d545cfb-9clf6\" (UID: \"6468c090-c0fd-49b2-b2f1-8830d600232e\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-33c6d1-f5d545cfb-9clf6"
Oct 31 20:26:14.102184 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:14.102162    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6468c090-c0fd-49b2-b2f1-8830d600232e-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-33c6d1-f5d545cfb-9clf6\" (UID: \"6468c090-c0fd-49b2-b2f1-8830d600232e\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-33c6d1-f5d545cfb-9clf6"
Oct 31 20:26:14.103859 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:14.103836    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6468c090-c0fd-49b2-b2f1-8830d600232e-metrics-certs\") pod \"maas-api-e2e-authcfg-http-33c6d1-f5d545cfb-9clf6\" (UID: \"6468c090-c0fd-49b2-b2f1-8830d600232e\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-33c6d1-f5d545cfb-9clf6"
Oct 31 20:26:14.103955 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:14.103898    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6468c090-c0fd-49b2-b2f1-8830d600232e-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-33c6d1-f5d545cfb-9clf6\" (UID: \"6468c090-c0fd-49b2-b2f1-8830d600232e\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-33c6d1-f5d545cfb-9clf6"
Oct 31 20:26:14.110648 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:14.110625    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:14.114733 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:14.114706    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwrhv\" (UniqueName: \"kubernetes.io/projected/6468c090-c0fd-49b2-b2f1-8830d600232e-kube-api-access-hwrhv\") pod \"maas-api-e2e-authcfg-http-33c6d1-f5d545cfb-9clf6\" (UID: \"6468c090-c0fd-49b2-b2f1-8830d600232e\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-33c6d1-f5d545cfb-9clf6"
Oct 31 20:26:14.149414 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:14.149366    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-33c6d1-6c98645495-6jzkt"]
Oct 31 20:26:14.149792 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:14.149780    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2cf6a6f-1268-4997-8728-e8ca29514c4f" containerName="istio-proxy"
Oct 31 20:26:14.152936 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:14.152920    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-33c6d1-6c98645495-6jzkt"
Oct 31 20:26:14.171438 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:14.171399    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-33c6d1-6c98645495-6jzkt"]
Oct 31 20:26:14.202953 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:14.202927    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a0f0c77-5a60-4d79-9635-f02b57c66677-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-33c6d1-6c98645495-6jzkt\" (UID: \"2a0f0c77-5a60-4d79-9635-f02b57c66677\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-33c6d1-6c98645495-6jzkt"
Oct 31 20:26:14.203047 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:14.203028    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgl95\" (UniqueName: \"kubernetes.io/projected/2a0f0c77-5a60-4d79-9635-f02b57c66677-kube-api-access-hgl95\") pod \"payload-pre-processing-e2e-authcfg-http-33c6d1-6c98645495-6jzkt\" (UID: \"2a0f0c77-5a60-4d79-9635-f02b57c66677\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-33c6d1-6c98645495-6jzkt"
Oct 31 20:26:14.216806 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:14.216783    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-33c6d1-f5d545cfb-9clf6"
Oct 31 20:26:14.238384 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:14.238363    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-33c6d1-dockercfg-jg7pd\""
Oct 31 20:26:14.247803 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:14.247783    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-33c6d1\""
Oct 31 20:26:14.301489 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:14.299233    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:14.305512 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:14.304988    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgl95\" (UniqueName: \"kubernetes.io/projected/2a0f0c77-5a60-4d79-9635-f02b57c66677-kube-api-access-hgl95\") pod \"payload-pre-processing-e2e-authcfg-http-33c6d1-6c98645495-6jzkt\" (UID: \"2a0f0c77-5a60-4d79-9635-f02b57c66677\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-33c6d1-6c98645495-6jzkt"
Oct 31 20:26:14.305512 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:14.305111    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a0f0c77-5a60-4d79-9635-f02b57c66677-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-33c6d1-6c98645495-6jzkt\" (UID: \"2a0f0c77-5a60-4d79-9635-f02b57c66677\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-33c6d1-6c98645495-6jzkt"
Oct 31 20:26:14.307375 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:14.306203    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a0f0c77-5a60-4d79-9635-f02b57c66677-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-33c6d1-6c98645495-6jzkt\" (UID: \"2a0f0c77-5a60-4d79-9635-f02b57c66677\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-33c6d1-6c98645495-6jzkt"
Oct 31 20:26:14.321803 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:14.321777    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgl95\" (UniqueName: \"kubernetes.io/projected/2a0f0c77-5a60-4d79-9635-f02b57c66677-kube-api-access-hgl95\") pod \"payload-pre-processing-e2e-authcfg-http-33c6d1-6c98645495-6jzkt\" (UID: \"2a0f0c77-5a60-4d79-9635-f02b57c66677\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-33c6d1-6c98645495-6jzkt"
Oct 31 20:26:14.468384 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:14.468347    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-33c6d1-6c98645495-6jzkt"
Oct 31 20:26:14.512858 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:14.512827    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-33c6d1-f5d545cfb-9clf6"]
Oct 31 20:26:14.518625 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:26:14.518581    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6468c090_c0fd_49b2_b2f1_8830d600232e.slice/crio-e49f005acc44e9f89389863d5eeafd4135a005a926d4e8c384f7c40b9895bcbc WatchSource:0}: Error finding container e49f005acc44e9f89389863d5eeafd4135a005a926d4e8c384f7c40b9895bcbc: Status 404 returned error can't find the container with id e49f005acc44e9f89389863d5eeafd4135a005a926d4e8c384f7c40b9895bcbc
Oct 31 20:26:14.746704 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:14.746669    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-33c6d1-6c98645495-6jzkt"]
Oct 31 20:26:14.748851 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:26:14.748824    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a0f0c77_5a60_4d79_9635_f02b57c66677.slice/crio-8d30a8092d318996ea7142af88b13207a913168e3673ad921ca6415625b4aeba WatchSource:0}: Error finding container 8d30a8092d318996ea7142af88b13207a913168e3673ad921ca6415625b4aeba: Status 404 returned error can't find the container with id 8d30a8092d318996ea7142af88b13207a913168e3673ad921ca6415625b4aeba
Oct 31 20:26:14.969348 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:14.969292    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:15.405201 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:15.405106    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-33c6d1-6c98645495-6jzkt" event={"ID":"2a0f0c77-5a60-4d79-9635-f02b57c66677","Type":"ContainerStarted","Data":"48f033530091982310ecc50102cf239014335090494b92a028729dfb6b178a8b"}
Oct 31 20:26:15.405201 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:15.405175    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-33c6d1-6c98645495-6jzkt" event={"ID":"2a0f0c77-5a60-4d79-9635-f02b57c66677","Type":"ContainerStarted","Data":"8d30a8092d318996ea7142af88b13207a913168e3673ad921ca6415625b4aeba"}
Oct 31 20:26:15.405489 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:15.405309    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-33c6d1-6c98645495-6jzkt"
Oct 31 20:26:15.406736 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:15.406713    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-33c6d1-f5d545cfb-9clf6" event={"ID":"6468c090-c0fd-49b2-b2f1-8830d600232e","Type":"ContainerStarted","Data":"81ebd1780fc156b5bd4e6fd7524678d3e4fc17aa6bef37e4a2dd8692fa0ac149"}
Oct 31 20:26:15.406736 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:15.406748    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-33c6d1-f5d545cfb-9clf6" event={"ID":"6468c090-c0fd-49b2-b2f1-8830d600232e","Type":"ContainerStarted","Data":"e49f005acc44e9f89389863d5eeafd4135a005a926d4e8c384f7c40b9895bcbc"}
Oct 31 20:26:15.406921 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:15.406783    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-33c6d1-f5d545cfb-9clf6"
Oct 31 20:26:15.426137 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:15.426096    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-33c6d1-6c98645495-6jzkt" podStartSLOduration=1.426085561 podStartE2EDuration="1.426085561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:26:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:26:15.42486923 +0000 UTC m=+4005.998103359" watchObservedRunningTime="2026-10-31 20:26:15.426085561 +0000 UTC m=+4005.999319689"
Oct 31 20:26:15.457851 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:15.457801    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-33c6d1-f5d545cfb-9clf6" podStartSLOduration=2.01370175 podStartE2EDuration="2.457789394s" totalImagesPullingTime="444.087644ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:26:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:26:15.456114474 +0000 UTC m=+4006.029348603" watchObservedRunningTime="2026-10-31 20:26:15.457789394 +0000 UTC m=+4006.031023522"
Oct 31 20:26:15.776712 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:15.776670    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:16.690069 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:16.690028    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:18.182442 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:18.182405    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:18.875352 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:18.872823    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:19.382019 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:19.381983    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:19.706600 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:19.706558    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:20.429003 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:20.428962    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:20.554984 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:20.554955    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:21.342923 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:21.342888    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:21.416697 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:21.416666    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-33c6d1-f5d545cfb-9clf6"
Oct 31 20:26:21.543285 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:21.543237    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:22.414659 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:22.414620    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:23.236191 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:23.236151    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:24.234284 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:24.234246    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:24.969183 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:24.969143    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:25.668324 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:25.668291    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:26.381774 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:26.381741    4645 patch_prober.go:29] interesting pod/e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.82:15021/healthz/ready\": dial tcp 10.133.1.82:15021: connect: connection refused" start-of-body=
Oct 31 20:26:26.382166 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:26.381791    4645 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww" podUID="d6bcb2da-7676-4255-a33a-084bb67f1bdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.82:15021/healthz/ready\": dial tcp 10.133.1.82:15021: connect: connection refused"
Oct 31 20:26:26.414708 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:26.414686    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-33c6d1-6c98645495-6jzkt"
Oct 31 20:26:26.462219 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:26.462052    4645 generic.go:356] "Generic (PLEG): container finished" podID="d6bcb2da-7676-4255-a33a-084bb67f1bdf" containerID="eb730b35f6ec8107fcb8c2b029c3b54924a37029598fcf041b82fc85da8b9f54" exitCode=0
Oct 31 20:26:26.462219 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:26.462075    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:26.462219 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:26.462181    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww" event={"ID":"d6bcb2da-7676-4255-a33a-084bb67f1bdf","Type":"ContainerDied","Data":"eb730b35f6ec8107fcb8c2b029c3b54924a37029598fcf041b82fc85da8b9f54"}
Oct 31 20:26:26.462649 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:26.462628    4645 scope.go:122] "RemoveContainer" containerID="eb730b35f6ec8107fcb8c2b029c3b54924a37029598fcf041b82fc85da8b9f54"
Oct 31 20:26:26.464032 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:26.463996    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:26:26.464143 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:26.464090    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:26:26.464214 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:26.464141    4645 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 31 20:26:27.345270 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:27.345229    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:27.485013 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:27.484974    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww" event={"ID":"d6bcb2da-7676-4255-a33a-084bb67f1bdf","Type":"ContainerStarted","Data":"4d7d0607c4886326086adc0a6d5368b37a3d0682eec8f18f0429e93a6c94a820"}
Oct 31 20:26:27.808627 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:27.808595    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:28.138593 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:28.138508    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:28.415781 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:28.415743    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww"
Oct 31 20:26:28.415990 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:28.415914    4645 patch_prober.go:29] interesting pod/e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.82:15021/healthz/ready\": dial tcp 10.133.1.82:15021: connect: connection refused" start-of-body=
Oct 31 20:26:28.415990 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:28.415967    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww" podUID="d6bcb2da-7676-4255-a33a-084bb67f1bdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.82:15021/healthz/ready\": dial tcp 10.133.1.82:15021: connect: connection refused"
Oct 31 20:26:28.854189 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:28.854101    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:29.416055 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:29.416011    4645 patch_prober.go:29] interesting pod/e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.82:15021/healthz/ready\": dial tcp 10.133.1.82:15021: connect: connection refused" start-of-body=
Oct 31 20:26:29.416262 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:29.416091    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww" podUID="d6bcb2da-7676-4255-a33a-084bb67f1bdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.82:15021/healthz/ready\": dial tcp 10.133.1.82:15021: connect: connection refused"
Oct 31 20:26:29.880839 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:29.880746    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:30.416284 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:30.416237    4645 patch_prober.go:29] interesting pod/e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.82:15021/healthz/ready\": dial tcp 10.133.1.82:15021: connect: connection refused" start-of-body=
Oct 31 20:26:30.416496 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:30.416301    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww" podUID="d6bcb2da-7676-4255-a33a-084bb67f1bdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.82:15021/healthz/ready\": dial tcp 10.133.1.82:15021: connect: connection refused"
Oct 31 20:26:30.674401 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:30.672929    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:31.415527 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:31.415496    4645 patch_prober.go:29] interesting pod/e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.82:15021/healthz/ready\": dial tcp 10.133.1.82:15021: connect: connection refused" start-of-body=
Oct 31 20:26:31.415986 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:31.415548    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww" podUID="d6bcb2da-7676-4255-a33a-084bb67f1bdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.82:15021/healthz/ready\": dial tcp 10.133.1.82:15021: connect: connection refused"
Oct 31 20:26:31.525863 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:31.525821    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:32.416077 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:32.416042    4645 patch_prober.go:29] interesting pod/e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.82:15021/healthz/ready\": dial tcp 10.133.1.82:15021: connect: connection refused" start-of-body=
Oct 31 20:26:32.416537 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:32.416096    4645 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww" podUID="d6bcb2da-7676-4255-a33a-084bb67f1bdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.82:15021/healthz/ready\": dial tcp 10.133.1.82:15021: connect: connection refused"
Oct 31 20:26:32.498524 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:32.498487    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:33.260423 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:33.260348    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:33.420129 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:33.420098    4645 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww"
Oct 31 20:26:33.420656 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:33.420495    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww"
Oct 31 20:26:33.421095 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:33.421074    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww"
Oct 31 20:26:34.054440 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:34.054403    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:34.858962 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:34.858925    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:35.789617 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:35.789575    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:36.245691 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:26:36.245652    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 20:26:36.567527 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:36.567450    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:37.312830 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:37.312787    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:38.165601 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:38.165566    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:38.873210 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:38.873174    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:39.636472 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:39.636430    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:40.497696 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:40.497665    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:41.076362 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:41.076309    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:41.357462 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:41.357372    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:41.960610 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:41.960572    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:42.647042 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:42.646997    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:43.350558 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:43.350521    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:44.151038 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:44.151001    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:44.850516 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:44.850480    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:45.558908 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:45.558875    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:46.254676 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:46.254644    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:46.951375 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:46.951318    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:47.665054 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:47.665010    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:48.363436 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:48.363400    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:49.143617 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:49.143583    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:49.988382 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:49.988314    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:50.863055 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:50.863012    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:51.634315 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:51.634278    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:52.360369 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:52.360322    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:53.132567 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:53.132532    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:53.844436 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:53.844394    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:54.553991 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:54.553955    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:55.253609 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:55.253565    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:56.105743 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:56.105702    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:56.907093 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:56.907054    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:57.663980 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:57.663946    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:58.460846 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:58.460805    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:26:59.272852 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:26:59.272818    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:00.200976 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:00.200937    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:00.975140 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:00.975101    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:01.658263 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:01.658228    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:02.369037 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:02.368992    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:03.224096 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:03.224059    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:04.051671 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:04.051627    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:04.753228 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:04.753178    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:05.640007 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:05.639971    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:06.246417 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:27:06.246381    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 20:27:06.463574 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:06.463545    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:07.303998 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:07.303946    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:08.050282 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:08.050247    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:08.866245 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:08.866209    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:09.662191 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:09.662145    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:10.548253 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:10.548193    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:11.287826 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:11.287785    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:12.074390 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:12.074317    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:12.780399 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:12.780352    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:13.559505 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:13.559472    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:14.257668 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:14.257638    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:14.953273 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:14.953233    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:15.747324 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:15.747284    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:16.464506 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:16.464469    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:17.175508 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:17.175464    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:18.048433 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:18.048394    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:18.955638 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:18.955603    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:19.385553 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:19.385469    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:19.719206 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:19.719176    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:20.469073 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:20.469037    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:20.576946 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:20.576915    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:21.150215 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:21.150176    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:21.297287 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:21.297253    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:21.405821 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:21.405718    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:21.564987 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:21.564956    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:22.096650 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:22.096617    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:22.952988 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:22.952951    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:23.864561 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:23.864524    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:24.715719 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:24.715680    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:25.449822 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:25.449781    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:26.286811 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:26.286775    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:27.056996 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:27.056944    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:27.881626 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:27.881568    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:28.739368 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:28.739311    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:29.579949 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:29.579919    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:30.378221 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:30.378184    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:31.156167 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:31.156131    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:32.074528 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:32.074479    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:32.794611 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:32.794568    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:33.576220 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:33.576173    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:34.380037 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:34.379989    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:35.265871 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:35.265829    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:36.047276 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:36.047229    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:36.245293 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:27:36.245252    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 20:27:36.842709 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:36.842672    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:37.562394 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:37.562350    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:38.370930 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:38.370888    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:39.142506 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:39.142465    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:39.864740 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:39.864702    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:40.657891 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:40.657853    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:41.475818 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:41.475773    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:42.286625 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:42.286580    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:43.083306 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:43.083250    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:43.896705 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:43.896669    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:44.674511 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:44.674477    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:45.387348 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:45.387296    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:46.141928 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:46.141879    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:46.849370 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:46.849316    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:47.638596 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:47.638561    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:48.038057 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:48.038024    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-33c6d1-wqs92"]
Oct 31 20:27:48.041556 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:48.041531    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-33c6d1-wqs92"
Oct 31 20:27:48.045135 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:48.045108    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:27:48.045535 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:48.045519    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qms6j\""
Oct 31 20:27:48.057688 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:48.057660    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-33c6d1-wqs92"]
Oct 31 20:27:48.202066 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:48.202029    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/996f384a-e32e-4284-a69b-9a24d2b93337-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-33c6d1-wqs92\" (UID: \"996f384a-e32e-4284-a69b-9a24d2b93337\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-33c6d1-wqs92"
Oct 31 20:27:48.202217 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:48.202093    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/996f384a-e32e-4284-a69b-9a24d2b93337-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-33c6d1-wqs92\" (UID: \"996f384a-e32e-4284-a69b-9a24d2b93337\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-33c6d1-wqs92"
Oct 31 20:27:48.202217 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:48.202194    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zss6k\" (UniqueName: \"kubernetes.io/projected/996f384a-e32e-4284-a69b-9a24d2b93337-kube-api-access-zss6k\") pod \"maas-api-revoke-keys-e2e-authcfg-http-33c6d1-wqs92\" (UID: \"996f384a-e32e-4284-a69b-9a24d2b93337\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-33c6d1-wqs92"
Oct 31 20:27:48.302790 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:48.302728    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/996f384a-e32e-4284-a69b-9a24d2b93337-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-33c6d1-wqs92\" (UID: \"996f384a-e32e-4284-a69b-9a24d2b93337\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-33c6d1-wqs92"
Oct 31 20:27:48.302790 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:48.302773    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zss6k\" (UniqueName: \"kubernetes.io/projected/996f384a-e32e-4284-a69b-9a24d2b93337-kube-api-access-zss6k\") pod \"maas-api-revoke-keys-e2e-authcfg-http-33c6d1-wqs92\" (UID: \"996f384a-e32e-4284-a69b-9a24d2b93337\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-33c6d1-wqs92"
Oct 31 20:27:48.302990 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:48.302831    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/996f384a-e32e-4284-a69b-9a24d2b93337-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-33c6d1-wqs92\" (UID: \"996f384a-e32e-4284-a69b-9a24d2b93337\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-33c6d1-wqs92"
Oct 31 20:27:48.303474 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:48.303449    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/996f384a-e32e-4284-a69b-9a24d2b93337-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-33c6d1-wqs92\" (UID: \"996f384a-e32e-4284-a69b-9a24d2b93337\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-33c6d1-wqs92"
Oct 31 20:27:48.305294 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:48.305270    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/996f384a-e32e-4284-a69b-9a24d2b93337-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-33c6d1-wqs92\" (UID: \"996f384a-e32e-4284-a69b-9a24d2b93337\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-33c6d1-wqs92"
Oct 31 20:27:48.314909 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:48.314880    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zss6k\" (UniqueName: \"kubernetes.io/projected/996f384a-e32e-4284-a69b-9a24d2b93337-kube-api-access-zss6k\") pod \"maas-api-revoke-keys-e2e-authcfg-http-33c6d1-wqs92\" (UID: \"996f384a-e32e-4284-a69b-9a24d2b93337\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-33c6d1-wqs92"
Oct 31 20:27:48.365869 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:48.365844    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-33c6d1-wqs92"
Oct 31 20:27:48.524274 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:48.524233    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:48.641685 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:48.641660    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-33c6d1-wqs92"]
Oct 31 20:27:48.644350 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:27:48.644301    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod996f384a_e32e_4284_a69b_9a24d2b93337.slice/crio-0157b8316cf18972ae99692b04787b74c25cf7f63cb7a96c92092e98e5c29061 WatchSource:0}: Error finding container 0157b8316cf18972ae99692b04787b74c25cf7f63cb7a96c92092e98e5c29061: Status 404 returned error can't find the container with id 0157b8316cf18972ae99692b04787b74c25cf7f63cb7a96c92092e98e5c29061
Oct 31 20:27:48.847729 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:48.847701    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-33c6d1-wqs92" event={"ID":"996f384a-e32e-4284-a69b-9a24d2b93337","Type":"ContainerStarted","Data":"0157b8316cf18972ae99692b04787b74c25cf7f63cb7a96c92092e98e5c29061"}
Oct 31 20:27:49.370548 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:49.370487    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:49.853537 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:49.853507    4645 generic.go:356] "Generic (PLEG): container finished" podID="996f384a-e32e-4284-a69b-9a24d2b93337" containerID="6baa06343e59d9087d08c035bacdd7a61bc5f53afb003ac5fd4142d0a54ae4f9" exitCode=0
Oct 31 20:27:49.853678 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:49.853590    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-33c6d1-wqs92" event={"ID":"996f384a-e32e-4284-a69b-9a24d2b93337","Type":"ContainerDied","Data":"6baa06343e59d9087d08c035bacdd7a61bc5f53afb003ac5fd4142d0a54ae4f9"}
Oct 31 20:27:50.340786 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:50.340755    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:50.993106 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:50.993081    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-33c6d1-wqs92"
Oct 31 20:27:51.129669 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:51.129580    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/996f384a-e32e-4284-a69b-9a24d2b93337-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/996f384a-e32e-4284-a69b-9a24d2b93337-maas-api-xks-ca\") pod \"996f384a-e32e-4284-a69b-9a24d2b93337\" (UID: \"996f384a-e32e-4284-a69b-9a24d2b93337\") "
Oct 31 20:27:51.129836 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:51.129694    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/996f384a-e32e-4284-a69b-9a24d2b93337-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/996f384a-e32e-4284-a69b-9a24d2b93337-maas-api-openshift-ca\") pod \"996f384a-e32e-4284-a69b-9a24d2b93337\" (UID: \"996f384a-e32e-4284-a69b-9a24d2b93337\") "
Oct 31 20:27:51.129836 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:51.129736    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/996f384a-e32e-4284-a69b-9a24d2b93337-kube-api-access-zss6k\" (UniqueName: \"kubernetes.io/projected/996f384a-e32e-4284-a69b-9a24d2b93337-kube-api-access-zss6k\") pod \"996f384a-e32e-4284-a69b-9a24d2b93337\" (UID: \"996f384a-e32e-4284-a69b-9a24d2b93337\") "
Oct 31 20:27:51.130150 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:51.130119    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/996f384a-e32e-4284-a69b-9a24d2b93337-maas-api-openshift-ca" pod "996f384a-e32e-4284-a69b-9a24d2b93337" (UID: "996f384a-e32e-4284-a69b-9a24d2b93337"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:27:51.131763 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:51.131732    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/996f384a-e32e-4284-a69b-9a24d2b93337-maas-api-xks-ca" pod "996f384a-e32e-4284-a69b-9a24d2b93337" (UID: "996f384a-e32e-4284-a69b-9a24d2b93337"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:27:51.131890 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:51.131869    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/996f384a-e32e-4284-a69b-9a24d2b93337-kube-api-access-zss6k" pod "996f384a-e32e-4284-a69b-9a24d2b93337" (UID: "996f384a-e32e-4284-a69b-9a24d2b93337"). InnerVolumeSpecName "kube-api-access-zss6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:27:51.155938 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:51.155912    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:51.230442 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:51.230415    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/996f384a-e32e-4284-a69b-9a24d2b93337-maas-api-openshift-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:27:51.230442 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:51.230441    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zss6k\" (UniqueName: \"kubernetes.io/projected/996f384a-e32e-4284-a69b-9a24d2b93337-kube-api-access-zss6k\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:27:51.230607 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:51.230452    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/996f384a-e32e-4284-a69b-9a24d2b93337-maas-api-xks-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:27:51.848816 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:51.848792    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:51.863520 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:51.863494    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-33c6d1-wqs92" event={"ID":"996f384a-e32e-4284-a69b-9a24d2b93337","Type":"ContainerDied","Data":"0157b8316cf18972ae99692b04787b74c25cf7f63cb7a96c92092e98e5c29061"}
Oct 31 20:27:51.863520 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:51.863511    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-33c6d1-wqs92"
Oct 31 20:27:51.863681 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:51.863538    4645 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0157b8316cf18972ae99692b04787b74c25cf7f63cb7a96c92092e98e5c29061"
Oct 31 20:27:52.679645 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:52.679606    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:53.458050 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:53.458013    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:54.155007 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:54.154974    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:54.843164 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:54.843122    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:55.552022 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:55.551980    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:56.363596 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:56.363563    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:57.141310 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:57.141272    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:57.951752 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:57.951713    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:58.214585 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:58.214501    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-33c6d1-6c98645495-6jzkt"]
Oct 31 20:27:58.292545 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:58.292510    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:58.417835 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:58.417803    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:27:58.860709 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:27:58.860674    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:28:03.215341 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:03.215254    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-33c6d1-6c98645495-6jzkt" podUID="2a0f0c77-5a60-4d79-9635-f02b57c66677" containerName="payload-pre-processing" containerID="cri-o://48f033530091982310ecc50102cf239014335090494b92a028729dfb6b178a8b" gracePeriod=25
Oct 31 20:28:03.464683 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:03.464654    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-33c6d1-6c98645495-6jzkt"
Oct 31 20:28:03.529221 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:03.529159    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a0f0c77-5a60-4d79-9635-f02b57c66677-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a0f0c77-5a60-4d79-9635-f02b57c66677-plugins-config-volume\") pod \"2a0f0c77-5a60-4d79-9635-f02b57c66677\" (UID: \"2a0f0c77-5a60-4d79-9635-f02b57c66677\") "
Oct 31 20:28:03.529221 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:03.529197    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a0f0c77-5a60-4d79-9635-f02b57c66677-kube-api-access-hgl95\" (UniqueName: \"kubernetes.io/projected/2a0f0c77-5a60-4d79-9635-f02b57c66677-kube-api-access-hgl95\") pod \"2a0f0c77-5a60-4d79-9635-f02b57c66677\" (UID: \"2a0f0c77-5a60-4d79-9635-f02b57c66677\") "
Oct 31 20:28:03.529537 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:03.529511    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a0f0c77-5a60-4d79-9635-f02b57c66677-plugins-config-volume" pod "2a0f0c77-5a60-4d79-9635-f02b57c66677" (UID: "2a0f0c77-5a60-4d79-9635-f02b57c66677"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:28:03.531182 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:03.531160    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a0f0c77-5a60-4d79-9635-f02b57c66677-kube-api-access-hgl95" pod "2a0f0c77-5a60-4d79-9635-f02b57c66677" (UID: "2a0f0c77-5a60-4d79-9635-f02b57c66677"). InnerVolumeSpecName "kube-api-access-hgl95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:28:03.630598 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:03.630569    4645 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a0f0c77-5a60-4d79-9635-f02b57c66677-plugins-config-volume\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:28:03.630598 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:03.630597    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgl95\" (UniqueName: \"kubernetes.io/projected/2a0f0c77-5a60-4d79-9635-f02b57c66677-kube-api-access-hgl95\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:28:03.922518 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:03.922484    4645 generic.go:356] "Generic (PLEG): container finished" podID="2a0f0c77-5a60-4d79-9635-f02b57c66677" containerID="48f033530091982310ecc50102cf239014335090494b92a028729dfb6b178a8b" exitCode=0
Oct 31 20:28:03.922653 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:03.922527    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-33c6d1-6c98645495-6jzkt"
Oct 31 20:28:03.922653 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:03.922564    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-33c6d1-6c98645495-6jzkt" event={"ID":"2a0f0c77-5a60-4d79-9635-f02b57c66677","Type":"ContainerDied","Data":"48f033530091982310ecc50102cf239014335090494b92a028729dfb6b178a8b"}
Oct 31 20:28:03.922653 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:03.922608    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-33c6d1-6c98645495-6jzkt" event={"ID":"2a0f0c77-5a60-4d79-9635-f02b57c66677","Type":"ContainerDied","Data":"8d30a8092d318996ea7142af88b13207a913168e3673ad921ca6415625b4aeba"}
Oct 31 20:28:03.922653 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:03.922630    4645 scope.go:122] "RemoveContainer" containerID="48f033530091982310ecc50102cf239014335090494b92a028729dfb6b178a8b"
Oct 31 20:28:03.931116 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:03.931095    4645 scope.go:122] "RemoveContainer" containerID="48f033530091982310ecc50102cf239014335090494b92a028729dfb6b178a8b"
Oct 31 20:28:03.931411 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:28:03.931390    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48f033530091982310ecc50102cf239014335090494b92a028729dfb6b178a8b\": container with ID starting with 48f033530091982310ecc50102cf239014335090494b92a028729dfb6b178a8b not found: ID does not exist" containerID="48f033530091982310ecc50102cf239014335090494b92a028729dfb6b178a8b"
Oct 31 20:28:03.931488 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:03.931419    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48f033530091982310ecc50102cf239014335090494b92a028729dfb6b178a8b"} err="failed to get container status \"48f033530091982310ecc50102cf239014335090494b92a028729dfb6b178a8b\": rpc error: code = NotFound desc = could not find container \"48f033530091982310ecc50102cf239014335090494b92a028729dfb6b178a8b\": container with ID starting with 48f033530091982310ecc50102cf239014335090494b92a028729dfb6b178a8b not found: ID does not exist"
Oct 31 20:28:03.941114 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:03.941093    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-33c6d1-6c98645495-6jzkt"]
Oct 31 20:28:03.943046 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:03.943025    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-33c6d1-6c98645495-6jzkt"]
Oct 31 20:28:04.528053 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:04.528019    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-33c6d1-f5d545cfb-9clf6"]
Oct 31 20:28:04.528583 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:04.528276    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-33c6d1-f5d545cfb-9clf6" podUID="6468c090-c0fd-49b2-b2f1-8830d600232e" containerName="maas-api" containerID="cri-o://81ebd1780fc156b5bd4e6fd7524678d3e4fc17aa6bef37e4a2dd8692fa0ac149" gracePeriod=30
Oct 31 20:28:04.583514 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:04.583485    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:28:04.757679 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:04.757637    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:28:04.788840 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:04.788784    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-33c6d1-f5d545cfb-9clf6"
Oct 31 20:28:04.810987 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:04.810956    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-33c6d1-wqs92"]
Oct 31 20:28:04.820984 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:04.820950    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-33c6d1-wqs92"]
Oct 31 20:28:04.844343 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:04.844296    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6468c090-c0fd-49b2-b2f1-8830d600232e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6468c090-c0fd-49b2-b2f1-8830d600232e-openshift-service-ca\") pod \"6468c090-c0fd-49b2-b2f1-8830d600232e\" (UID: \"6468c090-c0fd-49b2-b2f1-8830d600232e\") "
Oct 31 20:28:04.844545 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:04.844391    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6468c090-c0fd-49b2-b2f1-8830d600232e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6468c090-c0fd-49b2-b2f1-8830d600232e-metrics-certs\") pod \"6468c090-c0fd-49b2-b2f1-8830d600232e\" (UID: \"6468c090-c0fd-49b2-b2f1-8830d600232e\") "
Oct 31 20:28:04.844545 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:04.844443    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6468c090-c0fd-49b2-b2f1-8830d600232e-kube-api-access-hwrhv\" (UniqueName: \"kubernetes.io/projected/6468c090-c0fd-49b2-b2f1-8830d600232e-kube-api-access-hwrhv\") pod \"6468c090-c0fd-49b2-b2f1-8830d600232e\" (UID: \"6468c090-c0fd-49b2-b2f1-8830d600232e\") "
Oct 31 20:28:04.844545 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:04.844518    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6468c090-c0fd-49b2-b2f1-8830d600232e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6468c090-c0fd-49b2-b2f1-8830d600232e-maas-api-tls\") pod \"6468c090-c0fd-49b2-b2f1-8830d600232e\" (UID: \"6468c090-c0fd-49b2-b2f1-8830d600232e\") "
Oct 31 20:28:04.844763 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:04.844735    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6468c090-c0fd-49b2-b2f1-8830d600232e-openshift-service-ca" pod "6468c090-c0fd-49b2-b2f1-8830d600232e" (UID: "6468c090-c0fd-49b2-b2f1-8830d600232e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:28:04.844905 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:04.844885    4645 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6468c090-c0fd-49b2-b2f1-8830d600232e-openshift-service-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:28:04.846648 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:04.846616    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6468c090-c0fd-49b2-b2f1-8830d600232e-kube-api-access-hwrhv" pod "6468c090-c0fd-49b2-b2f1-8830d600232e" (UID: "6468c090-c0fd-49b2-b2f1-8830d600232e"). InnerVolumeSpecName "kube-api-access-hwrhv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:28:04.846790 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:04.846643    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6468c090-c0fd-49b2-b2f1-8830d600232e-metrics-certs" pod "6468c090-c0fd-49b2-b2f1-8830d600232e" (UID: "6468c090-c0fd-49b2-b2f1-8830d600232e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:28:04.846790 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:04.846776    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6468c090-c0fd-49b2-b2f1-8830d600232e-maas-api-tls" pod "6468c090-c0fd-49b2-b2f1-8830d600232e" (UID: "6468c090-c0fd-49b2-b2f1-8830d600232e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:28:04.869410 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:04.869388    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:28:04.929858 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:04.929821    4645 generic.go:356] "Generic (PLEG): container finished" podID="6468c090-c0fd-49b2-b2f1-8830d600232e" containerID="81ebd1780fc156b5bd4e6fd7524678d3e4fc17aa6bef37e4a2dd8692fa0ac149" exitCode=0
Oct 31 20:28:04.929988 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:04.929909    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-33c6d1-f5d545cfb-9clf6" event={"ID":"6468c090-c0fd-49b2-b2f1-8830d600232e","Type":"ContainerDied","Data":"81ebd1780fc156b5bd4e6fd7524678d3e4fc17aa6bef37e4a2dd8692fa0ac149"}
Oct 31 20:28:04.930058 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:04.930023    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-33c6d1-f5d545cfb-9clf6" event={"ID":"6468c090-c0fd-49b2-b2f1-8830d600232e","Type":"ContainerDied","Data":"e49f005acc44e9f89389863d5eeafd4135a005a926d4e8c384f7c40b9895bcbc"}
Oct 31 20:28:04.930117 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:04.930061    4645 scope.go:122] "RemoveContainer" containerID="81ebd1780fc156b5bd4e6fd7524678d3e4fc17aa6bef37e4a2dd8692fa0ac149"
Oct 31 20:28:04.930235 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:04.930215    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-33c6d1-f5d545cfb-9clf6"
Oct 31 20:28:04.940071 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:04.940050    4645 scope.go:122] "RemoveContainer" containerID="81ebd1780fc156b5bd4e6fd7524678d3e4fc17aa6bef37e4a2dd8692fa0ac149"
Oct 31 20:28:04.940343 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:28:04.940309    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81ebd1780fc156b5bd4e6fd7524678d3e4fc17aa6bef37e4a2dd8692fa0ac149\": container with ID starting with 81ebd1780fc156b5bd4e6fd7524678d3e4fc17aa6bef37e4a2dd8692fa0ac149 not found: ID does not exist" containerID="81ebd1780fc156b5bd4e6fd7524678d3e4fc17aa6bef37e4a2dd8692fa0ac149"
Oct 31 20:28:04.940428 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:04.940370    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81ebd1780fc156b5bd4e6fd7524678d3e4fc17aa6bef37e4a2dd8692fa0ac149"} err="failed to get container status \"81ebd1780fc156b5bd4e6fd7524678d3e4fc17aa6bef37e4a2dd8692fa0ac149\": rpc error: code = NotFound desc = could not find container \"81ebd1780fc156b5bd4e6fd7524678d3e4fc17aa6bef37e4a2dd8692fa0ac149\": container with ID starting with 81ebd1780fc156b5bd4e6fd7524678d3e4fc17aa6bef37e4a2dd8692fa0ac149 not found: ID does not exist"
Oct 31 20:28:04.945968 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:04.945947    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwrhv\" (UniqueName: \"kubernetes.io/projected/6468c090-c0fd-49b2-b2f1-8830d600232e-kube-api-access-hwrhv\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:28:04.945968 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:04.945969    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6468c090-c0fd-49b2-b2f1-8830d600232e-maas-api-tls\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:28:04.946101 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:04.945980    4645 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6468c090-c0fd-49b2-b2f1-8830d600232e-metrics-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:28:04.954360 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:04.954316    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-33c6d1-f5d545cfb-9clf6"]
Oct 31 20:28:04.958250 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:04.958230    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-33c6d1-f5d545cfb-9clf6"]
Oct 31 20:28:05.475818 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:05.475789    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:28:05.616341 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:05.616289    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a0f0c77-5a60-4d79-9635-f02b57c66677" path="/var/lib/kubelet/pods/2a0f0c77-5a60-4d79-9635-f02b57c66677/volumes"
Oct 31 20:28:05.616915 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:05.616888    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6468c090-c0fd-49b2-b2f1-8830d600232e" path="/var/lib/kubelet/pods/6468c090-c0fd-49b2-b2f1-8830d600232e/volumes"
Oct 31 20:28:05.617378 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:05.617356    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="996f384a-e32e-4284-a69b-9a24d2b93337" path="/var/lib/kubelet/pods/996f384a-e32e-4284-a69b-9a24d2b93337/volumes"
Oct 31 20:28:06.245406 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:28:06.245372    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 20:28:11.896379 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:11.896321    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:28:12.026428 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:12.026385    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:28:12.249936 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:12.249900    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww"]
Oct 31 20:28:12.250158 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:12.250136    4645 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww" podUID="d6bcb2da-7676-4255-a33a-084bb67f1bdf" containerName="istio-proxy" containerID="cri-o://4d7d0607c4886326086adc0a6d5368b37a3d0682eec8f18f0429e93a6c94a820" gracePeriod=30
Oct 31 20:28:12.289265 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:12.289236    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:28:17.746273 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:17.746249    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww"
Oct 31 20:28:17.783914 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:17.783888    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6bcb2da-7676-4255-a33a-084bb67f1bdf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6bcb2da-7676-4255-a33a-084bb67f1bdf-istio-envoy\") pod \"d6bcb2da-7676-4255-a33a-084bb67f1bdf\" (UID: \"d6bcb2da-7676-4255-a33a-084bb67f1bdf\") "
Oct 31 20:28:17.784050 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:17.783925    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6bcb2da-7676-4255-a33a-084bb67f1bdf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6bcb2da-7676-4255-a33a-084bb67f1bdf-istio-data\") pod \"d6bcb2da-7676-4255-a33a-084bb67f1bdf\" (UID: \"d6bcb2da-7676-4255-a33a-084bb67f1bdf\") "
Oct 31 20:28:17.784050 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:17.783963    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d6bcb2da-7676-4255-a33a-084bb67f1bdf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6bcb2da-7676-4255-a33a-084bb67f1bdf-istio-podinfo\") pod \"d6bcb2da-7676-4255-a33a-084bb67f1bdf\" (UID: \"d6bcb2da-7676-4255-a33a-084bb67f1bdf\") "
Oct 31 20:28:17.784050 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:17.783995    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6bcb2da-7676-4255-a33a-084bb67f1bdf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6bcb2da-7676-4255-a33a-084bb67f1bdf-credential-socket\") pod \"d6bcb2da-7676-4255-a33a-084bb67f1bdf\" (UID: \"d6bcb2da-7676-4255-a33a-084bb67f1bdf\") "
Oct 31 20:28:17.784050 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:17.784031    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6bcb2da-7676-4255-a33a-084bb67f1bdf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6bcb2da-7676-4255-a33a-084bb67f1bdf-workload-socket\") pod \"d6bcb2da-7676-4255-a33a-084bb67f1bdf\" (UID: \"d6bcb2da-7676-4255-a33a-084bb67f1bdf\") "
Oct 31 20:28:17.784296 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:17.784075    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6bcb2da-7676-4255-a33a-084bb67f1bdf-istio-token\" (UniqueName: \"kubernetes.io/projected/d6bcb2da-7676-4255-a33a-084bb67f1bdf-istio-token\") pod \"d6bcb2da-7676-4255-a33a-084bb67f1bdf\" (UID: \"d6bcb2da-7676-4255-a33a-084bb67f1bdf\") "
Oct 31 20:28:17.784296 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:17.784148    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6bcb2da-7676-4255-a33a-084bb67f1bdf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6bcb2da-7676-4255-a33a-084bb67f1bdf-workload-certs\") pod \"d6bcb2da-7676-4255-a33a-084bb67f1bdf\" (UID: \"d6bcb2da-7676-4255-a33a-084bb67f1bdf\") "
Oct 31 20:28:17.784296 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:17.784193    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6bcb2da-7676-4255-a33a-084bb67f1bdf-kube-api-access-2f468\" (UniqueName: \"kubernetes.io/projected/d6bcb2da-7676-4255-a33a-084bb67f1bdf-kube-api-access-2f468\") pod \"d6bcb2da-7676-4255-a33a-084bb67f1bdf\" (UID: \"d6bcb2da-7676-4255-a33a-084bb67f1bdf\") "
Oct 31 20:28:17.784296 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:17.784225    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6bcb2da-7676-4255-a33a-084bb67f1bdf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6bcb2da-7676-4255-a33a-084bb67f1bdf-istiod-ca-cert\") pod \"d6bcb2da-7676-4255-a33a-084bb67f1bdf\" (UID: \"d6bcb2da-7676-4255-a33a-084bb67f1bdf\") "
Oct 31 20:28:17.784296 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:17.784269    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6bcb2da-7676-4255-a33a-084bb67f1bdf-istio-data" pod "d6bcb2da-7676-4255-a33a-084bb67f1bdf" (UID: "d6bcb2da-7676-4255-a33a-084bb67f1bdf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:28:17.784592 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:17.784396    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6bcb2da-7676-4255-a33a-084bb67f1bdf-credential-socket" pod "d6bcb2da-7676-4255-a33a-084bb67f1bdf" (UID: "d6bcb2da-7676-4255-a33a-084bb67f1bdf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:28:17.784592 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:17.784499    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6bcb2da-7676-4255-a33a-084bb67f1bdf-workload-certs" pod "d6bcb2da-7676-4255-a33a-084bb67f1bdf" (UID: "d6bcb2da-7676-4255-a33a-084bb67f1bdf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:28:17.784706 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:17.784643    4645 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6bcb2da-7676-4255-a33a-084bb67f1bdf-workload-certs\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:28:17.784706 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:17.784665    4645 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6bcb2da-7676-4255-a33a-084bb67f1bdf-istio-data\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:28:17.784706 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:17.784681    4645 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6bcb2da-7676-4255-a33a-084bb67f1bdf-credential-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:28:17.784853 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:17.784703    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6bcb2da-7676-4255-a33a-084bb67f1bdf-istiod-ca-cert" pod "d6bcb2da-7676-4255-a33a-084bb67f1bdf" (UID: "d6bcb2da-7676-4255-a33a-084bb67f1bdf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:28:17.784853 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:17.784762    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6bcb2da-7676-4255-a33a-084bb67f1bdf-workload-socket" pod "d6bcb2da-7676-4255-a33a-084bb67f1bdf" (UID: "d6bcb2da-7676-4255-a33a-084bb67f1bdf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:28:17.786243 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:17.786220    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6bcb2da-7676-4255-a33a-084bb67f1bdf-istio-envoy" pod "d6bcb2da-7676-4255-a33a-084bb67f1bdf" (UID: "d6bcb2da-7676-4255-a33a-084bb67f1bdf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 20:28:17.786523 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:17.786504    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6bcb2da-7676-4255-a33a-084bb67f1bdf-kube-api-access-2f468" pod "d6bcb2da-7676-4255-a33a-084bb67f1bdf" (UID: "d6bcb2da-7676-4255-a33a-084bb67f1bdf"). InnerVolumeSpecName "kube-api-access-2f468". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:28:17.786721 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:17.786695    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d6bcb2da-7676-4255-a33a-084bb67f1bdf-istio-podinfo" pod "d6bcb2da-7676-4255-a33a-084bb67f1bdf" (UID: "d6bcb2da-7676-4255-a33a-084bb67f1bdf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 20:28:17.786768 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:17.786704    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6bcb2da-7676-4255-a33a-084bb67f1bdf-istio-token" pod "d6bcb2da-7676-4255-a33a-084bb67f1bdf" (UID: "d6bcb2da-7676-4255-a33a-084bb67f1bdf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:28:17.885179 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:17.885110    4645 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6bcb2da-7676-4255-a33a-084bb67f1bdf-workload-socket\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:28:17.885179 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:17.885139    4645 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6bcb2da-7676-4255-a33a-084bb67f1bdf-istio-token\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:28:17.885179 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:17.885151    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2f468\" (UniqueName: \"kubernetes.io/projected/d6bcb2da-7676-4255-a33a-084bb67f1bdf-kube-api-access-2f468\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:28:17.885179 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:17.885161    4645 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6bcb2da-7676-4255-a33a-084bb67f1bdf-istiod-ca-cert\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:28:17.885179 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:17.885170    4645 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6bcb2da-7676-4255-a33a-084bb67f1bdf-istio-envoy\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:28:17.885179 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:17.885178    4645 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6bcb2da-7676-4255-a33a-084bb67f1bdf-istio-podinfo\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:28:18.009945 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:18.009911    4645 generic.go:356] "Generic (PLEG): container finished" podID="d6bcb2da-7676-4255-a33a-084bb67f1bdf" containerID="4d7d0607c4886326086adc0a6d5368b37a3d0682eec8f18f0429e93a6c94a820" exitCode=0
Oct 31 20:28:18.010141 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:18.009978    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww"
Oct 31 20:28:18.010141 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:18.009996    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww" event={"ID":"d6bcb2da-7676-4255-a33a-084bb67f1bdf","Type":"ContainerDied","Data":"4d7d0607c4886326086adc0a6d5368b37a3d0682eec8f18f0429e93a6c94a820"}
Oct 31 20:28:18.010141 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:18.010068    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww" event={"ID":"d6bcb2da-7676-4255-a33a-084bb67f1bdf","Type":"ContainerDied","Data":"048e8242dca79f48f2c02114193700328be1946f81471b0b7cf4d3ecbe429eeb"}
Oct 31 20:28:18.010141 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:18.010099    4645 scope.go:122] "RemoveContainer" containerID="4d7d0607c4886326086adc0a6d5368b37a3d0682eec8f18f0429e93a6c94a820"
Oct 31 20:28:18.019476 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:18.019459    4645 scope.go:122] "RemoveContainer" containerID="eb730b35f6ec8107fcb8c2b029c3b54924a37029598fcf041b82fc85da8b9f54"
Oct 31 20:28:18.029062 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:18.029043    4645 scope.go:122] "RemoveContainer" containerID="4d7d0607c4886326086adc0a6d5368b37a3d0682eec8f18f0429e93a6c94a820"
Oct 31 20:28:18.029390 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:28:18.029370    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d7d0607c4886326086adc0a6d5368b37a3d0682eec8f18f0429e93a6c94a820\": container with ID starting with 4d7d0607c4886326086adc0a6d5368b37a3d0682eec8f18f0429e93a6c94a820 not found: ID does not exist" containerID="4d7d0607c4886326086adc0a6d5368b37a3d0682eec8f18f0429e93a6c94a820"
Oct 31 20:28:18.029471 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:18.029405    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d7d0607c4886326086adc0a6d5368b37a3d0682eec8f18f0429e93a6c94a820"} err="failed to get container status \"4d7d0607c4886326086adc0a6d5368b37a3d0682eec8f18f0429e93a6c94a820\": rpc error: code = NotFound desc = could not find container \"4d7d0607c4886326086adc0a6d5368b37a3d0682eec8f18f0429e93a6c94a820\": container with ID starting with 4d7d0607c4886326086adc0a6d5368b37a3d0682eec8f18f0429e93a6c94a820 not found: ID does not exist"
Oct 31 20:28:18.029471 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:18.029429    4645 scope.go:122] "RemoveContainer" containerID="eb730b35f6ec8107fcb8c2b029c3b54924a37029598fcf041b82fc85da8b9f54"
Oct 31 20:28:18.029767 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:28:18.029740    4645 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb730b35f6ec8107fcb8c2b029c3b54924a37029598fcf041b82fc85da8b9f54\": container with ID starting with eb730b35f6ec8107fcb8c2b029c3b54924a37029598fcf041b82fc85da8b9f54 not found: ID does not exist" containerID="eb730b35f6ec8107fcb8c2b029c3b54924a37029598fcf041b82fc85da8b9f54"
Oct 31 20:28:18.029865 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:18.029776    4645 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb730b35f6ec8107fcb8c2b029c3b54924a37029598fcf041b82fc85da8b9f54"} err="failed to get container status \"eb730b35f6ec8107fcb8c2b029c3b54924a37029598fcf041b82fc85da8b9f54\": rpc error: code = NotFound desc = could not find container \"eb730b35f6ec8107fcb8c2b029c3b54924a37029598fcf041b82fc85da8b9f54\": container with ID starting with eb730b35f6ec8107fcb8c2b029c3b54924a37029598fcf041b82fc85da8b9f54 not found: ID does not exist"
Oct 31 20:28:18.039856 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:18.039832    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww"]
Oct 31 20:28:18.042919 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:18.042894    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-authcfg-http-33c6d1-openshift-default-5547cf548f-btpww"]
Oct 31 20:28:19.422749 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:19.422710    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:28:19.616393 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:19.616354    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6bcb2da-7676-4255-a33a-084bb67f1bdf" path="/var/lib/kubelet/pods/d6bcb2da-7676-4255-a33a-084bb67f1bdf/volumes"
Oct 31 20:28:20.505362 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:20.505300    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:28:21.696352 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:21.682694    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:28:24.215025 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:24.214986    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-sj64m"]
Oct 31 20:28:24.215432 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:24.215351    4645 cpu_manager.go:414] "containerMap: removing container" podUID="d6bcb2da-7676-4255-a33a-084bb67f1bdf" containerName="istio-proxy"
Oct 31 20:28:24.215432 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:24.215364    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6bcb2da-7676-4255-a33a-084bb67f1bdf" containerName="istio-proxy"
Oct 31 20:28:24.215432 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:24.215388    4645 cpu_manager.go:414] "containerMap: removing container" podUID="996f384a-e32e-4284-a69b-9a24d2b93337" containerName="revoke-keys"
Oct 31 20:28:24.215432 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:24.215394    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="996f384a-e32e-4284-a69b-9a24d2b93337" containerName="revoke-keys"
Oct 31 20:28:24.215432 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:24.215407    4645 cpu_manager.go:414] "containerMap: removing container" podUID="2a0f0c77-5a60-4d79-9635-f02b57c66677" containerName="payload-pre-processing"
Oct 31 20:28:24.215432 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:24.215411    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a0f0c77-5a60-4d79-9635-f02b57c66677" containerName="payload-pre-processing"
Oct 31 20:28:24.215432 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:24.215422    4645 cpu_manager.go:414] "containerMap: removing container" podUID="6468c090-c0fd-49b2-b2f1-8830d600232e" containerName="maas-api"
Oct 31 20:28:24.215432 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:24.215427    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6468c090-c0fd-49b2-b2f1-8830d600232e" containerName="maas-api"
Oct 31 20:28:24.215734 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:24.215482    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6bcb2da-7676-4255-a33a-084bb67f1bdf" containerName="istio-proxy"
Oct 31 20:28:24.215734 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:24.215490    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="996f384a-e32e-4284-a69b-9a24d2b93337" containerName="revoke-keys"
Oct 31 20:28:24.215734 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:24.215496    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a0f0c77-5a60-4d79-9635-f02b57c66677" containerName="payload-pre-processing"
Oct 31 20:28:24.215734 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:24.215504    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="6468c090-c0fd-49b2-b2f1-8830d600232e" containerName="maas-api"
Oct 31 20:28:24.220043 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:24.220021    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-sj64m"
Oct 31 20:28:24.224435 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:24.224415    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 20:28:24.227019 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:24.226993    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 20:28:24.227884 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:24.227854    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:28:24.228041 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:24.228023    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qms6j\""
Oct 31 20:28:24.231271 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:24.231248    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-sj64m"]
Oct 31 20:28:24.281263 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:24.281234    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dffd37e-1fe7-4a81-bb92-b6cdcd2999cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-sj64m\" (UID: \"4dffd37e-1fe7-4a81-bb92-b6cdcd2999cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-sj64m"
Oct 31 20:28:24.281429 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:24.281271    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bg4sm\" (UniqueName: \"kubernetes.io/projected/4dffd37e-1fe7-4a81-bb92-b6cdcd2999cf-kube-api-access-bg4sm\") pod \"maas-api-revoke-sub-f03629db-sj64m\" (UID: \"4dffd37e-1fe7-4a81-bb92-b6cdcd2999cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-sj64m"
Oct 31 20:28:24.281429 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:24.281377    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dffd37e-1fe7-4a81-bb92-b6cdcd2999cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-sj64m\" (UID: \"4dffd37e-1fe7-4a81-bb92-b6cdcd2999cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-sj64m"
Oct 31 20:28:24.382472 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:24.382441    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dffd37e-1fe7-4a81-bb92-b6cdcd2999cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-sj64m\" (UID: \"4dffd37e-1fe7-4a81-bb92-b6cdcd2999cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-sj64m"
Oct 31 20:28:24.382584 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:24.382499    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dffd37e-1fe7-4a81-bb92-b6cdcd2999cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-sj64m\" (UID: \"4dffd37e-1fe7-4a81-bb92-b6cdcd2999cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-sj64m"
Oct 31 20:28:24.382584 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:24.382522    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bg4sm\" (UniqueName: \"kubernetes.io/projected/4dffd37e-1fe7-4a81-bb92-b6cdcd2999cf-kube-api-access-bg4sm\") pod \"maas-api-revoke-sub-f03629db-sj64m\" (UID: \"4dffd37e-1fe7-4a81-bb92-b6cdcd2999cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-sj64m"
Oct 31 20:28:24.383180 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:24.383154    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dffd37e-1fe7-4a81-bb92-b6cdcd2999cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-sj64m\" (UID: \"4dffd37e-1fe7-4a81-bb92-b6cdcd2999cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-sj64m"
Oct 31 20:28:24.384680 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:24.384662    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dffd37e-1fe7-4a81-bb92-b6cdcd2999cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-sj64m\" (UID: \"4dffd37e-1fe7-4a81-bb92-b6cdcd2999cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-sj64m"
Oct 31 20:28:24.393934 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:24.393912    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bg4sm\" (UniqueName: \"kubernetes.io/projected/4dffd37e-1fe7-4a81-bb92-b6cdcd2999cf-kube-api-access-bg4sm\") pod \"maas-api-revoke-sub-f03629db-sj64m\" (UID: \"4dffd37e-1fe7-4a81-bb92-b6cdcd2999cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-sj64m"
Oct 31 20:28:24.533834 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:24.533774    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-sj64m"
Oct 31 20:28:24.809522 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:24.809470    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-sj64m"]
Oct 31 20:28:24.812218 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:28:24.812191    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dffd37e_1fe7_4a81_bb92_b6cdcd2999cf.slice/crio-0cab142a52e440ed06db8c977c3017eb196b81a4c46bb679b662594f06caba30 WatchSource:0}: Error finding container 0cab142a52e440ed06db8c977c3017eb196b81a4c46bb679b662594f06caba30: Status 404 returned error can't find the container with id 0cab142a52e440ed06db8c977c3017eb196b81a4c46bb679b662594f06caba30
Oct 31 20:28:25.054975 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:25.054947    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-sj64m" event={"ID":"4dffd37e-1fe7-4a81-bb92-b6cdcd2999cf","Type":"ContainerStarted","Data":"0cab142a52e440ed06db8c977c3017eb196b81a4c46bb679b662594f06caba30"}
Oct 31 20:28:26.061858 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:26.061821    4645 generic.go:356] "Generic (PLEG): container finished" podID="4dffd37e-1fe7-4a81-bb92-b6cdcd2999cf" containerID="57d79e8cacc29da7b1c072eb4c45872fd9c335258f26a46e62f2ce50ecb0f028" exitCode=0
Oct 31 20:28:26.062316 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:26.061916    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-sj64m" event={"ID":"4dffd37e-1fe7-4a81-bb92-b6cdcd2999cf","Type":"ContainerDied","Data":"57d79e8cacc29da7b1c072eb4c45872fd9c335258f26a46e62f2ce50ecb0f028"}
Oct 31 20:28:27.197572 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:27.197547    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-sj64m"
Oct 31 20:28:27.206678 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:27.206654    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dffd37e-1fe7-4a81-bb92-b6cdcd2999cf-kube-api-access-bg4sm\" (UniqueName: \"kubernetes.io/projected/4dffd37e-1fe7-4a81-bb92-b6cdcd2999cf-kube-api-access-bg4sm\") pod \"4dffd37e-1fe7-4a81-bb92-b6cdcd2999cf\" (UID: \"4dffd37e-1fe7-4a81-bb92-b6cdcd2999cf\") "
Oct 31 20:28:27.206814 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:27.206710    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4dffd37e-1fe7-4a81-bb92-b6cdcd2999cf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dffd37e-1fe7-4a81-bb92-b6cdcd2999cf-maas-api-openshift-ca\") pod \"4dffd37e-1fe7-4a81-bb92-b6cdcd2999cf\" (UID: \"4dffd37e-1fe7-4a81-bb92-b6cdcd2999cf\") "
Oct 31 20:28:27.206814 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:27.206763    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4dffd37e-1fe7-4a81-bb92-b6cdcd2999cf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dffd37e-1fe7-4a81-bb92-b6cdcd2999cf-maas-api-xks-ca\") pod \"4dffd37e-1fe7-4a81-bb92-b6cdcd2999cf\" (UID: \"4dffd37e-1fe7-4a81-bb92-b6cdcd2999cf\") "
Oct 31 20:28:27.207000 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:27.206976    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dffd37e-1fe7-4a81-bb92-b6cdcd2999cf-maas-api-openshift-ca" pod "4dffd37e-1fe7-4a81-bb92-b6cdcd2999cf" (UID: "4dffd37e-1fe7-4a81-bb92-b6cdcd2999cf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:28:27.207088 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:27.207070    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dffd37e-1fe7-4a81-bb92-b6cdcd2999cf-maas-api-openshift-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:28:27.208741 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:27.208710    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4dffd37e-1fe7-4a81-bb92-b6cdcd2999cf-maas-api-xks-ca" pod "4dffd37e-1fe7-4a81-bb92-b6cdcd2999cf" (UID: "4dffd37e-1fe7-4a81-bb92-b6cdcd2999cf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:28:27.208741 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:27.208735    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dffd37e-1fe7-4a81-bb92-b6cdcd2999cf-kube-api-access-bg4sm" pod "4dffd37e-1fe7-4a81-bb92-b6cdcd2999cf" (UID: "4dffd37e-1fe7-4a81-bb92-b6cdcd2999cf"). InnerVolumeSpecName "kube-api-access-bg4sm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:28:27.307801 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:27.307774    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bg4sm\" (UniqueName: \"kubernetes.io/projected/4dffd37e-1fe7-4a81-bb92-b6cdcd2999cf-kube-api-access-bg4sm\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:28:27.307801 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:27.307799    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dffd37e-1fe7-4a81-bb92-b6cdcd2999cf-maas-api-xks-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:28:28.072054 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:28.072027    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-sj64m"
Oct 31 20:28:28.072224 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:28.072058    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-sj64m" event={"ID":"4dffd37e-1fe7-4a81-bb92-b6cdcd2999cf","Type":"ContainerDied","Data":"0cab142a52e440ed06db8c977c3017eb196b81a4c46bb679b662594f06caba30"}
Oct 31 20:28:28.072224 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:28.072102    4645 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0cab142a52e440ed06db8c977c3017eb196b81a4c46bb679b662594f06caba30"
Oct 31 20:28:36.248214 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:28:36.248167    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 20:28:40.133758 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:40.133716    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zn2n9"]
Oct 31 20:28:40.134196 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:40.134079    4645 cpu_manager.go:414] "containerMap: removing container" podUID="4dffd37e-1fe7-4a81-bb92-b6cdcd2999cf" containerName="revoke-keys"
Oct 31 20:28:40.134196 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:40.134092    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dffd37e-1fe7-4a81-bb92-b6cdcd2999cf" containerName="revoke-keys"
Oct 31 20:28:40.134196 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:40.134104    4645 cpu_manager.go:414] "containerMap: removing container" podUID="d6bcb2da-7676-4255-a33a-084bb67f1bdf" containerName="istio-proxy"
Oct 31 20:28:40.134196 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:40.134110    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6bcb2da-7676-4255-a33a-084bb67f1bdf" containerName="istio-proxy"
Oct 31 20:28:40.134196 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:40.134187    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6bcb2da-7676-4255-a33a-084bb67f1bdf" containerName="istio-proxy"
Oct 31 20:28:40.134196 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:40.134199    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dffd37e-1fe7-4a81-bb92-b6cdcd2999cf" containerName="revoke-keys"
Oct 31 20:28:40.137169 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:40.137147    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zn2n9"
Oct 31 20:28:40.140270 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:40.140233    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:28:40.140500 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:40.140481    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qms6j\""
Oct 31 20:28:40.140623 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:40.140608    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 20:28:40.140694 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:40.140674    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 20:28:40.160403 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:40.160370    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zn2n9"]
Oct 31 20:28:40.247819 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:40.247794    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bc61944-a725-40f0-9250-fee3c208822a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-zn2n9\" (UID: \"2bc61944-a725-40f0-9250-fee3c208822a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zn2n9"
Oct 31 20:28:40.247961 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:40.247828    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzbwr\" (UniqueName: \"kubernetes.io/projected/2bc61944-a725-40f0-9250-fee3c208822a-kube-api-access-lzbwr\") pod \"maas-api-revoke-sub-0165d7d2-zn2n9\" (UID: \"2bc61944-a725-40f0-9250-fee3c208822a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zn2n9"
Oct 31 20:28:40.247961 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:40.247853    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bc61944-a725-40f0-9250-fee3c208822a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-zn2n9\" (UID: \"2bc61944-a725-40f0-9250-fee3c208822a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zn2n9"
Oct 31 20:28:40.349112 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:40.349083    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bc61944-a725-40f0-9250-fee3c208822a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-zn2n9\" (UID: \"2bc61944-a725-40f0-9250-fee3c208822a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zn2n9"
Oct 31 20:28:40.349230 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:40.349128    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzbwr\" (UniqueName: \"kubernetes.io/projected/2bc61944-a725-40f0-9250-fee3c208822a-kube-api-access-lzbwr\") pod \"maas-api-revoke-sub-0165d7d2-zn2n9\" (UID: \"2bc61944-a725-40f0-9250-fee3c208822a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zn2n9"
Oct 31 20:28:40.349230 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:40.349163    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bc61944-a725-40f0-9250-fee3c208822a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-zn2n9\" (UID: \"2bc61944-a725-40f0-9250-fee3c208822a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zn2n9"
Oct 31 20:28:40.349866 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:40.349840    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bc61944-a725-40f0-9250-fee3c208822a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-zn2n9\" (UID: \"2bc61944-a725-40f0-9250-fee3c208822a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zn2n9"
Oct 31 20:28:40.351546 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:40.351525    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bc61944-a725-40f0-9250-fee3c208822a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-zn2n9\" (UID: \"2bc61944-a725-40f0-9250-fee3c208822a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zn2n9"
Oct 31 20:28:40.360753 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:40.360730    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzbwr\" (UniqueName: \"kubernetes.io/projected/2bc61944-a725-40f0-9250-fee3c208822a-kube-api-access-lzbwr\") pod \"maas-api-revoke-sub-0165d7d2-zn2n9\" (UID: \"2bc61944-a725-40f0-9250-fee3c208822a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zn2n9"
Oct 31 20:28:40.449757 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:40.449734    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zn2n9"
Oct 31 20:28:40.721829 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:40.721803    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zn2n9"]
Oct 31 20:28:40.723789 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:28:40.723759    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bc61944_a725_40f0_9250_fee3c208822a.slice/crio-807d8bd4fe0bfa2be5ab912e2aee98e45ce4c16bcf3b5f81b9d3673cb7dc37b0 WatchSource:0}: Error finding container 807d8bd4fe0bfa2be5ab912e2aee98e45ce4c16bcf3b5f81b9d3673cb7dc37b0: Status 404 returned error can't find the container with id 807d8bd4fe0bfa2be5ab912e2aee98e45ce4c16bcf3b5f81b9d3673cb7dc37b0
Oct 31 20:28:41.141784 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:41.141745    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zn2n9" event={"ID":"2bc61944-a725-40f0-9250-fee3c208822a","Type":"ContainerStarted","Data":"fa173437dd047cfed5a6e4112ed75780dfbff95d78a418b541cb6b3c1f2e9360"}
Oct 31 20:28:41.142094 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:41.141808    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zn2n9" event={"ID":"2bc61944-a725-40f0-9250-fee3c208822a","Type":"ContainerStarted","Data":"807d8bd4fe0bfa2be5ab912e2aee98e45ce4c16bcf3b5f81b9d3673cb7dc37b0"}
Oct 31 20:28:41.160765 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:41.160723    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zn2n9" podStartSLOduration=1.160709994 podStartE2EDuration="1.160709994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:28:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:28:41.160279873 +0000 UTC m=+4151.733514003" watchObservedRunningTime="2026-10-31 20:28:41.160709994 +0000 UTC m=+4151.733944123"
Oct 31 20:28:42.148218 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:42.148177    4645 generic.go:356] "Generic (PLEG): container finished" podID="2bc61944-a725-40f0-9250-fee3c208822a" containerID="fa173437dd047cfed5a6e4112ed75780dfbff95d78a418b541cb6b3c1f2e9360" exitCode=0
Oct 31 20:28:42.148648 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:42.148263    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zn2n9" event={"ID":"2bc61944-a725-40f0-9250-fee3c208822a","Type":"ContainerDied","Data":"fa173437dd047cfed5a6e4112ed75780dfbff95d78a418b541cb6b3c1f2e9360"}
Oct 31 20:28:43.285449 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:43.285426    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zn2n9"
Oct 31 20:28:43.372702 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:43.372673    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bc61944-a725-40f0-9250-fee3c208822a-kube-api-access-lzbwr\" (UniqueName: \"kubernetes.io/projected/2bc61944-a725-40f0-9250-fee3c208822a-kube-api-access-lzbwr\") pod \"2bc61944-a725-40f0-9250-fee3c208822a\" (UID: \"2bc61944-a725-40f0-9250-fee3c208822a\") "
Oct 31 20:28:43.372841 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:43.372720    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bc61944-a725-40f0-9250-fee3c208822a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bc61944-a725-40f0-9250-fee3c208822a-maas-api-xks-ca\") pod \"2bc61944-a725-40f0-9250-fee3c208822a\" (UID: \"2bc61944-a725-40f0-9250-fee3c208822a\") "
Oct 31 20:28:43.372841 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:43.372771    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bc61944-a725-40f0-9250-fee3c208822a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bc61944-a725-40f0-9250-fee3c208822a-maas-api-openshift-ca\") pod \"2bc61944-a725-40f0-9250-fee3c208822a\" (UID: \"2bc61944-a725-40f0-9250-fee3c208822a\") "
Oct 31 20:28:43.373195 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:43.373170    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bc61944-a725-40f0-9250-fee3c208822a-maas-api-openshift-ca" pod "2bc61944-a725-40f0-9250-fee3c208822a" (UID: "2bc61944-a725-40f0-9250-fee3c208822a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:28:43.374701 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:43.374676    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bc61944-a725-40f0-9250-fee3c208822a-maas-api-xks-ca" pod "2bc61944-a725-40f0-9250-fee3c208822a" (UID: "2bc61944-a725-40f0-9250-fee3c208822a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:28:43.374806 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:43.374768    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bc61944-a725-40f0-9250-fee3c208822a-kube-api-access-lzbwr" pod "2bc61944-a725-40f0-9250-fee3c208822a" (UID: "2bc61944-a725-40f0-9250-fee3c208822a"). InnerVolumeSpecName "kube-api-access-lzbwr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:28:43.474232 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:43.474208    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzbwr\" (UniqueName: \"kubernetes.io/projected/2bc61944-a725-40f0-9250-fee3c208822a-kube-api-access-lzbwr\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:28:43.474232 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:43.474233    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bc61944-a725-40f0-9250-fee3c208822a-maas-api-xks-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:28:43.474401 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:43.474243    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bc61944-a725-40f0-9250-fee3c208822a-maas-api-openshift-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:28:44.158243 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:44.158211    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zn2n9" event={"ID":"2bc61944-a725-40f0-9250-fee3c208822a","Type":"ContainerDied","Data":"807d8bd4fe0bfa2be5ab912e2aee98e45ce4c16bcf3b5f81b9d3673cb7dc37b0"}
Oct 31 20:28:44.158408 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:44.158266    4645 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="807d8bd4fe0bfa2be5ab912e2aee98e45ce4c16bcf3b5f81b9d3673cb7dc37b0"
Oct 31 20:28:44.158408 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:44.158223    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-zn2n9"
Oct 31 20:28:50.981679 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:50.981642    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-vz9fc"]
Oct 31 20:28:50.982090 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:50.982008    4645 cpu_manager.go:414] "containerMap: removing container" podUID="2bc61944-a725-40f0-9250-fee3c208822a" containerName="revoke-keys"
Oct 31 20:28:50.982090 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:50.982020    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bc61944-a725-40f0-9250-fee3c208822a" containerName="revoke-keys"
Oct 31 20:28:50.982090 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:50.982089    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bc61944-a725-40f0-9250-fee3c208822a" containerName="revoke-keys"
Oct 31 20:28:50.985291 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:50.985264    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-vz9fc"
Oct 31 20:28:50.989276 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:50.989247    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 20:28:50.989572 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:50.989550    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 20:28:50.989572 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:50.989551    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 20:28:50.989739 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:50.989644    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qms6j\""
Oct 31 20:28:51.000675 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:51.000643    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-vz9fc"]
Oct 31 20:28:51.033014 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:51.032988    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9164176b-3de6-4712-9a17-90e786b958c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-vz9fc\" (UID: \"9164176b-3de6-4712-9a17-90e786b958c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-vz9fc"
Oct 31 20:28:51.033150 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:51.033100    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwj5k\" (UniqueName: \"kubernetes.io/projected/9164176b-3de6-4712-9a17-90e786b958c8-kube-api-access-gwj5k\") pod \"maas-api-revoke-sub-6b19338f-vz9fc\" (UID: \"9164176b-3de6-4712-9a17-90e786b958c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-vz9fc"
Oct 31 20:28:51.033208 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:51.033182    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9164176b-3de6-4712-9a17-90e786b958c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-vz9fc\" (UID: \"9164176b-3de6-4712-9a17-90e786b958c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-vz9fc"
Oct 31 20:28:51.134082 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:51.134052    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwj5k\" (UniqueName: \"kubernetes.io/projected/9164176b-3de6-4712-9a17-90e786b958c8-kube-api-access-gwj5k\") pod \"maas-api-revoke-sub-6b19338f-vz9fc\" (UID: \"9164176b-3de6-4712-9a17-90e786b958c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-vz9fc"
Oct 31 20:28:51.134195 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:51.134110    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9164176b-3de6-4712-9a17-90e786b958c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-vz9fc\" (UID: \"9164176b-3de6-4712-9a17-90e786b958c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-vz9fc"
Oct 31 20:28:51.134195 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:51.134129    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9164176b-3de6-4712-9a17-90e786b958c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-vz9fc\" (UID: \"9164176b-3de6-4712-9a17-90e786b958c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-vz9fc"
Oct 31 20:28:51.134750 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:51.134722    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9164176b-3de6-4712-9a17-90e786b958c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-vz9fc\" (UID: \"9164176b-3de6-4712-9a17-90e786b958c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-vz9fc"
Oct 31 20:28:51.136483 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:51.136456    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9164176b-3de6-4712-9a17-90e786b958c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-vz9fc\" (UID: \"9164176b-3de6-4712-9a17-90e786b958c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-vz9fc"
Oct 31 20:28:51.145812 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:51.145790    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwj5k\" (UniqueName: \"kubernetes.io/projected/9164176b-3de6-4712-9a17-90e786b958c8-kube-api-access-gwj5k\") pod \"maas-api-revoke-sub-6b19338f-vz9fc\" (UID: \"9164176b-3de6-4712-9a17-90e786b958c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-vz9fc"
Oct 31 20:28:51.298297 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:51.298243    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-vz9fc"
Oct 31 20:28:51.572480 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:51.572284    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-vz9fc"]
Oct 31 20:28:51.575193 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:28:51.575160    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9164176b_3de6_4712_9a17_90e786b958c8.slice/crio-3c511c76310aded65379496904bfcbea975a956605ff8424daaa3bb444c601d2 WatchSource:0}: Error finding container 3c511c76310aded65379496904bfcbea975a956605ff8424daaa3bb444c601d2: Status 404 returned error can't find the container with id 3c511c76310aded65379496904bfcbea975a956605ff8424daaa3bb444c601d2
Oct 31 20:28:52.195784 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:52.195756    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-vz9fc" event={"ID":"9164176b-3de6-4712-9a17-90e786b958c8","Type":"ContainerStarted","Data":"ff216d664a34b5499eb09002cadf7f4432e4cdbc5726c48f9bea038bb9bc24ab"}
Oct 31 20:28:52.196189 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:52.195813    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-vz9fc" event={"ID":"9164176b-3de6-4712-9a17-90e786b958c8","Type":"ContainerStarted","Data":"3c511c76310aded65379496904bfcbea975a956605ff8424daaa3bb444c601d2"}
Oct 31 20:28:52.213710 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:52.213662    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-vz9fc" podStartSLOduration=2.213648516 podStartE2EDuration="2.213648516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:28:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:28:52.21241444 +0000 UTC m=+4162.785648581" watchObservedRunningTime="2026-10-31 20:28:52.213648516 +0000 UTC m=+4162.786882644"
Oct 31 20:28:53.201408 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:53.201372    4645 generic.go:356] "Generic (PLEG): container finished" podID="9164176b-3de6-4712-9a17-90e786b958c8" containerID="ff216d664a34b5499eb09002cadf7f4432e4cdbc5726c48f9bea038bb9bc24ab" exitCode=0
Oct 31 20:28:53.201846 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:53.201416    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-vz9fc" event={"ID":"9164176b-3de6-4712-9a17-90e786b958c8","Type":"ContainerDied","Data":"ff216d664a34b5499eb09002cadf7f4432e4cdbc5726c48f9bea038bb9bc24ab"}
Oct 31 20:28:54.345211 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:54.345186    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-vz9fc"
Oct 31 20:28:54.459515 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:54.459452    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9164176b-3de6-4712-9a17-90e786b958c8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9164176b-3de6-4712-9a17-90e786b958c8-maas-api-xks-ca\") pod \"9164176b-3de6-4712-9a17-90e786b958c8\" (UID: \"9164176b-3de6-4712-9a17-90e786b958c8\") "
Oct 31 20:28:54.459644 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:54.459554    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9164176b-3de6-4712-9a17-90e786b958c8-kube-api-access-gwj5k\" (UniqueName: \"kubernetes.io/projected/9164176b-3de6-4712-9a17-90e786b958c8-kube-api-access-gwj5k\") pod \"9164176b-3de6-4712-9a17-90e786b958c8\" (UID: \"9164176b-3de6-4712-9a17-90e786b958c8\") "
Oct 31 20:28:54.459644 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:54.459591    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9164176b-3de6-4712-9a17-90e786b958c8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9164176b-3de6-4712-9a17-90e786b958c8-maas-api-openshift-ca\") pod \"9164176b-3de6-4712-9a17-90e786b958c8\" (UID: \"9164176b-3de6-4712-9a17-90e786b958c8\") "
Oct 31 20:28:54.459946 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:54.459923    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9164176b-3de6-4712-9a17-90e786b958c8-maas-api-openshift-ca" pod "9164176b-3de6-4712-9a17-90e786b958c8" (UID: "9164176b-3de6-4712-9a17-90e786b958c8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:28:54.461561 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:54.461538    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9164176b-3de6-4712-9a17-90e786b958c8-kube-api-access-gwj5k" pod "9164176b-3de6-4712-9a17-90e786b958c8" (UID: "9164176b-3de6-4712-9a17-90e786b958c8"). InnerVolumeSpecName "kube-api-access-gwj5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:28:54.461642 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:54.461581    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9164176b-3de6-4712-9a17-90e786b958c8-maas-api-xks-ca" pod "9164176b-3de6-4712-9a17-90e786b958c8" (UID: "9164176b-3de6-4712-9a17-90e786b958c8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 20:28:54.560587 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:54.560551    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwj5k\" (UniqueName: \"kubernetes.io/projected/9164176b-3de6-4712-9a17-90e786b958c8-kube-api-access-gwj5k\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:28:54.560587 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:54.560584    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9164176b-3de6-4712-9a17-90e786b958c8-maas-api-openshift-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:28:54.560587 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:54.560595    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9164176b-3de6-4712-9a17-90e786b958c8-maas-api-xks-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:28:55.211976 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:55.211943    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-vz9fc"
Oct 31 20:28:55.212178 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:55.211983    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-vz9fc" event={"ID":"9164176b-3de6-4712-9a17-90e786b958c8","Type":"ContainerDied","Data":"3c511c76310aded65379496904bfcbea975a956605ff8424daaa3bb444c601d2"}
Oct 31 20:28:55.212178 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:28:55.212048    4645 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c511c76310aded65379496904bfcbea975a956605ff8424daaa3bb444c601d2"
Oct 31 20:29:00.204943 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:29:00.204898    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:29:00.306054 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:29:00.306019    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:29:02.144003 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:29:02.143966    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-559d65ddd-gx66l_c8726e82-9df2-47ab-bfbc-0babdddf5ee0/authorino/0.log"
Oct 31 20:29:06.244367 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:29:06.244315    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 20:29:13.094990 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:29:13.094850    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-c9tc7_9f12248a-17ea-4f35-9de4-d45b71d238cb/manager/0.log"
Oct 31 20:29:15.386919 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:29:15.386879    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-zn2n9_2bc61944-a725-40f0-9250-fee3c208822a/revoke-keys/0.log"
Oct 31 20:29:15.969391 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:29:15.969356    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-vz9fc_9164176b-3de6-4712-9a17-90e786b958c8/revoke-keys/0.log"
Oct 31 20:29:16.337500 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:29:16.337420    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-sj64m_4dffd37e-1fe7-4a81-bb92-b6cdcd2999cf/revoke-keys/0.log"
Oct 31 20:29:18.049071 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:29:18.049036    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-559d65ddd-gx66l_c8726e82-9df2-47ab-bfbc-0babdddf5ee0/authorino/0.log"
Oct 31 20:29:19.471395 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:29:19.471352    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:29:20.560484 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:29:20.560450    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:29:20.639383 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:29:20.639352    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-849d94dfbb-l28r8_ae933cbf-384e-4052-881d-800d405f8e7b/kube-auth-proxy/0.log"
Oct 31 20:29:20.821525 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:29:20.821470    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-68qbd_3a217b30-54c0-431f-90bc-0681901272f4/istio-proxy/0.log"
Oct 31 20:29:21.011634 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:29:21.011607    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-6rwmb_b92c4b09-c2c0-4602-af78-2f410c843118/istio-proxy/0.log"
Oct 31 20:29:21.585049 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:29:21.585008    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-lwvpn_3d401fef-1568-4d16-88fa-92aeaaea68b0/istio-proxy/0.log"
Oct 31 20:29:21.725341 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:29:21.725310    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:29:22.115526 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:29:22.115489    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-t27sn_f3994be2-19d9-4e2d-b3ce-03f63d2c9c32/istio-proxy/0.log"
Oct 31 20:29:23.034806 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:29:23.034776    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-76hjw_b14ee5b2-6bf8-4b41-a84d-d0a7dcc9b85a/payload-processing/0.log"
Oct 31 20:29:24.197094 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:29:24.197052    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-88sql_53abb174-56be-4e64-aadc-e871ec9c40f3/main/0.log"
Oct 31 20:29:24.378765 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:29:24.378734    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-gg8cz_7e0baa6d-56cc-4248-853b-540777aa7b9b/main/0.log"
Oct 31 20:29:24.556861 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:29:24.556795    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-k8zgg_7ac0d131-8660-4018-88c9-1e2d06f3c038/main/0.log"
Oct 31 20:29:24.739407 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:29:24.739383    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-wntcw_d6b653ca-0bd2-488f-925b-02a9a8dc509b/main/0.log"
Oct 31 20:29:24.920651 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:29:24.920630    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbsgn_280abee2-f10e-4a96-9735-a3aadde28b69/main/0.log"
Oct 31 20:29:25.456083 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:29:25.456054    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw_1287db54-a587-4c11-9eb6-1606de555894/main/0.log"
Oct 31 20:29:26.368425 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:29:26.368394    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-559d65ddd-gx66l_c8726e82-9df2-47ab-bfbc-0babdddf5ee0/authorino/0.log"
Oct 31 20:29:35.847865 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:29:35.847810    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-c9tc7_9f12248a-17ea-4f35-9de4-d45b71d238cb/manager/0.log"
Oct 31 20:29:36.245775 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:29:36.245738    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 20:29:37.847370 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:29:37.847317    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-zn2n9_2bc61944-a725-40f0-9250-fee3c208822a/revoke-keys/0.log"
Oct 31 20:29:38.403827 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:29:38.403799    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-vz9fc_9164176b-3de6-4712-9a17-90e786b958c8/revoke-keys/0.log"
Oct 31 20:29:38.766799 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:29:38.766764    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-sj64m_4dffd37e-1fe7-4a81-bb92-b6cdcd2999cf/revoke-keys/0.log"
Oct 31 20:29:40.102371 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:29:40.102316    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-559d65ddd-gx66l_c8726e82-9df2-47ab-bfbc-0babdddf5ee0/authorino/0.log"
Oct 31 20:29:42.090217 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:29:42.090175    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:29:42.405471 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:29:42.405371    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-849d94dfbb-l28r8_ae933cbf-384e-4052-881d-800d405f8e7b/kube-auth-proxy/0.log"
Oct 31 20:29:42.579255 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:29:42.579223    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-68qbd_3a217b30-54c0-431f-90bc-0681901272f4/istio-proxy/0.log"
Oct 31 20:29:42.767681 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:29:42.767655    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-6rwmb_b92c4b09-c2c0-4602-af78-2f410c843118/istio-proxy/0.log"
Oct 31 20:29:43.335335 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:29:43.335299    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-lwvpn_3d401fef-1568-4d16-88fa-92aeaaea68b0/istio-proxy/0.log"
Oct 31 20:29:43.883267 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:29:43.883238    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-t27sn_f3994be2-19d9-4e2d-b3ce-03f63d2c9c32/istio-proxy/0.log"
Oct 31 20:29:44.533405 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:29:44.533360    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:29:44.635728 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:29:44.635685    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:29:44.796641 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:29:44.796582    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-76hjw_b14ee5b2-6bf8-4b41-a84d-d0a7dcc9b85a/payload-processing/0.log"
Oct 31 20:29:45.526585 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:29:45.526549    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-88sql_53abb174-56be-4e64-aadc-e871ec9c40f3/main/0.log"
Oct 31 20:29:45.706363 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:29:45.706315    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-gg8cz_7e0baa6d-56cc-4248-853b-540777aa7b9b/main/0.log"
Oct 31 20:29:45.889741 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:29:45.889678    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-k8zgg_7ac0d131-8660-4018-88c9-1e2d06f3c038/main/0.log"
Oct 31 20:29:46.075403 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:29:46.075377    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-wntcw_d6b653ca-0bd2-488f-925b-02a9a8dc509b/main/0.log"
Oct 31 20:29:46.261421 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:29:46.261391    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8tbsgn_280abee2-f10e-4a96-9735-a3aadde28b69/main/0.log"
Oct 31 20:29:46.818282 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:29:46.818258    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-fhvrw_1287db54-a587-4c11-9eb6-1606de555894/main/0.log"
Oct 31 20:30:00.186455 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:30:00.186398    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891310-jndlf"]
Oct 31 20:30:00.186900 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:30:00.186801    4645 cpu_manager.go:414] "containerMap: removing container" podUID="9164176b-3de6-4712-9a17-90e786b958c8" containerName="revoke-keys"
Oct 31 20:30:00.186900 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:30:00.186814    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9164176b-3de6-4712-9a17-90e786b958c8" containerName="revoke-keys"
Oct 31 20:30:00.186900 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:30:00.186888    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="9164176b-3de6-4712-9a17-90e786b958c8" containerName="revoke-keys"
Oct 31 20:30:00.190140 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:30:00.190114    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891310-jndlf"
Oct 31 20:30:00.194078 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:30:00.194046    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 20:30:00.195547 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:30:00.195520    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 20:30:00.195696 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:30:00.195590    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qms6j\""
Oct 31 20:30:00.208460 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:30:00.208420    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891310-jndlf"]
Oct 31 20:30:00.355615 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:30:00.355569    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzjlj\" (UniqueName: \"kubernetes.io/projected/5e9b04fc-35eb-47ab-9cf7-1caaf4460313-kube-api-access-lzjlj\") pod \"maas-api-key-cleanup-29891310-jndlf\" (UID: \"5e9b04fc-35eb-47ab-9cf7-1caaf4460313\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891310-jndlf"
Oct 31 20:30:00.355837 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:30:00.355633    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e9b04fc-35eb-47ab-9cf7-1caaf4460313-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891310-jndlf\" (UID: \"5e9b04fc-35eb-47ab-9cf7-1caaf4460313\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891310-jndlf"
Oct 31 20:30:00.456543 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:30:00.456448    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e9b04fc-35eb-47ab-9cf7-1caaf4460313-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891310-jndlf\" (UID: \"5e9b04fc-35eb-47ab-9cf7-1caaf4460313\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891310-jndlf"
Oct 31 20:30:00.456717 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:30:00.456558    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzjlj\" (UniqueName: \"kubernetes.io/projected/5e9b04fc-35eb-47ab-9cf7-1caaf4460313-kube-api-access-lzjlj\") pod \"maas-api-key-cleanup-29891310-jndlf\" (UID: \"5e9b04fc-35eb-47ab-9cf7-1caaf4460313\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891310-jndlf"
Oct 31 20:30:00.457155 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:30:00.457131    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e9b04fc-35eb-47ab-9cf7-1caaf4460313-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891310-jndlf\" (UID: \"5e9b04fc-35eb-47ab-9cf7-1caaf4460313\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891310-jndlf"
Oct 31 20:30:00.469124 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:30:00.469084    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzjlj\" (UniqueName: \"kubernetes.io/projected/5e9b04fc-35eb-47ab-9cf7-1caaf4460313-kube-api-access-lzjlj\") pod \"maas-api-key-cleanup-29891310-jndlf\" (UID: \"5e9b04fc-35eb-47ab-9cf7-1caaf4460313\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891310-jndlf"
Oct 31 20:30:00.503963 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:30:00.503921    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891310-jndlf"
Oct 31 20:30:00.785317 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:30:00.785170    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891310-jndlf"]
Oct 31 20:30:00.788356 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:30:00.788289    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e9b04fc_35eb_47ab_9cf7_1caaf4460313.slice/crio-3defbdcbc9c3a426a45ecb1b24c1347ac1be4f39b865ad9a8e4d3fd21d060a54 WatchSource:0}: Error finding container 3defbdcbc9c3a426a45ecb1b24c1347ac1be4f39b865ad9a8e4d3fd21d060a54: Status 404 returned error can't find the container with id 3defbdcbc9c3a426a45ecb1b24c1347ac1be4f39b865ad9a8e4d3fd21d060a54
Oct 31 20:30:01.545833 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:30:01.545788    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891310-jndlf" event={"ID":"5e9b04fc-35eb-47ab-9cf7-1caaf4460313","Type":"ContainerStarted","Data":"aaf84648159a8e86fa9242830389038a5af3247383d6625f2564d6a791f8d1d8"}
Oct 31 20:30:01.546246 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:30:01.545890    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891310-jndlf" event={"ID":"5e9b04fc-35eb-47ab-9cf7-1caaf4460313","Type":"ContainerStarted","Data":"3defbdcbc9c3a426a45ecb1b24c1347ac1be4f39b865ad9a8e4d3fd21d060a54"}
Oct 31 20:30:01.569228 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:30:01.569181    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891310-jndlf" podStartSLOduration=1.569166916 podStartE2EDuration="1.569166916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:30:01.566376044 +0000 UTC m=+4232.139610170" watchObservedRunningTime="2026-10-31 20:30:01.569166916 +0000 UTC m=+4232.142401044"
Oct 31 20:30:02.552641 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:30:02.552602    4645 generic.go:356] "Generic (PLEG): container finished" podID="5e9b04fc-35eb-47ab-9cf7-1caaf4460313" containerID="aaf84648159a8e86fa9242830389038a5af3247383d6625f2564d6a791f8d1d8" exitCode=60
Oct 31 20:30:02.553079 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:30:02.552687    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891310-jndlf" event={"ID":"5e9b04fc-35eb-47ab-9cf7-1caaf4460313","Type":"ContainerDied","Data":"aaf84648159a8e86fa9242830389038a5af3247383d6625f2564d6a791f8d1d8"}
Oct 31 20:30:02.553079 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:30:02.552925    4645 scope.go:122] "RemoveContainer" containerID="aaf84648159a8e86fa9242830389038a5af3247383d6625f2564d6a791f8d1d8"
Oct 31 20:30:03.558961 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:30:03.558925    4645 generic.go:356] "Generic (PLEG): container finished" podID="5e9b04fc-35eb-47ab-9cf7-1caaf4460313" containerID="3f2ac2c3ecc44a798978342ca2518179045957048bc5fb1a6cd8d39394586ed4" exitCode=60
Oct 31 20:30:03.559320 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:30:03.558986    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891310-jndlf" event={"ID":"5e9b04fc-35eb-47ab-9cf7-1caaf4460313","Type":"ContainerDied","Data":"3f2ac2c3ecc44a798978342ca2518179045957048bc5fb1a6cd8d39394586ed4"}
Oct 31 20:30:03.559320 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:30:03.559047    4645 scope.go:122] "RemoveContainer" containerID="aaf84648159a8e86fa9242830389038a5af3247383d6625f2564d6a791f8d1d8"
Oct 31 20:30:03.559320 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:30:03.559233    4645 scope.go:122] "RemoveContainer" containerID="3f2ac2c3ecc44a798978342ca2518179045957048bc5fb1a6cd8d39394586ed4"
Oct 31 20:30:03.559495 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:30:03.559441    4645 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891310-jndlf_odh-ai-gateway-infra(5e9b04fc-35eb-47ab-9cf7-1caaf4460313)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891310-jndlf" podUID="5e9b04fc-35eb-47ab-9cf7-1caaf4460313"
Oct 31 20:30:04.566795 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:30:04.566762    4645 scope.go:122] "RemoveContainer" containerID="3f2ac2c3ecc44a798978342ca2518179045957048bc5fb1a6cd8d39394586ed4"
Oct 31 20:30:04.567238 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:30:04.566971    4645 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891310-jndlf_odh-ai-gateway-infra(5e9b04fc-35eb-47ab-9cf7-1caaf4460313)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891310-jndlf" podUID="5e9b04fc-35eb-47ab-9cf7-1caaf4460313"
Oct 31 20:30:06.247856 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:30:06.247822    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 20:30:14.611013 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:30:14.610974    4645 scope.go:122] "RemoveContainer" containerID="3f2ac2c3ecc44a798978342ca2518179045957048bc5fb1a6cd8d39394586ed4"
Oct 31 20:30:15.624184 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:30:15.624108    4645 generic.go:356] "Generic (PLEG): container finished" podID="5e9b04fc-35eb-47ab-9cf7-1caaf4460313" containerID="64ad4b3a2d18c4e73711648d1f88a2618d84695b233cb4ac55e11c2ef2ed4fb1" exitCode=60
Oct 31 20:30:15.624670 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:30:15.624177    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891310-jndlf" event={"ID":"5e9b04fc-35eb-47ab-9cf7-1caaf4460313","Type":"ContainerDied","Data":"64ad4b3a2d18c4e73711648d1f88a2618d84695b233cb4ac55e11c2ef2ed4fb1"}
Oct 31 20:30:15.624670 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:30:15.624262    4645 scope.go:122] "RemoveContainer" containerID="3f2ac2c3ecc44a798978342ca2518179045957048bc5fb1a6cd8d39394586ed4"
Oct 31 20:30:15.624788 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:30:15.624699    4645 scope.go:122] "RemoveContainer" containerID="64ad4b3a2d18c4e73711648d1f88a2618d84695b233cb4ac55e11c2ef2ed4fb1"
Oct 31 20:30:15.624958 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:30:15.624935    4645 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29891310-jndlf_odh-ai-gateway-infra(5e9b04fc-35eb-47ab-9cf7-1caaf4460313)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891310-jndlf" podUID="5e9b04fc-35eb-47ab-9cf7-1caaf4460313"
Oct 31 20:30:16.656999 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:30:16.656965    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891310-jndlf"]
Oct 31 20:30:16.794513 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:30:16.794486    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891310-jndlf"
Oct 31 20:30:16.891054 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:30:16.891014    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e9b04fc-35eb-47ab-9cf7-1caaf4460313-kube-api-access-lzjlj\" (UniqueName: \"kubernetes.io/projected/5e9b04fc-35eb-47ab-9cf7-1caaf4460313-kube-api-access-lzjlj\") pod \"5e9b04fc-35eb-47ab-9cf7-1caaf4460313\" (UID: \"5e9b04fc-35eb-47ab-9cf7-1caaf4460313\") "
Oct 31 20:30:16.891227 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:30:16.891163    4645 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e9b04fc-35eb-47ab-9cf7-1caaf4460313-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e9b04fc-35eb-47ab-9cf7-1caaf4460313-maas-api-service-ca\") pod \"5e9b04fc-35eb-47ab-9cf7-1caaf4460313\" (UID: \"5e9b04fc-35eb-47ab-9cf7-1caaf4460313\") "
Oct 31 20:30:16.891516 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:30:16.891492    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e9b04fc-35eb-47ab-9cf7-1caaf4460313-maas-api-service-ca" pod "5e9b04fc-35eb-47ab-9cf7-1caaf4460313" (UID: "5e9b04fc-35eb-47ab-9cf7-1caaf4460313"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 20:30:16.893136 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:30:16.893109    4645 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e9b04fc-35eb-47ab-9cf7-1caaf4460313-kube-api-access-lzjlj" pod "5e9b04fc-35eb-47ab-9cf7-1caaf4460313" (UID: "5e9b04fc-35eb-47ab-9cf7-1caaf4460313"). InnerVolumeSpecName "kube-api-access-lzjlj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 20:30:16.992838 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:30:16.992810    4645 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e9b04fc-35eb-47ab-9cf7-1caaf4460313-maas-api-service-ca\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:30:16.992838 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:30:16.992835    4645 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzjlj\" (UniqueName: \"kubernetes.io/projected/5e9b04fc-35eb-47ab-9cf7-1caaf4460313-kube-api-access-lzjlj\") on node \"ip-10-0-131-94.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 20:30:17.635361 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:30:17.635320    4645 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891310-jndlf"
Oct 31 20:30:17.635509 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:30:17.635366    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891310-jndlf" event={"ID":"5e9b04fc-35eb-47ab-9cf7-1caaf4460313","Type":"ContainerDied","Data":"3defbdcbc9c3a426a45ecb1b24c1347ac1be4f39b865ad9a8e4d3fd21d060a54"}
Oct 31 20:30:17.635509 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:30:17.635419    4645 scope.go:122] "RemoveContainer" containerID="64ad4b3a2d18c4e73711648d1f88a2618d84695b233cb4ac55e11c2ef2ed4fb1"
Oct 31 20:30:17.651116 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:30:17.651089    4645 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891310-jndlf"]
Oct 31 20:30:17.654968 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:30:17.654944    4645 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891310-jndlf"]
Oct 31 20:30:19.523030 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:30:19.522962    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:30:19.615717 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:30:19.615687    4645 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e9b04fc-35eb-47ab-9cf7-1caaf4460313" path="/var/lib/kubelet/pods/5e9b04fc-35eb-47ab-9cf7-1caaf4460313/volumes"
Oct 31 20:30:20.605147 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:30:20.605102    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:30:21.793499 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:30:21.793460    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:30:22.359464 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:30:22.359438    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-76hjw_b14ee5b2-6bf8-4b41-a84d-d0a7dcc9b85a/payload-processing/0.log"
Oct 31 20:30:22.548588 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:30:22.548560    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-76hjw_b14ee5b2-6bf8-4b41-a84d-d0a7dcc9b85a/payload-processing/0.log"
Oct 31 20:30:36.249228 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:30:36.249187    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 20:31:06.148818 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:06.148779    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-76hjw_b14ee5b2-6bf8-4b41-a84d-d0a7dcc9b85a/payload-processing/0.log"
Oct 31 20:31:06.247934 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:31:06.247895    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 20:31:06.354112 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:06.354085    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-76hjw_b14ee5b2-6bf8-4b41-a84d-d0a7dcc9b85a/payload-processing/0.log"
Oct 31 20:31:06.732067 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:06.732030    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-hk24f/must-gather-tbhnm"]
Oct 31 20:31:06.732417 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:06.732403    4645 cpu_manager.go:414] "containerMap: removing container" podUID="5e9b04fc-35eb-47ab-9cf7-1caaf4460313" containerName="cleanup"
Oct 31 20:31:06.732480 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:06.732419    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e9b04fc-35eb-47ab-9cf7-1caaf4460313" containerName="cleanup"
Oct 31 20:31:06.732480 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:06.732433    4645 cpu_manager.go:414] "containerMap: removing container" podUID="5e9b04fc-35eb-47ab-9cf7-1caaf4460313" containerName="cleanup"
Oct 31 20:31:06.732480 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:06.732438    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e9b04fc-35eb-47ab-9cf7-1caaf4460313" containerName="cleanup"
Oct 31 20:31:06.732480 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:06.732447    4645 cpu_manager.go:414] "containerMap: removing container" podUID="5e9b04fc-35eb-47ab-9cf7-1caaf4460313" containerName="cleanup"
Oct 31 20:31:06.732480 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:06.732452    4645 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e9b04fc-35eb-47ab-9cf7-1caaf4460313" containerName="cleanup"
Oct 31 20:31:06.732676 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:06.732513    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e9b04fc-35eb-47ab-9cf7-1caaf4460313" containerName="cleanup"
Oct 31 20:31:06.732676 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:06.732522    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e9b04fc-35eb-47ab-9cf7-1caaf4460313" containerName="cleanup"
Oct 31 20:31:06.732676 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:06.732529    4645 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e9b04fc-35eb-47ab-9cf7-1caaf4460313" containerName="cleanup"
Oct 31 20:31:06.735575 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:06.735558    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hk24f/must-gather-tbhnm"
Oct 31 20:31:06.752045 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:06.752024    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hk24f\"/\"kube-root-ca.crt\""
Oct 31 20:31:06.752197 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:06.752139    4645 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-hk24f\"/\"openshift-service-ca.crt\""
Oct 31 20:31:06.752197 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:06.752176    4645 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-hk24f\"/\"default-dockercfg-pj4k7\""
Oct 31 20:31:06.754252 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:06.754230    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hk24f/must-gather-tbhnm"]
Oct 31 20:31:06.787121 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:06.787090    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:31:06.901266 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:06.901237    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8b3c0bf6-9bbe-46f8-8b02-8f6acc690a57-must-gather-output\") pod \"must-gather-tbhnm\" (UID: \"8b3c0bf6-9bbe-46f8-8b02-8f6acc690a57\") " pod="openshift-must-gather-hk24f/must-gather-tbhnm"
Oct 31 20:31:06.901421 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:06.901294    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4m9ps\" (UniqueName: \"kubernetes.io/projected/8b3c0bf6-9bbe-46f8-8b02-8f6acc690a57-kube-api-access-4m9ps\") pod \"must-gather-tbhnm\" (UID: \"8b3c0bf6-9bbe-46f8-8b02-8f6acc690a57\") " pod="openshift-must-gather-hk24f/must-gather-tbhnm"
Oct 31 20:31:07.002032 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:07.001970    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8b3c0bf6-9bbe-46f8-8b02-8f6acc690a57-must-gather-output\") pod \"must-gather-tbhnm\" (UID: \"8b3c0bf6-9bbe-46f8-8b02-8f6acc690a57\") " pod="openshift-must-gather-hk24f/must-gather-tbhnm"
Oct 31 20:31:07.002032 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:07.002024    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m9ps\" (UniqueName: \"kubernetes.io/projected/8b3c0bf6-9bbe-46f8-8b02-8f6acc690a57-kube-api-access-4m9ps\") pod \"must-gather-tbhnm\" (UID: \"8b3c0bf6-9bbe-46f8-8b02-8f6acc690a57\") " pod="openshift-must-gather-hk24f/must-gather-tbhnm"
Oct 31 20:31:07.002321 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:07.002301    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8b3c0bf6-9bbe-46f8-8b02-8f6acc690a57-must-gather-output\") pod \"must-gather-tbhnm\" (UID: \"8b3c0bf6-9bbe-46f8-8b02-8f6acc690a57\") " pod="openshift-must-gather-hk24f/must-gather-tbhnm"
Oct 31 20:31:07.034005 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:07.033979    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m9ps\" (UniqueName: \"kubernetes.io/projected/8b3c0bf6-9bbe-46f8-8b02-8f6acc690a57-kube-api-access-4m9ps\") pod \"must-gather-tbhnm\" (UID: \"8b3c0bf6-9bbe-46f8-8b02-8f6acc690a57\") " pod="openshift-must-gather-hk24f/must-gather-tbhnm"
Oct 31 20:31:07.049017 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:07.048985    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hk24f/must-gather-tbhnm"
Oct 31 20:31:07.078009 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:07.077963    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:31:07.324907 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:07.324879    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hk24f/must-gather-tbhnm"]
Oct 31 20:31:07.326641 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:31:07.326602    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8b3c0bf6_9bbe_46f8_8b02_8f6acc690a57.slice/crio-13a4a16645258d1ddcff24b6b010e11279465958a8f33bd0ee447742b3ad2201 WatchSource:0}: Error finding container 13a4a16645258d1ddcff24b6b010e11279465958a8f33bd0ee447742b3ad2201: Status 404 returned error can't find the container with id 13a4a16645258d1ddcff24b6b010e11279465958a8f33bd0ee447742b3ad2201
Oct 31 20:31:07.328517 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:07.328496    4645 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 20:31:07.864507 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:07.864479    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hk24f/must-gather-tbhnm" event={"ID":"8b3c0bf6-9bbe-46f8-8b02-8f6acc690a57","Type":"ContainerStarted","Data":"13a4a16645258d1ddcff24b6b010e11279465958a8f33bd0ee447742b3ad2201"}
Oct 31 20:31:08.872722 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:08.872679    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hk24f/must-gather-tbhnm" event={"ID":"8b3c0bf6-9bbe-46f8-8b02-8f6acc690a57","Type":"ContainerStarted","Data":"b1e91c00c022ea17bfbb58de6e7d54ccb6fda84bf129c313e29390735bd7b1c6"}
Oct 31 20:31:08.873237 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:08.872738    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hk24f/must-gather-tbhnm" event={"ID":"8b3c0bf6-9bbe-46f8-8b02-8f6acc690a57","Type":"ContainerStarted","Data":"0d8a2d70b56f48a87deec6b24c463faf05186686027dffe5f62c45e5029a64dc"}
Oct 31 20:31:08.891973 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:08.891917    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-hk24f/must-gather-tbhnm" podStartSLOduration=1.8892698719999999 podStartE2EDuration="2.891898917s" totalImagesPullingTime="1.002629045s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 20:31:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:31:08.890862463 +0000 UTC m=+4299.464096593" watchObservedRunningTime="2026-10-31 20:31:08.891898917 +0000 UTC m=+4299.465133047"
Oct 31 20:31:10.386672 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:10.386634    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-4pntt_4d2c9df4-c306-4f0b-b6a0-a4fafe9cc761/global-pull-secret-syncer/0.log"
Oct 31 20:31:10.730517 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:10.730482    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log"
Oct 31 20:31:10.862581 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:10.862551    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-94.us-east-2.compute.internal_b8f50d0470923e3e6371360e86f8acbb/haproxy/0.log"
Oct 31 20:31:19.202351 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:19.202034    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-559d65ddd-gx66l_c8726e82-9df2-47ab-bfbc-0babdddf5ee0/authorino/0.log"
Oct 31 20:31:19.572221 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:19.572139    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:31:20.591123 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:20.590870    4645 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-hk24f/perf-node-gather-daemonset-w5c5b"]
Oct 31 20:31:20.599084 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:20.599050    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hk24f/perf-node-gather-daemonset-w5c5b"
Oct 31 20:31:20.612886 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:20.612778    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hk24f/perf-node-gather-daemonset-w5c5b"]
Oct 31 20:31:20.637588 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:20.637546    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a93807fe-76a9-426c-a839-8c366f37bf82-lib-modules\") pod \"perf-node-gather-daemonset-w5c5b\" (UID: \"a93807fe-76a9-426c-a839-8c366f37bf82\") " pod="openshift-must-gather-hk24f/perf-node-gather-daemonset-w5c5b"
Oct 31 20:31:20.637783 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:20.637610    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a93807fe-76a9-426c-a839-8c366f37bf82-podres\") pod \"perf-node-gather-daemonset-w5c5b\" (UID: \"a93807fe-76a9-426c-a839-8c366f37bf82\") " pod="openshift-must-gather-hk24f/perf-node-gather-daemonset-w5c5b"
Oct 31 20:31:20.638077 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:20.638043    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a93807fe-76a9-426c-a839-8c366f37bf82-proc\") pod \"perf-node-gather-daemonset-w5c5b\" (UID: \"a93807fe-76a9-426c-a839-8c366f37bf82\") " pod="openshift-must-gather-hk24f/perf-node-gather-daemonset-w5c5b"
Oct 31 20:31:20.638186 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:20.638121    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a93807fe-76a9-426c-a839-8c366f37bf82-sys\") pod \"perf-node-gather-daemonset-w5c5b\" (UID: \"a93807fe-76a9-426c-a839-8c366f37bf82\") " pod="openshift-must-gather-hk24f/perf-node-gather-daemonset-w5c5b"
Oct 31 20:31:20.638267 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:20.638184    4645 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c527m\" (UniqueName: \"kubernetes.io/projected/a93807fe-76a9-426c-a839-8c366f37bf82-kube-api-access-c527m\") pod \"perf-node-gather-daemonset-w5c5b\" (UID: \"a93807fe-76a9-426c-a839-8c366f37bf82\") " pod="openshift-must-gather-hk24f/perf-node-gather-daemonset-w5c5b"
Oct 31 20:31:20.668444 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:20.668406    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:31:20.738972 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:20.738928    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a93807fe-76a9-426c-a839-8c366f37bf82-proc\") pod \"perf-node-gather-daemonset-w5c5b\" (UID: \"a93807fe-76a9-426c-a839-8c366f37bf82\") " pod="openshift-must-gather-hk24f/perf-node-gather-daemonset-w5c5b"
Oct 31 20:31:20.739170 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:20.738992    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a93807fe-76a9-426c-a839-8c366f37bf82-sys\") pod \"perf-node-gather-daemonset-w5c5b\" (UID: \"a93807fe-76a9-426c-a839-8c366f37bf82\") " pod="openshift-must-gather-hk24f/perf-node-gather-daemonset-w5c5b"
Oct 31 20:31:20.739170 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:20.739042    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c527m\" (UniqueName: \"kubernetes.io/projected/a93807fe-76a9-426c-a839-8c366f37bf82-kube-api-access-c527m\") pod \"perf-node-gather-daemonset-w5c5b\" (UID: \"a93807fe-76a9-426c-a839-8c366f37bf82\") " pod="openshift-must-gather-hk24f/perf-node-gather-daemonset-w5c5b"
Oct 31 20:31:20.739170 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:20.739047    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a93807fe-76a9-426c-a839-8c366f37bf82-proc\") pod \"perf-node-gather-daemonset-w5c5b\" (UID: \"a93807fe-76a9-426c-a839-8c366f37bf82\") " pod="openshift-must-gather-hk24f/perf-node-gather-daemonset-w5c5b"
Oct 31 20:31:20.739170 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:20.739083    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a93807fe-76a9-426c-a839-8c366f37bf82-lib-modules\") pod \"perf-node-gather-daemonset-w5c5b\" (UID: \"a93807fe-76a9-426c-a839-8c366f37bf82\") " pod="openshift-must-gather-hk24f/perf-node-gather-daemonset-w5c5b"
Oct 31 20:31:20.739170 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:20.739112    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a93807fe-76a9-426c-a839-8c366f37bf82-sys\") pod \"perf-node-gather-daemonset-w5c5b\" (UID: \"a93807fe-76a9-426c-a839-8c366f37bf82\") " pod="openshift-must-gather-hk24f/perf-node-gather-daemonset-w5c5b"
Oct 31 20:31:20.739170 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:20.739114    4645 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a93807fe-76a9-426c-a839-8c366f37bf82-podres\") pod \"perf-node-gather-daemonset-w5c5b\" (UID: \"a93807fe-76a9-426c-a839-8c366f37bf82\") " pod="openshift-must-gather-hk24f/perf-node-gather-daemonset-w5c5b"
Oct 31 20:31:20.739581 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:20.739176    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a93807fe-76a9-426c-a839-8c366f37bf82-podres\") pod \"perf-node-gather-daemonset-w5c5b\" (UID: \"a93807fe-76a9-426c-a839-8c366f37bf82\") " pod="openshift-must-gather-hk24f/perf-node-gather-daemonset-w5c5b"
Oct 31 20:31:20.739581 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:20.739246    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a93807fe-76a9-426c-a839-8c366f37bf82-lib-modules\") pod \"perf-node-gather-daemonset-w5c5b\" (UID: \"a93807fe-76a9-426c-a839-8c366f37bf82\") " pod="openshift-must-gather-hk24f/perf-node-gather-daemonset-w5c5b"
Oct 31 20:31:20.754832 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:20.754804    4645 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c527m\" (UniqueName: \"kubernetes.io/projected/a93807fe-76a9-426c-a839-8c366f37bf82-kube-api-access-c527m\") pod \"perf-node-gather-daemonset-w5c5b\" (UID: \"a93807fe-76a9-426c-a839-8c366f37bf82\") " pod="openshift-must-gather-hk24f/perf-node-gather-daemonset-w5c5b"
Oct 31 20:31:20.929051 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:20.929013    4645 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-hk24f/perf-node-gather-daemonset-w5c5b"
Oct 31 20:31:21.024591 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:21.024508    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-hwpwv_b4b8cb87-d398-43fd-a6da-b56a4d3d8e9b/monitoring-plugin/0.log"
Oct 31 20:31:21.221216 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:21.221138    4645 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-hk24f/perf-node-gather-daemonset-w5c5b"]
Oct 31 20:31:21.228035 ip-10-0-131-94 kubenswrapper[4645]: W1031 20:31:21.228003    4645 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda93807fe_76a9_426c_a839_8c366f37bf82.slice/crio-9c66d144e6c2cb6b1df231af131b6f882583d051f299e47aad015582807e8021 WatchSource:0}: Error finding container 9c66d144e6c2cb6b1df231af131b6f882583d051f299e47aad015582807e8021: Status 404 returned error can't find the container with id 9c66d144e6c2cb6b1df231af131b6f882583d051f299e47aad015582807e8021
Oct 31 20:31:21.276797 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:21.276768    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tfk67_3e0e15bd-a239-4a71-8f6c-2fbe061f03fd/node-exporter/0.log"
Oct 31 20:31:21.306441 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:21.306417    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tfk67_3e0e15bd-a239-4a71-8f6c-2fbe061f03fd/kube-rbac-proxy/0.log"
Oct 31 20:31:21.337565 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:21.337539    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tfk67_3e0e15bd-a239-4a71-8f6c-2fbe061f03fd/init-textfile/0.log"
Oct 31 20:31:21.825347 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:21.825016    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-5zb27_a4864737-8f10-4291-8529-57acff0fe5c2/prometheus-operator-admission-webhook/0.log"
Oct 31 20:31:21.844349 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:21.841472    4645 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kc9mq_05ca774c-33f8-4368-a677-7c1a98b61bf7/konnectivity-agent/0.log" limit=1024
Oct 31 20:31:22.037251 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:22.037214    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hk24f/perf-node-gather-daemonset-w5c5b" event={"ID":"a93807fe-76a9-426c-a839-8c366f37bf82","Type":"ContainerStarted","Data":"b2793a2e92defaae0f702b3b76e4fe9aff048bb92a0bdfc806593f81f3b266f6"}
Oct 31 20:31:22.037479 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:22.037267    4645 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-hk24f/perf-node-gather-daemonset-w5c5b" event={"ID":"a93807fe-76a9-426c-a839-8c366f37bf82","Type":"ContainerStarted","Data":"9c66d144e6c2cb6b1df231af131b6f882583d051f299e47aad015582807e8021"}
Oct 31 20:31:22.038009 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:22.037984    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-hk24f/perf-node-gather-daemonset-w5c5b"
Oct 31 20:31:22.057490 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:22.057444    4645 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-hk24f/perf-node-gather-daemonset-w5c5b" podStartSLOduration=2.057432114 podStartE2EDuration="2.057432114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 20:31:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 20:31:22.053959198 +0000 UTC m=+4312.627193327" watchObservedRunningTime="2026-10-31 20:31:22.057432114 +0000 UTC m=+4312.630666242"
Oct 31 20:31:24.445730 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:24.445692    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-68496cf448-m7fgc_0e752859-51b7-481b-a5ba-81681289e311/console/0.log"
Oct 31 20:31:24.506761 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:24.506706    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-fs6gw_c62a9b4b-1882-4d88-b486-ec5f3a6d18db/download-server/0.log"
Oct 31 20:31:26.611448 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:26.611409    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kjlt7_bc386a66-cdd2-4f4b-9666-88d48e42bd33/dns/0.log"
Oct 31 20:31:26.638511 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:26.638483    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kjlt7_bc386a66-cdd2-4f4b-9666-88d48e42bd33/kube-rbac-proxy/0.log"
Oct 31 20:31:26.738081 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:26.738054    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-58f52_21dc10e8-ee61-46ae-83f7-67d1772f083c/dns-node-resolver/0.log"
Oct 31 20:31:27.455556 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:27.455525    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-779c8588cc-ns2s4_0f7455d5-9b9d-47f7-a0ad-df85dea8139c/registry/0.log"
Oct 31 20:31:27.508789 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:27.508744    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5mwk4_944a350f-aa7e-4921-8ddc-b0001fa05086/node-ca/0.log"
Oct 31 20:31:29.063913 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:29.063881    4645 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-hk24f/perf-node-gather-daemonset-w5c5b"
Oct 31 20:31:31.122278 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:31.122250    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-849d94dfbb-l28r8_ae933cbf-384e-4052-881d-800d405f8e7b/kube-auth-proxy/0.log"
Oct 31 20:31:31.166811 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:31.166777    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-68qbd_3a217b30-54c0-431f-90bc-0681901272f4/istio-proxy/0.log"
Oct 31 20:31:31.203752 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:31.203712    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-6rwmb_b92c4b09-c2c0-4602-af78-2f410c843118/istio-proxy/0.log"
Oct 31 20:31:31.328765 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:31.328726    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-lwvpn_3d401fef-1568-4d16-88fa-92aeaaea68b0/istio-proxy/0.log"
Oct 31 20:31:31.447985 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:31.447957    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-t27sn_f3994be2-19d9-4e2d-b3ce-03f63d2c9c32/istio-proxy/0.log"
Oct 31 20:31:31.623806 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:31.623766    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-76hjw_b14ee5b2-6bf8-4b41-a84d-d0a7dcc9b85a/payload-processing/0.log"
Oct 31 20:31:32.249665 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:32.249636    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-849d94dfbb-l28r8_ae933cbf-384e-4052-881d-800d405f8e7b/kube-auth-proxy/0.log"
Oct 31 20:31:32.273846 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:32.273814    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-68qbd_3a217b30-54c0-431f-90bc-0681901272f4/istio-proxy/0.log"
Oct 31 20:31:32.301532 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:32.301503    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-6rwmb_b92c4b09-c2c0-4602-af78-2f410c843118/istio-proxy/0.log"
Oct 31 20:31:32.360727 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:32.360704    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-lwvpn_3d401fef-1568-4d16-88fa-92aeaaea68b0/istio-proxy/0.log"
Oct 31 20:31:32.421578 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:32.421555    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-t27sn_f3994be2-19d9-4e2d-b3ce-03f63d2c9c32/istio-proxy/0.log"
Oct 31 20:31:32.525387 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:32.525289    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-76hjw_b14ee5b2-6bf8-4b41-a84d-d0a7dcc9b85a/payload-processing/0.log"
Oct 31 20:31:34.432167 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:34.432134    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2v47p_7306a726-d4c7-4ef5-b496-726e0bd488a1/serve-healthcheck-canary/0.log"
Oct 31 20:31:35.274401 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:35.274372    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kmlmk_a7702c04-56e6-4349-97f8-cd8c2efc76aa/kube-rbac-proxy/0.log"
Oct 31 20:31:35.305871 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:35.305844    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kmlmk_a7702c04-56e6-4349-97f8-cd8c2efc76aa/exporter/0.log"
Oct 31 20:31:35.343689 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:35.343657    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kmlmk_a7702c04-56e6-4349-97f8-cd8c2efc76aa/extractor/0.log"
Oct 31 20:31:35.494992 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:35.494953    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_periodic-gathering-j9l2c-hmgd8_611ab431-d82c-46e9-a108-4b221cc8c2cc/insights-gathering/0.log"
Oct 31 20:31:35.539437 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:35.539370    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_periodic-gathering-s8p5b-qtsxv_0d87fa56-1043-431f-9304-6ce9c42bfafb/insights-gathering/0.log"
Oct 31 20:31:36.246609 ip-10-0-131-94 kubenswrapper[4645]: E1031 20:31:36.246567    4645 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-kmlmk" containerName="extractor"
Oct 31 20:31:38.379853 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:38.379824    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-6bb8c5759f-mbfk5_e2ab4650-3c3b-4754-a1a8-f99673da37ac/manager/0.log"
Oct 31 20:31:38.413855 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:38.413824    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-6bb8c5759f-mbfk5_e2ab4650-3c3b-4754-a1a8-f99673da37ac/copy-manifests/0.log"
Oct 31 20:31:38.550212 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:38.550183    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-849d94dfbb-l28r8_ae933cbf-384e-4052-881d-800d405f8e7b/kube-auth-proxy/0.log"
Oct 31 20:31:38.576729 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:38.576697    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-68qbd_3a217b30-54c0-431f-90bc-0681901272f4/istio-proxy/0.log"
Oct 31 20:31:38.608766 ip-10-0-131-94 kubenswrapper[4645]: I1031 20:31:38.608734    4645 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-75d7c785f-6rwmb_b92c4b09-c2c0-4602-af78-2f410c843118/istio-proxy/0.log"